token --- Constants used with Python parse trees¶
Code source : Lib/token.py
This module provides constants which represent the numeric values of leaf nodes
of the parse tree (terminal tokens). Refer to the file Grammar/Tokens
in the Python distribution for the definitions of the names in the context of
the language grammar. The specific numeric values which the names map to may
change between Python versions.
Le module fournit également une correspondance entre les codes numériques et les noms et certaines fonctions. Les fonctions reflètent les définitions des fichiers d'en-tête C de Python.
Note that a token's value may depend on tokenizer options. For example, a
"+" token may be reported as either PLUS or OP, or
a "match" token may be either NAME or SOFT_KEYWORD.
- token.tok_name¶
Dictionnaire faisant correspondre les valeurs numériques des constantes définies dans ce module à leurs noms, permettant de générer une représentation plus humaine des arbres syntaxiques.
- token.ISTERMINAL(x)¶
Return
Truefor terminal token values.
- token.ISNONTERMINAL(x)¶
Return
Truefor non-terminal token values.
- token.ISEOF(x)¶
Return
Trueif x is the marker indicating the end of input.
Les constantes associées aux jetons sont :
- token.NAME¶
Token value that indicates an identifier. Note that keywords are also initially tokenized an
NAMEtokens.
- token.NUMBER¶
Token value that indicates a numeric literal
- token.STRING¶
Token value that indicates a string or byte literal, excluding formatted string literals. The token string is not interpreted: it includes the surrounding quotation marks and the prefix (if given); backslashes are included literally, without processing escape sequences.
- token.OP¶
A generic token value that indicates an operator or delimiter.
Particularité de l'implémentation CPython : This value is only reported by the
tokenizemodule. Internally, the tokenizer uses exact token types instead.
- token.COMMENT¶
Token value used to indicate a comment. The parser ignores
COMMENTtokens.
- token.NEWLINE¶
Token value that indicates the end of a logical line.
- token.NL¶
Token value used to indicate a non-terminating newline.
NLtokens are generated when a logical line of code is continued over multiple physical lines. The parser ignoresNLtokens.
- token.INDENT¶
Token value used at the beginning of a logical line to indicate the start of an indented block.
- token.DEDENT¶
Token value used at the beginning of a logical line to indicate the end of an indented block.
- token.FSTRING_START¶
Token value used to indicate the beginning of an f-string literal.
Particularité de l'implémentation CPython : The token string includes the prefix and the opening quote(s), but none of the contents of the literal.
- token.FSTRING_MIDDLE¶
Token value used for literal text inside an f-string literal, including format specifications.
Particularité de l'implémentation CPython : Replacement fields (that is, the non-literal parts of f-strings) use the same tokens as other expressions, and are delimited by
LBRACE,RBRACE,EXCLAMATIONandCOLONtokens.
- token.FSTRING_END¶
Token value used to indicate the end of a f-string.
Particularité de l'implémentation CPython : The token string contains the closing quote(s).
- token.ENDMARKER¶
Token value that indicates the end of input.
- token.ENCODING¶
Valeur de jeton qui indique l'encodage utilisé pour décoder le fichier initial. Le premier jeton renvoyé par
tokenize.tokenize()sera toujours un jetonENCODING.Particularité de l'implémentation CPython : This token type isn't used by the C tokenizer but is needed for the
tokenizemodule.
The following token types are not produced by the tokenize module,
and are defined for special uses in the tokenizer or parser:
- token.TYPE_IGNORE¶
Token value indicating that a
type: ignorecomment was recognized. Such tokens are produced instead of regularCOMMENTtokens only with thePyCF_TYPE_COMMENTSflag.
- token.TYPE_COMMENT¶
Token value indicating that a type comment was recognized. Such tokens are produced instead of regular
COMMENTtokens only with thePyCF_TYPE_COMMENTSflag.
- token.SOFT_KEYWORD¶
Token value indicating a soft keyword.
The tokenizer never produces this value. To check for a soft keyword, pass a
NAMEtoken's string tokeyword.issoftkeyword().
- token.ERRORTOKEN¶
Token value used to indicate wrong input.
The
tokenizemodule generally indicates errors by raising exceptions instead of emitting this token. It can also emit tokens such asOPorNAMEwith strings that are later rejected by the parser.
The remaining tokens represent specific operators and
delimiters.
(The tokenize module reports these as OP; see exact_type
in the tokenize documentation for details.)
Token |
Value |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The following non-token constants are provided:
- token.N_TOKENS¶
The number of token types defined in this module.
- token.EXACT_TOKEN_TYPES¶
A dictionary mapping the string representation of a token to its numeric code.
Ajouté dans la version 3.8.
Modifié dans la version 3.5: Added AWAIT and ASYNC tokens.
Modifié dans la version 3.7: Removed AWAIT and ASYNC tokens. "async" and "await" are
now tokenized as NAME tokens.
Modifié dans la version 3.8: Added TYPE_COMMENT, TYPE_IGNORE, COLONEQUAL.
Added AWAIT and ASYNC tokens back (they're needed
to support parsing older Python versions for ast.parse() with
feature_version set to 6 or lower).
Modifié dans la version 3.12: Added EXCLAMATION.
Modifié dans la version 3.13: Removed AWAIT and ASYNC tokens again.