29. Python Developer FAQ¶
- Where should I ask general Python questions?
- Where should I suggest new features and language changes?
- Where should I ask general questions about contributing to CPython?
- Where should I report specific problems?
- What if I’m not sure it is a bug?
- What if I disagree with an issue resolution on the tracker?
- How do I tell who is and isn’t a core developer?
- What standards of behaviour are expected in these communication channels?
- Version Control
- For everyone
- Where can I learn about the version control system used, Mercurial (hg)?
- I already know how to use Git, can I use that instead?
- What do I need to use Mercurial?
- What’s a working copy? What’s a repository?
- Which branches are in my local repository?
- Why are some branches marked “inactive”?
- Which branch is currently checked out in my working copy?
- How do I switch between branches inside my working copy?
- I want to keep a separate working copy per development branch, is it possible?
- How do I link my local repository to a particular remote repository?
- How do I create a shorthand alias for a remote repository?
- How do I compare my local repository to a remote repository?
- How do I update my local repository to be in sync with a remote repository?
- How do I update my working copy with the latest changes?
- How do I apply a patch?
- How do I solve conflicts when applying a patch fails?
- How do I add a file or directory to the repository?
- What’s the best way to split a file into several files?
- How do I delete a file or directory in the repository?
- What files are modified in my working copy?
- How do I revert a file I have modified back to the version in the repository?
- How do I find out who edited or what revision changed a line last?
- How can I see a list of log messages for a file or specific revision?
- How can I see the changeset graph in my repository?
- How do I update to a specific release tag?
- How do I find which changeset introduced a bug or regression?
- How come feature XYZ isn’t available in Mercurial?
- For core developers
- For everyone
The python-ideas mailing list is specifically intended for discussion of new features and language changes. Please don’t be disappointed if your idea isn’t met with universal approval: as the long list of Rejected and Withdrawn PEPs in the PEP Index attests, and as befits a reasonably mature programming language, getting significant changes into Python isn’t a simple task.
Sometimes core developers will differ in opinion, or merely be collectively unconvinced. When there isn’t an obvious victor then the Status Quo Wins a Stalemate as outlined in the linked post.
For some examples on language changes that were accepted please read Justifying Python Language Changes.
See also the Changing the Python Language section of this guide.
The Python Mentors program is specifically about encouraging
developers and others that would like to contribute to Python development in
general, rather than necessarily being focused on one particular issue.
Some core developers are also available on the
#python-dev IRC channel on
Specific problems should be posted to the issue tracker.
The general Python help locations listed above are the best place to start with that kind of question. If they agree it looks like a bug, then the next step is to either post it to the issue tracker or else to ask further on the core development mailing list, python-dev.
First, take some time to consider any comments made in association with the resolution of the tracker issue. On reflection, they may seem more reasonable than they first appeared.
If you still feel the resolution is incorrect, then raise the question on python-dev. If the consensus there supports the disputed resolution, please take any further objections to python-ideas (or some other forum). Further argument on python-dev after a consensus has been reached amongst the core developers is unlikely to win any converts.
You can check their name against the full list of developers with commit rights to the main source control repository.
On the issue tracker, most core developers will have the Python logo appear next to their name.
We try to foster environments of mutual respect, tolerance and encouragement, as described in the PSF’s Diversity Statement. Abiding by the guidelines in this document and asking questions or posting suggestions in the appropriate channels are an excellent way to get started on the mutual respect part, greatly increasing the chances of receiving tolerance and encouragement in return.
The following FAQs are intended for both core developers and contributors.
Mercurial‘s (also known as
hg) official web site is at
https://www.mercurial-scm.org/. A book on Mercurial published by
O’Reilly Media, Mercurial: The Definitive Guide, is available
for free online. Another resource is Hg Init: a Mercurial tutorial
by Joel Spolsky.
With Mercurial installed, you can run the help tool that comes with Mercurial to get help:
The man page for
hg provides a quick refresher on the details of
various commands, but doesn’t provide any guidance on overall
While the main workflow for core developers requires Mercurial, if
you just want to generate patches with
git diff and post them to the
issue tracker, there is a semi-offical read-only git mirror of the main
CPython repository. To create a local clone based on this mirror rather
than the main repository:
git clone git://github.com/python/cpython
The mirror’s master branch tracks the main repository’s default branch,
while the maintenance branch names (
3.4, etc) are mapped
Please only use this approach if you’re already an experienced Git user and don’t require assistance with the specifics of version control commands. All other parts of this developer’s guide assume the use of Mercurial for local version control.
First, you need to download Mercurial. Most UNIX-based operating systems have binary packages available. Most package management systems also have native Mercurial packages available.
If you have push rights, you need OpenSSH. This is needed to verify your identity when performing commits. As with Mercurial, binary packages are typically available either online or through the platform’s package management system.
Mercurial does not use its own compression via SSH
because it is better to enable compression at the SSH level. Enabling
SSH compression can make cloning a remote repository much faster.
You can configure it in your
~/.ssh/config file; for example:
Host hg.python.org Compression yes
The recommended option on Windows is to download TortoiseHg which
integrates with Windows Explorer and also bundles the command line client
(meaning you can type
hg in a DOS box). Note that most
entries in this FAQ only cover the command line client in detail - refer
to the TortoiseHg documentation for assistance with its graphical interface.
If you have push rights, you need to configure Mercurial to work with
your SSH keys. For that, open your Mercurial configuration file
(you can do so by opening the TortoiseHg Global Settings dialog and then
clicking “Edit File”). If there is no
[ui] section, create it by
typing just that on a line by itself. Then add the following line:
ssh = TortoisePlink.exe -ssh -2 -C -i C:\path\to\yourkey.ppk
C:\path\to\yourkey.ppk should be replaced with the actual path
to your SSH private key.
If your private key is in OpenSSH format, you must first convert it to PuTTY format by loading it into PuTTYgen.
Mercurial is a “distributed” version control system. This means that each
participant, even casual contributors, download a complete copy (called a
clone, since it is obtained by calling
hg clone) of the central
repository which can be treated as a stand-alone repository for all purposes.
That copy is called in the FAQ the local repository, to differentiate
with any remote repository you might also interact with.
But you don’t modify files directly in the local repository; Mercurial doesn’t allow for it. You modify files in what’s called the working copy associated with your local repository: you also run compilations and tests there. Once you are satisfied with your changes, you can commit them; committing records the changes as a new revision in the local repository.
Changes in your local repository don’t get automatically shared with the
rest of the world. Mercurial ensures that you have to do so explicitly
(this allows you to experiment quite freely with multiple branches of
development, all on your private computer). The main commands for doing
hg pull and
hg branches displays the open branches in your local repository:
$ hg branches default 93085:030fda7b1de8 2.7 93060:7ba47bbfe38d 3.5 99283:4d5417444961 (inactive) 3.4 93082:5fd481150b35 (inactive) 3.3 93079:cda907a02a80 (inactive) 3.2 92975:eac54f7a8018 (inactive)
Assuming you get the following output:
$ hg branches default 93085:030fda7b1de8 3.5 99283:4d5417444961 (inactive)
This means all changesets in the “3.5” branch have been merged into the “default” branch (or any other branch, if such exists).
$ hg branch default
Or to get more information:
$ hg summary parent: 68026:f12ef116dd10 tip In FTP.close() method, make sure to also close the socket object, not only the file. branch: default commit: (clean) update: (current)
hg update to checkout another branch in the current directory:
$ hg branch default $ hg update 3.6 86 files updated, 0 files merged, 11 files removed, 0 files unresolved $ hg branch 3.6
-v option to
hg update will list all updated files.
Note that, due to some previously built executables being used as a part of
the build process, you may sometimes run into issues when attempting to
switch between Python 2.x and Python 3.x branches. In these cases, it is
best to run a
make distclean to ensure that all previously built files
There are two ways:
- Use the “share extension” as described in the Multiple Clones Approach section;
- Create several clones of your local repository;
If you want to use the second way, you can do:
$ hg clone cpython py36 updating to branch default 3434 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd py35 $ hg update 3.6 86 files updated, 0 files merged, 11 files removed, 0 files unresolved
The current branch in a working copy is “sticky”: if you pull in some new
hg update will update to the head of the current branch.
In your global
.hgrc file add a section similar to the following:
[paths] dg = ssh://firstname.lastname@example.org/devguide
This example creates a
dg alias for the
hg.python.org. This allows “dg” to be entered instead of the
full URL for commands taking a repository argument (e.g.
hg pull dg or
hg outgoing dg).
<remote repository> is used in the commands in this
hg should accept an alias in place of a complete remote URL.
To display the list of changes that are in your local repository, but not in the remote, use:
hg outgoing <remote repository>
This is the list of changes that will be sent if you call
hg push <remote repository>. It does not include any uncommitted
changes in your working copy!
Conversely, for the list of changes that are in the remote repository but not in the local, use:
hg incoming <remote repository>
This is the list of changes that will be retrieved if you call
hg pull <remote repository>.
In most daily use, you will work against the default remote repository,
and therefore simply type
hg outgoing and
In this case, you can also get a synthetic summary using
hg summary --remote.
hg pull <remote repository>
from the repository you wish to pull the latest changes into. Most of the time, that repository is a clone of the repository you want to pull from, so you can simply type:
This doesn’t update your working copy, though. See below:
If you find yourself typing often
hg pull followed by
be aware that you can combine them in a single command:
hg pull -u
If you want to try out or review a patch generated using Mercurial, do:
patch -p1 < somework.patch
This will apply the changes in your working copy without committing them.
If the patch was not created by Mercurial (for example, a patch created by
Subversion and thus lacking any
b directory prefixes in the patch),
If the patch contains renames, deletions or copies, and you intend committing it after your review, you might prefer using:
hg import --no-commit somework.patch
If you want to work on the patch using mq (Mercurial Queues), type instead:
hg qimport somework.patch
This will create a patch in your queue with a name that matches the filename.
You can use the
-n argument to specify a different name. To have the
patch applied to the working copy, type:
Finally, to delete the patch, first un-apply it if necessary using
hg qdelete somework.patch
patch command, as well as
hg import, will produce
*.rej files when it fails applying parts of a patch.
We suggest you try the mpatch utility, which can help resolve a number of
common causes of patch rejects.
To make use of
mpatch transparent, you can define a shell alias in one
of your startup files. For example, if you want it to open the
merge program to fix failing patch hunks:
alias patch='mpatch --merge=kdiff3'
or if you want it to automatically solve conflicts by using heuristics:
alias patch='mpatch --auto --no-merge'
Simply specify the path to the file or directory to add and run:
hg add PATH
PATH is a directory, Mercurial will recursively add any files in that
directory and its descendants.
If you want Mercurial to figure out by itself which files should be added and/or removed, just run:
Be careful though, as it might add some files that are not desired in the repository (such as build products, cache files, or other data).
You will then need to run
hg commit (as discussed below) to commit
the file(s) to your local repository.
To split a file into several files (e.g. a module converted to a package or a
long doc file divided in two separate documents) use
hg copy module.rst module2.rst
and then remove the parts that are not necessary from
module2.rst. This allows Mercurial to know that the content of
module2.rst used to be in
module.rst, and will make subsequent merges
easier. If necessary, you can also use
hg copy several times.
If you simply create
module2.rst, add it with
hg add, and copy part of
the content from
module.rst, Mercurial won’t know that the two file are
Specify the path to be removed with:
hg remove PATH
This will remove the file or the directory from your working copy; you will have to commit your changes for the removal to be recorded in your local repository.
will list any pending changes in the working copy. These changes will get
committed to the local repository if you issue an
hg commit without
specifying any path.
Some key indicators that can appear in the first column of output are:
A Scheduled to be added R Scheduled to be removed M Modified locally ? Not under version control
If you want a line-by-line listing of the differences, use:
hg revert PATH
PATH to its version in the repository, throwing away any
changes you made locally. If you run:
hg revert -a
from the root of your working copy it will recursively restore everything to match up with the repository.
hg annotate PATH
This will output to stdout every line of the file along with which revision last modified that line. When you have the revision number, it is then easy to display it in detail.
To see the history of changes for a specific file, run:
hg log -v [PATH]
That will list all messages of revisions which modified the file specified
PATH is omitted, all revisions are listed.
If you want to display line-by-line differences for each revision as well,
hg log -vp [PATH]
If you want to view the differences for a specific revision, run:
hg log -vp -r <revision number>
In Mercurial repositories, changesets don’t form a simple list, but rather a graph: every changeset has one or two parents (it’s called a merge changeset in the latter case), and can have any number of children.
The graphlog extension is very useful for examining the structure of the changeset graph. It is bundled with Mercurial.
Graphical tools, such as TortoiseHG, will display the changeset graph by default.
to get a list of tags. To update your working copy to a specific tag, use:
hg update <tag>
hg bisect, as the name indicates, helps you do a bisection of a range of
You need two changesets to start the search: one that is “good” (doesn’t have the bug), and one that is “bad” (has the bug). Usually, you have just noticed the bug in your working copy, so you can start with:
hg bisect --bad
Then you must
update to a previous changeset that doesn’t have the bug.
You can conveniently choose a faraway changeset (for example a former release),
and check that it is indeed “good”. Then type:
hg bisect --good
Mercurial will automatically bisect so as to narrow the range of possible culprits, until a single changeset is isolated. Each time Mercurial presents you with a new changeset, re-compile Python and run the offending test, for example:
make -j2 ./python -m test -uall test_sometest
Then, type either
hg bisect --good or
hg bisect --bad depending on
whether the test succeeded or failed.
Mercurial comes with many bundled extensions which can be explicitly enabled.
You can get a list of them by typing
hg help extensions. Some of these
extensions, such as
color, can prettify output; others, such as
graphlog, add new Mercurial commands.
These FAQs are intended mainly for core developers.
To commit any changes to a file (which includes adding a new file or deleting an existing one), you use the command:
hg commit [PATH]
PATH is optional: if it is omitted, all changes in your working copy
will be committed to the local repository. When you commit, be sure that all
changes are desired by reviewing them first;
also, when making commits that you intend to push to public repositories,
you should not commit together unrelated changes.
To abort a commit that you are in the middle of, leave the message empty (i.e., close the text editor without adding any text for the message). Mercurial will then abort the commit operation so that you can try again later.
Once a change is committed to your local repository, it is still only visible by you. This means you are free to experiment with as many local commits you feel like.
If you do not like the default text editor Mercurial uses for
entering commit messages, you may specify a different editor,
either by changing the
EDITOR environment variable or by setting
a Mercurial-specific editor in your global
.hgrc with the
option in the
The easiest way is to install KDiff3 — Mercurial will open it automatically in case of conflicts, and you can then use it to solve the conflicts and save the resulting file(s). KDiff3 will also take care of marking the conflicts as resolved.
If you don’t use a merge tool, you can use
hg resolve --list to list the
conflicting files, resolve the conflicts manually, and the use
hg resolve --mark <file path> to mark these conflicts as resolved.
You can also use
hg resolve -am to mark all the conflicts as resolved.
Mercurial will use KDiff3 automatically if it’s installed and it can find it — you don’t need to change any settings. KDiff3 is also already included in the installer of TortoiseHg. For more information, see https://www.mercurial-scm.org/wiki/KDiff3.
If you committed something (e.g. on 3.6) that shouldn’t be ported on newer branches (e.g. on default), you have to do a null merge:
cd 3.x hg merge 3.6 hg revert -ar default hg resolve -am # needed only if the merge created conflicts hg ci -m '#12345: null merge with 3.6.'
hg status should list all the merged files as
hg diff should produce no output. This will record the merge without
actually changing the content of the files.
If you see this message while pushing, it means that you committed something on a clone that was not up to date, thus creating a new head. This usually happens for two reasons:
- You forgot to run
hg upbefore committing;
- Someone else pushed on the main repo just before you, causing a push race;
First of all you should pull the new changesets using
hg pull. Then you can
hg heads to see which branches have multiple heads.
If only one branch has multiple heads, you can do:
cd default hg heads . hg up csid-of-the-other-head hg merge hg ci -m 'Merge heads.'
hg heads . will show you the two heads of the current branch: the one you
pulled and the one you created with your commit (you can also specify a branch
hg heads <branch>). While not strictly necessary, it is highly
recommended to switch to the other head before merging. This way you will be
merging only your changeset with the rest, and in case of conflicts it will be
a lot easier.
If more than one branch has multiple heads, you have to repeat these steps for
each branch. Since this creates new changesets, you will also have to
merge them between branches. For example, if both
default have multiple heads, you should first merge heads in
then merge heads in
default, and finally merge
hg merge 3.6 as usual.
In order to avoid this, you should always remember to pull and update before committing.
First, this should not happen if you take the habit of reviewing changes before committing them.
In any case, run:
hg backout <revision number>
This will modify your working copy so that all changes in
(including added or deleted files) are undone. You then need to commit these changes so that the backout gets permanently recorded.
These instructions are for Mercurial 1.7 and higher.
hg backout has
a slightly different behaviour in versions before 1.7.
All generated SSH keys should be sent to email@example.com for adding to the list of keys. DSA keys are unacceptable. To avoid a spam classification, give a non-empty subject and body. A reply will be sent, usually within a day or two, when the key can be used.
ssh-keygen -t ed25519
This will generate two files; your public key and your private key. The public
key is in file ending in
ssh-add to register your private key with SSH for
your current session. The simplest solution, though, is to use KeyChain,
which is a shell script that will handle
ssh-add for you
once per login instead of per session.
The Pageant program is bundled with TortoiseHg. You can find it in its
installation directory (usually
C:\Program Files (x86)\TortoiseHg\);
you can also download it separately.
Running Pageant will prevent you from having to type your password constantly. If you add a shortcut to Pageant to your Autostart group and edit the shortcut so that the command line includes an argument to your private key then Pageant will load the key every time you log in.
You can make commits from any machine, since they will be recorded in your local repository.
However, to push these changes to the remote server, you will need proper credentials. All you need is to make sure that the machine you want to push changes from has both the public and private keys in the standard place that ssh will look for them (i.e. ~/.ssh on Unix machines). Please note that although the key file ending in .pub contains your user name and machine name in it, that information is not used by the verification process, therefore these key files can be moved to a different computer and used for verification. Please guard your keys and never share your private key with anyone. If you lose the media on which your keys are stored or the machine on which your keys are stored, be sure to report this to firstname.lastname@example.org at the same time that you change your keys.
If a change is made to Python which relies on some POSIX system-specific
functionality (such as using a new system call), it is necessary to update the
configure script to test for availability of the functionality.
configure script is generated from
configure.ac using Autoconf.
Instead of editing
configure.ac and then run
autoreconf to regenerate
configure and a number of other files (such as
When submitting a patch with changes made to
configure.ac, it is preferred
to leave out the generated files as differences between Autoconf versions
frequently results in many spurious changes cluttering the patch. Instead,
remind any potential reviewers on the tracker to run
Note that running
autoreconf is not the same as running
autoconf by itself will not regenerate
autoconf and a number of other tools repeatedly as is
configure.ac script typically requires a specific version of
Autoconf. At the moment, this reads:
If the system copy of Autoconf does not match this version, you will need to install your own copy of Autoconf.
test_gdb attempts to automatically load additional Python specific
hooks into gdb in order to test them. Unfortunately, the command line
options it uses to do this aren’t always supported correctly.
test_gdb is being skipped with an “auto-loading has been declined”
message, then it is necessary to identify any Python build directories as
auto-load safe. One way to achieve this is to add a line like the following
~/.gdbinit (edit the specific list of paths as appropriate):
The first step is to familiarize yourself with the development toolchain on the platform in question, notably the C compiler. Make sure you can compile and run a hello-world program using the target compiler.
Next, learn how to compile and run the Python interpreter on a platform to
which it has already been ported; preferably Unix, but Windows will
do, too. The build process for Python, in particular the
Makefile in the
source distribution, will give you a hint on which files to compile
for Python. Not all source files are relevant: some are platform
specific, others are only used in emergencies (e.g.
It is not recommended to start porting Python without at least medium-level understanding of your target platform; i.e. how it is generally used, how to write platform specific apps, etc. Also, some Python knowledge is required, or you will be unable to verify that your port is working correctly.
You will need a
pyconfig.h file tailored for your platform. You can
pyconfig.h.in, read the comments, and turn on definitions that
apply to your platform. Also, you will need a
config.c file, which lists
the built-in modules you support. Again, starting with
Modules/config.c.in is recommended.
Finally, you will run into some things that are not supported on your
target platform. Forget about the
posix module in the beginning. You can
simply comment it out of the
Keep working on it until you get a
>>> prompt. You may have to disable the
site.py by passing the
-S option. When you have a prompt,
bang on it until it executes very simple Python statements.
At some point you will want to use the
os module; this is the time to start
thinking about what to do with the
posix module. It is okay to simply
comment out functions in the
posix module that cause problems; the
remaining ones will be quite useful.
Before you are done, it is highly recommended to run the Python regression test suite, as described in Running & Writing Tests.