The class provided by packaging.pypi.simple can access project indexes and provide useful information about distributions. PyPI, other indexes and local indexes are supported.
You should use this module to search distributions by name and versions, process index external pages and download distributions. It is not suited for things that will end up in too long index processing (like “finding all distributions with a specific version, no matter the name”); use packaging.pypi.xmlrpc for that.
index_url is the address of the index to use for requests.
The first two parameters control the query results. prefer_final indicates whether a final version (not alpha, beta or candidate) is to be preferred over a newer but non-final version (for example, whether to pick up 1.0 over 2.0a3). It is used only for queries that don’t give a version argument. Likewise, prefer_source tells whether to prefer a source distribution over a binary one, if no distribution argument was prodived.
Other parameters are related to external links (that is links that go outside the simple index): hosts is a list of hosts allowed to be processed if follow_externals is true (default behavior is to follow all hosts), follow_externals enables or disables following external links (default is false, meaning disabled).
The remaining parameters are related to the mirroring infrastructure defined in PEP 381. mirrors_url gives a URL to look on for DNS records giving mirror adresses; mirrors is a list of mirror URLs (see the PEP). If both mirrors and mirrors_url are given, mirrors_url will only be used if mirrors is set to None. timeout is the time (in seconds) to wait before considering a URL has timed out; mirrors_max_tries” is the number of times to try requesting informations on mirrors before switching.
The following methods are defined:
- get_distributions(project_name, version)¶
Return the distributions found in the index for the given release.
- get_metadata(project_name, version)¶
Return the metadata found on the index for this project name and version. Currently downloads and unpacks a distribution to read the PKG-INFO file.
- get_release(requirements, prefer_final=None)¶
Return one release that fulfills the given requirements.
- get_releases(requirements, prefer_final=None, force_update=False)¶
Search for releases and return a ReleasesList object containing the results.
- search_projects(name=None)¶
Search the index for projects containing the given name and return a list of matching names.
See also the base class packaging.pypi.base.BaseClient for inherited methods.
The address used by default by the crawler class. It is currently 'http://a.pypi.python.org/simple/', the main PyPI installation.
To help you understand how using the Crawler class, here are some basic usages.
Supposing you want to scan an index to get a list of distributions for the “foobar” project. You can use the “get_releases” method for that. The get_releases method will browse the project page, and return ReleaseInfo objects for each found link that rely on downloads.
>>> from packaging.pypi.simple import Crawler
>>> crawler = Crawler()
>>> crawler.get_releases("FooBar")
[<ReleaseInfo "Foobar 1.1">, <ReleaseInfo "Foobar 1.2">]
Note that you also can request the client about specific versions, using version specifiers (described in PEP 345):
>>> client.get_releases("FooBar < 1.2")
[<ReleaseInfo "FooBar 1.1">, ]
get_releases returns a list of ReleaseInfo, but you also can get the best distribution that fullfil your requirements, using “get_release”:
>>> client.get_release("FooBar < 1.2")
<ReleaseInfo "FooBar 1.1">
As it can get the urls of distributions provided by PyPI, the Crawler client also can download the distributions and put it for you in a temporary destination:
>>> client.download("foobar")
/tmp/temp_dir/foobar-1.2.tar.gz
You also can specify the directory you want to download to:
>>> client.download("foobar", "/path/to/my/dir")
/path/to/my/dir/foobar-1.2.tar.gz
While downloading, the md5 of the archive will be checked, if not matches, it will try another time, then if fails again, raise MD5HashDoesNotMatchError.
Internally, that’s not the Crawler which download the distributions, but the DistributionInfo class. Please refer to this documentation for more details.
The default behavior for packaging is to not follow the links provided by HTML pages in the “simple index”, to find distributions related downloads.
It’s possible to tell the PyPIClient to follow external links by setting the follow_externals attribute, on instantiation or after:
>>> client = Crawler(follow_externals=True)
or
>>> client = Crawler()
>>> client.follow_externals = True
The default Crawler behavior is to rely on the Python Package index stored on PyPI (http://pypi.python.org/simple).
As you can need to work with a local index, or private indexes, you can specify it using the index_url parameter:
>>> client = Crawler(index_url="file://filesystem/path/")
or
>>> client = Crawler(index_url="http://some.specific.url/")
You also can specify mirrors to fallback on in case the first index_url you provided doesnt respond, or not correctly. The default behavior for Crawler is to use the list provided by Python.org DNS records, as described in the PEP 381 about mirroring infrastructure.
If you don’t want to rely on these, you could specify the list of mirrors you want to try by specifying the mirrors attribute. It’s a simple iterable:
>>> mirrors = ["http://first.mirror","http://second.mirror"]
>>> client = Crawler(mirrors=mirrors)
It’s possible to search for projects with specific names in the package index. Assuming you want to find all projects containing the “distutils” keyword:
>>> c.search_projects("distutils")
[<Project "collective.recipe.distutils">, <Project "Distutils">, <Project
"Packaging">, <Project "distutilscross">, <Project "lpdistutils">, <Project
"taras.recipe.distutils">, <Project "zerokspot.recipe.distutils">]
You can also search the projects starting with a specific text, or ending with that text, using a wildcard:
>>> c.search_projects("distutils*")
[<Project "Distutils">, <Project "Packaging">, <Project "distutilscross">]
>>> c.search_projects("*distutils")
[<Project "collective.recipe.distutils">, <Project "Distutils">, <Project
"lpdistutils">, <Project "taras.recipe.distutils">, <Project
"zerokspot.recipe.distutils">]