13.3. robotparser
— Parser for robots.txt¶
注解
The robotparser
module has been renamed urllib.robotparser
in
Python 3.
The 2to3 tool will automatically adapt imports when converting
your sources to Python 3.
This module provides a single class, RobotFileParser
, which answers
questions about whether or not a particular user agent can fetch a URL on the
Web site that published the robots.txt
file. For more details on the
structure of robots.txt
files, see http://www.robotstxt.org/orig.html.
-
class
robotparser.
RobotFileParser
(url='')¶ This class provides methods to read, parse and answer questions about the
robots.txt
file at url.-
set_url
(url)¶ Sets the URL referring to a
robots.txt
file.
-
read
()¶ Reads the
robots.txt
URL and feeds it to the parser.
-
parse
(lines)¶ Parses the lines argument.
-
can_fetch
(useragent, url)¶ Returns
True
if the useragent is allowed to fetch the url according to the rules contained in the parsedrobots.txt
file.
-
mtime
()¶ Returns the time the
robots.txt
file was last fetched. This is useful for long-running web spiders that need to check for newrobots.txt
files periodically.
-
modified
()¶ Sets the time the
robots.txt
file was last fetched to the current time.
-
The following example demonstrates basic use of the RobotFileParser class.
>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://www.musi-cal.com/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
False
>>> rp.can_fetch("*", "http://www.musi-cal.com/")
True