pickle
--- Python object serialization¶
Source code: Lib/pickle.py
The pickle
module implements binary protocols for serializing and
de-serializing a Python object structure. "Pickling" is the process
whereby a Python object hierarchy is converted into a byte stream, and
"unpickling" is the inverse operation, whereby a byte stream
(from a binary file or bytes-like object) is converted
back into an object hierarchy. Pickling (and unpickling) is alternatively
known as "serialization", "marshalling," 1 or "flattening"; however, to
avoid confusion, the terms used here are "pickling" and "unpickling".
警告
pickle
模块**并不安全**。你只应该对你信任的数据进行unpickle操作。
构建恶意的 pickle 数据来**在解封时执行任意代码**是可能的。绝对不要对不信任来源的数据和可能被篡改过的数据进行解封。
请考虑使用 hmac
来对数据进行签名,确保数据没有被篡改。
在你处理不信任数据时,更安全的序列化格式如 json
可能更为适合。参见 Comparison with json 。
Relationship to other Python modules¶
Comparison with marshal
¶
Python has a more primitive serialization module called marshal
, but in
general pickle
should always be the preferred way to serialize Python
objects. marshal
exists primarily to support Python's .pyc
files.
The pickle
module differs from marshal
in several significant ways:
The
pickle
module keeps track of the objects it has already serialized, so that later references to the same object won't be serialized again.marshal
doesn't do this.This has implications both for recursive objects and object sharing. Recursive objects are objects that contain references to themselves. These are not handled by marshal, and in fact, attempting to marshal recursive objects will crash your Python interpreter. Object sharing happens when there are multiple references to the same object in different places in the object hierarchy being serialized.
pickle
stores such objects only once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects.marshal
cannot be used to serialize user-defined classes and their instances.pickle
can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.同样用于序列化的
marshal
格式不保证数据能移植到不同的 Python 版本中。因为它的主要任务是支持.pyc
文件,必要时会以破坏向后兼容的方式更改这种序列化格式,为此 Python 的实现者保留了更改格式的权利。pickle
序列化格式可以在不同版本的 Python 中实现向后兼容,前提是选择了合适的 pickle 协议。如果你的数据要在 Python 2 与 Python 3 之间跨越传递,封存和解封的代码在 2 和 3 之间也是不同的。
Comparison with json
¶
There are fundamental differences between the pickle protocols and JSON (JavaScript Object Notation):
JSON is a text serialization format (it outputs unicode text, although most of the time it is then encoded to
utf-8
), while pickle is a binary serialization format;JSON is human-readable, while pickle is not;
JSON is interoperable and widely used outside of the Python ecosystem, while pickle is Python-specific;
默认情况下,JSON 只能表示 Python 内置类型的子集,不能表示自定义的类;但 pickle 可以表示大量的 Python 数据类型(可以合理使用 Python 的对象内省功能自动地表示大多数类型,复杂情况可以通过实现 specific object APIs 来解决)。
不像pickle,对一个不信任的JSON进行反序列化的操作本身不会造成任意代码执行漏洞。
也參考
The json
module: a standard library module allowing JSON
serialization and deserialization.
Data stream format¶
The data format used by pickle
is Python-specific. This has the
advantage that there are no restrictions imposed by external standards such as
JSON or XDR (which can't represent pointer sharing); however it means that
non-Python programs may not be able to reconstruct pickled Python objects.
By default, the pickle
data format uses a relatively compact binary
representation. If you need optimal size characteristics, you can efficiently
compress pickled data.
The module pickletools
contains tools for analyzing data streams
generated by pickle
. pickletools
source code has extensive
comments about opcodes used by pickle protocols.
There are currently 5 different protocols which can be used for pickling. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.
Protocol version 0 is the original "human-readable" protocol and is backwards compatible with earlier versions of Python.
Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.
Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes. Refer to PEP 307 for information about improvements brought by protocol 2.
v3 版协议是在 Python 3.0 中引入的。 它显式地支持
bytes
字节对象,不能使用 Python 2.x 解封。这是 Python 3.0-3.7 的默认协议。v4 版协议添加于 Python 3.4。它支持存储非常大的对象,能存储更多种类的对象,还包括一些针对数据格式的优化。它是Python 3.8使用的默认协议。有关第 4 版协议带来改进的信息,请参阅 PEP 3154。
備註
Serialization is a more primitive notion than persistence; although
pickle
reads and writes file objects, it does not handle the issue of
naming persistent objects, nor the (even more complicated) issue of concurrent
access to persistent objects. The pickle
module can transform a complex
object into a byte stream and it can transform the byte stream into an object
with the same internal structure. Perhaps the most obvious thing to do with
these byte streams is to write them onto a file, but it is also conceivable to
send them across a network or store them in a database. The shelve
module provides a simple interface to pickle and unpickle objects on
DBM-style database files.
Module Interface¶
To serialize an object hierarchy, you simply call the dumps()
function.
Similarly, to de-serialize a data stream, you call the loads()
function.
However, if you want more control over serialization and de-serialization,
you can create a Pickler
or an Unpickler
object, respectively.
The pickle
module provides the following constants:
-
pickle.
HIGHEST_PROTOCOL
¶ An integer, the highest protocol version available. This value can be passed as a protocol value to functions
dump()
anddumps()
as well as thePickler
constructor.
-
pickle.
DEFAULT_PROTOCOL
¶ 整数,用于 pickle 数据的默认 协议版本。它可能小于
HIGHEST_PROTOCOL
。当前默认协议是 v4,它在 Python 3.4 中首次引入,与之前的版本不兼容。3.0 版更變: 默认协议版本是 3。
3.8 版更變: 默认协议版本是 4。
The pickle
module provides the following functions to make the pickling
process more convenient:
-
pickle.
dump
(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)¶ 将对象 obj 封存以后的对象写入已打开的 file object file。它等同于
Pickler(file, protocol).dump(obj)
。参数 file、protocol、fix_imports 和 buffer_callback 的含义与它们在
Pickler
的构造函数中的含义相同。3.8 版更變: 加入了 buffer_callback 参数。
-
pickle.
dumps
(obj, protocol=None, *, fix_imports=True, buffer_callback=None)¶ 将 obj 封存以后的对象作为
bytes
类型直接返回,而不是将其写入到文件。参数 protocol、fix_imports 和 buffer_callback 的含义与它们在
Pickler
的构造函数中的含义相同。3.8 版更變: 加入了 buffer_callback 参数。
-
pickle.
load
(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)¶ 从已打开的 file object 文件 中读取封存后的对象,重建其中特定对象的层次结构并返回。它相当于
Unpickler(file).load()
。Pickle 协议版本是自动检测出来的,所以不需要参数来指定协议。封存对象以外的其他字节将被忽略。
参数 file、fix_imports、encoding、errors、strict 和 buffers 的含义与它们在
Unpickler
的构造函数中的含义相同。3.8 版更變: 加入了 buffers 参数。
-
pickle.
loads
(bytes_object, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)¶ 对于封存生成的对象 bytes_object,还原出原对象的结构并返回。
Pickle 协议版本是自动检测出来的,所以不需要参数来指定协议。封存对象以外的其他字节将被忽略。
参数 file、fix_imports、encoding、errors、strict 和 buffers 的含义与它们在
Unpickler
的构造函数中的含义相同。3.8 版更變: 加入了 buffers 参数。
The pickle
module defines three exceptions:
-
exception
pickle.
PickleError
¶ Common base class for the other pickling exceptions. It inherits
Exception
.
-
exception
pickle.
PicklingError
¶ Error raised when an unpicklable object is encountered by
Pickler
. It inheritsPickleError
.Refer to What can be pickled and unpickled? to learn what kinds of objects can be pickled.
-
exception
pickle.
UnpicklingError
¶ Error raised when there is a problem unpickling an object, such as a data corruption or a security violation. It inherits
PickleError
.Note that other exceptions may also be raised during unpickling, including (but not necessarily limited to) AttributeError, EOFError, ImportError, and IndexError.
pickle
模块包含了 3 个类,Pickler
、Unpickler
和 PickleBuffer
:
-
class
pickle.
Pickler
(file, protocol=None, *, fix_imports=True, buffer_callback=None)¶ This takes a binary file for writing a pickle data stream.
The optional protocol argument, an integer, tells the pickler to use the given protocol; supported protocols are 0 to
HIGHEST_PROTOCOL
. If not specified, the default isDEFAULT_PROTOCOL
. If a negative number is specified,HIGHEST_PROTOCOL
is selected.The file argument must have a write() method that accepts a single bytes argument. It can thus be an on-disk file opened for binary writing, an
io.BytesIO
instance, or any other custom object that meets this interface.If fix_imports is true and protocol is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.
如果 buffer_callback 为 None(默认情况),缓冲区视图(buffer view)将会作为 pickle 流的一部分被序列化到 file 中。
如果 buffer_callback 不为 None,那它可以用缓冲区视图调用任意次。如果某次调用返回了 False 值(例如 None),则给定的缓冲区是 带外的,否则缓冲区是带内的(例如保存在了 pickle 流里面)。
如果 buffer_callback 不是 None 且 protocol 是 None 或小于 5,就会出错。
3.8 版更變: 加入了 buffer_callback 参数。
-
dump
(obj)¶ 将 obj 封存后的内容写入已打开的文件对象,该文件对象已经在构造函数中指定。
-
persistent_id
(obj)¶ Do nothing by default. This exists so a subclass can override it.
If
persistent_id()
returnsNone
, obj is pickled as usual. Any other value causesPickler
to emit the returned value as a persistent ID for obj. The meaning of this persistent ID should be defined byUnpickler.persistent_load()
. Note that the value returned bypersistent_id()
cannot itself have a persistent ID.See Persistence of External Objects for details and examples of uses.
-
dispatch_table
¶ A pickler object's dispatch table is a registry of reduction functions of the kind which can be declared using
copyreg.pickle()
. It is a mapping whose keys are classes and whose values are reduction functions. A reduction function takes a single argument of the associated class and should conform to the same interface as a__reduce__()
method.By default, a pickler object will not have a
dispatch_table
attribute, and it will instead use the global dispatch table managed by thecopyreg
module. However, to customize the pickling for a specific pickler object one can set thedispatch_table
attribute to a dict-like object. Alternatively, if a subclass ofPickler
has adispatch_table
attribute then this will be used as the default dispatch table for instances of that class.See Dispatch Tables for usage examples.
3.3 版新加入.
-
reducer_override
(self, obj)¶ 可以在
Pickler
的子类中定义的特殊 reducer。此方法的优先级高于dispatch_table
中的任何 reducer。它应该与__reduce__()
方法遵循相同的接口,它也可以返回NotImplemented
,这将使用dispatch_table
里注册的 reducer 来封存obj
。参阅 类型,函数和其他对象的自定义归约 获取详细的示例。
3.8 版新加入.
-
fast
¶ Deprecated. Enable fast mode if set to a true value. The fast mode disables the usage of memo, therefore speeding the pickling process by not generating superfluous PUT opcodes. It should not be used with self-referential objects, doing otherwise will cause
Pickler
to recurse infinitely.Use
pickletools.optimize()
if you need more compact pickles.
-
-
class
pickle.
Unpickler
(file, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)¶ This takes a binary file for reading a pickle data stream.
The protocol version of the pickle is detected automatically, so no protocol argument is needed.
参数 file 必须有三个方法,read() 方法接受一个整数参数,readinto() 方法接受一个缓冲区作为参数,readline() 方法不需要参数,这与
io.BufferedIOBase
里定义的接口是相同的。因此 file 可以是一个磁盘上用于二进制读取的文件,也可以是一个io.BytesIO
实例,也可以是满足这一接口的其他任何自定义对象。可选的参数是 fix_imports, encoding 和 errors,用于控制由Python 2 生成的 pickle 流的兼容性。如果 fix_imports 为 True,则 pickle 将尝试将旧的 Python 2 名称映射到 Python 3 中对应的新名称。encoding 和 errors 参数告诉 pickle 如何解码 Python 2 存储的 8 位字符串实例;这两个参数默认分别为 'ASCII' 和 'strict'。encoding 参数可置为 'bytes' 来将这些 8 位字符串实例读取为字节对象。读取 NumPy array 和 Python 2 存储的
datetime
、date
和time
实例时,请使用encoding='latin1'
。如果 buffers 为 None(默认值),则反序列化所需的所有数据都必须包含在 pickle 流中。这意味着在实例化
Pickler
时(或调用dump()
或dumps()
时),参数 buffer_callback 为 None。如果 buffers 不为 None,则每次 pickle 流引用 带外 缓冲区视图时,消耗的对象都应该是可迭代的启用缓冲区的对象。这样的缓冲区应该按顺序地提供给 Pickler 对象的 buffer_callback 方法。
3.8 版更變: 加入了 buffers 参数。
-
load
()¶ 从构造函数中指定的文件对象里读取封存好的对象,重建其中特定对象的层次结构并返回。封存对象以外的其他字节将被忽略。
-
persistent_load
(pid)¶ Raise an
UnpicklingError
by default.If defined,
persistent_load()
should return the object specified by the persistent ID pid. If an invalid persistent ID is encountered, anUnpicklingError
should be raised.See Persistence of External Objects for details and examples of uses.
-
find_class
(module, name)¶ Import module if necessary and return the object called name from it, where the module and name arguments are
str
objects. Note, unlike its name suggests,find_class()
is also used for finding functions.Subclasses may override this to gain control over what type of objects and how they can be loaded, potentially reducing security risks. Refer to Restricting Globals for details.
使用
module
,name
参数会引发 审核事件pickle.find_class
。
-
-
class
pickle.
PickleBuffer
(buffer)¶ 缓冲区的包装器 (wrapper),缓冲区中包含着可封存的数据。buffer 必须是一个 buffer-providing 对象,比如 bytes-like object 或多维数组。
PickleBuffer
本身就可以生成缓冲区对象,因此可以将其传递给需要缓冲区生成器的其他 API,比如memoryview
。PickleBuffer
对象只能用 pickle 版本 5 及以上协议进行序列化。它们符合 带外序列化 的条件。3.8 版新加入.
-
raw
()¶ 返回该缓冲区底层内存区域的
memoryview
。 返回的对象是一维的、C 连续布局的 memoryview,格式为B
(无符号字节)。 如果缓冲区既不是 C 连续布局也不是 Fortran 连续布局的,则抛出BufferError
异常。
-
release
()¶ 释放由 PickleBuffer 占用的底层缓冲区。
-
What can be pickled and unpickled?¶
The following types can be pickled:
None
,True
, andFalse
integers, floating point numbers, complex numbers
strings, bytes, bytearrays
tuples, lists, sets, and dictionaries containing only picklable objects
functions defined at the top level of a module (using
def
, notlambda
)built-in functions defined at the top level of a module
classes that are defined at the top level of a module
instances of such classes whose
__dict__
or the result of calling__getstate__()
is picklable (see section Pickling Class Instances for details).
Attempts to pickle unpicklable objects will raise the PicklingError
exception; when this happens, an unspecified number of bytes may have already
been written to the underlying file. Trying to pickle a highly recursive data
structure may exceed the maximum recursion depth, a RecursionError
will be
raised in this case. You can carefully raise this limit with
sys.setrecursionlimit()
.
Note that functions (built-in and user-defined) are pickled by "fully qualified" name reference, not by value. 2 This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. 3
Similarly, classes are pickled by named reference, so the same restrictions in
the unpickling environment apply. Note that none of the class's code or data is
pickled, so in the following example the class attribute attr
is not
restored in the unpickling environment:
class Foo:
attr = 'A class attribute'
picklestring = pickle.dumps(Foo)
These restrictions are why picklable functions and classes must be defined in the top level of a module.
Similarly, when class instances are pickled, their class's code and data are not
pickled along with them. Only the instance data are pickled. This is done on
purpose, so you can fix bugs in a class or add methods to the class and still
load objects that were created with an earlier version of the class. If you
plan to have long-lived objects that will see many versions of a class, it may
be worthwhile to put a version number in the objects so that suitable
conversions can be made by the class's __setstate__()
method.
Pickling Class Instances¶
In this section, we describe the general mechanisms available to you to define, customize, and control how class instances are pickled and unpickled.
In most cases, no additional code is needed to make instances picklable. By
default, pickle will retrieve the class and the attributes of an instance via
introspection. When a class instance is unpickled, its __init__()
method
is usually not invoked. The default behaviour first creates an uninitialized
instance and then restores the saved attributes. The following code shows an
implementation of this behaviour:
def save(obj):
return (obj.__class__, obj.__dict__)
def load(cls, attributes):
obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj
Classes can alter the default behaviour by providing one or several special methods:
-
object.
__getnewargs_ex__
()¶ In protocols 2 and newer, classes that implements the
__getnewargs_ex__()
method can dictate the values passed to the__new__()
method upon unpickling. The method must return a pair(args, kwargs)
where args is a tuple of positional arguments and kwargs a dictionary of named arguments for constructing the object. Those will be passed to the__new__()
method upon unpickling.You should implement this method if the
__new__()
method of your class requires keyword-only arguments. Otherwise, it is recommended for compatibility to implement__getnewargs__()
.3.6 版更變:
__getnewargs_ex__()
is now used in protocols 2 and 3.
-
object.
__getnewargs__
()¶ This method serves a similar purpose as
__getnewargs_ex__()
, but supports only positional arguments. It must return a tuple of argumentsargs
which will be passed to the__new__()
method upon unpickling.__getnewargs__()
will not be called if__getnewargs_ex__()
is defined.3.6 版更變: Before Python 3.6,
__getnewargs__()
was called instead of__getnewargs_ex__()
in protocols 2 and 3.
-
object.
__getstate__
()¶ Classes can further influence how their instances are pickled; if the class defines the method
__getstate__()
, it is called and the returned object is pickled as the contents for the instance, instead of the contents of the instance's dictionary. If the__getstate__()
method is absent, the instance's__dict__
is pickled as usual.
-
object.
__setstate__
(state)¶ Upon unpickling, if the class defines
__setstate__()
, it is called with the unpickled state. In that case, there is no requirement for the state object to be a dictionary. Otherwise, the pickled state must be a dictionary and its items are assigned to the new instance's dictionary.備註
If
__getstate__()
returns a false value, the__setstate__()
method will not be called upon unpickling.
Refer to the section Handling Stateful Objects for more information about how to use
the methods __getstate__()
and __setstate__()
.
備註
At unpickling time, some methods like __getattr__()
,
__getattribute__()
, or __setattr__()
may be called upon the
instance. In case those methods rely on some internal invariant being
true, the type should implement __getnewargs__()
or
__getnewargs_ex__()
to establish such an invariant; otherwise,
neither __new__()
nor __init__()
will be called.
As we shall see, pickle does not use directly the methods described above. In
fact, these methods are part of the copy protocol which implements the
__reduce__()
special method. The copy protocol provides a unified
interface for retrieving the data necessary for pickling and copying
objects. 4
Although powerful, implementing __reduce__()
directly in your classes is
error prone. For this reason, class designers should use the high-level
interface (i.e., __getnewargs_ex__()
, __getstate__()
and
__setstate__()
) whenever possible. We will show, however, cases where
using __reduce__()
is the only option or leads to more efficient pickling
or both.
-
object.
__reduce__
()¶ The interface is currently defined as follows. The
__reduce__()
method takes no argument and shall return either a string or preferably a tuple (the returned object is often referred to as the "reduce value").If a string is returned, the string should be interpreted as the name of a global variable. It should be the object's local name relative to its module; the pickle module searches the module namespace to determine the object's module. This behaviour is typically useful for singletons.
如果返回的是元组,则应当包含 2 到 6 个元素,可选元素可以省略或设置为
None
。每个元素代表的意义如下:A callable object that will be called to create the initial version of the object.
A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument.
Optionally, the object's state, which will be passed to the object's
__setstate__()
method as previously described. If the object has no such method then, the value must be a dictionary and it will be added to the object's__dict__
attribute.Optionally, an iterator (and not a sequence) yielding successive items. These items will be appended to the object either using
obj.append(item)
or, in batch, usingobj.extend(list_of_items)
. This is primarily used for list subclasses, but may be used by other classes as long as they haveappend()
andextend()
methods with the appropriate signature. (Whetherappend()
orextend()
is used depends on which pickle protocol version is used as well as the number of items to append, so both must be supported.)Optionally, an iterator (not a sequence) yielding successive key-value pairs. These items will be stored to the object using
obj[key] = value
. This is primarily used for dictionary subclasses, but may be used by other classes as long as they implement__setitem__()
.可选元素,一个带有
(obj, state)
签名的可调用对象。该可调用对象允许用户以编程方式控制特定对象的状态更新行为,而不是使用obj
的静态__setstate__()
方法。如果此处不是None
,则此可调用对象的优先级高于obj
的__setstate__()
。3.8 版新加入: 新增了元组的第 6 项,可选元素
(obj, state)
。
-
object.
__reduce_ex__
(protocol)¶ Alternatively, a
__reduce_ex__()
method may be defined. The only difference is this method should take a single integer argument, the protocol version. When defined, pickle will prefer it over the__reduce__()
method. In addition,__reduce__()
automatically becomes a synonym for the extended version. The main use for this method is to provide backwards-compatible reduce values for older Python releases.
Persistence of External Objects¶
For the benefit of object persistence, the pickle
module supports the
notion of a reference to an object outside the pickled data stream. Such
objects are referenced by a persistent ID, which should be either a string of
alphanumeric characters (for protocol 0) 5 or just an arbitrary object (for
any newer protocol).
pickle
模块不提供对持久化 ID 的解析工作,它将解析工作分配给用户定义的方法,分别是 pickler 中的 persistent_id()
方法和 unpickler 中的 persistent_load()
方法。
要通过持久化 ID 将外部对象封存,必须在 pickler 中实现 persistent_id()
方法,该方法接受需要被封存的对象作为参数,返回一个 None
或返回该对象的持久化 ID。如果返回 None
,该对象会被按照默认方式封存为数据流。如果返回字符串形式的持久化 ID,则会封存这个字符串并加上一个标记,这样 unpickler 才能将其识别为持久化 ID。
To unpickle external objects, the unpickler must have a custom
persistent_load()
method that takes a persistent ID object and
returns the referenced object.
Here is a comprehensive example presenting how persistent ID can be used to pickle external objects by reference.
# Simple example presenting how persistent ID can be used to pickle
# external objects by reference.
import pickle
import sqlite3
from collections import namedtuple
# Simple class representing a record in our database.
MemoRecord = namedtuple("MemoRecord", "key, task")
class DBPickler(pickle.Pickler):
def persistent_id(self, obj):
# Instead of pickling MemoRecord as a regular class instance, we emit a
# persistent ID.
if isinstance(obj, MemoRecord):
# Here, our persistent ID is simply a tuple, containing a tag and a
# key, which refers to a specific record in the database.
return ("MemoRecord", obj.key)
else:
# If obj does not have a persistent ID, return None. This means obj
# needs to be pickled as usual.
return None
class DBUnpickler(pickle.Unpickler):
def __init__(self, file, connection):
super().__init__(file)
self.connection = connection
def persistent_load(self, pid):
# This method is invoked whenever a persistent ID is encountered.
# Here, pid is the tuple returned by DBPickler.
cursor = self.connection.cursor()
type_tag, key_id = pid
if type_tag == "MemoRecord":
# Fetch the referenced record from the database and return it.
cursor.execute("SELECT * FROM memos WHERE key=?", (str(key_id),))
key, task = cursor.fetchone()
return MemoRecord(key, task)
else:
# Always raises an error if you cannot return the correct object.
# Otherwise, the unpickler will think None is the object referenced
# by the persistent ID.
raise pickle.UnpicklingError("unsupported persistent object")
def main():
import io
import pprint
# Initialize and populate our database.
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
cursor.execute("CREATE TABLE memos(key INTEGER PRIMARY KEY, task TEXT)")
tasks = (
'give food to fish',
'prepare group meeting',
'fight with a zebra',
)
for task in tasks:
cursor.execute("INSERT INTO memos VALUES(NULL, ?)", (task,))
# Fetch the records to be pickled.
cursor.execute("SELECT * FROM memos")
memos = [MemoRecord(key, task) for key, task in cursor]
# Save the records using our custom DBPickler.
file = io.BytesIO()
DBPickler(file).dump(memos)
print("Pickled records:")
pprint.pprint(memos)
# Update a record, just for good measure.
cursor.execute("UPDATE memos SET task='learn italian' WHERE key=1")
# Load the records from the pickle data stream.
file.seek(0)
memos = DBUnpickler(file, conn).load()
print("Unpickled records:")
pprint.pprint(memos)
if __name__ == '__main__':
main()
Dispatch Tables¶
If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.
The global dispatch table managed by the copyreg
module is
available as copyreg.dispatch_table
. Therefore, one may
choose to use a modified copy of copyreg.dispatch_table
as a
private dispatch table.
For example
f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass
creates an instance of pickle.Pickler
with a private dispatch
table which handles the SomeClass
class specially. Alternatively,
the code
class MyPickler(pickle.Pickler):
dispatch_table = copyreg.dispatch_table.copy()
dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)
does the same, but all instances of MyPickler
will by default
share the same dispatch table. The equivalent code using the
copyreg
module is
copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)
Handling Stateful Objects¶
Here's an example that shows how to modify pickling behavior for a class.
The TextReader
class opens a text file, and returns the line number and
line contents each time its readline()
method is called. If a
TextReader
instance is pickled, all attributes except the file object
member are saved. When the instance is unpickled, the file is reopened, and
reading resumes from the last location. The __setstate__()
and
__getstate__()
methods are used to implement this behavior.
class TextReader:
"""Print and number lines in a text file."""
def __init__(self, filename):
self.filename = filename
self.file = open(filename)
self.lineno = 0
def readline(self):
self.lineno += 1
line = self.file.readline()
if not line:
return None
if line.endswith('\n'):
line = line[:-1]
return "%i: %s" % (self.lineno, line)
def __getstate__(self):
# Copy the object's state from self.__dict__ which contains
# all our instance attributes. Always use the dict.copy()
# method to avoid modifying the original state.
state = self.__dict__.copy()
# Remove the unpicklable entries.
del state['file']
return state
def __setstate__(self, state):
# Restore instance attributes (i.e., filename and lineno).
self.__dict__.update(state)
# Restore the previously opened file's state. To do so, we need to
# reopen it and read from it until the line count is restored.
file = open(self.filename)
for _ in range(self.lineno):
file.readline()
# Finally, save the file.
self.file = file
A sample usage might be something like this:
>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'
类型,函数和其他对象的自定义归约¶
3.8 版新加入.
有时,dispatch_table
可能不够灵活。 特别是当我们想要基于对象类型以外的其他规则来对封存进行定制,或是当我们想要对函数和类的封存进行定制的时候。
对于那些情况,可能要基于 Pickler
类进行子类化并实现 reducer_override()
方法。 此方法可返回任意的归约元组 (参见 __reduce__()
)。 它也可以选择返回 NotImplemented
来回退到传统行为。
如果同时定义了 dispatch_table
和 reducer_override()
,则 reducer_override()
方法具有优先权。
備註
出于性能理由,可能不会为以下对象调用 reducer_override()
: None
, True
, False
, 以及 int
, float
, bytes
, str
, dict
, set
, frozenset
, list
和 tuple
的具体实例。
以下是一个简单的例子,其中我们允许封存并重新构建一个给定的类:
import io
import pickle
class MyClass:
my_attribute = 1
class MyPickler(pickle.Pickler):
def reducer_override(self, obj):
"""Custom reducer for MyClass."""
if getattr(obj, "__name__", None) == "MyClass":
return type, (obj.__name__, obj.__bases__,
{'my_attribute': obj.my_attribute})
else:
# For any other object, fallback to usual reduction
return NotImplemented
f = io.BytesIO()
p = MyPickler(f)
p.dump(MyClass)
del MyClass
unpickled_class = pickle.loads(f.getvalue())
assert isinstance(unpickled_class, type)
assert unpickled_class.__name__ == "MyClass"
assert unpickled_class.my_attribute == 1
外部缓冲区¶
3.8 版新加入.
在某些场景中,pickle
模块会被用来传输海量的数据。 因此,最小化内存复制次数以保证性能和节省资源是很重要的。 但是 pickle
模块的正常运作会将图类对象结构转换为字节序列流,因此在本质上就要从封存流中来回复制数据。
如果 provider (待传输对象类型的实现) 和 consumer (通信系统的实现) 都支持 pickle 第 5 版或更高版本所提供的外部传输功能,则此约束可以被撤销。
提供方 API¶
大的待封存数据对象必须实现协议 5 及以上版本专属的 __reduce_ex__()
方法,该方法将为任意大的数据返回一个 PickleBuffer
实例(而不是 bytes
对象等)。
PickleBuffer
对象会 表明 底层缓冲区可被用于外部数据传输。 那些对象仍将保持与 pickle
模块的正常用法兼容。 但是,使用方也可以选择告知 pickle
它们将自行处理那些缓冲区。
使用方 API¶
当序列化一个对象图时,通信系统可以启用对所生成 PickleBuffer
对象的定制处理。
发送端需要传递 buffer_callback 参数到 Pickler
(或是到 dump()
或 dumps()
函数),该回调函数将在封存对象图时附带每个所生成的 PickleBuffer
被调用。 由 buffer_callback 所累积的缓冲区的数据将不会被拷贝到 pickle 流,而是仅插入一个简单的标记。
接收端需要传递 buffers 参数到 Unpickler
(或是到 load()
或 loads()
函数),其值是一个由缓冲区组成的可迭代对象,它会被传递给 buffer_callback。 该可迭代对象应当按其被传递给 buffer_callback 时的顺序产生缓冲区。 这些缓冲区将提供对象重构造器所期望的数据,对这些数据的封存产生了原本的 PickleBuffer
对象。
在发送端和接受端之间,通信系统可以自由地实现它自己用于外部缓冲区的传输机制。 潜在的优化包括使用共享内存或基于特定数据类型的压缩等。
示例¶
下面是一个小例子,在其中我们实现了一个 bytearray
的子类,能够用于外部缓冲区封存:
class ZeroCopyByteArray(bytearray):
def __reduce_ex__(self, protocol):
if protocol >= 5:
return type(self)._reconstruct, (PickleBuffer(self),), None
else:
# PickleBuffer is forbidden with pickle protocols <= 4.
return type(self)._reconstruct, (bytearray(self),)
@classmethod
def _reconstruct(cls, obj):
with memoryview(obj) as m:
# Get a handle over the original buffer object
obj = m.obj
if type(obj) is cls:
# Original buffer object is a ZeroCopyByteArray, return it
# as-is.
return obj
else:
return cls(obj)
重构造器 (_reconstruct
类方法) 会在缓冲区的提供对象具有正确类型时返回该对象。 在此小示例中这是模拟零拷贝行为的便捷方式。
在使用方,我们可以按通常方式封存那些对象,它们在反序列化时将提供原始对象的一个副本:
b = ZeroCopyByteArray(b"abc")
data = pickle.dumps(b, protocol=5)
new_b = pickle.loads(data)
print(b == new_b) # True
print(b is new_b) # False: a copy was made
但是如果我们传入 buffer_callback 然后在反序列化时给回累积的缓冲区,我们就能够取回原始对象:
b = ZeroCopyByteArray(b"abc")
buffers = []
data = pickle.dumps(b, protocol=5, buffer_callback=buffers.append)
new_b = pickle.loads(data, buffers=buffers)
print(b == new_b) # True
print(b is new_b) # True: no copy was made
这个例子受限于 bytearray
会自行分配内存这一事实:你无法基于另一个对象的内存创建 bytearray
的实例。 但是,第三方数据类型例如 NumPy 数组则没有这种限制,允许在单独进程或系统间传输时使用零拷贝的封存(或是尽可能少地拷贝) 。
也參考
PEP 574 -- 带有外部数据缓冲区的 pickle 协议 5
Restricting Globals¶
By default, unpickling will import any class or function that it finds in the pickle data. For many applications, this behaviour is unacceptable as it permits the unpickler to import and invoke arbitrary code. Just consider what this hand-crafted pickle data stream does when loaded:
>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0
In this example, the unpickler imports the os.system()
function and then
apply the string argument "echo hello world". Although this example is
inoffensive, it is not difficult to imagine one that could damage your system.
For this reason, you may want to control what gets unpickled by customizing
Unpickler.find_class()
. Unlike its name suggests,
Unpickler.find_class()
is called whenever a global (i.e., a class or
a function) is requested. Thus it is possible to either completely forbid
globals or restrict them to a safe subset.
Here is an example of an unpickler allowing only few safe classes from the
builtins
module to be loaded:
import builtins
import io
import pickle
safe_builtins = {
'range',
'complex',
'set',
'frozenset',
'slice',
}
class RestrictedUnpickler(pickle.Unpickler):
def find_class(self, module, name):
# Only allow safe classes from builtins.
if module == "builtins" and name in safe_builtins:
return getattr(builtins, name)
# Forbid everything else.
raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
(module, name))
def restricted_loads(s):
"""Helper function analogous to pickle.loads()."""
return RestrictedUnpickler(io.BytesIO(s)).load()
A sample usage of our unpickler working has intended:
>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
... b'(S\'getattr(__import__("os"), "system")'
... b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
...
pickle.UnpicklingError: global 'builtins.eval' is forbidden
As our examples shows, you have to be careful with what you allow to be
unpickled. Therefore if security is a concern, you may want to consider
alternatives such as the marshalling API in xmlrpc.client
or
third-party solutions.
Performance¶
Recent versions of the pickle protocol (from protocol 2 and upwards) feature
efficient binary encodings for several common features and built-in types.
Also, the pickle
module has a transparent optimizer written in C.
Examples¶
For the simplest code, use the dump()
and load()
functions.
import pickle
# An arbitrary collection of objects supported by pickle.
data = {
'a': [1, 2.0, 3, 4+6j],
'b': ("character string", b"byte string"),
'c': {None, True, False}
}
with open('data.pickle', 'wb') as f:
# Pickle the 'data' dictionary using the highest protocol available.
pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
The following example reads the resulting pickled data.
import pickle
with open('data.pickle', 'rb') as f:
# The protocol version used is detected automatically, so we do not
# have to specify it.
data = pickle.load(f)
也參考
- Module
copyreg
Pickle interface constructor registration for extension types.
- Module
pickletools
Tools for working with and analyzing pickled data.
- Module
shelve
Indexed databases of objects; uses
pickle
.- Module
copy
Shallow and deep object copying.
- Module
marshal
High-performance serialization of built-in types.
註解
- 1
Don't confuse this with the
marshal
module- 2
这就是为什么
lambda
函数不可以被封存:所有的匿名函数都有同一个名字:<lambda>
。- 3
The exception raised will likely be an
ImportError
or anAttributeError
but it could be something else.- 4
The
copy
module uses this protocol for shallow and deep copying operations.- 5
The limitation on alphanumeric characters is due to the fact the persistent IDs, in protocol 0, are delimited by the newline character. Therefore if any kind of newline characters occurs in persistent IDs, the resulting pickle will become unreadable.