Python引用計數(shù)(Reference Count)

Summary

引用計數(shù)記錄指向?qū)ο笠玫膫€數(shù)，當變?yōu)?，則被釋放。總結(jié)了引用計數(shù)的注意點和如何使用。更新：weakref(弱引用)、用弱引用解決引用環(huán)問題

簡介

It counts how many different places there are that have a reference to an object.When an object’s reference count becomes zero, the object is deallocated

記錄指向?qū)ο笠玫膫€數(shù)，當變?yōu)?，則被釋放

使用引用計數(shù)的實質(zhì)

The only real reason to use the reference count is to prevent the object from being deallocated as long as our variable is pointing to it

使用引用計數(shù)的唯一理由就是只要還有變量指向就應當阻止對象被釋放

引用計數(shù)的實現(xiàn)

  • PyObject* 是什么?
  • 引用計數(shù)變量 ob_refcnt
  • 如何操作ob_refcnt (Py_INCREF and Py_DECREF)

0x00. PyObject* 是什么?

This type is a pointer to an opaque data type representing an arbitrary Python object. Since all Python object types are treated the same way by the Python language in most situations (e.g., assignments, scope rules, and argument passing), it is only fitting that they should be represented by a single C type. Almost all Python objects live on the heap: you never declare an automatic or static variable of type PyObject, only pointer variables of type *PyObject **can be declared.

這種數(shù)據(jù)類型是可以表示任意Python對象的封裝數(shù)據(jù)類型，因此Python對象類型在大多數(shù)情況下以相同方式處理。幾乎全部的Python的對象存儲在heap堆(由程序員分配malloc)，因此不可以聲明一個PyObject的類型對象(局部變量值保存在stack)，只能是*PyObject **

0x01. 引用計數(shù)變量 **ob_refcnt** 

typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
} PyObject;

0x02. 如何操作ob_refcnt (Py_INCREF and Py_DECREF) 

#define Py_INCREF(op) (                         \
    _Py_INC_REFTOTAL  _Py_REF_DEBUG_COMMA       \
    ((PyObject *)(op))->ob_refcnt++)

#define Py_DECREF(op)                                   \
    do {                                                \
        PyObject *_py_decref_tmp = (PyObject *)(op);    \
        if (_Py_DEC_REFTOTAL  _Py_REF_DEBUG_COMMA       \
        --(_py_decref_tmp)->ob_refcnt != 0)             \
            _Py_CHECK_REFCNT(_py_decref_tmp)            \
        else                                            \
        _Py_Dealloc(_py_decref_tmp);                    \
    } while (0)

什么時候操作引用計數(shù)

僅有當你需要保護這個變量不被釋放時才使用INCREF

具體：

  • 創(chuàng)建一個Object* 對象
  • 處理函數(shù)返回的對象
  • 借用引用(borrow)
  • 偷取引用(Steal)

注: 不需要對每個本地變量(stack變量)的引用+1，因為當一個變量創(chuàng)建并且有一個指針指向時，默認INC，然而當變量失去作用范圍(stack棧)又會DEC，兩者抵消。

0x00. 創(chuàng)建一個Object* 對象

Example1: 源碼分析

0x0000.針對long類型變量分析

PyObject *l, *x;
x = PyLong_FromLong(1L);

0x0001. PyLong_FromLong() [longobject.c]

調(diào)用_PyLong_New()創(chuàng)建新long對象

PyObject *
PyLong_FromLong(long ival)
{
    PyLongObject *v;
    ...
    //something done
        v = _PyLong_New(1);
    ...
    //something done
    return (PyObject *)v;
}

0x0002. _PyLong_New() [longobject.c]

調(diào)用PyObject_MALLOC()分配內(nèi)存，調(diào)用PyObject_INIT_VAR()初始化為PyLongOject*類型，完成PyObject*相關項的初始化，比如類型項等等。

注：關于PyObject_MALLOC()[obmalloc.c]，采用內(nèi)存池進行內(nèi)存管理，此處不詳細介紹
Python內(nèi)存管理

PyLongObject *
_PyLong_New(Py_ssize_t size)
{
    PyLongObject *result;
    if (size > (Py_ssize_t)MAX_LONG_DIGITS) {
        PyErr_SetString(PyExc_OverflowError,
                        "too many digits in integer");
        return NULL;
    }
    result = PyObject_MALLOC(offsetof(PyLongObject, ob_digit) +
                             size*sizeof(digit));
    if (!result) {
        PyErr_NoMemory();
        return NULL;
    }
    return (PyLongObject*)PyObject_INIT_VAR(result, &PyLong_Type, size);
}

0x0003. PyObject_INIT_VAR() [objimp.h]

#define PyObject_INIT(op, typeobj) \
    ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) )
#define PyObject_INIT_VAR(op, typeobj, size) \
    ( Py_SIZE(op) = (size), PyObject_INIT((op), (typeobj)) )

0x0004. 宏展開(Macro Expansion) Py_TYPE() _Py_NewReference() [object.h]

最終發(fā)現(xiàn)調(diào)用創(chuàng)建PyObject*變量時 初始化op_refcnt為1

#define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)
#define Py_REFCNT(ob)           (((PyObject*)(ob))->ob_refcnt)
#define _Py_NewReference(op) (                          \
    _Py_INC_TPALLOCS(op) _Py_COUNT_ALLOCS_COMMA         \
    _Py_INC_REFTOTAL  _Py_REF_DEBUG_COMMA               \
    Py_REFCNT(op) = 1)

0x01. 處理函數(shù)返回的對象

很多函數(shù)在返回之前會調(diào)用Py_INCREF()，因此該函數(shù)的caller需要調(diào)用Py_DECREF(),以防內(nèi)存泄露(memory leak)

Example1: MyCode必須處理pyo，調(diào)用Py_DECREF

void MyCode(arguments)
{
    PyObject* pyo;
    ...
    pyo = Py_Something(args);
    //Py_DECREF(pyo);
}

Example2: 如果MyCode傳遞pyo的所有權，則不能調(diào)用Py_DECREF

PyObject* MyCode(arguments) {
    PyObject* pyo;
    ...
    pyo = Py_Something(args);
    ...
    return pyo;
}

注: 函數(shù)返回None則返回之前需要Py_INCREF(Py_None)

Py_INCREF(Py_None) from stackoverflow

Py_INCREF(Py_None);
return Py_None;

0x02. 借用引用(borrow)

僅獲得拷貝，引用計數(shù)不增加

產(chǎn)生借用:

  • 返回借用引用對象的函數(shù)[borrow]
  • 傳遞給函數(shù)的對象[borrow]

0x000. 返回借用引用對象的函數(shù)[borrow]

  • PyTuple_GetItem()
  • PyList_GetItem()
  • PyList_GET_ITEM()
  • PyList_SET_ITEM()
  • PyDict_GetItem()
  • PyDict_GetItemString()
  • PyErr_Occurred()
  • PyFile_Name()
  • PyImport_GetModuleDict()
  • PyModule_GetDict()
  • PyImport_AddModule()
  • PyObject_Init()
  • Py_InitModule()
  • Py_InitModule3()
  • Py_InitModule4()
  • PySequence_Fast_GET_ITEM()

Example1: PyList_GetItem僅獲得對應項目拷貝，不增加引用計數(shù)

long sum_list(PyObject *list)
{
 int i, n;
 long total = 0;
 PyObject *item;

 n = PyList_Size(list);
 if (n < 0)
     return -1; /* Not a list */
     /* Caller should use PyErr_Occurred() if a -1 is returned. */
 for (i = 0; i < n; i++) {
     /* PyList_GetItem does not INCREF "item".
        "item" is unprotected and borrowed. IMPORTANT!!! */
     item = PyList_GetItem(list, i); /* Can't fail */
     if (PyInt_Check(item))
         total += PyInt_AsLong(item);
 }
 return total;
}

Example2: PySequence_GetItem獲得對象所有權，其返回對象+1，因而每次循環(huán)后需要-1.

long sum_sequence(PyObject *sequence)
{
 int i, n;
 long total = 0;
 PyObject *item;
 n = PySequence_Length(sequence);
 if (n < 0)
     return -1; /* Has no length. */
     /* Caller should use PyErr_Occurred() if a -1 is returned. */
 for (i = 0; i < n; i++) {
     /* PySequence_GetItem INCREFs item.  IMPORTANT!!!*/
     item = PySequence_GetItem(sequence, i);
     if (item == NULL)
         return -1; /* Not a sequence, or other failure */
     if (PyInt_Check(item))
         total += PyInt_AsLong(item);
     Py_DECREF(item);
 }
 return total;
}

0x001. 傳遞給函數(shù)的對象[borrow]

Most functions assume that the arguments passed to them are already protected.Therefore Py_INCREF() is not called inside Function unless Function wants the argument to continue to exist after Caller exits. In the documentation, Function is said to borrow a reference:
大多數(shù)函數(shù)假定傳入函數(shù)的參數(shù)都是受保護的不需要INCREF，除非希望參數(shù)在函數(shù)exit后仍然存在。官方文檔說法是函數(shù)借用引用

When you pass an object reference into another function, in general, the function borrows the reference from you if it needs to store it, it will use Py_INCREF() to become an independent owner.
你傳遞對象給一個函數(shù)，一般情況來說是函數(shù)借用引用，如果你希望保存那么請INCREF，將其變?yōu)楠毩碛姓摺?/strong>

PyTuple_SetItem()和PyList_SetItem()除外，它們接管傳入對象所有權(take over responsibility) or 偷取引用(steal a reference)
詳細見下

0x03. 偷取引用(Steal)

PyTuple_SetItem(tuple,i,item)和PyList_SetItem()接管所有權(take over responsibility) or 偷取引用(steal a reference) item引用，but not to the tuple or list into which the item is put，即僅僅偷取item引用.

  • PyDict_SetItem()非借用，既然是store變量到dict，因此PyDict_SetItem() INCREF它的kye和value
  • 但是PyTuple_SetItem()和PyList_SetItem()比較特殊，接管所有權(take over responsibility) or 偷取引用(steal a reference)
  • PyTuple_SetItem(tuple,i,item)實現(xiàn)：如果tuple[i]存在PyObject則DECREF，然后tuple[i]設置為item。并且Item并沒有INCREF
  • PyTuple_SetItem(tuple,i,item)既然是steal，那么Item之前必須有所有權
  • 如果PyTuple_SetItem()插入item失敗，則DECREF item引用計數(shù)
  • PyTuple_SetItem()是設置Tuple中item的唯一方法

Example1:

你不需要調(diào)用DECREF(x)，PyTuple_SetItem()已經(jīng)自動調(diào)用了,當Tuple被DECREF時，它的item也會被DECREF

PyObjetc *t;
PyObject *x;
x=PyIntFromLong(1L);
PyTuple_SetItem(t,0,x);

總結(jié)

  • 許多從其他對象上提取子對象的函數(shù)，通過引用傳遞所有權，但有一些例外，PyTuple_GetItem(),PyList_GetItem(),PyDict_GetItem()，和PyDict_GetItemString()，這些返回的引用是從tuple，list或dict中借用的.(借用僅獲得拷貝，引用計數(shù)并不增加)
  • 當你傳遞一個對象引用給其他函數(shù)，這個函數(shù)會從借用這個引用，如果需要保存它應該使用Py_INCREF()轉(zhuǎn)換為獨立擁有者。但是有例外，PyTuple_SetItem()和PyList_SetItem()，直接傳遞對象所有權
  • Python調(diào)用一個C函數(shù)的返回對象必須擁有引用所有權傳遞給它的調(diào)用者

常見問題

0x00. INCREF不可馬虎

常見的情況是從list中提取對象，一些操作符可以能會替換或者移除list中某個對象，并且假如這個向?qū)ο笫怯脩糇远x的calss，包含del，然而這個del可以執(zhí)行任意的code，但是這些操作可能會無意的DEC 該list[0]的引用計數(shù)，導致free。

bug(PyObject *list) {
/*item利用PyList_GetItem借用list引用*/
 PyObject *item = PyList_GetItem(list, 0);
 //修改措施Py_INCREF(item); /* Protect item. */
 /* This function “steals” a reference to item and discards a reference to an item already in the list at the affected position.
 可能引起list中原list[1]中__del__，導致DEClist[0]*/
 PyList_SetItem(list, 1, PyInt_FromLong(0L));
 PyObject_Print(item, stdout, 0); /* BUG! */
 //修改措施:Py_DECREF(item);
}

0x01. 偷取和借用對比在build list or tuple方面

Example1: steal a referfence(take over responsibilty)

PyObjetc *t;
PyObject *x;
x=PyIntFromLong(1L);
PyTuple_SetItem(t,0,x);
//Dont't Need Py_DECREF()

Example2 borrow a reference

/*Better way*/

PyObject *l, *x;
l = PyList_New(3);
x = PyInt_FromLong(1L);
PySequence_SetItem(l, 0, x); Py_DECREF(x);
x = PyInt_FromLong(2L);
PySequence_SetItem(l, 1, x); Py_DECREF(x);
x = PyString_FromString("three");
PySequence_SetItem(l, 2, x); Py_DECREF(x);

注: 更常見的創(chuàng)建list和tuple的方法

More Common Way to Populate a tuple or list

PyObject *t, *l; 
t = Py_BuildValue("(iis)", 1, 2, "three");
l = Py_BuildValue("[iis]", 1, 2, "three");

Refer:

Reference Counting in Python

Extending Python with C or C++

Two Examples

Example 1

This is a pretty standard example of C code using the Python API.

PyObject*
    MyFunction(void)
    {
        PyObject* temporary_list=NULL;
        PyObject* return_this=NULL;

        temporary_list = PyList_New(1);          /* Note 1 */
        if (temporary_list == NULL)
            return NULL;

        return_this = PyList_New(1);             /* Note 1 */
        if (return_this == NULL)
            Py_DECREF(temporary_list);           /* Note 2 */
            return NULL;
        }

        Py_DECREF(temporary_list);               /* Note 2 */
        return return_this;
    }
  • Note 1: The object returned by PyList_New has a reference count of 1.
  • Note 2: Since temporary_list should disappear when MyFunction exits, it must be DECREFed before any return from the function. If a return can be reached both before or after temporary_list is created, then initialize temporary_list to NULL and use Py_XDECREF().

Example 2

This is the same as Example 1 except PyTuple_GetItem() is used.
    PyObject*
    MyFunction(void)
    {
        PyObject* temporary=NULL;
        PyObject* return_this=NULL;
        PyObject* tup;
        PyObject* num;
        int err;
        tup = PyTuple_New(2);
        if (tup == NULL)
            return NULL;
        err = PyTuple_SetItem(tup, 0, PyInt_FromLong(222L));
        /* Note 1 */
        if (err) {
            Py_DECREF(tup);
            return NULL;
        }
        err = PyTuple_SetItem(tup, 1, PyInt_FromLong(333L));
        /* Note 1 */
        if (err) {
            Py_DECREF(tup);
            return NULL;
        }
        temporary = PyTuple_Getitem(tup, 0);
        /* Note 2 */
        if (temporary == NULL) {
            Py_DECREF(tup);
            return NULL;
        }
        return_this = PyTuple_Getitem(tup, 1);
        /* Note 3 */
        if (return_this == NULL) {
            Py_DECREF(tup);
            /* Note 3 */
            return NULL;
        }
        /* Note 3 */
        Py_DECREF(tup);
        return return_this;
    }
  • Note 1: If PyTuple_SetItem fails or if the tuple it created is DECREFed to 0, then the object returned by PyInt_FromLong is DECREFed.
  • Note 2: PyTuple_Getitem does not increment the reference count for the object it returns.
  • Note 3: You have no responsibility for DECFREFing temporary.

更新

1. 弱引用

弱引用與強引用相對，指不能確保其引用對象不會被垃圾回收器回收，一個對象若只被弱引用所引用，則被認為是不可訪問的。 --wiki

弱引用解決引用環(huán)問題

# python2.x會出現(xiàn)，python3.x做了改進
class LeakTest(object):
   def __init__(self):
     print 'Object with id %d born here.' % id(self)
   def __del__(self):
     print 'Object with id %d dead here.' % id(self)

def foo():
   A = LeakTest()
   B = LeakTest()
   A.b = B
   B.a = A
if __name__ = ="__main__": 
  foo()
  
RESULT:
Object with id 10462448 born here.
Object with id 10462832 born here.

相互引用導致形成環(huán)，當對象只有被弱引用時，同樣會被回收,因此可做如下修改

import weakref
class LeakTest(object):
   def __init__(self):
     print 'Object with id %d born here.' % id(self)
   def __del__(self):
     print 'Object with id %d dead here.' % id(self)

def foo():
   A = LeakTest()
   B = LeakTest()
   A.b = weakref.proxy(B)
   B.a = weakref.proxy(A)
if __name__ = ="__main__": 
  foo()

弱引用對象使用

弱引用和代理對象都可以設置callback，在沒有強引用時，python要進行銷毀時調(diào)用。

>>> from socket import *
>>> import weakref
>>> s=socket(AF_INET,SOCK_STREAM) 
>>> ref=weakref.ref(s) # 通過調(diào)用弱引用來獲取被弱引用的對象
>>> pref=weakref.proxy(s) #代理對象就是弱引用對象
>>> s
<socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
>>> ref
<weakref at 0x103275598; to 'socket' at 0x10325aee8>
>>> ref()
<socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>
>>> pref
<weakproxy at 0x103275548 to socket at 0x10325aee8>
最后編輯于 ：
?著作權歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務。

相關閱讀更多精彩內(nèi)容

  • The Python Data Model If you learned another object-orien...
    plutoese閱讀 1,976評論 0贊 51
  • Python是一門功能強大的高級腳本語言，它的強大不僅表現(xiàn)在其自身的功能上，而且還表現(xiàn)在其良好的可擴展性上，正因如...
    蝴蝶蘭玫瑰閱讀 1,709評論 0贊 17
  • C++調(diào)用python 在C/C++中嵌入Python，可以使用Python提供的強大功能，通過嵌入Python可...
    Bruce_Szh閱讀 14,005評論 1贊 7
  • 小謹這一節(jié)的帶讀，了解了如果想要和孩子分享一個大概念，光說道理、看書是不足夠的，從游戲中體驗可以有親身的深刻的體...
    阿姬_NLP心智提升閱讀 230評論 0贊 0
  • 下接上期： 嗬，這塊水晶晶瑩剔透，走上去看看，都看得見人影！可這塊水晶如此亮，一定有蹊蹺，這也是田田奇怪的...
    書自由閱讀 307評論 0贊 0

友情鏈接更多精彩內(nèi)容