Skip to content

Commit 3ef5e16

Browse files
authored
[memory-ownership] Honour ownership attributes when taking and releasing memory (#114)
* [memory-ownership] Honour ownership attributes when taking and releasing memory Added an AllocType field on CPPMethod, cached from Cpp::IsAllocator, that now decides the ownership of a returned object. The overloads of a group share fMethodInfo, so the answer of the called overload stays in a local instead of going into the shared kIsCreator flag, which nothing ever clears. The analyzer is not called; this is the attribute-checker path only. AllocType was collapsed into kIsCreator and never reached op_dealloc_nofree, so malloc'd, new[]'d and raw operator-new'd memory were all freed with a scalar delete. It now rides on the instance as kIsArrayAlloc, kIsNoConstruct and kIsMalloc; op_dealloc_nofree branches on them to delete[], free, ::operator delete or ::operator delete[], with no destructor for the raw cases. All flags clear keeps the old delete path. bind_object carries those flags to the new proxy along with the ownership, as BindCppObjectNoCast() masks them out, so a cast object frees its memory the way it was allocated. Mirrored that with Cpp::IsDeallocator, cached the same way, and passed the call arguments to HandleReturn: handing a Python-owned instance to an ownership-taking function now gives ownership back to C++ instead of leaving it to be freed twice. IsDeallocator reports one DeallocType per parameter, so every argument of a call is matched to the parameter that the attribute labels. A positional argument finds its slot from its place in the args array, and a keyword argument from its name in the signature. Two shifts turn either number into a slot, because an implicit object parameter and the shape of the call each move it: GetDeallocArgShift() answers for the args array and GetDeallocParamShift() for the declaration. CPPMethod, CPPFunction and CPPClassMethod answer both for their own kind of callee, and the defaults on PyCallable keep every other callable out of this path. The interop layer carries what the above needs. interop::AllocType and interop::DeallocType name the CppInterOp enums, and IsAllocator() and IsDeallocator() join their neighbours in the wrapper, each holding the interpreter mutex. interop::Destruct() takes an object count, which it used to pin to 0; a count other than 0 makes the CppInterOp destructor wrapper run delete[] in place of delete, which is what the array path wants. CPPINTEROP_GIT_TAG moves to the CppInterOp commit that carries those two entry points, and test/Makefile builds the memory_analysis dictionary. Ownership is only released, never nullified: the instance keeps a stale pointer after the callee frees the memory. GetAllocBehaviour is restricted to raw-pointer returns. Exposed read-only __is_array_alloc__ / __is_no_construct__ / __is_malloc__ and added tests for the deallocation paths and for ownership release via free, via an ownership_takes function, and the negative case. An attribute reaches the tests three ways: a clang::annotate on the declaration, a redeclaration that carries it, and APINotes behind a module map. The release tests cover the bound, unbound and stored-bound forms of a member method, a static method and a free function, each in a positional and a keyword shape, a keyword shape with the names reversed, an argument that is no object at all, a cppDeallocNone parameter, a Python overload that carries no attribute, and a cast through bind_object. * [memory-ownership] Find the argument shifts without virtual functions PyCallable carried four virtual functions for the two slot shifts. Only CPPClassMethod needed a real computation, so HandleReturn now picks the shifts with a dynamic_cast chain. CPPClassMethod::Call and HandleReturn share one ShouldDropSelf, which ends the repeated condition. As an optimization, the loop also skips an argument that C++ already owns.
1 parent 8a20f95 commit 3ef5e16

18 files changed

Lines changed: 948 additions & 34 deletions

‎src/cpyrt/CPPClassMethod.cxx‎

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -32,15 +32,11 @@ PyObject* cpyrt::CPPClassMethod::Call(CPPInstance*& self, cpyrt_PyArgs_t args,
3232
// first argument matches the enclosing scope of the class method and it isn't
3333
// needed for the call, then assume that the method was (incorrectly) bound
3434
// and so drop that instance from args.
35-
int nargs = (int)cpyrt_PyArgs_GET_SIZE(args, nargsf);
36-
if ((!self || (PyObject*)self == Py_None) && nargs) {
37-
PyObject* arg0 = cpyrt_PyArgs_GET_ITEM(args, 0);
38-
if (CPPInstance_Check(arg0) && fArgsRequired <= nargs - 1 &&
39-
interop::IsSubclass(reinterpret_cast<CPPInstance*>(arg0)->ObjectIsA(),
40-
GetScope())) {
41-
args += 1; // drops first argument
42-
nargsf -= 1;
43-
}
35+
Py_ssize_t nargs = cpyrt_PyArgs_GET_SIZE(args, nargsf);
36+
const bool isBound = self && (PyObject*)self != Py_None;
37+
if (ShouldDropSelf(isBound, args, nargs)) {
38+
args += 1; // drops first argument
39+
nargsf -= 1;
4440
}
4541

4642
if (!this->ConvertAndSetArgs(args, nargsf, ctxt))
@@ -50,6 +46,21 @@ PyObject* cpyrt::CPPClassMethod::Call(CPPInstance*& self, cpyrt_PyArgs_t args,
5046
return this->Execute(nullptr, 0, ctxt);
5147
}
5248

49+
//----------------------------------------------------------------------------
50+
bool cpyrt::CPPClassMethod::ShouldDropSelf(bool isBound, PyObject* const* args,
51+
Py_ssize_t nargs) {
52+
// A bound call keeps self outside of args
53+
// Also if there is no arg in call, there is no self to drop
54+
if (isBound || nargs == 0)
55+
return false;
56+
57+
if (CPPInstance_Check(args[0]) && fArgsRequired <= nargs - 1 &&
58+
interop::IsSubclass(((CPPInstance*)args[0])->ObjectIsA(), GetScope()))
59+
return true;
60+
61+
return false;
62+
}
63+
5364
//----------------------------------------------------------------------------
5465
PyObject* cpyrt::CPPClassMethod::GetTypeName() {
5566
PyObject* cppname =

‎src/cpyrt/CPPClassMethod.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ class CPPClassMethod : public CPPMethod {
1414
PyObject* GetTypeName() override;
1515

1616
public:
17+
bool ShouldDropSelf(bool isBound, PyObject* const* args, Py_ssize_t nargs);
18+
1719
PyCallable* Clone() override { return new CPPClassMethod(*this); }
1820
PyObject* Call(CPPInstance*& self, cpyrt_PyArgs_t args, size_t nargsf,
1921
PyObject* kwds, CallContext* ctxt = nullptr) override;

‎src/cpyrt/CPPInstance.cxx‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ using namespace cppjit;
1717

1818
// Standard
1919
#include <algorithm>
20+
#include <cstdlib>
2021
#include <sstream>
2122

2223
//- data _____________________________________________________________________
@@ -227,7 +228,17 @@ void cpyrt::op_dealloc_nofree(CPPInstance* pyobj) {
227228
if (pyobj->fFlags & CPPInstance::kIsValue) {
228229
interop::CallDestructor(klass, cppobj);
229230
interop::Deallocate(klass, cppobj);
230-
} else
231+
} else if (pyobj->fFlags & CPPInstance::kIsMalloc)
232+
std::free(cppobj);
233+
else if (pyobj->fFlags & CPPInstance::kIsNoConstruct) {
234+
if (pyobj->fFlags & CPPInstance::kIsArrayAlloc)
235+
::operator delete[](cppobj);
236+
else
237+
::operator delete(cppobj);
238+
} else if (pyobj->fFlags & CPPInstance::kIsArrayAlloc)
239+
interop::Destruct(klass, cppobj, 1);
240+
// Default case: just kIsOwner set in all of memory-ownership flags
241+
else
231242
interop::Destruct(klass, cppobj);
232243
}
233244
cppobj = nullptr;
@@ -938,10 +949,30 @@ static int op_setownership(CPPInstance* pyobj, PyObject* value, void*) {
938949
return 0;
939950
}
940951

952+
// Added for testing purposes
953+
//-----------------------------------------------------------------------------
954+
static PyObject* op_get_array_alloc(CPPInstance* pyobj, void*) {
955+
return PyBool_FromLong((long)(pyobj->fFlags & CPPInstance::kIsArrayAlloc));
956+
}
957+
//-----------------------------------------------------------------------------
958+
static PyObject* op_get_no_construct(CPPInstance* pyobj, void*) {
959+
return PyBool_FromLong((long)(pyobj->fFlags & CPPInstance::kIsNoConstruct));
960+
}
961+
//-----------------------------------------------------------------------------
962+
static PyObject* op_get_malloc(CPPInstance* pyobj, void*) {
963+
return PyBool_FromLong((long)(pyobj->fFlags & CPPInstance::kIsMalloc));
964+
}
941965
//-----------------------------------------------------------------------------
942966
static PyGetSetDef op_getset[] = {
943967
{(char*)"__python_owns__", (getter)op_getownership, (setter)op_setownership,
944968
(char*)"If true, python manages the life time of this object", nullptr},
969+
{(char*)"__is_array_alloc__", (getter)op_get_array_alloc, nullptr,
970+
(char*)"If true, the object was allocated with new[]/operator new[]",
971+
nullptr},
972+
{(char*)"__is_no_construct__", (getter)op_get_no_construct, nullptr,
973+
(char*)"If true, the memory is raw: no constructor was run", nullptr},
974+
{(char*)"__is_malloc__", (getter)op_get_malloc, nullptr,
975+
(char*)"If true, the object was allocated with malloc", nullptr},
945976
{(char*)nullptr, nullptr, nullptr, nullptr, nullptr}};
946977

947978
//= cpyrt type number stubs to allow dynamic overrides =====================

‎src/cpyrt/CPPInstance.h‎

Lines changed: 18 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -29,20 +29,24 @@ class CPPInstance {
2929
public:
3030
enum EFlags {
3131
kDefault = 0x0000,
32-
kNoWrapConv = 0x0001, // use type as-is (eg. no smart ptr wrap)
33-
kIsOwner = 0x0002, // Python instance owns C++ object/memory
34-
kIsExtended = 0x0004, // has extended data
35-
kIsValue = 0x0008, // was created from a by-value return
36-
kIsReference = 0x0010, // represents one indirection
37-
kIsArray = 0x0020, // represents an array of objects
38-
kIsSmartPtr = 0x0040, // is or embeds a smart pointer
39-
kIsPtrPtr = 0x0080, // represents two indirections
40-
kIsRValue = 0x0100, // can be used as an r-value
41-
kIsLValue = 0x0200, // can be used as an l-value
42-
kNoMemReg = 0x0400, // do not register with memory regulator
43-
kIsRegulated = 0x0800, // is registered with memory regulator
44-
kIsActual = 0x1000, // has been downcasted to actual type
45-
kHasLifeLine = 0x2000, // has a life line set
32+
kNoWrapConv = 0x0001, // use type as-is (eg. no smart ptr wrap)
33+
kIsOwner = 0x0002, // Python instance owns C++ object/memory
34+
kIsExtended = 0x0004, // has extended data
35+
kIsValue = 0x0008, // was created from a by-value return
36+
kIsReference = 0x0010, // represents one indirection
37+
kIsArray = 0x0020, // represents an array of objects
38+
kIsSmartPtr = 0x0040, // is or embeds a smart pointer
39+
kIsPtrPtr = 0x0080, // represents two indirections
40+
kIsRValue = 0x0100, // can be used as an r-value
41+
kIsLValue = 0x0200, // can be used as an l-value
42+
kNoMemReg = 0x0400, // do not register with memory regulator
43+
kIsRegulated = 0x0800, // is registered with memory regulator
44+
kIsActual = 0x1000, // has been downcasted to actual type
45+
kHasLifeLine = 0x2000, // has a life line set
46+
kIsArrayAlloc = 0x4000, // represents a heap allocated array of objects
47+
kIsNoConstruct =
48+
0x8000, // represents constructor is not called in the allocation
49+
kIsMalloc = 0x10000, // is allocated with malloc
4650
};
4751

4852
public: // public, as the python C-API works with C structs

‎src/cpyrt/CPPMethod.cxx‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,8 @@ inline void cpyrt::CPPMethod::Copy_(const CPPMethod& /* other */) {
8585
fExecutor = nullptr;
8686
fArgIndices = nullptr;
8787
fArgsRequired = -1;
88+
fAllocType.reset();
89+
fDeallocType.reset();
8890
}
8991

9092
//----------------------------------------------------------------------------
@@ -703,6 +705,37 @@ PyObject* cpyrt::CPPMethod::GetArgDefault(int iarg, bool silent) {
703705

704706
bool cpyrt::CPPMethod::IsConst() { return interop::IsConstMethod(GetMethod()); }
705707

708+
//----------------------------------------------------------------------------
709+
interop::AllocType cpyrt::CPPMethod::GetAllocBehaviour() {
710+
if (fAllocType.has_value())
711+
return *fAllocType;
712+
interop::TCppMethod_t method = GetMethod();
713+
if (!method) {
714+
fAllocType = interop::AllocType::None;
715+
return interop::AllocType::None;
716+
}
717+
if (!IsPointerType(interop::GetMethodReturnType(method))) {
718+
fAllocType = interop::AllocType::None;
719+
return interop::AllocType::None;
720+
}
721+
interop::AllocType attrResult = interop::IsAllocator(method);
722+
fAllocType = attrResult;
723+
return attrResult;
724+
}
725+
726+
//----------------------------------------------------------------------------
727+
interop::DeallocType cpyrt::CPPMethod::GetDeallocBehaviour(size_t index) {
728+
if (!fDeallocType) {
729+
fDeallocType.emplace();
730+
if (interop::TCppMethod_t method = GetMethod())
731+
interop::IsDeallocator(method, *fDeallocType);
732+
}
733+
if (index < fDeallocType->size()) {
734+
return (*fDeallocType)[index];
735+
}
736+
return interop::DeallocType::Opaque;
737+
}
738+
706739
//----------------------------------------------------------------------------
707740
PyObject* cpyrt::CPPMethod::GetScopeProxy() {
708741
// Get or build the scope of this method.

‎src/cpyrt/CPPMethod.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
#include "PyCallable.h"
66

77
// Standard
8+
#include <optional>
89
#include <string>
910
#include <unordered_map>
1011
#include <vector>
@@ -62,6 +63,8 @@ class CPPMethod : public PyCallable {
6263
PyObject* GetCoVarNames() override;
6364
PyObject* GetArgDefault(int iarg, bool silent = true) override;
6465
bool IsConst() override;
66+
interop::AllocType GetAllocBehaviour() override;
67+
interop::DeallocType GetDeallocBehaviour(size_t index) override;
6568

6669
PyObject* GetScopeProxy() override;
6770
interop::TCppFuncAddr_t GetFunctionAddress() override;
@@ -121,6 +124,8 @@ class CPPMethod : public PyCallable {
121124
protected:
122125
// cached value that doubles as initialized flag (uninitialized if -1)
123126
int fArgsRequired;
127+
std::optional<interop::AllocType> fAllocType;
128+
std::optional<std::vector<interop::DeallocType>> fDeallocType;
124129
};
125130

126131
} // namespace cppjit::cpyrt

‎src/cpyrt/CPPOverload.cxx‎

Lines changed: 109 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
#include "cpyrt.h"
33

44
using namespace cppjit;
5+
#include "CPPClassMethod.h"
6+
#include "CPPFunction.h"
57
#include "CPPInstance.h"
68
#include "CPPOverload.h"
79
#include "CallContext.h"
@@ -142,14 +144,23 @@ static inline void ResetCallState(CPPInstance* descr_self,
142144

143145
// helper to factor out return logic of mp_call / mp_vectorcall
144146
static inline PyObject* HandleReturn(CPPOverload* pymeth, CPPInstance* im_self,
145-
PyObject* result) {
147+
PyObject* result, PyCallable* method,
148+
PyObject* const* args, size_t nargsf,
149+
PyObject* kwds) {
146150
// special case for python exceptions, propagated through C++ layer
147151
if (result) {
148152
CPPInstance* cppres =
149153
(CPPInstance*)(CPPInstance_Check(result) ? result : nullptr);
150154

155+
interop::AllocType AT = method->GetAllocBehaviour();
156+
// the overloads of a group share fMethodInfo, so the answer of this one
157+
// stays in a local instead of going into the shared flags
158+
const bool allocates = AT != interop::AllocType::None &&
159+
AT != interop::AllocType::Null &&
160+
AT != interop::AllocType::Unknown;
161+
151162
// if this method creates new objects, always take ownership
152-
if (IsCreator(pymeth->fMethodInfo->fFlags)) {
163+
if (allocates || IsCreator(pymeth->fMethodInfo->fFlags)) {
153164

154165
// either be a constructor with a fresh object proxy self ...
155166
if (IsConstructor(pymeth->fMethodInfo->fFlags)) {
@@ -158,9 +169,98 @@ static inline PyObject* HandleReturn(CPPOverload* pymeth, CPPInstance* im_self,
158169
}
159170

160171
// ... or be a regular method with an object proxy return value
161-
else if (cppres)
172+
else if (cppres) {
162173
cppres->PythonOwns();
174+
// After giving ownership, set proper flags to indicate allocation
175+
// method/func
176+
switch (AT) {
177+
case interop::AllocType::Malloc:
178+
cppres->fFlags |= CPPInstance::kIsMalloc;
179+
break;
180+
case interop::AllocType::NewArr:
181+
cppres->fFlags |= CPPInstance::kIsArrayAlloc;
182+
break;
183+
case interop::AllocType::OperatorNew:
184+
cppres->fFlags |= CPPInstance::kIsNoConstruct;
185+
break;
186+
case interop::AllocType::OperatorNewArr:
187+
cppres->fFlags |= CPPInstance::kIsArrayAlloc;
188+
cppres->fFlags |= CPPInstance::kIsNoConstruct;
189+
break;
190+
default:
191+
break;
192+
}
193+
}
194+
}
195+
196+
// an argument that the called function releases belongs to C++ from now
197+
// on, else Python frees it a second time
198+
const Py_ssize_t nargs = cpyrt_PyArgs_GET_SIZE(args, nargsf);
199+
const Py_ssize_t numKwds =
200+
kwds && PyTuple_CheckExact(kwds) ? PyTuple_GET_SIZE(kwds) : 0;
201+
// the keyword values follow the positional ones in call order, so their
202+
// names are what tell which parameter each one belongs to
203+
PyObject* argnames = numKwds ? method->GetSignatureNames() : nullptr;
204+
205+
const bool isBound = pymeth->fSelf != nullptr;
206+
// Tells if there is an implicit `this`
207+
Py_ssize_t paramShift = 0;
208+
// Tells if self parameter should be dropped (may be bound function or
209+
// static with self in) or added
210+
Py_ssize_t argShift = 0;
211+
if (dynamic_cast<CPPFunction*>(method)) {
212+
// For a plain func, only case there is a self is when it is bound
213+
argShift = isBound ? 1 : 0;
214+
// There is no implicit `this` in plain functions
215+
paramShift = 0;
216+
} else if (auto* CPPCM = dynamic_cast<CPPClassMethod*>(method)) {
217+
// Check ShouldDropSelf
218+
argShift = CPPCM->ShouldDropSelf(isBound, args, nargs) ? -1 : 0;
219+
// Static methods do not carry implicit `this`
220+
paramShift = 0;
221+
}
222+
// Only remain possibility is CPPMethod and its other children which acts as
223+
// CPPMethod
224+
else {
225+
// If method is bound, self is carried outside of args
226+
argShift = isBound ? 1 : 0;
227+
// A non-static method always keeps implicit `this`
228+
paramShift = 1;
229+
}
230+
231+
// the array holds the positional values first, then the keyword ones
232+
for (Py_ssize_t i = 0; i < nargs + numKwds; i++) {
233+
// only a C++ object can hold memory that the callee releases
234+
if (!CPPInstance_Check(args[i]) ||
235+
!(((CPPInstance*)args[i])->fFlags & CPPInstance::kIsOwner))
236+
continue;
237+
238+
Py_ssize_t slot = -1; // no slot of its own
239+
if (i < nargs)
240+
slot = i + argShift;
241+
else {
242+
// The place of a keyword value means nothing, its name picks the
243+
// parameter. kwds has the names in the same order, so kwds[i - nargs]
244+
// names args[i].
245+
for (Py_ssize_t iarg = 0; iarg < PyTuple_GET_SIZE(argnames); iarg++) {
246+
if (PyUnicode_Compare(PyTuple_GET_ITEM(argnames, iarg),
247+
PyTuple_GET_ITEM(kwds, i - nargs)) == 0) {
248+
slot = iarg + paramShift;
249+
break;
250+
}
251+
}
252+
}
253+
// the callee drops this argument, or no name of it matches a parameter
254+
if (slot < 0)
255+
continue;
256+
257+
const interop::DeallocType DT = method->GetDeallocBehaviour((size_t)slot);
258+
if (DT == interop::DeallocType::None ||
259+
DT == interop::DeallocType::Opaque)
260+
continue;
261+
((CPPInstance*)args[i])->CppOwns();
163262
}
263+
Py_XDECREF(argnames);
164264

165265
// if this new object falls inside self, make sure its lifetime is proper
166266
if (!(pymeth->fMethodInfo->fFlags & CallContext::kNeverLifeLine)) {
@@ -576,7 +676,8 @@ static PyObject* mp_vectorcall(CPPOverload* pymeth, PyObject* const* args,
576676
if (!NoImplicit(&ctxt))
577677
ctxt.fFlags |= CallContext::kAllowImplicit; // no two rounds needed
578678
PyObject* result = methods[0]->Call(im_self, args, nargsf, kwds, &ctxt);
579-
return HandleReturn(pymeth, im_self, result);
679+
return HandleReturn(pymeth, im_self, result, methods[0], args, nargsf,
680+
kwds);
580681
}
581682

582683
// otherwise, handle overloading
@@ -599,7 +700,8 @@ static PyObject* mp_vectorcall(CPPOverload* pymeth, PyObject* const* args,
599700
ctxt.fFlags |= CallContext::kAllowImplicit;
600701
PyObject* result = memoized_pc->Call(im_self, args, nargsf, kwds, &ctxt);
601702
if (result)
602-
return HandleReturn(pymeth, im_self, result);
703+
return HandleReturn(pymeth, im_self, result, memoized_pc, args, nargsf,
704+
kwds);
603705

604706
// fall through: python is dynamic, and so, the hashing isn't infallible
605707
ctxt.fFlags &= ~(CallContext::kAllowImplicit | CallContext::kPyException |
@@ -656,7 +758,8 @@ static PyObject* mp_vectorcall(CPPOverload* pymeth, PyObject* const* args,
656758
}
657759
}
658760

659-
return HandleReturn(pymeth, im_self, result);
761+
return HandleReturn(pymeth, im_self, result, methods[i], args, nargsf,
762+
kwds);
660763
}
661764

662765
// else failure ..

‎src/cpyrt/PyCallable.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,12 @@ class PyCallable {
3737
virtual PyObject* GetCoVarNames() = 0;
3838
virtual PyObject* GetArgDefault(int /* iarg */, bool silent = true) = 0;
3939
virtual bool IsConst() { return false; }
40+
virtual interop::AllocType GetAllocBehaviour() {
41+
return interop::AllocType::None;
42+
}
43+
virtual interop::DeallocType GetDeallocBehaviour(size_t /* index */) {
44+
return interop::DeallocType::Opaque;
45+
}
4046

4147
virtual PyObject* GetScopeProxy() = 0;
4248
virtual interop::TCppFuncAddr_t GetFunctionAddress() = 0;

0 commit comments

Comments
 (0)