Failed to build a class from a custom metaclass

[NaleRaphael]

Just a memo for this issue.

Take the following code as an example.

class MyMeta(type):
    def __new__(cls, name, bases, dct):
        x = super().__new__(cls, name, bases, dct)
        x.name = 'Foo'
        return x

class Foo(metaclass=MyMeta):
    def greet(self):
        print('hello from {}!'.format(self.name))

if __name__ == '__main__':
    Foo().greet()

After running this snippet, we will get an error: TypeError: __classcell__ must be a nonlocal cell, not <class 'bytefall.objects.cellobject.Cell'>.

It is because __classcell__ in namespace should be a built-in cell, not the one implemented in Python, while we are going to make a new class from metaclass. (see also cpython/typeobject.c::type_new and the following code)

# @bytefall/ops.py
def build_class(func, name, *bases, **kwds):
    # ... omitted code
    cls = metaclass(name, bases, namespace)
    # ... omitted code

To fix it, we can replace the cell by the one actually created from internal. Like the solution taken by byterun:

def make_cell(value):
    fn = (lambda x: lambda: x)(value)
    return fn.__closure__[0]

def build_class(func, name, *bases, **kwds):
    # ... omitted code
    if '__classcell__' in namespace:
        real_cell = make_cell(None)
        real_cell.cell_contents = namespace['__classcell__'].contents
        namespace['__classcell__'] = real_cell

    cls = metaclass(name, bases, namespace)
    # ... omitted code

But here comes another problem. super() in MyMeta.__new__() is failed to be executed, we got an error RuntimeError: super(): __class__ cell not found.

After looking into CPython source code, that message is written in the function typeobject.c::super_init().

// @cpython/Objects/typeobject.c::super_init()
static int
super_init(PyObject *self, PyObject *args, PyObject *kwds)
{
    // ... omitted code
    if (type == NULL) {
        /* Call super(), without args -- fill in from __class__
           and first local variable on the stack. */
        PyFrameObject *f;
        PyCodeObject *co;
        Py_ssize_t i, n;
        f = PyThreadState_GET()->frame;     // 1.

        // ... omitted code

        for (i = 0; i < n; i++) {
            PyObject *name = PyTuple_GET_ITEM(co->co_freevars, i);  // 2.
            assert(PyUnicode_Check(name));
            if (_PyUnicode_EqualToASCIIId(name, &PyId___class__)) {
                // code for finding cell, and cast cell to a type object
                Py_ssize_t index = co->co_nlocals +
                    PyTuple_GET_SIZE(co->co_cellvars) + i;
                PyObject *cell = f->f_localsplus[index];
                if (cell == NULL || !PyCell_Check(cell)) {
                    PyErr_SetString(PyExc_RuntimeError,
                      "super(): bad __class__ cell");
                    return -1;
                }
                type = (PyTypeObject *) PyCell_GET(cell);
                // ... omitted code
            }
        }
        if (type == NULL) {     // 3.
            PyErr_SetString(PyExc_RuntimeError,
                            "super(): __class__ cell not found");
            return -1;
        }
    }
    // ... omitted code

In the simplified code above:

1. f is a frame hold by interpreter.
2. In that loop, we are going to find __class__ in co.co_freevars. Once we found that, we can know the index of cell in f.f_localsplus. Then we can get type object from it.
3. If no __class__ in co.co_freevars, cell won't be availabe. Hence that we cannot get type.

However, in our implementation, all information is stored in Frame implemented by us instead of built-in frame. Therefore, frame f retrieved by PyThreadState_GET()->frame in step 1 is not the one with correct data for execution, and that's why step 2 and 3 failed.

A possible solution for this is re-implementing super_init and type_new, but we still need to go deeper into it.