Random stealing calls to child initializer

I found a way to pass a list into the Random's inheritor and use it in __init__.

import random
from typing import List


class MyRandom(random.Random):
    internal_list: List

    def __init__(self, x=None):
        if type(x) is list:
            print(f"Access to the list from `__init__`: {MyRandom.internal_list}")
            super().__init__(MyRandom.internal_list[0])
        else:
            super().__init__(x)

    def __new__(cls, x):
        cls.internal_list = x
        return super().__new__(cls)

    def new_method(self):
        print(f"Access to the list from `new_method`: {MyRandom.internal_list}")

r1 = MyRandom([1, 2])
r1.new_method()
print(r1.random())

r2 = MyRandom([3, 4])
r2.new_method()
print(r2.random())

Output:

Access to the list from `__init__`: [1, 2]
Access to the list from `new_method`: [1, 2]
0.13436424411240122
Access to the list from `__init__`: [3, 4]
Access to the list from `new_method`: [3, 4]
0.23796462709189137

For example purpose, I used MyRandom.internal_list[0] to initialize the PRNG. Of course, it's needed to check if the first element exists.

I'm not sure why __new__ is used when you init MyRandom. It's definitely not documented, because in PyCharm implementation I found this:

    @staticmethod # known case of __new__
    def __new__(*args, **kwargs): # real signature unknown
        """ Create and return a new object.  See help(type) for accurate signature. """
        pass

So there is already a good answer showing how to work-around this issue, but it got me curious as to why this happens. I couldn't get to a definitive answer, but posting my findings here for anyone who wants to follow through.

So we know that when a new instance is created, first __new__ is called - creating the actual instance (assigning the memory on C-level). Then the newly created instance is passed to the class' __init__ method.

Now, as the print of "Before" didn't even happen, it is safe to assume that the problem is in the __new__ method. Indeed, when I override it like:

def __new__(cls, *args, **kwargs):
    print("in new")
    return super().__new__(cls)

No errors were raised and an expected print-out of:

in new
Before
After

Once I added *args to the super call:

return super().__new__(cls, *args)

The same error was back. So this must be an issue in Random's __new__.

Inspecting the code with Pycharm, Random doesn't override its __new__ method, but the class signature is:

class Random(_random.Random):

Trying to inspect this parent class shows a bunch of methods only containing pass in them. This seemed weird but after a quick search, I found out (for some it is probably not a surprise) that modules starting with _ are C implementations. And _random's C implementation is _randommodule.c.

Now, I don't have much knowledge or experience with inspecting C implementations of Python, but I found what seems to be the basic slots of the Random class:

static PyType_Slot Random_Type_slots[] = {
    {Py_tp_doc, (void *)random_doc},
    {Py_tp_methods, random_methods},
    {Py_tp_new, PyType_GenericNew},
    {Py_tp_init, random_init},
    {Py_tp_free, PyObject_Free},
    {0, 0},
};

My own understanding from this is that the class' __init__ is mapped to random_init, and the class' __new__ is mapped to PyType_GenericNew. But as its name suggests, PyType_GenericNew is just a generic object creator assigning the necessary amount of memory for the object. Its body is the single line:

PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    return type->tp_alloc(type, 0);
}

The args are not even used.

On the other hand, the random_init function calls random_seed which does have some hashing in it:

Py_hash_t hash = PyObject_Hash(arg);

But then again, we established that the __init__ is not even called yet, at which point I'm stumped...

Instantiating a class causes its __new__ method to be called. It is passed the name of the class and the arguments in the constructor call. So MyRandom([1, 2]) results in the call MyRandom.__new__(MyRandom, [1, 2]). (3.9.10 documentation).

Because there isn't a MyRandom.__new__() method, the base classes are searched. random.Random does have a __new__() method (see random_new() in _randommodule.c). So we get a call something like this random_new(MyRandom, [1, 2]).

Looking at the C code for random_new(), it calls random_seed(self, [1, 2]). Because the second argument isn't Null, or None, or an int, or a subclass of int, the code calls PyObject_Hash([1, 2]). But a list isn't hashable, hence the error.

If __new__() returns a instance of the class, then the __init__() method is called with the arguments in the constuctor call.

One possible fix is to define a MyRandom.__new__() method, which calls super().__new__() but only passes the appropriate args.

class MyRandom(random.Random):
    def __new__(cls, *args, **kwargs):
        #print(f"In __new__: {args=}, {kwargs=}")

        # Random.__new__ expects an optional seed. We are going to 
        # implement out own RNG, so ignore args and kwargs. Pass in a 
        # junk integer value so that Random.__new__ doesn't waste time
        # trying to access urandom or calling time to initialize the MT RNG
        # since we aren't going to use it anyway.
        return super().__new__(cls, 123)
    
    def __init__(cls, *args, **kwargs):
        #print(f"In __init__: {args=}, {kwargs=}")

        # initialize your custom RNG here
        pass

Also override the methods: random(), seed(), getstate(), setstate(), and optionally getrandbits().

An alternative fix is to only use keyword arguments in the __init__() methods of the subclasses. The C code for random_new() checks to see if a an instance of random.Random is being created. If true, the code throws and error if there are any keyword arguments. However, if a subclass is being created, any keyword arguments are ignored by random_new(), but can be used in the subclass __init__().

class MyRandom(random.Random):
    def __init__(self, *, x):  # make x a keyword only argument
        print("Before")
        super().__init__()  # Nothing passed to parent
        print("After")

MyRandom(x=[])

Interestingly, in Python 3.10, the code for random_new has been changed to raise an error if more that 1 positional argument is supplied.