Understanding Memory Addressing in Python

user_5658788 0
Viewed 374

I am writing this to expand my understanding of how pointers and memory-addressing work in python.

Python being a high level language, I have learned that the memory management is taken care of by python.

I have also learned that the pointers can be identified with id() object.

Assume I have two variables a and b, and these two variables can be swapped with the below command:

a,b=b,a

The change in identity of these variables can be found using the below methods.

>>> a=2
>>> b=3
>>> id(a)
140732857872688
>>> id(b)
140732857872720
>>> a,b=b,a
>>> a
3
>>> b
2
>>> id(a)
140732857872720
>>> id(b)
140732857872688

As you can see the ids are swapped (as expected) , and in the other words we can say the idsassigned to variable are swapped. How is this happening in python?

And in one more example,

>>> i=0
>>> id(i)
140732857872624
>>> i+=1
>>> id(i)
140732857872656

When i is incremented by 1 , how is a new variable( I am calling it new because the identity is changing) with the same name created under the hood?

I am adding one more example also to further expand my question!

Operation 1:

>>> i,j=0,0
>>> id(i)
140732857872624
>>> id(j)
140732857872624
>>> i,j = i+1,i+1
>>> i
1
>>> j
1
>>> id(i)
140732857872656
>>> id(j)
140732857872656

Operation 2:

>>> i,j=0,0
>>> i=i+1
>>> j=i+1
>>> i
1
>>> j
2
>>> id(i)
140732857872656
>>> id(j)
140732857872688

Of course operation 1 and operation 2 are different. How do both of these operations occur at a low level.

And once there is no more use for an already assigned pointer, how will it get reused ? if a memory-address can not be reused in a given program and new memory-address has to be given for each new operation(in the same program) , is it the reason behind memory overflow when recursion with a high depth is used?

I hope I have made my questions clear.

If my understanding is wrong anywhere , please correct me. Coming from Mechanical Engineering background, although I have been programming for quite some time , this thing is really confusing to me.

Regards

memory-address pointers python
3 Answers

Use dis to see what's happening under the hood.

In [1]: import dis

In [2]: def swap(): a=1; b=2; a,b=b,a

In [3]: dis.dis(swap)
  1           0 LOAD_CONST               1 (1)
              2 STORE_FAST               0 (a)
              4 LOAD_CONST               2 (2)
              6 STORE_FAST               1 (b)
              8 LOAD_FAST                1 (b)
             10 LOAD_FAST                0 (a)
             12 ROT_TWO
             14 STORE_FAST               0 (a)
             16 STORE_FAST               1 (b)
             18 LOAD_CONST               0 (None)
             20 RETURN_VALUE
user_67967310

To understand what is happening a simple mental model is

  1. Every python variable is a pointer.
    Even after i=1 internally in Python i is a pointer to an object that contains the value 1. Even integers are normal objects (this is not very common as in other languages normally variables end up containing the values directly for small integers).

  2. For efficiency reasons some immutable objects are cached and shared.
    Small integers are cached so in a Python program if you all the variables with value 0 will be pointers pointing to the same object. In the second case you are observing different ids because the value is different (i will be pointing to 1 and j will be pointing to 2).
    Note that sharing is done only for immutable objects because, except by using id the sharing cannot be observed in the code. In Python this is done for small numbers and some small strings (those that end up being "interned").

user_3207260

in python, variables are memory references. when you create

a=2

python memory manager will place this value 2 inside a slot of memory heap, and assign the address of this memory slot to the variable. Essentially

a=adress_of_this_memory_slot

Behind the scene whenever you call a, python memory manager looks at the address and retrieves the value.

Python memory manager automatically reuses the memory references if the objects are immutable. In your case integers are immutable objects. So python reusing objects on demand and this is called INTERNING in python.

At start-up python preloads or caches the a global list of integers in the range [-5,256]. Anytime an integer is referenced in this range Python will use the cached version of that object.

So based on your first example, you created:

a=2
b=3

So python memory manager will add those values into two different slots of memory heap. in your example their address are:

id(a) ==> 140732857872688 
id(b) ==> 140732857872720

After this point any time you want to use int 2,3 python will use this memory addresses for the optimization purpose.

try your examples with integers out of this range [-5,256]. you will see that python will recreate a new address for this variable, in other words python will not share references for these values.

user_102628050
Related