Why does deleting columns or parts of a DataFrame increase memory usage, and how to ensure garbage collection on unused slices of DataFrame

When dealing with large DataFrames, you need to be careful with memory usage (for example you might want to download large data in chunks, process the chunks, and from then on delete all the unnecessary parts from memory).

I can't find any resources on the best procedures to deal with garbage collection in pandas, but I tried the following and got surprising results:

import os, psutil, gc
import pandas as pd

def get_process_mem_usage():
    process = psutil.Process(os.getpid())
    print("{:.3f} GB".format(process.memory_info().rss / 1e9))

get_process_mem_usage()
# Out: 0.146 GB
cdf = pd.DataFrame({i:np.random.rand(int(1e7)) for i in range(10)})
get_process_mem_usage()
# Out: 0.946 GB

With the following globals() and their memory usage:

          Size
cdf   781.25MB
_iii    1.05KB
_i1     1.05KB
_oh    240.00B

When I try to delete something, I get:

del cdf[1]
gc.collect()
get_process_mem_usage()
# Out: 1.668 GB

with a high process memory usage, but the following globals()

          Size
cdf   703.13MB
_i1     1.05KB
Out    240.00B
_oh    240.00B

so some memory is still allocated but not used by any object in globals().

I've also seen weird results when doing something like

cdf2 = cdf.iloc[:,:5]
del cdf

which sometimes creates a new global with a name like "_5" and more memory usage than cdf had before (I'm not sure what this global refers to, perhaps some sort of object containing the no-longer referenced columns from cdf, but why is it larger?

Another option is to "delete" columns through one of:

cdf = cdf.iloc[:, :5]
# or
cdf = cdf.drop(columns=[...])

where the columns are no longer referenced by any object so they get dropped. But for me this doesn't seem to happen every time; I could swear I've seen my process take up the same amount of memory after this operation, even when I call gc.collect() afterwards. Though when I try to recreate this in a notebook it doesn't happen.

So I guess my question is:

• Why does the above happen with deleting resulting in more memory usage
• What is the best way to ensure that no-longer needed columns are deleted from memory and properly garbage cleaned?