source_pos = get_list_index_or_none(col_list, SOURCE_COLUMN)
# This line below makes copy of the column and allocates
# additional memory, which is not ideal
source_only = filtered_df.iloc[:, source_pos:source_pos + 1:].iloc[:, 0]
rows_with_many_words_in_source = source_only.astype(str).str.count(
' ') + 1 > SOURCE_WORDS_LIMIT
I want to count the amount of rows that has a string that consists of multiple words (i.e. contains a space) in a specific column. How can I do that without allocating memory for pd.Series, like it does in the example code?
This is memory usage before that line:
2022-07-29 12:27:56,974 - DEBUG - Total memory usage: 102.21875 MB
2022-07-29 12:27:57,539 - DEBUG - Total objects allocated: 438113
types | # objects | total size
======================================= | =========== | ============
pandas.core.frame.DataFrame | 2 | 24.61 MB
str | 144600 | 23.60 MB
dict | 40957 | 15.36 MB
code | 36800 | 6.26 MB
type | 5342 | 4.92 MB
list | 31955 | 3.04 MB
tuple | 34692 | 2.01 MB
set | 2222 | 1.42 MB
numpy.ndarray | 86 | 752.26 KB
weakref | 9220 | 648.28 KB
collections.OrderedDict | 269 | 459.09 KB
abc.ABCMeta | 421 | 453.01 KB
openpyxl.descriptors.MetaSerialisable | 424 | 440.56 KB
int | 14551 | 426.82 KB
builtin_function_or_method | 5516 | 387.84 KB
This is memory usage after:
2022-07-29 12:28:10,295 - DEBUG - Total memory usage: 111.296875 MB
2022-07-29 12:28:10,793 - DEBUG - Total objects allocated: 439022
types | # objects | total size
======================================= | =========== | ============
pandas.core.series.Series | 48 | 24.93 MB
pandas.core.frame.DataFrame | 2 | 24.61 MB
str | 144605 | 23.60 MB
dict | 41326 | 15.42 MB
code | 36800 | 6.26 MB
type | 5342 | 4.92 MB
list | 31998 | 3.04 MB
tuple | 34785 | 2.02 MB
set | 2223 | 1.42 MB
numpy.ndarray | 133 | 757.03 KB
weakref | 9269 | 651.73 KB
collections.OrderedDict | 269 | 459.09 KB
abc.ABCMeta | 421 | 453.01 KB
openpyxl.descriptors.MetaSerialisable | 424 | 440.56 KB
int | 14597 | 428.08 KB