I have a question regarding the performance of the loc method in Pandas.
I tried the following code and measured its time to execute:
import timeit
import pandas as pd
data = {"id": [str(i) for i in range(100_000)]}
data_frame = pd.DataFrame(data=data)
def test_loc_read():
print(pd.__version__)
data_frame["value"] = 0
for label, _ in data_frame.iterrows():
data_frame.loc[label, "value"] = 1
if __name__ == "__main__":
print(timeit.timeit(test_loc_read, number=3))
It seems that the performance increased from Pandas 1.3.5 to Pandas 1.4.2.
I measured approx. 14 sec. with Pandas 1.4.2 and approx. 50 sec. with Pandas 1.3.5.
Can somebody explain where this performance gain is coming from? I already checked the Release Notes but couldn't figure it out.
Thanks!