How to create new dataframe with only rows that have been changed in the another dataframe?

Try storing the original values and comparing them later:

original = df['Part Number'].copy()

#...Changes happen here...

new_df = df[df['Part Number'] != original].join(original, rsuffix = " Changed")

As the two answer above addressed well the question, by curious, I make a small simulation of data and compare the run time of the two approaches. I post it here in case it may be useful for someone else.

executing time for a dataframe of shape (400000, 2):

• executed in 17.8s, finished 09:24:38 2021-01-14
• executed in 17.5s, finished 09:24:56 2021-01-14

So basically, the executing time is comparable.

Thanks Pablo C and Jezrael for sharing. Kr.

Method proposed by Pablo C:

import pandas as pd

df = pd.DataFrame({'Part Number': ['aa', 'bb'], 'other':[1, 2]})
# print(df)
df = pd.concat([df] * 200000, ignore_index=True)

original = df['Part Number'].copy()

#...Changes happen here...
df['Part Number'] = df['Part Number'].apply(lambda x: "ab" if "aa" in x else x)


#print('after change', df)
new_df = df[df['Part Number'] != original].copy()
print( new_df.head(3))

  Part Number  other
0          ab      1
2          ab      1
4          ab      1

Method proposed by Jezrael:

df = pd.DataFrame({'Part Number': ['aa', 'bb'], 'other':[1, 2]})
df = pd.concat([df] * 200000, ignore_index=True)
# print(df)
orig = df['Part Number'].astype(str)

#...Changes happen here...
df['Part Number'] = df['Part Number'].apply(lambda x: "ab" if "aa" in x else x)

df1 = df[orig.str.contains('|'.join(d1.keys()))]
print(df1.head(3))

  Part Number  other
0          ab      1
2          ab      1
4          ab      1

This worked for me taking reference from Pablo C:

import pandas as pd
df = pd.read_csv('sample.csv')
original = df['Part Number'].copy()

final_df = df.reindex(['ID','Part Number','Changed Part Number'],axis=1)
df['Part Number']= df['Part Number'].astype(str).str.replace('=', 'Equals')
final_df['Changed Part Number'] = final_df['Changed Part Number'].fillna(df['Part Number'])

new_df = final_df[df['Part Number'] != original].copy()
new_df

First you can simplify replacement by create dictionary, escape regex in dictioanry comprehension and pass to Series.replace:

df = pd.DataFrame({'Part Number':['8/()2-2','3+/5==3-\\5[]',
                                '3-]%[ [";','5']})
        
import re    

d = {'+':'-PLUS',
     '/':'-SLASH-',
     '\\':'-SLASH-',
     ' ':'-',
     '_':'-',
     '.':'-',
    '"':'',
    '(':'',
    ')':'',
    '%':'-'
}
   
d1 = {rf"{re.escape(k)}":v for k, v in d.items()}

orig = df['Part Number'].astype(str)
df['Part Number'] = orig.replace(d1, regex=True)
print (df)
                    Part Number
0                   8-SLASH-2-2
1  3-PLUS-SLASH-5==3--SLASH-5[]
2                      3-]-[-[;
3                             5

And then for filter replaced rows in boolean indexing with Series.ne:

df1 = df.assign(Original = orig)[df['Part Number'].ne(orig)]
print (df1)
                    Part Number      Original
0                   8-SLASH-2-2       8/()2-2
1  3-PLUS-SLASH-5==3--SLASH-5[]  3+/5==3-\5[]
2                      3-]-[-[;     3-]%[ [";