With my current code, I am trying to skip a csv file if it does not contain a value within the actual data that I am looking for.
basically if it has "PROD_NAME" as a column, then it looks for that string and replaces it with the second string in that statement, but the first file in my folder does not have this column name and so the script fails. I've looked into ways to skip but have only seen ways to skip based on the filename itself and not the data within a file not having the correct information. Any help would be appreciated. Thanks!
def worker(files):
filenames = glob.glob(dest_dir + '\\*.csv')
for filename in filenames:
my_file = Path(os.path.join(dest_dir, filename))
#read header
with open(filename) as f:
read_data = f.read()
header = read_data[:read_data.find('!1')]
idx = header.find('\n')
# read data
df1 = pd.read_csv(filename, skiprows=1, encoding='ISO-8859-1', nrows=1) # read column header only - to get the list of columns
dtypes = {}
for col in df1.columns:# make all columns text, to avoid formatting errors
dtypes[col] = 'str'
df1 = pd.read_csv(filename, dtype=dtypes, skiprows=1, encoding='ISO-8859-1', quotechar="'", delimiter='\t')
df1.loc[df1['PROD_NAME'].str.contains('NA_NRF'), 'PROD_NAME'] = 'FA_GUAR'
file_count += 1 # count the fil
worker(files)