Python Pandas uses a different seperator depending on if I use debug mode or normal mode in Spyder

user_19434206 0
Viewed 28

I am importing some csv Files with measurement data. These Files have different names depending on the date of measurement and the measurement device. I append the Files corresponding to the same device chronologically to each other.

I have the following Code:

import numpy as np
import pandas as pd
import os

sonicpath = *PathtoFiles*
def import_sonic():
    Dates = np.array([1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029,
                      1030, 1031, 1101, 1102])+20210000
    Sensors = ['UwGilHs40', 'UwGilHs135', 'UwmetuS20', 'UwmetuS20', 'UwMetuS60',
               'UwMetuS100', 'UwMetuS188', 'UwThi3DH80', 'UwThi3DH140']
    Ext = 'csv'

    dic_sonics = {}
    for Sensor in Sensors:
        Filenames = []
        dic_sonics[Sensor] = pd.DataFrame([])
        for Date in Dates:
            Filenames.append('.'.join(['_'.join([str(Date), Sensor]), Ext]))
        for Filename in Filenames:
            with open(os.path.join(sonicpath, Filename)) as f:
                lines = f.readlines()
                count = 0
            for line in lines:
                if 'Timestamp' not in line:
                    count = count+1
                else:
                    skip = count-1
                    break
        
            dic_sonics[Sensor] = dic_sonics[Sensor].append(
                pd.read_csv(os.path.join(sonicpath, Filename),
                            skiprows = skip, header=1, sep = ';', decimal='.',
                            ), ignore_index = True)
    
        timestamps = dic_sonics[Sensor].loc[:, 'Timestamp']
        #   round to seconds and convert to unix Timestamp
        for its in timestamps.index:
            dic_sonics[Sensor].loc[its, 'Timestamp'] = int((timestamps.loc[
                its].round('S') - pd.Timestamp("1970-01-01")).total_seconds())
    return(dic_sonics)

I am sure there are better ways to achieve what I am doing but now my issue:

dic_sonics[Sensor] = dic_sonics[Sensor].append(
            pd.read_csv(os.path.join(sonicpath, Filename),
                        skiprows = skip, header=1, sep = ';', decimal='.',
                        ), ignore_index = True)

When I execute these lines of code with the Run File option everything works as expected. I get the following DataFrame columns:

Output

Index(['Timestamp', 'Counts', 'x_mean', 'x_min', 'x_max', 'x_sigma', 'y_mean', 'y_min', 'y_max', 'y_sigma', 'z_mean', 'z_min', 'z_max', 'z_sigma', 'T_mean', 'T_min', 'T_max', 'T_sigma', 'vel_mean', 'vel_min', 'vel_max', 'vel_sigma', 'dir_mean', 'dir_min', 'dir_max', 'dir_sigma', >'IncX_mean', 'IncX_min', 'IncX_max', 'IncY_mean', 'IncY_min', 'IncY_max'], dtype='object')

When I execute these lines with Debug File pandas somehow uses e as the seperator. I get the following DataFrame columns (And puts out an error because it cant find the column to parse dates):

Output

Index(['Tim', 'stamp;Counts;x_m', 'an;x_min;x_max;x_sigma;y_m', 'an;y_min;y_max;y_sigma;z_m', 'an;z_min;z_max;z_sigma;T_m', 'an;T_min;T_max;T_sigma;v', 'l_m', 'an;v', 'l_min;v', 'l_max;v', 'l_sigma;dir_m', 'an;dir_min;dir_max;dir_sigma'], dtype='object')

When I do Debug File and set a breakpoint before the import occurs and execute that line of code over the console it also works.

I really dont understand why that is happening. Why is pandas suddenly using a different seperator?

Thank you for your help!

Additional information:

Spyder 5.3.0

Python 3.7.8

pandas 1.2.3

numpy 1.21.3

pandas spyder python
1 Answers

Thanks for the Disclaimer. I already found a solution to my issue. If anyone wants to reproduce it and maybe provide additional information on why exactly this is happening feel free to do so.

First we create the File test.csv and fill it with data:

SerialNumber=
AverageInterval=600
Timestamp;Counts

Here is the simplified Code:

import pandas as pd

df = pd.read_csv('./test.csv', skiprows = 1, header = 1, sep = ';')

print(df.columns)

Output

Index(['Timestamp', 'Counts'], dtype='object')

Output when debugging

Index(['Tim', 'stamp;Counts'], dtype='object')

I could solve my issue by changing the input variables:

import pandas as pd

df = pd.read_csv('./test.csv', skiprows = 2, header = 0, sep = ';')

print(df.columns)

Output

Index(['Timestamp', 'Counts'], dtype='object')

Output when debugging

Index(['Timestamp', 'Counts'], dtype='object')

user_194342060
Related