Pandas failure in Linux, not occuring in Windows - missing _data attribute

Viewed 202

I run my python scripts in RHEL Linux, and I get the following error:

Traceback (most recent call last):
  File "main.py", line 162, in <module>
    find_deltas(logging, snapshot_id)
  File "/ariel/python_scripts/ariel_deltas/deltas.py", line 71, in find_deltas
    data = prepare_frames(logging, file_extracts)
  File "/ariel/python_scripts/ariel_deltas/deltas.py", line 606, in prepare_frames
    logging.info("df_old has %d records", len(df_old))
  File "/ariel/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 1041, in __len__
    return len(self.index)
  File "/ariel/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5270, in __getattr__
    return object.__getattribute__(self, name)
  File "pandas/_libs/properties.pyx", line 63, in pandas._libs.properties.AxisProperty.__get__
  File "/ariel/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5270, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute '_data'
Traceback (most recent call last):
  File "main.py", line 162, in <module>
    find_deltas(logging, snapshot_id)
  File "/ariel/python_scripts/ariel_deltas/deltas.py", line 71, in find_deltas
    data = prepare_frames(logging, file_extracts)
  File "/ariel/python_scripts/ariel_deltas/deltas.py", line 606, in prepare_frames
    logging.info("df_old has %d records", len(df_old))
  File "/ariel/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 1041, in __len__
    return len(self.index)
  File "/ariel/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5270, in __getattr__
    return object.__getattribute__(self, name)
  File "pandas/_libs/properties.pyx", line 63, in pandas._libs.properties.AxisProperty.__get__
  File "/ariel/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5270, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute '_data'

I am effectively reading in a dataframe from Oracle, writing it to a pickle file, and then reading in the pickle file, also reading in yesterdays pickle file, and then doing a join on the primary key.

Why on earth would Linux generate an error about a missing "_data" attribute, when the code runs fine on the exact same data set in Windows?!

Reading in the pickle file in Linux, the columns are as expected.

>>> df.columns
Index(['AS_OF_DT', 'VARIATION_REQUEST_ID', 'LU_NUMBER', 'LU_TITLE', 'COUNTRY',
       'ARCHIVED', 'APPLIED', 'LU_DESCRIPTION', 'HA_LU_REF_NO', 'REMARKS',
       'LU_CATEGORY', 'VARIATION_TYPE', 'INSERT_UPDATE_TIME',
       'INSERT_UPDATE_USER', 'MERGED', 'REVISION_NUMBER', 'VERSION_SEQ',
       'RECORD_ID', 'IMPLEMENTED_SEQ', 'RMS_VERSION_SEQ',
       'REASON_FOR_LOCAL_UPDATE', 'C_ECTD_SEQUENCE_NO', 'INSERT_TIME',
       'ARCHIVED_DATE', 'REASON_FOR_MERGE', 'SCRN_NO'],
      dtype='object')
>>>

The function generating the issue is below:

def prepare_frames(logging, file_extracts):
    # file_extracts is a tuple of dictionaries
    # old_file
    # new_file
    # file_info

    # file_info is a dict describing the file master record including the join keys
    # {"file_id":file_id, "file_desc": r.FILE_DESC, "file_prefix": r.FILE_PREFIX, "compare_col": r.COMPARE_COL}

    # old_file and new_file dictionaries describes the file name of the older snapshot file to be compared
    # old_file["new_old"] = "old"
    # old_file["extract_id"] = extract_id
    # old_file["file_id"] = file_id
    # old_file["file_name"] = file_name
    # old_file["snapshot_id"] = snapshot_id
    # old_file["num_records"] = num_records

    # Strip columns which we know will be different, to remove false positives such as AS_OF_DT

    logging.info("Start: Reading in DataFrames for analysis from pickle files.")

    data = []

    for extract in file_extracts:
        old_file = extract[0]
        new_file = extract[1]
        file_info = extract[2]  # the dictionary

        old_file_name = old_file["file_name"]
        new_file_name = new_file["file_name"]
        logging.info("Reading in old snapshot from pickle file: %s", old_file_name)
        df_old = pd.read_pickle('snapshots/' + old_file_name)
        logging.info("Reading in new snapshot from pickle file: %s", new_file_name)
        df_new = pd.read_pickle('snapshots/' + new_file_name)

        logging.info("df_old has %d records", len(df_old))
        logging.info("df_new has %d records", len(df_new))




        # before we do any comparisons we need to remove as_of_dt type values as this will produce false deltas
        #if "AS_OF_DT" in df_new.columns:
        #    del df_new["AS_OF_DT"]
        #    del df_old["AS_OF_DT"]

        #if "AS_OF_DATE" in df_new.columns:
        #    del df_new["AS_OF_DATE"]
        #    del df_old["AS_OF_DATE"]

        data.append((df_old, df_new, old_file, new_file, file_info))

    logging.info("End: Reading in DataFrames for analysis from pickle files.")

    return data

Line 606 is this one:

logging.info("df_old has %d records", len(df_old))

df_old and df_new are basically pickle files read into a dataframe. I copy the same pickle files to windows, and no issue at all

UPDATE: Looks like it was a logic error, the dataframe was actually empty!

1 Answers

I had the same issue. I was using pandas=1.0.4 within conda environment. Updating pandas to 1.1.0 solved my problem.

Hope that works.

Related