Converting bdf to tab delimited using python - dbfpy3 error: "dbf fields definition is corrupt, fields start does not match"

CJ Knoble 0
Viewed 27

I am EXTREMELY new to Python and have been handed a deprecated script for conducting a batch spatial analysis using arcpy. I've updated everything successfully except for the last few lines, which are used to export the dbf to a tab delimited format. I've been banging my head against this wall for too long now and could really use some help.

The code I'm having trouble with is as follows (please let me know if there's any other info I can provide too):

import arcpy
import os
from arcpy import env
import csv
from dbfpy3 import dbf
import sys

p = arcpy.mp.ArcGISProject(r"C:/Users/...project.aprx")
m = p.listMaps("Layers - Data")[0]


#set workspace to Output Folder 
arcpy.env.workspace = r"C:/Users/...gis/output"
os.chdir(r"C:/Users/...gis/output")


tableList = arcpy.ListTables("*.dbf") # List of all dbf files found in output folder

for table in tableList:
   
    dbf_fn = table 
    csv_fn = table[:-4] + ".txt"
    print (csv_fn)

    in_db = dbf.Dbf(dbf_fn) ### <- ERROR APPEARS HERE 
    out_csv = csv.writer(open(csv_fn, 'wb'), delimiter='\t', quotechar='"', quoting=csv.QUOTE_NONE)

    names = []
    for field in in_db.header.fields:
        names.append(field.name)
    
    out_csv.writerow(names)

    for rec in in_db:
        out_csv.writerow(rec.fieldData)

    in_db.close()

print ("text files complete")

The error I receive is:

Traceback (most recent call last):
  File "C:\Users\...\AppData\Local\ESRI\conda\envs\my_envir\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 363, in RunScript
    exec(codeObject, __main__.__dict__)
  File "C:\Users\...\script.py", line 33, in <module>
    in_db = dbf.Dbf(dbf_fn, True)
  File "C:\Users\...\AppData\Roaming\Python\Python39\site-packages\dbfpy3\dbf.py", line 112, in __init__
    self.header = DbfHeader.parse(self.stream)
  File "C:\Users\...\AppData\Roaming\Python\Python39\site-packages\dbfpy3\header.py", line 158, in parse
    raise ValueError(
ValueError: dbf fields definition is corrupt, fields start does not match.

The goal was to batch convert all dbf files in the output folder to tab delimited .txt files.

I can open the dbf files in ArcGIS Pro with no issues, so they appear to function well enough. I tried using arcpy.conversion.TableToTable() instead of working with the dbf directly, but received an error with the same message. The column headers of the dbfs are things like "MESSAGE_ID", "MMSI", "LATITUDE", "LONGITUDE", so I don't see any reason they should be causing errors. I can't identify what "fields start" it requires to match, so I'm struggling to figure out what the issue could be.

I only somewhat understand this portion of the script overall, but I've spent hours digging into this dbf error in particular with no luck. Any help would be greatly appreciated!

dbf arcpy python
1 Answers

The only issue I could see in your dbf files is that they are set for the Arabic code page, but the data is encoded in utf-8.

Using the dbf library you can override the used code page and condense your code to:

for table_name in tableList:

    with dbf.Table(table_name, codepage='utf8', default_data_types='enhanced') as table:
        dbf.export(table)

Using the enhanced data types means the character fields are automatically stripped of trailing whitespace, which makes the tab-delimited file much smaller.

Disclosure: I am the author of the dbf library.

Ethan Furman0
Related