Pandas / Dask Reading a semi-tabular text

user_5199427 0
Viewed 36

I have a text file that looks like this:

Version:23
Developer: Ali

NAME AGE IN
- Carol 22 no
- Kyle 31 yes
...

I ma reading it using Dask dataframe (which should be similar to Pandas). The result table should be dataframe look like this:

NAME AGE IN
Carol 22 no
Kyle 31 yes

I am having trouble to get rid of the dash in each row ('-') below the column name '-'. I tried

dd.read_csv(filepath, header = 3, sep="\s+")

which results in a dataframe with has different row size and brings more problems, and I also tried using multiple delimiters but still giving errors.

dd.read_csv(filepath, header = 3, sep="\s-\s+")
dask pandas dataframe parsing
1 Answers

dask.dataframe assumes your data is already in a tabular format. If you insist on using dask, then you will get further with dask.bag, which will load the file line by line. You can then filter out the lines that do not start with a dash, and process the ones that do, encoding them as a json object/dict, which you later convert to dataframe with .to_dataframe().

user_106935960
Related