dataprep.eda TypeError: Please provide npartitions as an int, or possibly as None if you specify chunksize

user_15988103 0
Viewed 77

Struggling to understand this TypeError coming out of the dataprep package. My setup is very simple and as follows:

import pandas as pd
import numpy as np

df = pd.DataFrame(
    {
        "phone": [
            "555-234-5678",
            "(555) 234-5678",
            "555.234.5678",
            "555/234/5678",
            15551234567,
            "(1) 555-234-5678",
            "+1 (234) 567-8901 x. 1234",
            "2345678901 extension 1234",
            "2345678",
            "800-299-JUNK",
            "1-866-4ZIPCAR",
            "123 ABC COMPANY",
            "+66 91 889 8948",
            "hello",
            np.nan,
            "NULL",
        ]
    }
)

from dataprep.clean import clean_phone
clean_phone(df, "phone")

The resulting error message gets thrown in the terminal (I've omitted file paths and replaced sensitive values with x for security purposes) :

Traceback (most recent call last):
  File "c:\Users\x\x\Documents\Repositories\test.py", line 14, in <module>
    clean_phone(df, "phone")
  File "C:\Users\x\Anaconda3\envs\myenv\lib\site-packages\dataprep\clean\clean_phone.py", line 150, in clean_phone
    df = to_dask(df)
  File "C:\Users\x\Anaconda3\envs\myenv\lib\site-packages\dataprep\clean\utils.py", line 73, in to_dask
    return dd.from_pandas(df, npartitions=npartitions)
  File "C:\Users\x\Anaconda3\envs\myenv\lib\site-packages\dask\dataframe\io\io.py", line 236, in from_pandas
    raise TypeError(
TypeError: Please provide npartitions as an int, or possibly as None if you specify chunksize.

This is a direct attempt to replicate the tutorial shown by the dataprep package team found at: https://docs.dataprep.ai/user_guide/clean/clean_phone.html

The expected output is below, as per the tutorial:

Expected output.

Posting this as the TypeError only shows one semi-relevant result when Googled.

dask-dataframe dask pandas dataframe python
1 Answers

There is a small bug in dataprep package, you can track it in this PR.

In the meantime, one option to avoid the bug is to explicitly convert data to a dask dataframe and pass that into the function:

import numpy as np
import pandas as pd
from dask.dataframe import from_pandas
from dataprep.clean import clean_phone

df = pd.DataFrame(
    {
        "phone": [
            "555-234-5678",
            "(555) 234-5678",
            "555.234.5678",
            "555/234/5678",
            15551234567,
            "(1) 555-234-5678",
            "+1 (234) 567-8901 x. 1234",
            "2345678901 extension 1234",
            "2345678",
            "800-299-JUNK",
            "1-866-4ZIPCAR",
            "123 ABC COMPANY",
            "+66 91 889 8948",
            "hello",
            np.nan,
            "NULL",
        ]
    }
)

# to avoid the bug we are passing ddf, not df
ddf = from_pandas(df, npartitions=2)
clean_phone(ddf, "phone")
user_106935960
Related