Assign list as Dask-Dataframe column

Assigning a list to a Dask dataframe is not trivial

A Dask dataframe is split into partitions. Those partitions may be of different lengths and you do not know those lengths until you actually compute the dataframe.

Thus you cannot simply assign a list: you do not know how to distribute the list across the partitions.

How to assign a list

Knowing the above, we know the missing piece of the puzzle: the partition lengths. Here is a reproducible example:

import dask
import dask.array as da

# Sample data
df = dask.datasets.timeseries()
mylist = list(range(len(df)))

# Work out the lengths of each partition
chunks = df.map_partitions(lambda x: len(x)).compute().to_numpy()

# Build a Dask array with the same partition sizes
myarray = da.from_array(mylist, chunks=tuple(chunks))

# Now you can assign!
df['test'] = myarray

What to do instead

While the above will work, it needs to compute the whole dataframe. That is typically a bad idea, as you only want to compute some smaller final result.

It's quite possible that you can rewrite your code such that the result of your computation (the one producing your list with dates) produces e.g. Dask series and thus does so lazily.

You can try to rephrase your question (or maybe ask another, since this is a reasonable standalone question already) with more details, possibly utilizing dask.datasets.timeseries() for a reproducible example.