List of lists to array conversion. Mixed strings and floats

user_15835406 0
Viewed 67

I have an array (150,40) that looks like:

list_of_lists= ['name_1' 0.0123
        'name_2' 0.1234
   ... 'name_40' 0.213241
Name: 2015-03-26 16:02:42.117000, dtype: float64,

and so on, 149 more ]

I have two questions:

The 40 names are all the same for all 150 lists, how can I convert these to the columns of an array, and the values (whcih differ across all 150 lists) to rows corresponding to each column name ? Example:

array= [ 'name_1', 'name_2',... 'name_40
          0.0123,    0.1234,     0.213241]

Second, the Name: 2015-03-26 16:02:42.117000 is actually a timestamp, and I would need this to be the 0 column, with 150 rows like:

array= [ 'timestamp'     'name_1', 'name_2',... 'name_40
        16:02:42.117000   0.0123,    0.1234,     0.213241]

I have no clue why the timestamp is the Name of the list in the first place And I have no idea how to convert this to an array for further processing

list numpy arrays python
1 Answers

In regards the first question, one way you could do this, strictly using only numpy would be to create a new numpy structured array and create a set of two for each loops (for x in list), the top level to loop over array then the nested one to loop over each array element, appending it to the new structured array. That being said this approach will result in code that has poor readability and overly complicated.

A better option is to use a pandas DataFrame. You should be able to convert all your numpy structured arrays directly into pandas as simply as this pandas.DataFrame(mylist) or read in the data directly into a dataframe if you're reading it in from external files.

The approach below is starting from your structured arrays (but you could just swap pd.DataFrame to something like pd.read_csv(...) if using csv files or another function for other file formats.

import pandas as pd

# need a list of those arrays to allow easy looping
listofArrays = [array1, array2, ...] 

# now using list comprehension, converting all those to dataframes
listofDataFrames = [pd.DataFrame(arrayX) for arrayX in listofArrays]

# now we can just use pd.Concat to put all those together
completeSet = pd.concat(listofDataFrames)

And with that you have a dataframe with everything pulled together. Which if you strictly need it, can be converted back to a purely numpy data structure.

In regards to the second question, this can be solved using regular expression, or since its a fairly simple string, using split and join.

Its not clear on what your input data format, but if I'm reading correctly this is not in an array yet. So I take it the input looks something like

name: date1 timestamp       data: 213123 12313
name2: date2 timestamp2     data: 21311223 313

Even if it is different, the same principle applies.

Since this is relatively simple, split and join would be the easiest approach

originalnamestring = "Name: 2015-03-26 16:02:42.117000"

splitbyspace = originalnamestring.split(" ")

name = splitbyspace[0]

# and if you want to remove the : at end
name = name[:-1]

# if you want separate date and time

date = splitbyspace[1]
time = splitbyspace[2]

# or combined timestamp

timestamp = " ".join(splitbyspace[1:])
user_164022350
Related