Numpy-recarray, C# structured data

user_13766037 0
Viewed 66

I have written a consumer of data in C# that reads string data from python API script. The best performing method I have found so far is to get the data into a numpy-recarray in python, print it to the console and read it with StreamReader.

I end up with the following string in the reading C# routine. This is one long string (7,000 records) of stock quotes, (Date, Open, High, Low, Close). No carriage returns, no line feeds.

[(datetime.date(2020, 6, 1), 29.80, 30.65, 29.45, 30.24), (datetime.date(2020, 6, 2), 30.49, 31.43, 30.19, 31.17)...]

I now need to parse this somehow into something that I can easily use to insert this data into a SQL database. I know how to do the insert from a list, array, etc. I can see a brut force parsing routing that reads this character by character and rebuilds the records into a List. Then the SQL bulk insert is easy.

I feel like there might be a better approach to alter the python console writer to write this long string in a format that could be converted to a List in C# with Linq or some other assignment. I am a little confused on where to start. Any suggestions?

numpy python c#
1 Answers

You might be happier with a pandas produced csv:

Just to crudely take you list of tuples:

In [25]: alist = [(datetime.date(2020, 6, 1), 29.80, 30.65, 29.45, 30.24), (date
    ...: time.date(2020, 6, 2), 30.49, 31.43, 30.19, 31.17)]                    
In [26]: alist                                                                  
Out[26]: 
[(datetime.date(2020, 6, 1), 29.8, 30.65, 29.45, 30.24),
 (datetime.date(2020, 6, 2), 30.49, 31.43, 30.19, 31.17)]

In [28]: import pandas as pd                                                    
In [29]: df=pd.DataFrame(alist)                                                 
In [30]: df                                                                     
Out[30]: 
            0      1      2      3      4
0  2020-06-01  29.80  30.65  29.45  30.24
1  2020-06-02  30.49  31.43  30.19  31.17
In [31]: df.to_csv('test.csv')                                                  
In [32]: cat test.csv                                                           
,0,1,2,3,4
0,2020-06-01,29.8,30.65,29.45,30.24
1,2020-06-02,30.49,31.43,30.19,31.17

You can refine that output with or without the column headers and row indices.

There is also a numpy savetxt that could be more work to use with a structured array like this.

user_9019250
Related