Read csv stream line by line to create an array for Excel Range

Viewed 775

This is my first post - I will be happy to make any corrections required for any mistakes made in the post.

I have been looking through the forums here for a few months and have learned a lot but I cannot seem to accomplish my goal with what I have found.

I need to read a CSV file (Read-Only) when it changes and place the resulting array into and active and open Excel 2016 Tab. I can do this using com and system.io.watcherchangetypes but this is too slow and requires copy paste.

I need to read the csv as fast as possible (under a second) and convert the lines into a usable array for Excel. This whole process has to take under 2 seconds MAX. Some of the CSV's will exceed 180,000 lines as the day goes on.

I work for a Trading Company.

I would be happy with a single column, Tab delimited and multiple Rows. I cant get the multiple rows.

I have to write the range line by line and that takes too long.

I was looking at this one but I am not clear on how to make the whole thing dynamic. There is no set amount of headers and the rows will change as well. I cannot work with any static data at all.

This is the post which prompted me to ask for help: How to use powershell to reorder CSV columns

$export = "\\UNC\to\file\Name.csv"
#$excel = New-Object -ComObject Excel.Application
#$excel.visible = $true
#$workbook = $excel.Workbooks.Add()

$reader = [System.IO.File]::OpenText($export)
$writer = New-Object System.IO.StreamWriter "data2.csv"
for(;;) {
    $line = $reader.ReadLine()
    if ($null -eq $line) {
        break
    }
    $i=1
    $data = $line.Split(",") | %{
    if($_ -ne $null)
    {
        Write-Host $_ $i
        ++$i
    }
    }
    [void]$data.Length
   # $data.GetValue()
    #$writer.WriteLine('{0},{1},{2}', $data[0], $data[1], $data[2])
}
$reader.Close()
#$writer.Close()

Any help will be greatly appreciated!

UPDATE:

I figured it out. The result is probably not the most efficient but it gets me what I need for now while i explore how to better accomplish it with what I have learned.

(Measure-Command { $data = [System.io.File]::Open($export, 'Open', 'Read', 'ReadWrite')
$reader = New-Object System.IO.StreamReader($data)
$count = 0
While($text = $reader.Readline())
{
    If($text -eq $null)
    {
        $reader.Close()
        $data.close()
    }
    ++$count
}
}).TotalSeconds

$array2 = New-Object 'object[,]' $count,1
$end = ++$count
$file = New-Object System.IO.StreamReader -ArgumentList $export
$stringBuilder = New-Object System.Text.StringBuilder
$list = New-Object System.Collections.Generic.List[System.String]
$a = 0
Measure-Command {
    While ($i = $file.ReadLine() -Replace ",","`t")
    {
        if ($i -eq $null)
        {
            $file.close()
            break loop
        }
        $null = $stringBuilder.Append($i)
        $list.Add($i)
        $array2[$a,0] = $i
        ++$a
    }

    $outputString = $stringBuilder.ToString()
    $array = $list.ToArray()
}
1 Answers

You can do something like this

data = pd.read_csv("data1.csv", sep='\s+',header=None)
dataarraynew13phse = np.array(data)
dataarraynew13phse=dataarraynew13phse.flatten()

sep = '\s+' can be useful to decode tabs, in multiple lines And then flatten() can make it in a single row or array

Related