Reformat a text file so it can be used a numpy array, in python?

user_13989644 0
Viewed 178

I have a small chunk of code that i'm using to find the confidence interval from a data set.

from scipy import stats
import numpy as np

a = np.loadtxt("test1.txt")
mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean,
    scale=sigma)

print(conf_int)

However, my text file (test1.txt) is a list of numbers that a) has a square bracket at the start and finish b)is not in equal columns.

"[-10.197663 -22.970129 -15.678419 -15.306197
-12.09961 -11.845362 -18.10553 -25.370747
-19.34831 -22.45586]

np.loadtxt really doesn't seem to like this, so is there any way i can use a function to either read and use the data as is or reformat it?

Any help would be greatly appreciated!

Update so i manged to remove my brackets with the code below

with open('test1.txt', 'r') as my_file:
text = my_file.read()
text = text.replace("[", "")
text = text.replace("]", "")


with open('clean.txt', 'w') as my_file:
my_file.write(text)


a = np.loadtxt("clean.txt")
mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean,
   scale=sigma)

print(conf_int)

Just need to reformat clean.txt so its in one single column now so np.array will recognise it.

Final update

I managed to get it working, using @David Hoffman suggested code and my long work around from above; see below

from scipy import stats
import numpy as np

with open('test1.txt', 'r') as my_file:
    text = my_file.read()
    text = text.replace("[", "")
    text = text.replace("]", "")


with open('clean.txt', 'w') as my_file:
    my_file.write(text)


a = np.array(list(map(float, text.strip("[]").split())))
mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean,
   scale=sigma)

print(conf_int)

Thank you to everyone for taking the time to help, it was very much appreciated, especially to a new coder like me.

numpy python
2 Answers

You can read it as string then replace space with , to make it like list and use eval to convert string list to list type and at last to numpy array.
For your given dummy input

li = """[-10.197663 -22.970129 -15.678419 -15.306197
-12.09961 -11.845362 -18.10553 -25.370747
-19.34831 -22.45586]"""

np.array(eval(li.replace(' ',',')))
array([-10.197663, -22.970129, -15.678419, -27.405807, -11.845362,
       -18.10553 , -44.719057, -22.45586 ])

For given input file - here solution would be

import re
li = open('test1.txt', 'r').read()

np.array(eval(re.sub(r'(\n +| +)',',',li)))
array([-10.197663  , -22.970129  , -15.678419  , -15.306197  ,
        -0.38851437, -12.09961   , -11.845362  , -18.10553   ,
       -25.370747  , -20.575884  , -19.34831   , -22.45586   ,
       -31.209     , -19.68507   , -31.07194   , -28.4792    ,
        ...])
user_60756990

This is what I would do:

import numpy as np
from scipy import stats
import requests

link = "https://pastebin.pl/view/raw/929f5228"

response = requests.get(link)
text = response.text

# with open("test1.txt", "r") as my_file:
#     text = my_file.read()

a = np.array(list(map(float, text.strip("[]").split())))

mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean, scale=sigma)

print(conf_int)

The commented lines are for if you have a file.

There's a lot packed into the string handling line:

  1. The text string is cleaned (removing brackets)
  2. The clean text is split by white space (any length of consecutive whitespace characters are treated as delimiters)
  3. Each split token is converted to a float (this is the map part)
  4. The map generator is converted to a list and passed to the numpy array function

As @Dishin said, there's some weirdness with how your input file is formatted. If you have any control over how the file is written (say from a LabVIEW program or other Python script) it might be worth formatting the data in a more widely accepted format like .csv so that functions like np.loadtxt (or programs like Excel) can read it more easily.

If you're stuck with the files as is you can just make a little utility function like:

def loader(filename):
    with open(filename, "r") as my_file:
        text = my_file.read()

    return np.array(list(map(float, text.strip("[]").split())))

to reuse in your scripts.

user_50300140
Related