Reading data from text file to postgres with python

Admin 1
Viewed 1233

I can read the content of a test file test.plt in the root directory into a postgres table tempo using this script:

$cat test.plt
Geolife trajectory
WGS 84
Altitude is in Feet
Reserved 3
0,2,255,My Track,0,0,2,8421376
0
39.9756783,116.3308383,0,131.2,39717.4473148148,2008-09-26,10:44:08
39.9756649,116.3308749,0,131.2,39717.4473842593,2008-09-26,10:44:14
39.97564,116.3308749,0,131.2,39717.4474189815,2008-09-26,10:44:17
39.9756533,116.3308583,0,131.2,39717.4474537037,2008-09-26,10:44:20
39.9756316,116.3308299,0,131.2,39717.4474884259,2008-09-26,10:44:23
39.9753166,116.3306299,0,131.2,39717.4480324074,2008-09-26,10:45:10
39.9753566,116.3305916,0,131.2,39717.4480671296,2008-09-26,10:45:13
39.9753516,116.3305249,0,131.2,39717.4481018518,2008-09-26,10:45:16

Python script:

import psycopg2
from config import config
import os
import glob

query = "INSERT INTO tempo (lat, lon, flag, alt, passeddate, gpsdate, gpstime) VALUES (%s, %s, %s, %s, %s, %s, %s)"

path = '~/Desktop/Data/'

conn = None
try:
    #read the connection parameters
    params = config()
    # connect to the PostgreSQL server
    conn = psycopg2.connect(**params)
    cur = conn.cursor()

    # INSERRT data to the database
    with open('test.plt') as file:
        file_content = file.readlines()[6:]
        values = [line.strip().split(',') for line in file_content]
        cur.executemany(query, values)

    cur.close()
        # commit the changes
    conn.commit()
except (Exception, psycopg2.DatabaseError) as error:
    print(error)
finally:
    if conn is not None:
        conn.close()

Results:

postgres=> SELECT * FROM tempo;
 id |    lat     |     lon     | flag |  alt  |    passeddate    |  gpsdate   | gpstime  
----+------------+-------------+------+-------+------------------+------------+----------
    | 39.9756783 | 116.3308383 |    0 | 131.2 | 39717.4473148148 | 2008-09-26 | 10:44:08
    | 39.9756649 | 116.3308749 |    0 | 131.2 | 39717.4473842593 | 2008-09-26 | 10:44:14
    |   39.97564 | 116.3308749 |    0 | 131.2 | 39717.4474189815 | 2008-09-26 | 10:44:17
    | 39.9756533 | 116.3308583 |    0 | 131.2 | 39717.4474537037 | 2008-09-26 | 10:44:20
    | 39.9756316 | 116.3308299 |    0 | 131.2 | 39717.4474884259 | 2008-09-26 | 10:44:23
    | 39.9753166 | 116.3306299 |    0 | 131.2 | 39717.4480324074 | 2008-09-26 | 10:45:10
    | 39.9753566 | 116.3305916 |    0 | 131.2 | 39717.4480671296 | 2008-09-26 | 10:45:13
    | 39.9753516 | 116.3305249 |    0 | 131.2 | 39717.4481018518 | 2008-09-26 | 10:45:16
(8 rows)

I can also get the names of all files in all sub-directories with the .plt extension (stripping the .plt ext ) by replacing the insert statement in the python script with:

for root, dirs, files in os.walk(path):
        for file in files:
            if file.endswith(".plt"):
                print(file.strip('.plt'))

Output:

20081210001529
20081113121334
20081205143505
20081029234123
20081202145929
20081204142253
20081111234235
20081118003844
20081105110052
20081023055305

As you can see, files are named in figures. The goal is to take each file name, insert it into the id field of the tempotable, then its contents into the rest of columns. Repeating this for each in file in all sub-directories.

  1. How do I modify my code so that the filename(e.g. 20081210001529) is added to the insert query (get inserted to the table)

Using the code below, with the intent to read data from all files in sub-directories (i.e. having listed all file in code above) gives error with only the first file in the first sub-directory, listed.

query = "INSERT INTO tempo (lat, lon, flag, alt, passeddate, gpsdate, gpstime) VALUES (%s, %s, %s, %s, %s, %s, %s)"

path = '~/Desktop/Data/'
#Establish connection to postgres
conn = None
try:
    #read the connection parameters
    params = config()
    # connect to the PostgreSQL server
    conn = psycopg2.connect(**params)
    cur = conn.cursor()

    for root, dirs, files in os.walk(path):
        for file in files:
            if file.endswith(".plt"):
                f = open(file, 'r')
                content = f.readlines()[6:]
                values = [lines.strip().split(',') for line in content]
                cur.executemany(query, values)

    cur.close()
        # commit the changes
    conn.commit()
except (Exception, psycopg2.DatabaseError) as error:
    print(error)
finally:
    if conn is not None:
        conn.close()
[Errno 2] No such file or directory: '20081210001529.plt'

I would appreciate your help on this task.

psycopg2 postgresql file python
1 Answers

file just holds the filename, you need to create a full path (os.path.join).

I just put you code into 2 functions (first one to find all plt files, second one to insert each file's contents) that makes error handling easier:

def findFiles(rootDir):
    pltFiles = []
    for path, subdirs, files in os.walk(rootDir):
        for x in files:
            if x.endswith(".plt"):
                pltFiles.append(os.path.join(path, x))  # create full path!
    return pltFiles

def insert(pltFilePath):
    query = "INSERT INTO tempo (lat, lon, flag, alt, passeddate, gpsdate, gpstime) VALUES (%s, %s, %s, %s, %s, %s, %s)"
    conn = psycopg2.connect(**params)
    cur = conn.cursor()

    f = open(file, 'r')
    content = f.readlines()[6:]
    values = [lines.strip().split(',') for line in content]
    status = False
    try:
        cur.executemany(query, values)
        status = True
    except (Exception, psycopg2.DatabaseError) as error:
        print(error)
    finally:
        if conn is not None:
            conn.close()
    return status


path = '/tmp/data'
pltFiles = findFiles(path)

if not pltFiles:
    print("No plt files found!")

for pltFile in pltFiles:
    print("Processing: %s" % pltFile)
    res = insert(pltFile)
    # res holds true or false, so you could rename, move or delete the file:
    #if res:
    #   os.remove(fullPath)
user_72168650
Related