How to edit each line of file and write to file after each edit

user_9904275 0
Viewed 93

I have a file that is very large. I need to rearrange something in the file but the file is too large to load into memory. I was thinking of ways to achieve this goal and what I came up with was just editing the file line by line. So what I need to do is read the file, remove certain columns, and then write the file. As I mentioned before, the file is very large so I need to write the file as the script runs. I will give an example data set and the code that I am using

Here is an example dataset

{'CHROM': {0: 'chr1', 1: 'chr1'}, 'POS': {0: 10397, 1: 12719}, 'ID': {0: '.', 1: '.'}, 'REF': {0: 'CCCCTAA', 1: 'G'}, 'ALT': {0: 'C', 1: 'C'}, 'QUAL': {0: 943.64, 1: 255.34}, 'FILTER': {0: 'VQSRTrancheINDEL99.00to100.00', 1: 'VQSRTrancheSNP99.80to100.00'}, 'INFO': {0: 'AC=1;AF=0.5;AN=2;BaseQRankSum=1.07;ClippingRankSum=-0.322;DP=11;ExcessHet=0.2139;FS=1.056;InbreedingCoeff=0.1828;MQ=27.81;MQ0=0;MQRankSum=1.59;NEGATIVE_TRAIN_SITE;QD=25.5;ReadPosRankSum=0.572;SOR=0.922;VQSLOD=-2.735;culprit=DP', 1: 'AC=1;AF=0.5;AN=2;BaseQRankSum=-0.922;ClippingRankSum=-0.198;DP=7;ExcessHet=0.0067;FS=0;InbreedingCoeff=0.4331;MQ=24.5;MQ0=0;MQRankSum=-1.495;QD=17.02;ReadPosRankSum=1.5;SOR=3.126;VQSLOD=-28.96;culprit=MQ'}, 'FORMAT': {0: 'GT:AD:DP:GQ:PL', 1: 'GT:AB:AD:DP:GQ:PL'}, 'CGND-HDA-03201': {0: '0/1:5,6:11:99:224,0,156', 1: '0/1:0.29:2,5:7:42:126,0,42'}, 'CGND-HDA-03202': {0: '0/1:5,6:11:99:224,0,156', 1: '0/1:0.29:2,5:7:42:126,0,42'}}

Here is the code that I am using

n = 0
for line in open(input, "r+"):
    li=line.strip()
    if li.startswith("#"):
        n = n+1
    if not li.startswith("#"):
        test = li.split("\t")
        test2 = (f"{test[0]}\t{test[1]}\t{test[9:]}")
        with open("output.txt","w") as out:
            out.write(test2, sep='\t')

I have a few problems here

  1. the output contain on one single line, the last line that was read. For example, the file contains one line like this

    chr1 1021791 ['0/1:0.56:22,17:39:99:414,0,623', '0/1:0.56:22,17:39:99:414,0,623']

  2. I don't want the output to have any bracket. I need the output to look more like this

    chr1 1021791 0/1:0.56:22,17:39:99:414,0,623 0/1:0.56:22,17:39:99:414,0,623

  3. I need the ouput file to be tab delimited

Is there a way I can continuously write to a file after each line is edited?

The original file does contain #'s that I didn't show here, so the part of the script that ignores lines with #'s is required I know this can easily be done using bash but I am looking for a solution using python.

python
2 Answers

seperator should be \n instead of \t while writing to the file. Also opening file in w mode inside the for loop overrides the previous contents of the file.

import ast
n = 0
with open("output.txt","w") as out:
    for line in open(input, "r+"):
        li=line.strip()
        if li.startswith("#"):
            n = n+1
        elif not li.startswith("#"):
            test = li.split("\t")
            new_test = ast.literal_eval(str(test[9:]))
            test2 = (f"{test[0]}\t{test[1]}\t{new_test[0]}\t{new_test[1]}\n")       
            out.write(test2)
user_135731680

Prakash Dahal solution works great. After playing with the data a bit more, I realized that test[9:] will likely be a large number of extracted values so I wouldn't want to have to list each of the newvalue's in test2 = (f"{test[0]}\t{test[1]}\t{new_test[0]}\t{new_test[1]}\n")

Since new_test = test[9:] is a list, I combine the list to make a tab-delimited string. This results in the overall desired output

n = 0
with open("output.txt","w") as out:
    for line in open(input, "r+"):
        li=line.strip()
        if li.startswith("#"):
         n = n+1
        elif not li.startswith("#"):
            test = li.split("\t")
            new_test = test[9:]
            new_test = '\t'.join(new_test)
            test2 = (f"{test[0]}\t{test[1]}\t{new_test}\n")       
            out.write(test2)
user_99042750
Related