How to read and manipulate a txt file into json with Apache beam in Python

user_18527011 0
Viewed 166

I have a .txt file which has the some sort of data scrap. I want to read, manipulate and restructure the file into json format, How Can I do this in Python with Apache Beam?

the txt file is something like this.

IDX|99214764|085500|00010541|1|084500|1|ALSX |SG | |00000016325.00|000000000500|000000000500|D|000000006385|00000014400.00|000000004600|00000014425.00|000000000600|000000000000|\'c7\uc0\u8800

IDX|70120724|085500|00010542|1|084500|1|IDFL |LG | |00000007100.00|000000000800|000000000800|D|000000006386|00000006625.00|000000010400|00000006650.00|000000027800|000000000000|\'ff\'e8

already tried something like this but didntwork

import apache_beam as beam
import re
with beam.Pipeline() as pipe:
    #convert txt to json with beam apache
    header = (pipe
        | 'Read' >> beam.io.ReadFromText('DLS.txt', skip_header_lines=(9))
        | 'Find words' >> beam.FlatMap(lambda x: re.findall(r'[A-Za-z\']+', x))
        | 'beam.Filter' >> beam.Filter(lambda x: x != '|')
        | 'Write' >> beam.io.WriteToText('DESS.json',
                                            file_name_suffix='',
                                            num_shards=1,
                                            shard_name_template=''))
apache-beam python
1 Answers

You can definitely use Beam to convert text to json, but it's a bit unclear what you want your json output to look like. Let's say your input is of the form

[nine header lines]
apples | red | $1
bananas | yellow | $3
...

You could do something like

def line_to_dict(line):
    name, color, price = line.split('|')
    return {"name": name.strip(), "color": color.strip(), "price", price.strip()}

with beam.Pipeline() as pipe:
    header = (pipe
        | 'Read' >> beam.io.ReadFromText('fruit.txt', skip_header_lines=9)
        | 'ConvertToDict' >> beam.Map(line_to_dict)
        | 'FormatAsJson' >> beam.Map(json.dumps)
        | 'Write' >> beam.io.WriteToText('fruit.json',
                                            file_name_suffix='',
                                            num_shards=1,
                                            shard_name_template=''))

which would result in a file looking like

{"name": apple, "color": red, "price": "$1"}
{"name": banana, "color": yellow, "price": "$3"}
...
user_5823330
Related