How can I create a python dictionary from unstructured text?

user_7112012 0
Viewed 485

I have a set of broken link checker results that exist in a text file:

Getting links from: https://www.foo.com/
├───OK─── http://www.this.com/
├───OK─── http://www.is.com/
├─BROKEN─ http://www.broken.com/
├───OK─── http://www.set.com/
├───OK─── http://www.one.com/
5 links found. 0 excluded. 1 broken.

Getting links from: https://www.bar.com/
├───OK─── http://www.this.com/
├───OK─── http://www.is.com/
├─BROKEN─ http://www.broken.com/
3 links found. 0 excluded. 1 broken.

Getting links from: https://www.boo.com/
├───OK─── http://www.this.com/
├───OK─── http://www.is.com/
2 links found. 0 excluded. 0 broken.

I'm trying to write a script that reads in the file and creates a list of dictionaries with each root link as the key, and its children as the values (including the summary line).

The output I am trying to achieve looks like this:

{"Getting links from: https://www.foo.com/": ["├───OK─── http://www.this.com/", "├───OK─── http://www.is.com/", "├─BROKEN─ http://www.broken.com/", "├───OK─── http://www.set.com/", "├───OK─── http://www.one.com/", "5 links found. 0 excluded. 1 broken."], 
"Getting links from: https://www.bar.com/": ["├───OK─── http://www.this.com/", "├───OK─── http://www.is.com/", "├─BROKEN─ http://www.broken.com/", "3 links found. 0 excluded. 1 broken."],
"Getting links from: https://www.boo.com/": ["├───OK─── http://www.this.com/", "├───OK─── http://www.is.com/", "2 links found. 0 excluded. 0 broken."] }

Here is what I have so far:

result_list = []

with open('link_checker_result.txt', 'r') as f:
    temp_list = f.readlines()
    for line in temp_list:
        result_list.append(line)

Which gives me the output:

['Getting links from: https://www.foo.com/', '├───OK─── http://www.this.com/', '├───OK─── http://www.is.com/', '├─BROKEN─ http://www.broken.com/', '├───OK─── http://www.set.com/', '├───OK─── http://www.one.com/', '5 links found. 0 excluded. 1 broken.', 'Getting links from: https://www.bar.com/', '├───OK─── http://www.this.com/', '├───OK─── http://www.is.com/', '...'  ]

I recognize there are some features that each of these sets share, for instance, a blank line in between, or the fact that they start with "Getting...". Is this something I should try splitting on before writing to a dictionary?

I'm new-ish to Python, so I admit I'm not even sure if I am going in the right direction. Would really appreciate some expert eyes on this! Thanks in advance!

python-2.7 dictionary python
2 Answers

This can be quite short actually, within 4 lines of code:

finalDict = {}
with open('link_checker_result.txt', 'r') as f:
    lines = list(map(lambda line: line.split('\n'),f.read().split('\n\n')))
    finalDict = dict((elem[0],elem[1:]) for elem in lines)
print(finalDict)

Output:

{'Getting links from: https://www.foo.com/': ['+---OK--- http://www.this.com/', '+---OK--- http://www.is.com/', '+-BROKEN- http://www.broken.com/', '+---OK--- http://www.set.com/', '+---OK--- http://www.one.com/'], 'Getting links from: https://www.bar.com/': ['+---OK--- http://www.this.com/', '+---OK--- http://www.is.com/', '+-BROKEN- http://www.broken.com/'], 'Getting links from: https://www.boo.com/': ['+---OK--- http://www.this.com/', '+---OK--- http://www.is.com/']}

What the above code does is, reading the input file and splitting it using two consecutive newline \n characters in order to get the links for each url.

Finally it creates tuples of the first element and the rest of each list and transforms them into key-value pairs in the finalDict dictionary.

An easier to comprehend way is the one below:

finalDict = {}
with open('link_checker_result.txt', 'r') as f:
    # Getting data and splitting in order to get each url and its links as a unique list element.
    data = f.read().split('\n\n')
    # Splitting each of the above created elements and discarding the last one which is redundant.
    links = [line.split('\n') for line in data]
    # Transforming these elements into key-value pairs and inserting them in the dictionary.
    finalDict = dict((elem[0],elem[1:]) for elem in links)
print(finalDict)
user_87102650

This will produce the result you want:

result = {}

with open('link_checker_result.txt', 'r') as f:
    temp_list = f.readlines()
    key = ''
    value = []
    for line in temp_list:
        if not line:
            result[key] = value
            key = ''
            value = []
        elif not key:
            key = line
        else:
            value.append(line)

    if key:
      result[key] = value
user_7532370
Related