I am building a database of concepts and neighbour concepts from a certain set of text files (arbitral awards). So far, I use lists, as I believe (not an expert here) it is the computable most simple and efficient way to store and retreive the information I'm using. The structure of those lists is:
memory = [
[tag1,number1,
[
[tag11,number11],
[tag12,number12],
...
]
],
...
]
When feed with a string (usually by the lenght of a paragraph or a single page) my script will look for the tag and then for the sub tags of every tag. Basically:
for tag in memory:
if tag[0] in text:
tag[1]+=1
else:
for subtag in tag[2]:
if subtag[0] in text:
subtag[1]+=1
tag[1]+=1
There are some extra rules to break the search and to avoid repetitions, but you can imagine they don't solve the 'for in for in for' loop problem! (accounting for the 'in text' part)
My purpose is to build a semantic structure by enumeration, by relating tags with sub tags. For example: 'Greetings', with: 'Hi', 'Hello', 'Good Morning', and so on. The numbers say how often a tag is found in the text files I use and the sub-tags numbers say how often a certain sub tag is related with its parent tag.
My problem is that, after a few months using this structure, I have a fair amount of tags and sub tags as to make my script run slow every time it has to search every tag and every sub tag within a given string, and I'm looking for options to solve this problem.
One option I have is to migrate to numpy arrays, but I wouldn't know where to start to be sure I will gain on efficiency and not just translate my problem into a more fancy structure. I am familiar with matrix multiplication and tensor products with numpy, which seem to be applicable here with some sort of convolutional algorithm (just guessing over what I've done before), but I'm not sure if they work as well with strings as I've seen them work with numbers, because I would be multipliying small matrices (tag list) with large matrices (strings), and I have been told that numpy is more usefull with large to large multiplications.
The other option is to use dictionaries, at least as an intermediate step, which I sincerely don't get how whould make things go faster, but they were suggested by an engineer (I'm a lawyer so... no idea). The problem with the last is how to keep track of ocurrencies of keys. I can see how I can translate my list into dictionaries, but even tough I can think of ways to translate and update the ocurrency information, I just feel it won't be more efficient, as the same for loops would have to be executed.
One option would be to set the ocurrencies as the first value of every key... but again, not sure how it would make my script go faster.
I get that the increasing amount of computations won't just go away, but as an amateur-intermediate enthusiast of programming, I understand that my method can be sensibily improved by going further than simple for loops.
Therefore, as a non expert, I would appreciate some information, guidance or just references about how to translate my list into dictionaries or numpy arrays which at the same time help me to loop over tags and sub-tags faster than the basic for loop.