Admin
1
I have decided to work with Python for processing graphs. I use NetworkX for loading the edge list to construct the graph. When I try to load edges with nx.read_edgelist I receive MemoryError.
The graph has 4 million nodes and 34 million edges and my PC has 12GB of RAM. Is it possible to load that graph into 12GB? Is there any optimal solution?
NetworkX uses dictionaries to implement graph data structures. Each edge uses at least 100 bytes of memory. Hence, with 34 million edges, you would would likely need more than 40 GB.
One solution would be to use cloud computing and scale up memory as needed.
An alternative to NetworkX is graph-tool, which is a Python module that implements core data structures and algorithms in C++.
