There is a TLDR at the bottom. I have a python class as follows which is used to make nodes for D* Pathfinding (Pathfinding process not included):
import numpy as np
from tqdm import tqdm
import sys
class Node():
"""A node class for D* Pathfinding"""
def __init__(self,parent, position):
self.parent = parent
self.position = position
self.tag = "New"
self.state = "Empty"
self.h = 0
self.k = 0
self.neighbours = []
def __eq__(self, other):
if type(other) != Node : return False
else: return self.position == other.position
def __lt__(self, other):
return self.k < other.k
def makenodes(grid):
"""Given all the enpoints, make nodes for them"""
endpoints = getendpoints(grid)
nodes = [Node(None,pos) for pos in endpoints]
t = tqdm(nodes, desc='Making Nodes') #Just creates a progress bar
for node in t:
t.refresh()
node.neighbours = getneighbours(node,grid,nodes)
return nodes
def getneighbours(current_node,grid,spots):
'''Given the current node, link it to it's neighbours'''
neighbours = []
for new_position in [(0, -1), (0, 1), (-1, 0), (1, 0)]: # Adjacent nodes
# Get node position
node_position = (int(current_node.position[0] + new_position[0]), int(current_node.position[1] + new_position[1]))
# Make sure within range
if node_position[0] > (len(maze) - 1) or node_position[0] < 0 or node_position[1] > (len(maze[len(maze)-1]) -1) or node_position[1] < 0:
continue
# Create new node
new_node = spots[spots.index(Node(None, node_position))]
neighbours.append(new_node)
return neighbours
def getendpoints(grid):
"""returns all locations on the grid"""
x,y = np.where(grid == 0)
endpoints = [(x,y) for (x,y) in zip(x,y)]
return endpoints
"""Actually Making The Nodes Goes as Follows"""
grid = np.zeros((100,100))
spots = makenodes(grid)
sys.setrecursionlimit(40000)
grid = np.zeros((100,100))
spots = makenodes(grid)
with open('100x100 Nodes Init 4 Connected', 'wb') as f:
pickle.dump(spots, f)
print('Done 100x100')
The program runs well however this node making process takes 3min on 100x100 but over a week on 1000x1000. To counter this, I am using pickle.dump() to save the all the nodes, then when I run my program I can use spots = pickle.load(...). On grid=np.zeros((40,50)) I have to set the recursion limit to around 16000, on 100x100, I increase the recursion limit to 40000 but the kernel dies.
To help you visualise how what the nodes are and how they are connected to neighbours, refer to the image I drew where black: Node and white: connection. 
If I change, getneighbours() to return tuples of (x,y) coordinates (instead of node objects) then I am able to pickle the nodes as they're no longer recursive. This approach slows down the rest of the program because each time I want to refer to a node I have to reconstruct it and search for it in the list spots. (I have simplified this part of the explanation as the question is already very long).
How can I save these node objects while maintaining their neighbours as nodes for larger grids?
TLDR: I have a node class that is recursive in that each node contains a reference to it's neighbours making it highly recursive. I can save the nodes using pickle for grid = np.zeros((40,50)) and setting a fairly high recursion limit of 16000. For larger grids I reach a point where I reach the max system recursion when calling pickle.dump