I was recently reading a book as prep for an interview and came across the following question:
What will you do when your crawler runs into a honey pot that generates an infinite subgraph for you to wander about?
I wanted to get some solutions to this qn. Personally, I would some form of depth limited search to prevent traversing continuously. Or perhaps use some form of machine learning to detect patterns. Thoughts?