I am right now searching and saving the graph structure of english wiktionary, saving each page as a vertex and its hyperlinked pages as another nodes that are connected with unweighted edges.
So I had coded an traversing algorithm and implemented it with Python, so the traverser first visit a page then crawls every hyperlinks on that arbitrarily given page then decides randomly which page to visit next.
This algorithm become exponentially inefficient after it crawls 20,000 nodes(out of 1,000,000) since it keep visiting already visited nodes.
So I had added another function that before visit, check whether the node has been visited or not.
However, still, it is too slow. It crawls 200 pages only during an hour on average.
Now, let's mathematically formulate the problem I want to solve.
Suppose there $\exists G(V,E)$ such that $|V|, |E| \in \Bbb N$ which are finite. Then with each algorithm a traverser can reveal the structure of $G(V,E) $ most efficiently where the 'efficient' defined by smallest number of steps from $V_i \ \text{to} \ V_j$ where $i,j \in V$ ? The traverser already knows the set $V$ but doesn't know $E$ at all. The $G$ is directed and unweighted.
The traverser keep increasing its partial information about Graph. Then I roughly guess the optimal traversing strategy might keep changing. I'd like to discuss about that. Simply speaking,
How would the algorithm be affected while the traverser keep increasing its knowledge over the graph?