How do I download and work with wikipedia data dumps?

user_6645587 0
Viewed 5418

I want to count entities/categories in wiki dump of a particular language, say English. The official documentation is very tough to find/follow for a beginner. What I have understood till now is that I can download an XML dump (What do I download out of all the available different files), and parse it (?) to count entities (The article topics) and categories.

This information, if available, is very difficult to find. Please help with some instructions as to how do I work with it or resources where I can learn about it.

Thanks!

wikidata knowledge-graph information-retrieval wikipedia
2 Answers

The dumps are rather unwieldy: Even the small "truthy" dump is 25G. And because RDF is rather verbose, that expands to >100G. So my generic advice is to avoid the dumps.

If you can't help it, https://wdumps.toolforge.org/dumps allows you to create customised subsets of dumps with just the languages/properties/entities you want.

Then, just read it line-by-line and ... do something with each line

Matt_Winkelmann_44940

The exact instructions what do to differ a lot based on your usecase. You can either download the dumps from https://dumps.wikimedia.org/enwiki/ and parse them locally, or you can also contact the API.

If you want to parse the dumps, https://jamesthorne.co.uk/blog/processing-wikipedia-in-a-couple-of-hours/ is a good article that shows how one could do that.

However, parsing the dumps isn't always the best solution. If you want to know the three largest pages, for instance, you could use https://en.wikipedia.org/wiki/Special:LongPages.

In addition to all of this, you can also use https://quarry.wmflabs.org to query the live replica of Wikipedia's database. An example can be found at https://quarry.wmflabs.org/query/38441.

user_103581210
Related