Despite being impressive in many aspects, Blazegraph's official support has been abandonned years ago. Wikimedia had to manage their own version of Blazegraph.

To remain the closest to Wikipedia's database, we use their provided tools to ingest the data. We favor an approach based on the CLI. Although it cannot benefit from the ingestion of multiple files at a time, it is easily monitorable, and feels more reliable.

However, we had trouble compiling the tools: Lombok was not properly generating constructors; and the Scala compiler dependency threw unintelligible errors. So instead, we used munge.sh provided online; and restricted our goal to create a self-contained blazegraph.jar that contains Wikidata's home-made vocabulary.

• We added a shade target so DataLoader is part of the exported jar;
• We added the dependency to log4j 1.2.17 which the old DataLoader depends on.

Wikimedia recommends to munge.sh the compressed turtle file. From the original file, it creates a large number of compressed turtle files:

./munge.sh -c 50000 \
           -f /DATA/datasets/latest-all.ttl.gz \
           -d /DATA/datasets/wikidata-munged/

…
04:22:45.108 … - Processed 102100000 entities at (2899, 2652, 2354)
04:22:45.108 … - Switching to /DATA/datasets/wikidata-munged//wikidump-000002043.ttl.gz
04:22:47.697 … - Processed 102110000 entities at (2979, 2672, 2363)
04:22:50.619 … - Processed 102120000 entities at (2979, 2672, 2363)
04:22:54.360 … - Processed 102130000 entities at (3022, 2686, 2369)
04:22:57.630 … - Processed 102140000 entities at (2989, 2685, 2371)
04:23:00.375 … - Processed 102150000 entities at (2989, 2685, 2371)
04:23:00.376 … - Switching to /DATA/datasets/wikidata-munged//wikidump-000002044.ttl.gz
04:23:02.485 … - Processed 102160000 entities at (3036, 2700, 2377)

It creates 2044 files formated as wikidump-000000001.ttl.gz. To plan the ingestion of all these files, we checked the idempotency of the operation, i.e., ingesting a statement multiple times has no more effect that ingesting it once. Thus, the ingestion can be paused and resumed easily, if needed.

Finally, we can start the sequential ingestion of the 100GB of downloaded data. The following scripts run the ingester for each file in the targeted $SOURCE_FOLDER. FROM_FILE and TO_FILE allow resuming the ingestion if needed. RWStore.properties is the property file defining Blazegraph's engine and storage. To keep the log of the operation, we redirect every output towards ingestion_log.dat. We allocate -Xmx32G of memory to the JVM, although it does not seem to require more than 10GB.

SOURCE_FOLDER="/DATA/datasets/wikidata-munged"
FROM_FILE="/DATA/datasets/wikidata-munged/wikidump-000000257.ttl.gz" # included
TO_FILE="/DATA/datasets/wikidata-munged/wikidump-000003000.ttl.gz" # excluded

for file in "$SOURCE_FOLDER"/*.ttl.gz; do
    if ([[ "$file" > "$FROM_FILE" ]] || [[ "$file" == "$FROM_FILE" ]]) && [[ "$file" < "$TO_FILE" ]]; then
      java -Xmx32g -cp blazegraph-0.3.154-shaded.jar com.bigdata.rdf.store.DataLoader RWStore.properties "$file" &>> ingestion_log.dat
    fi
done

…
Will load from: /DATA/datasets/wikidata-munged/wikidump-000000589.ttl.gz
Journal file: wikidata.jnl
loading: 15264095 stmts added in 1745.302 secs, rate= 8745, commitLatency=0ms, {failSet=0,goodSet=0}
Load: 15264095 stmts added in 1745.302 secs, rate= 8745, commitLatency=0ms, {failSet=0,goodSet=1}
Total elapsed=1800641ms
…

Using the output log, we plot the number of statements and ingestion time of each compressed turtle file.

The top figure shows that the number of statements widely differ from one file to another, but overall remains constant during the whole process. Put in relation with the bottom figure, a large number of statements often means a long ingestion (as expected).

The bottom figure about ingestion times is more informative: ingestion times are increasing over time. We suspect that it comes from the underlying balanced tree data structure used by Blazegraph for its indexes. Not only the depth of the tree increases, but dichotomic searches to find the insertion location take longer and longer.

Unfortunately, even for us, this is too slow. Resources are largely underexploited: CPU usage remains very low, RAM usage is below 10GB. To avoid opening the journal that now takes 150s, and to hopefully get better multithreading capabilities, we load folders of 100 compressed turtle files.