"Cannot re-add index information" when loading large dataset

Support Bug
1

Hi I am loading an ontology (all entities are owl classes) that has ~ 100 000 000 triples. At the end of loading using the following command on stardog version 5.0.5.1:

bin/stardog-admin db create -n kb1-inf -o strict.parsing=false -- /kb1/kb-dump-inferred/kb-dump-inferred.nq

I get this message in the logs:

Cannot re-add index information

Which looks like this in the stardog.log file:

INFO  2017-12-18 13:09:36,422 [Stardog.Executor-6283] com.complexible.stardog.StardogKernel:printInternal(314): Computing statistics: 83% complete in 00:03:17 (2173.9K triples/sec)
INFO  2017-12-18 13:09:37,281 [Stardog.Executor-6246] com.complexible.stardog.StardogKernel:printInternal(314): Computing statistics: 84% complete in 00:04:19 (1678.5K triples/sec)
INFO  2017-12-18 13:09:41,197 [XNIO-1 task-8] com.complexible.stardog.StardogKernel:printInternal(314): Computing statistics: 100% complete in 00:04:23 (1968.4K triples/sec)
INFO  2017-12-18 13:09:41,197 [XNIO-1 task-8] com.complexible.stardog.StardogKernel:stop(326):
INFO  2017-12-18 13:09:41,198 [XNIO-1 task-8] com.complexible.stardog.StardogKernel:stop(329): Computing statistics finished in 00:04:23.360
INFO  2017-12-18 13:09:45,695 [Stardog.Executor-6283] com.complexible.stardog.StardogKernel:printInternal(314): Computing statistics: 84% complete in 00:03:27 (2101.7K triples/sec)
INFO  2017-12-18 13:09:49,452 [XNIO-1 task-4] com.complexible.stardog.StardogKernel:printInternal(314): Computing statistics: 100% complete in 00:03:30 (2457.4K triples/sec)
INFO  2017-12-18 13:09:49,453 [XNIO-1 task-4] com.complexible.stardog.StardogKernel:stop(326):
INFO  2017-12-18 13:09:49,453 [XNIO-1 task-4] com.complexible.stardog.StardogKernel:stop(329): Computing statistics finished in 00:03:30.956
ERROR 2017-12-18 13:09:49,454 [XNIO-1 task-4] com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder:build(121): Error computing the selectivity stati
stics
com.complexible.common.io.impl.PageNotFoundException: Page not found: 2
        at com.complexible.common.io.impl.AbstractPagedFile.getPage(AbstractPagedFile.java:145) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.PageManagerImpl.getPageInternal(PageManagerImpl.java:174) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.PageManagerImpl.getPage(PageManagerImpl.java:157) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.MultiPage.freeLinkedPages2(MultiPage.java:641) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.MultiPage.access$100(MultiPage.java:34) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.MultiPage$MultiPageWriter.<init>(MultiPage.java:261) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.MultiPage$MultiPageWriter.<init>(MultiPage.java:249) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.common.io.impl.MultiPage.write(MultiPage.java:228) ~[stardog-utils-common-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.btree.impl.PagedKeysIO.write(PagedKeysIO.java:66) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.btree.impl.ArrayKeys.persist(ArrayKeys.java:74) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.btree.impl.BPlusTreeImpl.persist(BPlusTreeImpl.java:364) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.btree.impl.BPlusTreeBackedTrie.finish(BPlusTreeBackedTrie.java:82) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSets.finish(DiskCharacteristicSets.java:65) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsBuilder.persist(DiskCharacteristicSetsStatisticsBuilder.java:133) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsBuilder.build(DiskCharacteristicSetsStatisticsBuilder.java:116) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.DiskIndex.<init>(DiskIndex.java:57) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.ConcurrentDiskIndexLoader.build(ConcurrentDiskIndexLoader.java:156) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.index.disk.ConcurrentDiskIndexLoader.build(ConcurrentDiskIndexLoader.java:56) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.StardogKernel.bulkLoad(StardogKernel.java:1328) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.StardogKernel.createDatabase(StardogKernel.java:1456) ~[stardog-5.0.5.1.jar:?]
        at com.complexible.stardog.protocols.http.server.AdminDatabaseService.createNewDatabase(AdminDatabaseService.java:433) ~[stardog-protocols-http-server-5.0.5.1.jar:?]
        at com.stardog.http.server.undertow.jaxrs.ExtractRoutes.lambda$handleIt$5(ExtractRoutes.java:192) ~[stardog-protocols-http-server-5.0.5.1.jar:?]
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) ~[shiro-core-1.2.3.jar:1.2.3]
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) ~[shiro-core-1.2.3.jar:1.2.3]
        at com.stardog.http.server.undertow.ErrorHandling.lambda$safeDispatch$1(ErrorHandling.java:71) ~[stardog-protocols-http-server-5.0.5.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
INFO  2017-12-18 13:09:49,987 [XNIO-1 task-4] com.complexible.stardog.docs.db.StardocsConnectableFactory:createDocumentStore(117): Creating document store for kb1-inf at file:////st
ardog/kb1-inf/docs
INFO  2017-12-18 13:09:49,991 [XNIO-1 task-4] com.complexible.stardog.StardogKernel:write(77): Loaded 518,409,379 triples to kb1-inf from 1 file(s) in 03:08:24.243 @ 45.9K triples/s
ec.
INFO  2017-12-18 13:09:49,991 [XNIO-1 task-4] com.complexible.stardog.StardogKernel:write(77): Successfully created database 'kb1-inf'.
ERROR 2017-12-18 13:10:03,199 [XNIO-1 task-8] com.complexible.stardog.protocols.http.server.StardogHttpServiceLoader:accept(228): An exception was handled by the server: Cannot re-a
dd index information

From the log it looks like it is pretty much done, but I do not understand what has caused the error. I have got strict parsing off to avoid an error in one of the datetimes in the input file. I will try to manually delete the triple and try again in the meantime. I think the manual says this type of thing may lead to index corruption.

Thanks,

Sheldon

2

Hi Sheldon,

If you run a stardog-admin db optimize command to re-calculate the statistics, do you again see this error? Did this crash your server, or are you able to successfully run queries against this db?

3

Hi Stephen,

I restarted stardog and lost the dataset unfortunately. I will try load again and the recompute with the command you suggested. It didn't actually crash the server and I could run queries.

Sheldon

4

It appears to have just worked the second time. I will leave it up to you guys whether to close the issue or not.