To whom it may concern,
I am trying to load a ttl file into the stardog but it fails. The ttl file has around 10 million triples. When I use the bulk load command, which is “stardog-admin db create –n mydb data.ttl” I receive an error which is “exception during index bulk loading”. I also could not load it from the workbench. What are the appropriate approaches to resolve the issue?
Regards,
Shahrzad
Hi Shahrzad,
Can you share the full stack trace from your stardog.log for this error?
Best,
Clark
Hi,
yes, sure. Here is the log for the error.
INFO 2021-12-13 17:05:22,738 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 17% complete in 00:00:06 (1.6M triples - 261.9K triples/sec)
INFO 2021-12-13 17:05:23,911 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 20% complete in 00:00:07 (1.9M triples - 260.9K triples/sec)
INFO 2021-12-13 17:05:25,515 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 24% complete in 00:00:08 (2.3M triples - 257.7K triples/sec)
INFO 2021-12-13 17:05:27,051 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 28% complete in 00:00:10 (2.7M triples - 256.2K triples/sec)
INFO 2021-12-13 17:05:28,216 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 32% complete in 00:00:11 (3.1M triples - 264.1K triples/sec)
INFO 2021-12-13 17:05:29,594 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 35% complete in 00:00:13 (3.4M triples - 259.2K triples/sec)
INFO 2021-12-13 17:05:30,821 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 38% complete in 00:00:14 (3.7M triples - 257.2K triples/sec)
INFO 2021-12-13 17:05:31,895 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 41% complete in 00:00:15 (4.0M triples - 258.7K triples/sec)
INFO 2021-12-13 17:05:33,018 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 43% complete in 00:00:16 (4.2M triples - 252.6K triples/sec)
INFO 2021-12-13 17:05:34,627 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 46% complete in 00:00:18 (4.5M triples - 246.7K triples/sec)
INFO 2021-12-13 17:05:36,470 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 48% complete in 00:00:19 (4.7M triples - 233.9K triples/sec)
INFO 2021-12-13 17:05:37,758 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 51% complete in 00:00:21 (5.0M triples - 233.8K triples/sec)
INFO 2021-12-13 17:05:38,816 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 53% complete in 00:00:22 (5.2M triples - 231.7K triples/sec)
INFO 2021-12-13 17:05:40,141 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 56% complete in 00:00:23 (5.4M triples - 231.0K triples/sec)
INFO 2021-12-13 17:05:41,470 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 58% complete in 00:00:24 (5.6M triples - 226.3K triples/sec)
INFO 2021-12-13 17:05:42,642 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 61% complete in 00:00:26 (5.9M triples - 227.2K triples/sec)
INFO 2021-12-13 17:05:43,682 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 63% complete in 00:00:27 (6.1M triples - 225.5K triples/sec)
INFO 2021-12-13 17:05:44,770 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 66% complete in 00:00:28 (6.4M triples - 227.1K triples/sec)
INFO 2021-12-13 17:05:46,109 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 68% complete in 00:00:29 (6.6M triples - 223.2K triples/sec)
INFO 2021-12-13 17:05:47,473 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 71% complete in 00:00:30 (6.9M triples - 223.1K triples/sec)
INFO 2021-12-13 17:05:48,625 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 73% complete in 00:00:32 (7.1M triples - 221.3K triples/sec)
INFO 2021-12-13 17:05:49,930 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 76% complete in 00:00:33 (7.4M triples - 221.4K triples/sec)
INFO 2021-12-13 17:05:51,414 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 78% complete in 00:00:34 (7.6M triples - 217.7K triples/sec)
INFO 2021-12-13 17:05:52,678 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 81% complete in 00:00:36 (7.9M triples - 218.1K triples/sec)
INFO 2021-12-13 17:05:53,859 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 83% complete in 00:00:37 (8.1M triples - 216.3K triples/sec)
INFO 2021-12-13 17:05:55,235 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 86% complete in 00:00:38 (8.4M triples - 216.1K triples/sec)
INFO 2021-12-13 17:05:56,309 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 88% complete in 00:00:39 (8.5M triples - 215.0K triples/sec)
INFO 2021-12-13 17:05:57,853 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 89% complete in 00:00:41 (8.6M triples - 209.4K triples/sec)
INFO 2021-12-13 17:05:59,659 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 91% complete in 00:00:43 (8.8M triples - 205.0K triples/sec)
INFO 2021-12-13 17:06:01,152 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 92% complete in 00:00:44 (9.0M triples - 202.0K triples/sec)
INFO 2021-12-13 17:06:05,946 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 94% complete in 00:00:49 (9.1M triples - 184.6K triples/sec)
INFO 2021-12-13 17:06:07,091 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 96% complete in 00:00:50 (9.3M triples - 184.2K triples/sec)
INFO 2021-12-13 17:06:08,405 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 98% complete in 00:00:51 (9.5M triples - 183.2K triples/sec)
INFO 2021-12-13 17:06:10,768 [Stardog.Executor-90] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 100% complete in 00:00:54 (9.7M triples - 178.6K triples/sec)
INFO 2021-12-13 17:06:12,291 [stardog-user-6] com.complexible.stardog.index.Index:printInternal(319): Parsing triples: 100% complete in 00:00:55 (9.7M triples - 173.8K triples/sec)
INFO 2021-12-13 17:06:12,292 [stardog-user-6] com.complexible.stardog.index.Index:stop(331):
INFO 2021-12-13 17:06:12,292 [stardog-user-6] com.complexible.stardog.index.Index:stop(334): Parsing triples finished in 00:00:55.700
WARN 2021-12-13 17:06:15,488 [stardog-user-6] com.complexible.stardog.protocols.http.server.StardogUndertowErrorHandler:accept(68): Unexpected exception was handled by the server
com.stardog.starrocks.CorruptionException: Corruption: file is too short (35489269 bytes) to be an sstableH:\Apps\stardog\stardog-latest\stardog-7.8.1\startdog_home\data/000514.sst
at com.stardog.starrocks.NativeKVStore.importFiles(Native Method) ~[stardog-7.8.1.jar:?]
at com.stardog.starrocks.NativeKVStore.importFiles(NativeKVStore.java:152) ~[stardog-7.8.1.jar:?]
at com.stardog.starrocks.BaseSstFileWriter.commit(BaseSstFileWriter.java:50) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.dht.dictionary.BaseLoadValueFile.persist(BaseLoadValueFile.java:56) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.dht.dictionary.BaseHashDictionary.persist(BaseHashDictionary.java:191) ~[stardog-7.8.1.jar:?]
at com.complexible.common.base.AutoCloser.close(AutoCloser.java:50) ~[stardog-utils-common-7.8.1.jar:?]
at com.complexible.common.base.AutoCloser.close(AutoCloser.java:37) ~[stardog-utils-common-7.8.1.jar:?]
at com.complexible.stardog.dht.dictionary.BaseHashDictionary.close(BaseHashDictionary.java:153) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.dht.rocks.RocksHashDictionary$Builder.build(RocksHashDictionary.java:489) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.index.dictionary.InliningMappingDictionary$Builder.build(InliningMappingDictionary.java:80) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.index.disk.rocks.RocksConcurrentIndexLoader.build(RocksConcurrentIndexLoader.java:314) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.index.disk.rocks.RocksConcurrentIndexLoader.build(RocksConcurrentIndexLoader.java:279) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.action.CreateDatabaseKernelAction.bulkLoad(CreateDatabaseKernelAction.java:538) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.action.CreateDatabaseKernelAction.loadDatabase(CreateDatabaseKernelAction.java:371) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.action.CreateDatabaseKernelAction.execute(CreateDatabaseKernelAction.java:264) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.StardogKernel.createDatabase(StardogKernel.java:1346) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.StardogKernel.createDatabase(StardogKernel.java:1439) ~[stardog-7.8.1.jar:?]
at com.complexible.stardog.protocols.http.server.AdminDatabaseService.createNewDatabase(AdminDatabaseService.java:803) ~[stardog-protocols-http-server-7.8.1.jar:?]
at com.stardog.http.server.undertow.jaxrs.ExtractRoutes.lambda$handleIt$5(ExtractRoutes.java:192) ~[stardog-protocols-http-server-7.8.1.jar:?]
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [shiro-core-1.7.1.jar:1.7.1]
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [shiro-core-1.7.1.jar:1.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_291]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_291]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_291]
INFO 2021-12-13 17:06:20,880 [memory-monitor] com.complexible.stardog.api.NativeMemoryMonitor:reportStatus(127): Memory usage NaN% - initiating detailed analysis - will check again in 1m
INFO 2021-12-13 17:06:20,881 [memory-monitor] com.complexible.stardog.api.NativeMemoryMonitor:reportStatus(128): Stardog JVM memory usage
OSMemory{ virtualSize=0B, residentSize=0B, residentSizePeak=0B, regionCount=0, pinnedSize=0B, vmTotal=0B, vmAvailable=0B, pageSize=0B}
INFO 2021-12-13 17:06:20,886 [memory-monitor] com.complexible.stardog.api.NativeMemoryMonitor:reportStatus(133): Stardog Metrics
Regards,
Shahrzad
Hi Shahrzad,
This error possibly indicates that you ran out of space either in your %STARDOG_HOME% directory or in java.io.tmpdir. Can you ensure that there is enough space in both of these locations when loading the file?
Best,
Clark
Hi Clark,
finally, I managed to load the data into the database but I ran into another trouble that I really do not know what I should do.
as the documentation said, I followed all the steps again to create the database. In the column of the database in the workbench, I click on the load data and after some seconds it says that the data is loaded completely. Now I restart the database and in that column, it shows that the number of the triple counts is 10M. Now the problem is that when I run a query, even a very simple triple pattern, nothing is returned as a result. I am really confused that why it happens. I have attached the screenshot maybe it helps better.
Regards,
Shahrzad
Hi Shahrzad,
Could you send us your full stardog.log, starrocks.log, and every file in $STARDOG_HOME/data of the form LOG*?
Best,
Clark
Hi,
Here is the another log file. I could not upload them all together.
Regards,
Shahrzad
starrocks.log (110.5 KB)
Hi,
Here are the other three log files that were available in the Stardog_Home/data folder.
I just need to mention that I tried to import the data through the python code using the pystardog library but it also did not work.
Regards,
Shahrzad
LOG.old.1639574410964796.txt (324.3 KB)
LOG.old.1639644913678732.txt (355.3 KB)
LOG.txt (854.5 KB)
Hi Shahrzad,
I was able to reproduce the first issue you ran into but not the second. Would you be willing to send the data you are trying to load or trying the process again on a clean STARDOG_HOME directory and sending me the same logs I asked for before (stardog.log, starrocks.log, and all of the LOG* files)?
Best,
Clark
Hi Shahrzad,
I may actually have a workaround for this issue. Can you try adding storage.envoptions.use_mmap_writes = false to you stardog.properties file? If that does not work, I have an additional thing we can try.
Best,
Clark
Hi Clark,
I ran into other problems. I tried to start everything from scratch, now the problem I face is that I receive an error regarding an invalid license key although my license key is not expired yet.
I have decided to start stardog and loading the data via docker on Linux. The problem is that when I run the docker command, which is " docker run -it -v ~/stardog-home/:/var/opt/stardog -p 5820:5820 stardog/stardog", I receive an error regarding "stardog_home 'var/opt/stardog 'is not writable by the current user". Does it mean that the problem is my license? or there are other issues?
Thanks for your help in advance.
Regards,
Shahrzad
Hi,
I assume I have solved the problem. In the docker command I used -u root and it worked.
Regards,
Shahrzad
Hi Shahrzad,
The problem you were running into there was a permissions error. You solved it by running as root but it could also be solved by running an appropriate chmod or chown command on the directory. You could also solve it by using a different directory that you have appropriate permissions for.
Best,
Clark
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
