There was a fatal failure during preparation of 6bdebde5-eb4e-4337-8d87-03832b3bd471 RDF index prepare failed

Support Bug
1

When loading the data, I got the message as follows.

There was a fatal failure during preparation of 6bdebde5-eb4e-4337-8d87-03832b3bd471 RDF index prepare failed

I don't know how to handle this problem.
I'd be glad if you could tell me what I should do to work around this.
Currently, the database holds 5.3 billion triples, and I issued the following command.

stardog data add -g <graph> --server-side http://localhost:1111/MyDB <.nt files>

Thanks.

2

How much memory do you have allocated to Stardog? How many triples are you attempting to add when you get the error?

You may want to also verify that you have enough temp space Home | Stardog Documentation Latest

I suspect that you may be running out of memory or temp space.

3

If that is not the case, can you find a stack trace in your stardog.log that might help us out?

4

Here I got some info. concerning the memory environment.
VIRT RES SHR S %CPU %MEM TIME+ COMMAND 104g 84g 2.2g S 4.6 6.7 29664:45 java -Xms16g -Xmx16g -XX:MaxDirectMemorySize=256g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UseParallelOldGC -XX:+UseCompressedOops -Djavax.xml.datatype.DatatypeFactory=org.apache.xerces.jaxp.datatype.DatatypeFactoryImpl ...

$ cat /proc/meminfo MemTotal: 1323593976 kB MemFree: 16801948 kB Buffers: 495264 kB Cached: 1110134488 kB SwapCached: 116352 kB

$ df -h /tmp Filesystem Size Used Avail Use% Mounted on /dev/sda3 2.0T 475G 1.4T 25% /

Hope this help.

5

Here is the log concerning the problem.
https://gist.github.com/yayamamo/31c7f060e8f2040849ee02b86dee331c

6

When rebooting the stardog process, I’ve gotten the following message.
WARN 2017-04-13 12:23:32,306 [main] com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsManager:doLoad(99): Statistics for database DiskIndex(MyDB) cannot be loaded due to missing files (probably need to be rebuilt) Loading Databases: 100% complete in 00:00:12 Loading Databases: 100% complete in 00:00:12 Loading Databases finished in 00:00:12.539
In addition, the following messages are also recorded in the log file.
INFO 2017-04-13 12:24:38,879 [main] com.complexible.stardog.cli.impl.ServerStart:call(249): Memory options INFO 2017-04-13 12:24:38,879 [main] com.complexible.stardog.cli.impl.ServerStart:call(250): Min Heap Size: 16G INFO 2017-04-13 12:24:38,879 [main] com.complexible.stardog.cli.impl.ServerStart:call(251): Max Heap Size: 15G INFO 2017-04-13 12:24:38,879 [main] com.complexible.stardog.cli.impl.ServerStart:call(252): Max Off-Heap : 256G INFO 2017-04-13 12:24:38,880 [main] com.complexible.stardog.cli.impl.ServerStart:call(253): System Memory: 1.2T

7

Does performing a stardog-admin db repair myDb fix the issue?

8

Am I reading the right number of digits there, you have 1.3Tb of total memory? With 256Gb of memory allocated and 1.4T in temp space I would say that running out of resources is unlikely to be your problem.

9

Running out of disk space where STARDOG_HOME is stored is indeed the most likely explanation. The error message Write error: 8184 != 8191 means Stardog was trying to write 8191 bytes to a file but only 8184 bytes were written.

If there is enough disk space then it might be some other hardware related issue. How many triple were you trying to add? Did you try this more than once and do you get the same error every time?

Best,
Evren

10

Just hard to believe you’d run out of disk space on a monster machine like that but maybe it’s on a small partition or disk quota perhaps?

What is the out put of

$> quota -u user_running_stardog

and

$> df -h

You can just include the filesystem that hosts STARDOG_HOME

and

$> echo $STARDOG_HOME

11

Thank you for your information. Now I realized that at least the process is running out of disk space.
I’m preparing another disk space, and try to load the data again.

Currently, the repair db process is running and going to take a long time.
(Took more than one hour to complete two percent.)

Is there any way to interrupt this process, or must I wait for the completion?

12

Do you have a recent backup or dump? How long did the initial load take? It would be interesting to know if there is a big difference between the time to restore and time to repair. Even if they’re the same a restore would be preferable to a repair since you don’t know if a repair is going to be successful. This obviously depends on if you would lose any data since your last backup.

13

I have a set of RDF data to be loaded, which has about 8 billion triples in total, of which 5.3 billion was loaded.
The total amount of time to load was very long, I think it took more than 20 hours in total.

14

It looks like your on track for 50+ hrs for the repair. (assuming the progress counter is linear. Maybe that's a bad assumption)