Stardog7 Out Of Space errors

onto_rory · May 7, 2019, 9:10am

Hi

Been test driving the new v7 of the server and been creating several DBs from a reasonably large dataset (489m quads)

The server was started with the following:

export STARDOG_HOME=/home/some-user/stardog7
export STARDOG_SERVER_JAVA_ARGS="-Xms31g -Xmx31g -XX:MaxDirectMemorySize=256g"
bin/stardog-admin server start

And the data is loaded via:

./bin/stardog-admin db create --name murtyX -- /home/some-user/data/quads.trig.gz

I created 7 DBs worth of data OK (X = 1-7) - one after the other, then went for creating 2 at the same time (staggered by about ~5mins). At this point I got errors relating to com.complexible.stardog.index.IndexException: No space left on device

The system did have lots of space left:

df -lh
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-home  2.9T  325G  2.6T  12% /home

Is the space related to the MaxDirectMemorySize setting or some other OS level setting I need to update?

The stack trace is as follows:

WARN  2019-05-03 17:55:48,990 [Stardog.Executor-5584] com.complexible.common.rdf.rio.RDFStreamProcessor:setException(602): Error during loading /home/some-user/data/quads.trig.gz: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
com.stardog.stark.io.RDFHandlerException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
      at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:102) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:49) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.common.rdf.rio.RDFStreamProcessor$Consumer.work(RDFStreamProcessor.java:1060) [stardog-utils-rdf-7.0.0-beta1.jar:?]
      at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:811) [stardog-utils-rdf-7.0.0-beta1.jar:?]
      at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:800) [stardog-utils-rdf-7.0.0-beta1.jar:?]
      at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [guava-26.0-jre.jar:?]
      at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) [guava-26.0-jre.jar:?]
      at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [guava-26.0-jre.jar:?]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
      at com.complexible.stardog.index.impl.ConcurrentIndexUpdater.values(ConcurrentIndexUpdater.java:79) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:93) ~[stardog-7.0.0-beta1.jar:?]
      ... 10 more
Caused by: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
      at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) ~[guava-26.0-jre.jar:?]
      at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:512) ~[guava-26.0-jre.jar:?]
      at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-26.0-jre.jar:?]
      at com.complexible.common.util.concurrent.ExecutionGroup$1.executeAndWait(ExecutionGroup.java:80) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.impl.ConcurrentIndexUpdater.values(ConcurrentIndexUpdater.java:76) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:93) ~[stardog-7.0.0-beta1.jar:?]
      ... 10 more
      Suppressed: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
              at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-26.0-jre.jar:?]
              at com.complexible.common.util.concurrent.ExecutionGroup$1.executeAndWait(ExecutionGroup.java:80) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.impl.ConcurrentIndexUpdater.values(ConcurrentIndexUpdater.java:76) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:93) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:49) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.common.rdf.rio.RDFStreamProcessor$Consumer.work(RDFStreamProcessor.java:1060) [stardog-utils-rdf-7.0.0-beta1.jar:?]
              at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:811) [stardog-utils-rdf-7.0.0-beta1.jar:?]
              at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:800) [stardog-utils-rdf-7.0.0-beta1.jar:?]
              at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) [guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [guava-26.0-jre.jar:?]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
      Caused by: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.waitForFlush(DiskExternalSorter.java:153) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.values(DiskExternalSorter.java:113) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.impl.ConcurrentIndexUpdater$CallableUpdater.lambda$copyData$0(ConcurrentIndexUpdater.java:102) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Caused by: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: No space left on device
              at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-26.0-jre.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.waitForFlush(DiskExternalSorter.java:150) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.values(DiskExternalSorter.java:113) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.impl.ConcurrentIndexUpdater$CallableUpdater.lambda$copyData$0(ConcurrentIndexUpdater.java:102) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Caused by: com.complexible.stardog.index.IndexException: No space left on device
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.flushNow(DiskExternalSorter.java:178) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.lambda$flush$0(DiskExternalSorter.java:142) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Caused by: java.io.IOException: No space left on device
              at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) ~[?:1.8.0_181]
              at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) ~[?:1.8.0_181]
              at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) ~[?:1.8.0_181]
              at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_181]
              at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:778) ~[?:1.8.0_181]
              at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:764) ~[?:1.8.0_181]
              at com.complexible.common.nio.Channels2.writeFullyAtPosition(Channels2.java:48) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.disk.ChannelFile.writeInternal(ChannelFile.java:180) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.disk.ChannelFile.write(ChannelFile.java:176) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.SinglePage$Writer.flush(SinglePage.java:228) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.compression.AbstractDataCompressor$2.flush(AbstractDataCompressor.java:273) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.PageObjectWriterImpl.flushPage(PageObjectWriterImpl.java:108) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.PageObjectWriterImpl.write(PageObjectWriterImpl.java:71) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.AbstractObjectWriter.write(AbstractObjectWriter.java:25) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.writeBucket(DiskExternalSorter.java:183) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.flushNow(DiskExternalSorter.java:168) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.lambda$flush$0(DiskExternalSorter.java:142) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Suppressed: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
              at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-26.0-jre.jar:?]
              at com.complexible.common.util.concurrent.ExecutionGroup$1.executeAndWait(ExecutionGroup.java:80) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.impl.ConcurrentIndexUpdater.values(ConcurrentIndexUpdater.java:76) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:93) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.IndexUpdaterHandler.handleStatements(IndexUpdaterHandler.java:49) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.common.rdf.rio.RDFStreamProcessor$Consumer.work(RDFStreamProcessor.java:1060) [stardog-utils-rdf-7.0.0-beta1.jar:?]
              at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:811) [stardog-utils-rdf-7.0.0-beta1.jar:?]
              at com.complexible.common.rdf.rio.RDFStreamProcessor$Worker.call(RDFStreamProcessor.java:800) [stardog-utils-rdf-7.0.0-beta1.jar:?]
              at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) [guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [guava-26.0-jre.jar:?]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
      Caused by: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.waitForFlush(DiskExternalSorter.java:153) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.values(DiskExternalSorter.java:113) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.impl.ConcurrentIndexUpdater$CallableUpdater.lambda$copyData$0(ConcurrentIndexUpdater.java:102) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Caused by: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: No space left on device
              at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) ~[guava-26.0-jre.jar:?]
              at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-26.0-jre.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.waitForFlush(DiskExternalSorter.java:150) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.values(DiskExternalSorter.java:113) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.impl.ConcurrentIndexUpdater$CallableUpdater.lambda$copyData$0(ConcurrentIndexUpdater.java:102) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Caused by: com.complexible.stardog.index.IndexException: No space left on device
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.flushNow(DiskExternalSorter.java:178) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.lambda$flush$0(DiskExternalSorter.java:142) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
      Caused by: java.io.IOException: No space left on device
              at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) ~[?:1.8.0_181]
              at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) ~[?:1.8.0_181]
              at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) ~[?:1.8.0_181]
              at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_181]
              at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:778) ~[?:1.8.0_181]
              at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:764) ~[?:1.8.0_181]
              at com.complexible.common.nio.Channels2.writeFullyAtPosition(Channels2.java:48) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.disk.ChannelFile.writeInternal(ChannelFile.java:180) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.disk.ChannelFile.write(ChannelFile.java:176) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.SinglePage$Writer.flush(SinglePage.java:228) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.compression.AbstractDataCompressor$2.flush(AbstractDataCompressor.java:273) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.PageObjectWriterImpl.flushPage(PageObjectWriterImpl.java:108) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.PageObjectWriterImpl.write(PageObjectWriterImpl.java:71) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.common.io.impl.AbstractObjectWriter.write(AbstractObjectWriter.java:25) ~[stardog-utils-common-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.writeBucket(DiskExternalSorter.java:183) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.flushNow(DiskExternalSorter.java:168) ~[stardog-7.0.0-beta1.jar:?]
              at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.lambda$flush$0(DiskExternalSorter.java:142) ~[stardog-7.0.0-beta1.jar:?]
              ... 6 more
Caused by: com.complexible.stardog.index.IndexException: com.complexible.stardog.index.IndexException: No space left on device
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.waitForFlush(DiskExternalSorter.java:153) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.values(DiskExternalSorter.java:113) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.impl.ConcurrentIndexUpdater$CallableUpdater.lambda$copyData$0(ConcurrentIndexUpdater.java:102) ~[stardog-7.0.0-beta1.jar:?]
      ... 6 more
Caused by: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: No space left on device
      at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) ~[guava-26.0-jre.jar:?]
      at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) ~[guava-26.0-jre.jar:?]
      at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-26.0-jre.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.waitForFlush(DiskExternalSorter.java:150) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.values(DiskExternalSorter.java:113) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.impl.ConcurrentIndexUpdater$CallableUpdater.lambda$copyData$0(ConcurrentIndexUpdater.java:102) ~[stardog-7.0.0-beta1.jar:?]
      ... 6 more
Caused by: com.complexible.stardog.index.IndexException: No space left on device
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.flushNow(DiskExternalSorter.java:178) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.lambda$flush$0(DiskExternalSorter.java:142) ~[stardog-7.0.0-beta1.jar:?]
      ... 6 more
Caused by: java.io.IOException: No space left on device
      at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) ~[?:1.8.0_181]
      at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) ~[?:1.8.0_181]
      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) ~[?:1.8.0_181]
      at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_181]
      at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:778) ~[?:1.8.0_181]
      at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:764) ~[?:1.8.0_181]
      at com.complexible.common.nio.Channels2.writeFullyAtPosition(Channels2.java:48) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.common.io.disk.ChannelFile.writeInternal(ChannelFile.java:180) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.common.io.disk.ChannelFile.write(ChannelFile.java:176) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.common.io.impl.SinglePage$Writer.flush(SinglePage.java:228) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.compression.AbstractDataCompressor$2.flush(AbstractDataCompressor.java:273) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.common.io.impl.PageObjectWriterImpl.flushPage(PageObjectWriterImpl.java:108) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.common.io.impl.PageObjectWriterImpl.write(PageObjectWriterImpl.java:71) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.common.io.impl.AbstractObjectWriter.write(AbstractObjectWriter.java:25) ~[stardog-utils-common-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.writeBucket(DiskExternalSorter.java:183) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.flushNow(DiskExternalSorter.java:168) ~[stardog-7.0.0-beta1.jar:?]
      at com.complexible.stardog.index.disk.btree.impl.DiskExternalSorter.lambda$flush$0(DiskExternalSorter.java:142) ~[stardog-7.0.0-beta1.jar:?]
      ... 6 more

zachary.whitley · May 7, 2019, 10:17am

You showed STARDOG_HOME set to STARDOG_HOME=/some/path/stardog7. Is some/path, home? How much space do you have on /some/path? Could you possibility that you have a disk quota setup?

onto_rory · May 7, 2019, 10:28am

sorry - they're just made up values to hide real locations/system names. They are in /home and I'll update them above

onto_rory · May 7, 2019, 10:34am

checking disk quota setup now ...

can't see anything under /etc/fstab or via xfs_quota (/home is using XFS) but put out request out to sysadmin who setup the box for final confirmation

onto_rory · May 7, 2019, 11:08am

Might be something todo with /tmp maybe? I had this for the root partition - could the load be doing some transient stuff in there?

df -lh
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   44G    6G  88% /

Virtually all of that was in /tmp - I've cleaned it out and I'll try the loading again and monitor

stephen · May 7, 2019, 11:16am

The "No space left on device" error does appear when your java.io.tmpdir is full, not just when the disk itself is full. That is very likely the culprit.

zachary.whitley · May 7, 2019, 12:52pm

I was going to suggest that but I only know that Stardog uses tmp not what or when exactly it uses it for. That might be a nice addition to the docs. The section on tmp space is here

You can set java.io.tempdir JVM arg with STARDOG_SERVER_JAVA_ARGS

onto_rory · May 7, 2019, 3:42pm

I think it definitely is the culprit. There was quite a lot of cruft in the /tmp that I cleared out that took it down to

df -lh
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  7.1G   43G  15% /

When I ran 2 parallel DB creations for the 489m datasets, it max'd the /tmp to about 92% usage during the parsing stage (each tempDir its created was using up about 20G) and then dropped down again on the indexing stage of the load - I just got unlucky in that cruft + 2DBs = no space left

Switched java.io.tmpdir to be in /home and running 3 parallel loads - and it looks fine

Cheers for the help

system · May 21, 2019, 3:42pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Endpoint not responding, "No space left on device" Support	7	230	August 8, 2024
How to recover after a "No space left on device" error Stardog Cloud	5	1092	November 7, 2023
Running out of space when loading Wikidata Support	7	348	August 14, 2023
java.lang.OutOfMemoryError: Java heap space Support	19	2927	May 12, 2017
Stardog Cloud, Java IO exception "No space left on device" Support	6	299	February 14, 2023

Stardog7 Out Of Space errors

Related topics