Memory error after server upgrade

Support Bug
1

Hi after upgrading to 5.3.3 on a server I run into the following error, the system does seem to function as expected though

Computing statistics: 51% complete in 00:01:11 (183,1K triples/sec)
ERROR 2018-08-27 20:42:57,052 [main] com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder:build(121): Error computing the selectivity statistics
com.complexible.stardog.index.IndexException: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.build(ConcurrentCharacteristicSetStatisticsBuilder.java:93) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsBuilder.build(DiskCharacteristicSetsStatisticsBuilder.java:114) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsLoader.rebuild(DiskCharacteristicSetsStatisticsLoader.java:107) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsLoader.load(DiskCharacteristicSetsStatisticsLoader.java:64) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.DiskIndex.<init>(DiskIndex.java:43) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.DefaultDiskIndexReader.read(DefaultDiskIndexReader.java:129) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.io.IndexIO.read(IndexIO.java:324) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.db.DatabaseFactoryImpl.read(DatabaseFactoryImpl.java:126) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.db.DatabaseFactoryImpl.read(DatabaseFactoryImpl.java:55) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.StardogKernel.openDatabase(StardogKernel.java:2708) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.StardogKernel.initDatabases(StardogKernel.java:2432) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.StardogKernel.start(StardogKernel.java:2335) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.StardogKernel.initialize(StardogKernel.java:799) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.Stardog.initKernel(Stardog.java:214) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.Stardog.<init>(Stardog.java:206) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.Stardog.<init>(Stardog.java:60) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.Stardog$StardogBuilder.create(Stardog.java:597) [stardog-5.3.3.jar:?]
	at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:162) [stardog-cli-5.3.3.jar:?]
	at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:41) [stardog-cli-5.3.3.jar:?]
	at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:55) [stardog-cli-5.3.3.jar:?]
	at com.complexible.stardog.cli.admin.CLI.main(CLI.java:186) [stardog-cli-5.3.3.jar:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_181]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_181]
	at com.complexible.common.util.concurrent.ExecutionGroup$1.executeAndWait(ExecutionGroup.java:78) ~[stardog-utils-common-5.3.3.jar:?]
	at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.build(ConcurrentCharacteristicSetStatisticsBuilder.java:88) ~[stardog-5.3.3.jar:?]
	... 20 more
	Suppressed: java.util.concurrent.ExecutionException: com.complexible.stardog.index.IndexException: java.lang.OutOfMemoryError: GC overhead limit exceeded
		at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_181]
		at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_181]
		at com.complexible.common.util.concurrent.ExecutionGroup$1.executeAndWait(ExecutionGroup.java:78) ~[stardog-utils-common-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.build(ConcurrentCharacteristicSetStatisticsBuilder.java:88) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsBuilder.build(DiskCharacteristicSetsStatisticsBuilder.java:114) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsLoader.rebuild(DiskCharacteristicSetsStatisticsLoader.java:107) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsLoader.load(DiskCharacteristicSetsStatisticsLoader.java:64) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.disk.DiskIndex.<init>(DiskIndex.java:43) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.disk.DefaultDiskIndexReader.read(DefaultDiskIndexReader.java:129) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.io.IndexIO.read(IndexIO.java:324) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.db.DatabaseFactoryImpl.read(DatabaseFactoryImpl.java:126) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.db.DatabaseFactoryImpl.read(DatabaseFactoryImpl.java:55) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.StardogKernel.openDatabase(StardogKernel.java:2708) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.StardogKernel.initDatabases(StardogKernel.java:2432) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.StardogKernel.start(StardogKernel.java:2335) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.StardogKernel.initialize(StardogKernel.java:799) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.Stardog.initKernel(Stardog.java:214) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.Stardog.<init>(Stardog.java:206) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.Stardog.<init>(Stardog.java:60) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.Stardog$StardogBuilder.create(Stardog.java:597) [stardog-5.3.3.jar:?]
		at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:162) [stardog-cli-5.3.3.jar:?]
		at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:41) [stardog-cli-5.3.3.jar:?]
		at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:55) [stardog-cli-5.3.3.jar:?]
		at com.complexible.stardog.cli.admin.CLI.main(CLI.java:186) [stardog-cli-5.3.3.jar:?]
	Caused by: com.complexible.stardog.index.IndexException: java.lang.OutOfMemoryError: GC overhead limit exceeded
		at com.complexible.stardog.index.statistics.ComputePropertyStatistics.compute(ComputePropertyStatistics.java:44) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.computePropertyStatistics(ConcurrentCharacteristicSetStatisticsBuilder.java:49) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.lambda$build$1(ConcurrentCharacteristicSetStatisticsBuilder.java:85) ~[stardog-5.3.3.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
		at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
	Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
		at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_181]
		at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_181]
		at com.complexible.common.util.concurrent.ExecutionGroup$1.executeAndWait(ExecutionGroup.java:78) ~[stardog-utils-common-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ComputePropertyStatistics.compute(ComputePropertyStatistics.java:41) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.computePropertyStatistics(ConcurrentCharacteristicSetStatisticsBuilder.java:49) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.lambda$build$1(ConcurrentCharacteristicSetStatisticsBuilder.java:85) ~[stardog-5.3.3.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
		at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
	Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
		at com.complexible.stardog.index.BinaryCounts.create(BinaryCounts.java:89) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.disk.compression.BinaryTransformers$2.create(BinaryTransformers.java:78) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.IndexOrders.lambda$null$1(IndexOrders.java:79) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.IndexOrders$$Lambda$243/1799535763.apply(Unknown Source) ~[?:?]
		at com.complexible.stardog.index.StreamingCountingIndexIterator.computeNext(StreamingCountingIndexIterator.java:67) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.StreamingCountingIndexIterator.computeNext(StreamingCountingIndexIterator.java:17) ~[stardog-5.3.3.jar:?]
		at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-5.3.3.jar:?]
		at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ComputePropertyStatistics.computeDegrees(ComputePropertyStatistics.java:99) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ComputePropertyStatistics.computeDomains(ComputePropertyStatistics.java:71) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ComputePropertyStatistics.lambda$compute$0(ComputePropertyStatistics.java:38) ~[stardog-5.3.3.jar:?]
		at com.complexible.stardog.index.statistics.ComputePropertyStatistics$$Lambda$238/56711263.call(Unknown Source) ~[?:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
		at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at com.complexible.common.trie.Trie$Node.<init>(Trie.java:366) ~[stardog-utils-common-5.3.3.jar:?]
	at com.complexible.common.trie.Trie$Node.addChild(Trie.java:371) ~[stardog-utils-common-5.3.3.jar:?]
	at com.complexible.common.trie.Trie.put(Trie.java:57) ~[stardog-utils-common-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.DiskTrie.put(DiskTrie.java:94) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.btree.impl.BPlusTreeBackedTrie.lambda$addAll$0(BPlusTreeBackedTrie.java:44) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.btree.impl.BPlusTreeBackedTrie$$Lambda$267/2127412082.accept(Unknown Source) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_181]
	at com.complexible.stardog.index.disk.btree.impl.BPlusTreeBackedTrie.addAll(BPlusTreeBackedTrie.java:44) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSets.addAll(DiskCharacteristicSets.java:49) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsBuilder.computeCharacteristicSets(DiskCharacteristicSetsStatisticsBuilder.java:85) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.disk.statistics.DiskCharacteristicSetsStatisticsBuilder.computeCharacteristicSets(DiskCharacteristicSetsStatisticsBuilder.java:36) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder.lambda$build$0(ConcurrentCharacteristicSetStatisticsBuilder.java:82) ~[stardog-5.3.3.jar:?]
	at com.complexible.stardog.index.statistics.ConcurrentCharacteristicSetStatisticsBuilder$$Lambda$235/1389426051.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
2

This indicates that selectivity statistics wasn’t computed for one or more databases due to insufficient amount of heap. This may result in slower queries. Ideally you’d bump the memory ($STARDOG_SERVER_JAVA_ARGS) and restart the server. In theory you can also manually run stardog-admin db optimize <db name> on your database (or all of them).

Cheers,
Pavel

3

What if my JAVA_ARGS are maxed out to the physical available memory ?

4

How large are your databases and what’s in $STARDOG_SERVER_JAVA_ARGS?

5

I’ve got about 33G on disk data, system memory is 4G

6

Do you mean 33G of data in one database or all of them? If that's one database, my guess would be it's over 100M triples and then your server has considerably less memory than suggested in [1] (which also tends to be on the lower side). What does stardog-admin db status print?

[1] Home | Stardog Documentation Latest

7

I’m talking about all the databases, some of them are not really active, but can be relatively large ( although below 100M )

You need to work on a command to get this info for all databases :slight_smile:

8

If large databases are not used but your server has only 4G, then it’s reasonable to take them out (e.g. back up and then remove). They still use some resources even if not really used. Normally it’s not a big deal but in your situation it might help.

Cheers,
Pavel

9

Bart,

You can use the stardog-admin server status command which will print the size of each database along with other statistics.

Best,
Evren

10

When I try to do the optimize per database the command line states 'succesfully optimezed ’ but the server console signals a GC error.

A new server restart results in the same error as before.