Stardog 5.3.1 embedded for testing - suddenly requiring 2+ GB of ram


(Hmottestad) #1

Hi,

We use an embedded version of stardog for running unit tests. This has been running pretty smoothly since 5.0 was released.

With the upgrade to Stardog 5.3.1 (from 5.3.0) our automated test server is running out of memory because it looks like the embedded Stardog want’s to reserve a lot of memory:

There was an error initializing Stardog; one or more dependencies could not be satisfied. Please verify your classpath is correct.
The initialization errors were: 
*	Error injecting constructor, com.complexible.memory.structure.impl.tape.OutOfMemoryException: Unable to reserve 2,237,899,296 bytes, direct memory exhausted
		com.complexible.stardog.StardogKernel.<init>()
		com.complexible.stardog.StardogKernel
	Caused by:
com.complexible.memory.structure.impl.tape.OutOfMemoryException: Unable to reserve 2,237,899,296 bytes, direct memory exhausted
	at com.complexible.memory.util.MMBits$BitsJDK7Or8.reserveMemory(MMBits.java:189)
	at com.complexible.memory.util.MMBits.reserveMemory(MMBits.java:81)
	at com.complexible.memory.memoryblock.MemoryBlockPool.allocateAll(MemoryBlockPool.java:181)
	at com.complexible.memory.memoryblock.MemoryBlockPool.<init>(MemoryBlockPool.java:50)
	at com.complexible.memory.memoryblock.MemoryContext.createMemoryBlockPool(MemoryContext.java:179)
	at com.complexible.memory.memoryblock.MemoryContext.<init>(MemoryContext.java:64)
	at com.complexible.stardog.StardogKernel.initMemoryContext(StardogKernel.java:699)
	at com.complexible.stardog.StardogKernel.<init>(StardogKernel.java:480)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.google.inject.internal.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:86)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
	at com.complexible.stardog.Stardog.initKernel(Stardog.java:213)
	at com.complexible.stardog.Stardog.<init>(Stardog.java:206)
	at com.complexible.stardog.Stardog.<init>(Stardog.java:60)
	at com.complexible.stardog.Stardog$StardogBuilder.create(Stardog.java:597)
	at no.difi.einnsyn.stardog.AbstractStardogServiceTest.beforeClass(AbstractStardogServiceTest.java:54)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)

Is there anything I can do to stop stardog from reserving that much memory for our embedded unit tests? Or is there anything I should call after my tests to tell stardog to release memory?

Cheers,
Håvard


(Hmottestad) #2

Managed to get it running by switching to JVM memory management (from default NATIVE).

Though now my logs are riddled with:

[WARN] com.complexible.stardog.plan.optimizer.FixJoinType - Invalid join detected
MergeJoin(key=1 card=1[ACCURATE] cost=2.2 LeftOuter)
Scan[1[ACCURATE]](PSOC[SortType[0]], Var(0), http://data.einnsyn.no/virksomhetmeta/kontaktpunkt, Var(1), null, scope=Default)
Bind("true"^^xsd:boolean AS 2)
Scan[1[ACCURATE]](POSC[SortType[1]], Var(1), http://www.w3.org/1999/02/22-rdf-syntax-ns#type, http://data.einnsyn.no/virksomhetmeta/Kontaktpunkt, null, scope=Default)

(system) #3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.