Stardog running out of memory when using connection pool

Support Bug
1

Hi,

I'm using the connection pool from java against Stardog. When using the pool Stardog will eventually run out of memory. Switching to creating and closing connections for each query fixes this.

Here is how Stardog looks when running out of memory:

Screen Shot 2017-05-02 at 16.02.57.png2384×1434 282 KB

The small white sections are when stardog is unresponsive because of GC. The light green, dark green beginning of the memory is how stardog usually runs, only minor GC.

I've been debugging to see if I'm forgetting to close something against stardog, but I can't see that anything is left open.

I used reflection to get hold of all the variables inside the connection, and also the size of the pool.

mClient : com.complexible.stardog.protocols.http.client.HttpClientImpl@65215184
mContext : com.complexible.stardog.protocols.SPECMessageContext@117a4c
mServerURL : http://stardog:5820/
mDBMS : com.complexible.stardog.api.admin.AdminConnectionImpl@11c1e8a
LOGGER : Logger[com.complexible.stardog.api.Connection]
THRESHOLD : 100000
mOpenQueryResults : []
mStreamSourceCreator : com.complexible.common.rdf.impl.StreamStatementSources$OneTimeSourceCreator@72949cd6
mOpen : true
mInTx : false
mPendingChanges : com.complexible.common.base.ChangeList@1dd65e71
mNumPending : 0
mConnectionViews : com.complexible.common.view.Views@6c5446bc
pool size: 0

This is from one the services that is causing Stardog to run out of memory. As you can see there is no transaction left open and all query results are closed. Pool size is 0, because this connection hasn't been returned to the pool yet.

How do I proceed to debug stardog to find out why the connection is taking more and more memory in Stardog?

Cheers,
Håvard

2

Håvard,

Sorry for the delay, but have you tried this with the 5.0 beta? If it’s still not good there, can you check to make sure you aren’t leaving something like TupleQueryResult open? We have a few classes that extend AutoCloseable that you might want to use try-with-resources on just to ensure they don’t ever remain unclosed.

3

I’ll give it a try in 5.0 beta as soon as I have time.

4

Trying this on 5.0 beta today. The web ui isn’t showing any memory info. I can see that there are some new fields in the memory info from the server, but they are not shown. Also it seems that the heap usage is always 0 mb, which seems strange.

5

Also, number of running queries is always 0. And the query management interface doesn’t show anything.

6

stardog-admin server status is giving me the correct info about the heap.

7

@stephen I’ve run an overnight torture test and I can’t reproduce this issue on 5.0 beta. So everything seems to be good.

1 Like
8

That’s great to hear! Apologies for your problems with some of the graphical components of the web console. Everything front-end is a bit in flux at the moment with 5.0

9

Hello,
I'm creating connections using try-with-resources, but I'm having the following issue:

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:798)
        at org.apache.http.impl.client.IdleConnectionEvictor.start(IdleConnectionEvictor.java:96)
        at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1238)
        at com.complexible.stardog.protocols.http.client.BaseHttpClient.<init>(BaseHttpClient.java:194)
        at com.complexible.stardog.protocols.http.client.BaseHttpClient.<init>(BaseHttpClient.java:207)
        at com.complexible.stardog.protocols.http.client.HttpClientImpl.<init>(HttpClientImpl.java:147)
        at com.complexible.stardog.protocols.http.client.HttpConnection.create(HttpConnection.java:106)
        at com.complexible.stardog.protocols.http.client.HttpDriver.connect(HttpDriver.java:84)
        at com.complexible.stardog.api.DriverManager.getConnection(DriverManager.java:90)
        at com.complexible.stardog.api.ConnectionConfiguration.connect(ConnectionConfiguration.java:204)
        at com.complexible.stardog.api.ConnectionPoolableObjectFactory.makeObject(ConnectionPoolableObjectFactory.java:30)
        at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1691)
        at org.apache.commons.pool.PoolUtils.prefill(PoolUtils.java:315)
        at com.complexible.common.pool.ApachePoolImpl.<init>(ApachePoolImpl.java:36)
        at com.complexible.stardog.api.ConnectionPoolConfig$ApacheConnectionPool.<init>(ConnectionPoolConfig.java:120)
        at com.complexible.stardog.api.ConnectionPoolConfig.create(ConnectionPoolConfig.java:113)
        at com.complexible.stardog.rdf4j.StardogRepository.initializeInternal(StardogRepository.java:115)
        at org.eclipse.rdf4j.repository.base.AbstractRepository.init(AbstractRepository.java:39)
        at org.eclipse.rdf4j.repository.base.AbstractRepository.initialize(AbstractRepository.java:31)
        at com.complexible.stardog.rdf4j.StardogRepository.<init>(StardogRepository.java:57)

My Stardog version is 7.7.1

Example of my code to replicate the issue:

public Repository getRepository(String repositoryID) {
		return new StardogRepository(ConnectionConfiguration.from(address + "/" + repositoryID).credentials(user, pwd));
	}
	public void queryContructorOrDescribe(String repositoryID, String query,
			org.eclipse.rdf4j.rio.RDFFormat format) {
       try (RepositoryConnection coon = this.getRepository(repositoryID).getConnection()) {
			GraphQuery q = coon.prepareGraphQuery(query);
			try (StringWriter rdf = new StringWriter()) {
				RDFHandler writer = Rio.createWriter(format, rdf);
				q.evaluate(writer);
				System.out.println(rdf.toString());
			} catch (IOException e) {
				log.error("Error while reading sparql response: ", e);
			} 
		}
}

This is executed +- 400k times

Any help would be appreciated :slight_smile: