Virtual import fails with Out of Memory error

I am trying to import the geonames allcountries data (which is quite large, >11M lines) and get the following error:

Exception in thread "Connection evictor" java.lang.OutOfMemoryError: Java heap space
  at java.util.LinkedList.listIterator(LinkedList.java:868)
  at java.util.AbstractList.listIterator(AbstractList.java:299)
  at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:239)
  at org.apache.http.pool.AbstractConnPool.enumAvailable(AbstractConnPool.java:560)
  at org.apache.http.impl.conn.CPool.enumAvailable(CPool.java:76)
  at org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:619)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:423)
  at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:6)
  at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
  at java.util.regex.Pattern$BnM.optimize(Pattern.java:5411)
  at java.util.regex.Pattern.compile(Pattern.java:1711)
  at java.util.regex.Pattern.<init>(Pattern.java:1351)
  at java.util.regex.Pattern.compile(Pattern.java:1054)
  at java.lang.String.replace(String.java:2239)
  at com.complexible.stardog.virtual.api.RDFGenerator$TemplateValueGenerator.applyTemplate(RDFGenerator.java:340)
  at com.complexible.stardog.virtual.api.RDFGenerator$TemplateLiteralGenerator.generate(RDFGenerator.java:369)
  at com.complexible.stardog.virtual.api.RDFGenerator$TemplateLiteralGenerator.generate(RDFGenerator.java:357)
  at com.complexible.stardog.virtual.api.RDFGenerator$StatementGenerator.generate(RDFGenerator.java:225)
  at com.complexible.stardog.virtual.api.RDFGenerator.generate(RDFGenerator.java:62)
  at com.complexible.stardog.virtual.api.CSVMapper.map(CSVMapper.java:134)
  at com.complexible.stardog.virtual.cli.VirtualGraphImport.execute(VirtualGraphImport.java:124)
  at com.complexible.stardog.cli.impl.ConnectionCommand.call(ConnectionCommand.java:70)
  at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:55)
  at com.complexible.stardog.cli.admin.CLI.main(CLI.java:186)

My STARDOG_SERVER_JAVA_ARGS = “-Xms16g -Xmx16g -XX:MaxDirectMemorySize=16g”.

Any suggestions?
Andrea

Hi Andrea,

The CSV is turned into RDF on the client side. You can increased memory on the client by using STARDOG_JAVA_ARGS. You shouldn’t need a huge amount of RAM allocated to the client as CSV import is committed in batches.

Jess

Thanks.

As an aside, it might be warranted to document STARDOG_JAVA_ARGS on the stardog.com/docs page, since only STARDOG_SERVER_JAVA_ARGS is described. (There is one mention of STARDOG_JAVA_ARGS wrt Stardog 6 alpha.)

Andrea

Good point. We’ll make note of this in the docs. Thanks.

Jess

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.