# Can Stardog load files larger than the DirectMemorySize limit? How? **URL:** \ **Category:** Support\ **Created:** [September 3, 2018, 8:06pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277 "2018-09-03T20:06:10Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [September 3, 2018, 8:06pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/1 "2018-09-03T20:06:10Z")
Stardog Server 5.3.3 crashed on loading the file larger than the DirectMomorySize. > WARN 2018-09-04 05:26:11,700 [Stardog.Executor-3] com.complexible.stardog.dht.impl.PagedDiskHashTable:createPage(1123): Available direct memory is low, will try to reduce memory usage > WARN 2018-09-04 05:26:13,817 [Stardog.Executor-3] com.complexible.common.rdf.rio.RDFStreamProcessor:setException(586): Error during loading /data/Downloads/yago3.1\_entire\_ttl/yagoSources.ttl: java.lang.OutOfMemoryError: Direct buffer memory > java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory then > : Parsing triples: 100% complete in 11:37:25 (475.0M triples - 11.4K triples/sec) > INFO 2018-09-04 05:26:13,842 [stardog-user-1] com.complexible.stardog.index.Index:stop(326): > INFO 2018-09-04 05:26:13,842 [stardog-user-1] com.complexible.stardog.index.Index:stop(329): Parsing triples finished in 11:37:25.777 > ERROR 2018-09-04 05:26:13,856 [stardog-user-1] com.stardog.http.server.undertow.ErrorHandling:writeError(138): Unexpected error on the server > java.lang.NoClassDefFoundError: Could not initialize class io.netty.util.internal.Cleaner0 ---
**Author:** ![stephen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/stephen/32/27_2.png) [@stephen](https://community.stardog.com/u/stephen)\ **Post date:** [September 4, 2018, 1:41pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/2 "2018-09-04T13:41:57Z")
Hi Alex, Can we get some more details on what you’re trying to do? Can you also let us know how big the file is and how much direct/total memory there is on the machine? ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [September 4, 2018, 1:58pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/3 "2018-09-04T13:58:12Z")
Hi Stephen, I am trying to load YAGO ---
**Author:** ![stephen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/stephen/32/27_2.png) [@stephen](https://community.stardog.com/u/stephen)\ **Post date:** [September 4, 2018, 2:14pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/4 "2018-09-04T14:14:35Z")
How big is the yagoSources.ttl file you’re loading? How much memory does your machine have? How much memory is allocated to Stardog? ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [September 4, 2018, 10:08pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/5 "2018-09-04T22:08:47Z")
The biggest file (yagoSources.ttl) is 30GB. The computer has 32Gb of RAM, plus 16Gb of swap. I try not to use swap, it is there to prevent OOM killing The primary hard drive is 256Gb NvME. The data drive is 2Tb spinning drive ---
**Author:** ![stephen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/stephen/32/27_2.png) [@stephen](https://community.stardog.com/u/stephen)\ **Post date:** [September 5, 2018, 11:23am UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/6 "2018-09-05T11:23:20Z")
How about the memory (heap/direct) allocated to the Stardog process, either through STARDOG\_JAVA\_ARGS or STARDOG\_SERVER\_JAVA\_ARGS? ---
**Author:** ![pierluigi](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/pierluigi/32/472_2.png) [@pierluigi](https://community.stardog.com/u/pierluigi)\ **Post date:** [September 5, 2018, 3:38pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/7 "2018-09-05T15:38:26Z")
Alex, just reporting my experience with YAGO and other large KBs: Yes (to your original question), it’s definitely possible. In experimental (=non-production) mode, I at one point loaded both a substantial subset of YAGO and the entirety of FreeBase (the last publicly available version) on a machine with 48G RAM. Stardog is not an in-memory DB (by default at least) and “loading” doesn’t mean that every bit of deserialized stuff needs to be in memory at the same time. I wasn’t interested in seeing “how low you can go” on RAM, so I don’t know whether it could have worked in 32 or 16G, but there you have it. For my case, I found that 48G for loading all that stuff were barely working. Queries otoh were much less happy (but still perfectly possible once you tweaked the env parameters Stephen is pointing to). For actual usage, I expanded at some point RAM on the virtual machine to 92G, and set about 3/4 of that for direct access. The data loading was much more efficient if you change the default memory mode parameters to the value expressing “optimized for updates” (it’s in the doc), and that was really helpful in practical terms. ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [September 6, 2018, 9:40pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/8 "2018-09-06T21:40:21Z")
Hi Stephen, I use 12Gb for Heap and 12Gb for External Memory…Somehow it ends up using 29Gb. I also use bulk\_load. Should I use write\_optimised instead? Because it looks like it loads first 30% very fast, then it becomes non-stop memory management. ---
**Author:** ![zachary.whitley](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/zachary.whitley/32/94_2.png) [@zachary.whitley](https://community.stardog.com/u/zachary.whitley)\ **Post date:** [September 6, 2018, 10:11pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/9 "2018-09-06T22:11:45Z")
What do you have your swappiness set to? (Kernel param not Stardog) ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [September 6, 2018, 10:13pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/10 "2018-09-06T22:13:13Z")
The swap file is 16Gb, but I can increase it, there is plenty of space on the NvME drive. ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [September 6, 2018, 10:14pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/11 "2018-09-06T22:14:07Z")
The swap file is 16Gb, but I can increase it, there is plenty of space on the NvME drive. Sorry, first posted it without reply ---
**Author:** ![zachary.whitley](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/zachary.whitley/32/94_2.png) [@zachary.whitley](https://community.stardog.com/u/zachary.whitley)\ **Post date:** [September 6, 2018, 10:16pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/12 "2018-09-06T22:16:35Z")
Not the amount of swap but the vm.swappiness kernel param. You might want to set it to 1 NvME is fast but you still want to avoid swapping if you can help it. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex030/uploads/stardog/original/2X/e/ed66b48a616f505106a4acde3c9cee7e6da9bc67.svg) [@system](https://community.stardog.com/u/system)\ **Post date:** [September 20, 2018, 10:16pm UTC](https://community.stardog.com/t/can-stardog-load-files-larger-than-the-directmemorysize-limit-how/1277/13 "2018-09-20T22:16:37Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.