# Performance deteriorates while loading large dataset **URL:** \ **Category:** Support\ **Created:** [August 28, 2018, 8:51am UTC](https://community.stardog.com/t/performance-deteriorates-while-loading-large-dataset/1260 "2018-08-28T08:51:46Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [August 28, 2018, 8:51am UTC](https://community.stardog.com/t/performance-deteriorates-while-loading-large-dataset/1260/1 "2018-08-28T08:51:46Z")
Loading YAGO, about 170Gb of ttl. The status of the Server: > ``` > Backup Storage Directory : .backup > CPU Load : 0.15 % > Connection Timeout : 10m > Export Storage Directory : .exports > Memory Direct : 7.7G (Max: 8.0G) > Memory Direct Buffers : 7.7G (Max: 7.5G) > Memory Direct Mapped : 16M (Max: 78M) > Memory Heap : 2.5G (Max: 7.4G) > Memory Mode : BULK_LOAD > Platform Arch : amd64 > Platform OS : Linux 4.15.0-32-generic, Java 1.8.0_181 > Query All Graphs : false > Query Timeout : 5m > Security Disabled : false > Stardog Home : /data/stardog > Stardog Version : 5.3.3 > Strict Parsing : true > Uptime : 16 hours 8 minutes 43 seconds > Databases : > +----------+-------+-------------+------------------------------------------------------+--------------------------------------------+ > | | | | Transactions | Queries | > +----------+-------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+ > | Database | Size | Connections | Open | Total | Avg. Size | Avg. Time (s) | Rate/sec | Running | Total | Avg. Time (s) | Rate/sec | > +----------+-------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+ > | stardog | 0 | 0 | 0 | 0 | 0 | 0.000 | 0.000 | 0 | 0 | 0.000 | 0.000 | > +----------+-------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+ > > ``` Time for parsing million triples reached over an hour > INFO 2018-08-28 08:14:52,243 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 31% complete in 05:42:57 (593.0M triples - 28.8K triples/sec) > INFO 2018-08-28 08:20:09,470 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 31% complete in 05:48:14 (594.0M triples - 28.4K triples/sec) > INFO 2018-08-28 08:21:49,945 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 05:49:55 (594.4M triples - 28.3K triples/sec) > INFO 2018-08-28 08:30:26,079 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 05:58:31 (595.0M triples - 27.7K triples/sec) > INFO 2018-08-28 08:48:10,996 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 06:16:16 (596.0M triples - 26.4K triples/sec) > INFO 2018-08-28 09:16:01,572 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 06:44:06 (597.0M triples - 24.6K triples/sec) > INFO 2018-08-28 09:56:36,708 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 07:24:41 (598.0M triples - 22.4K triples/sec) > INFO 2018-08-28 10:20:04,428 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 07:48:09 (599.0M triples - 21.3K triples/sec) > INFO 2018-08-28 10:55:55,800 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 08:24:00 (600.0M triples - 19.8K triples/sec) > INFO 2018-08-28 11:24:07,305 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 08:52:12 (601.0M triples - 18.8K triples/sec) > INFO 2018-08-28 12:04:02,300 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 09:32:07 (602.0M triples - 17.5K triples/sec) > INFO 2018-08-28 12:43:58,405 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 10:12:03 (603.0M triples - 16.4K triples/sec) > INFO 2018-08-28 13:30:12,661 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 10:58:17 (604.0M triples - 15.3K triples/sec) > INFO 2018-08-28 14:15:03,520 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 11:43:08 (605.0M triples - 14.3K triples/sec) > INFO 2018-08-28 14:48:40,217 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 12:16:45 (606.0M triples - 13.7K triples/sec) > INFO 2018-08-28 15:44:13,948 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 13:12:19 (607.0M triples - 12.8K triples/sec) > INFO 2018-08-28 17:06:10,606 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 14:34:15 (608.0M triples - 11.6K triples/sec) > INFO 2018-08-28 18:20:02,745 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 15:48:07 (609.0M triples - 10.7K triples/sec) The computer is a 6-core laptop with 32Gb of DDR4 RAM, however the db is on the secondary HDD which is a spinning disk. ---
**Author:** ![stephen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/stephen/32/27_2.png) [@stephen](https://community.stardog.com/u/stephen)\ **Post date:** [August 28, 2018, 11:39am UTC](https://community.stardog.com/t/performance-deteriorates-while-loading-large-dataset/1260/2 "2018-08-28T11:39:38Z")
> [@alexj](#): > > Memory Direct : 7.7G (Max: 8.0G) > Memory Direct Buffers : 7.7G (Max: 7.5G) > Memory Direct Mapped : 16M (Max: 78M) > Memory Heap : 2.5G (Max: 7.4G) Hi, from the looks of your output here it seems that while the machine has 32G of RAM, only about half of that is being dedicated to the Stardog process. You could try setting STARDOG\_SERVER\_JAVA\_ARGS to something like `-Xms12g -Xmx12g -XX:MaxDirectMemorySize=16g` when you start Stardog so more of the memory is allocated for use by the Stardog JVM. You should also try setting `memory.mode=bulk_load` in your stardog.properties file to make the most efficient use of that memory. ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [August 28, 2018, 4:31pm UTC](https://community.stardog.com/t/performance-deteriorates-while-loading-large-dataset/1260/3 "2018-08-28T16:31:39Z")
Hi Stephen, After working for some time, it started analysing memory, and never again restarted parsing triples: > INFO 2018-08-28 18:20:02,745 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 15:48:07 (609.0M triples - 10.7K triples/sec) > > INFO 2018-08-28 20:02:04,806 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 17:30:09 (610.0M triples - 9.7K triples/sec) > > INFO 2018-08-28 21:08:39,168 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Memory usage 92% - initiating detailed analysis - will check again in 150s > > INFO 2018-08-28 21:08:39,190 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Stardog JVM > > RssAnon = 16G > > RssFile = 9.8M > > RssShmem = 0B > > VmData = 16G > > VmExe = 4.0K > > VmHWM = 16G > > VmLck = 0B > > VmLib = 19M > > VmPTE = 33M > > VmPeak = 23G > > VmPin = 0B > > VmRSS = 16G > > VmSize = 23G > > VmStk = 136K > > VmSwap = 0B > > INFO 2018-08-28 21:08:39,333 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): 4 value file mappings occupy 0B of memory > > INFO 2018-08-28 21:08:39,335 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Stardog Metrics > > dbms.mem.direct.buffer.max = 7.5G > > dbms.mem.direct.buffer.used = 7.7G > > dbms.mem.direct.max = 8.0G > > dbms.mem.direct.pool.max = 0B > > dbms.mem.direct.pool.used = 0B > > dbms.mem.direct.used = 7.7G > > dbms.mem.heap.max = 7.4G > > dbms.mem.heap.pool.max = 0B > > dbms.mem.heap.pool.used = 0B > > dbms.mem.heap.used = 2.6G > > dbms.mem.mapped.max = 78M > > dbms.mem.mapped.used = 16M > > dbms.mem.system.rss = 16G > > dbms.memory.direct.reserve = > > dbms.memory.heap.reserve = > > dbms.memory.managed.direct = > > dbms.memory.managed.heap = > > dbms.memory.management = NATIVE > > dbms.memory.mode = BULK\_LOAD > > dbms.memory.monitor.interval = 600000 > > dbms.memory.monitor.report.threshold = 0.9 > > INFO 2018-08-28 21:08:39,336 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): System mem > > Active = 28552589312 > > Active(anon) = 26969939968 > > Active(file) = 1582649344 > > AnonHugePages = 0 > > AnonPages = 28215037952 > > Bounce = 0 > > Buffers = 129515520 > > Cached = 3711053824 > > CmaFree = 0 > > CmaTotal = 0 > > CommitLimit = 18936037376 > > Committed\_AS = 49939787776 > > DirectMap1G = 0 > > DirectMap2M = 31725715456 > > DirectMap4k = 2481446912 > > Dirty = 493125632 > > HardwareCorrupted = 0 > > Hugepagesize = 2097152 > > Inactive = 3512745984 > > Inactive(anon) = 2607915008 > > Inactive(file) = 904830976 > > KernelStack = 44744704 > > Mapped = 1021927424 > > MemAvailable = 2699071488 > > MemFree = 278822912 > > MemTotal = 33577119744 > > Mlocked = 1277952 > > NFS\_Unstable = 0 > > PageTables = 240017408 > > SReclaimable = 414076928 > > SUnreclaim = 238104576 > > Shmem = 1353056256 > > ShmemHugePages = 0 > > ShmemPmdMapped = 0 > > Slab = 652181504 > > SwapCached = 21872640 > > SwapFree = 1324961792 > > SwapTotal = 2147479552 > > Unevictable = 1277952 > > VmallocChunk = 0 > > VmallocTotal = 35184372087808 > > VmallocUsed = 0 > > Writeback = 942080 > > WritebackTmp = 0 > > INFO 2018-08-28 21:08:39,357 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Other processes > > F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND > > 4 1000 2531 2476 20 0 212160 352 poll\_s Ssl+ tty2 0:00 /usr/lib/gdm3/gdm-x-session --run-script env GNOME\_SHELL\_SESSION\_MODE=ubuntu gnome-session --session=ubuntu ---
**Author:** ![alexj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/alexj/32/202_2.png) [@alexj](https://community.stardog.com/u/alexj)\ **Post date:** [August 28, 2018, 10:19pm UTC](https://community.stardog.com/t/performance-deteriorates-while-loading-large-dataset/1260/4 "2018-08-28T22:19:14Z")
Hi Stephen, After setting parameters as you said, it started to work nicely, but eventually the server crashed. I created a separated topic ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex030/uploads/stardog/original/2X/e/ed66b48a616f505106a4acde3c9cee7e6da9bc67.svg) [@system](https://community.stardog.com/u/system)\ **Post date:** [September 11, 2018, 10:19pm UTC](https://community.stardog.com/t/performance-deteriorates-while-loading-large-dataset/1260/5 "2018-09-11T22:19:19Z")
Closed on September 11, 2018, 10:19pm UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.