Performance deteriorates while loading large dataset

Support
1

Loading YAGO, about 170Gb of ttl.
The status of the Server:

Backup Storage Directory : .backup
CPU Load                 : 0.15 %
Connection Timeout       : 10m
Export Storage Directory : .exports
Memory Direct            : 7.7G (Max: 8.0G)
Memory Direct Buffers    : 7.7G (Max: 7.5G)
Memory Direct Mapped     :  16M (Max:  78M)
Memory Heap              : 2.5G (Max: 7.4G)
Memory Mode              : BULK_LOAD
Platform Arch            : amd64
Platform OS              : Linux 4.15.0-32-generic, Java 1.8.0_181
Query All Graphs         : false
Query Timeout            : 5m
Security Disabled        : false
Stardog Home             : /data/stardog
Stardog Version          : 5.3.3
Strict Parsing           : true
Uptime                   : 16 hours 8 minutes 43 seconds
Databases                :
+----------+-------+-------------+------------------------------------------------------+--------------------------------------------+
|          |       |             |                     Transactions                     |                  Queries                   |
+----------+-------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+
| Database | Size  | Connections | Open  | Total | Avg. Size | Avg. Time (s) | Rate/sec | Running | Total | Avg. Time (s) | Rate/sec |
+----------+-------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+
| stardog  |     0 |           0 |     0 |     0 |         0 |         0.000 |    0.000 |       0 |     0 |         0.000 |    0.000 |
+----------+-------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+

Time for parsing million triples reached over an hour

INFO 2018-08-28 08:14:52,243 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 31% complete in 05:42:57 (593.0M triples - 28.8K triples/sec)
INFO 2018-08-28 08:20:09,470 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 31% complete in 05:48:14 (594.0M triples - 28.4K triples/sec)
INFO 2018-08-28 08:21:49,945 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 05:49:55 (594.4M triples - 28.3K triples/sec)
INFO 2018-08-28 08:30:26,079 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 05:58:31 (595.0M triples - 27.7K triples/sec)
INFO 2018-08-28 08:48:10,996 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 06:16:16 (596.0M triples - 26.4K triples/sec)
INFO 2018-08-28 09:16:01,572 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 06:44:06 (597.0M triples - 24.6K triples/sec)
INFO 2018-08-28 09:56:36,708 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 07:24:41 (598.0M triples - 22.4K triples/sec)
INFO 2018-08-28 10:20:04,428 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 07:48:09 (599.0M triples - 21.3K triples/sec)
INFO 2018-08-28 10:55:55,800 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 08:24:00 (600.0M triples - 19.8K triples/sec)
INFO 2018-08-28 11:24:07,305 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 08:52:12 (601.0M triples - 18.8K triples/sec)
INFO 2018-08-28 12:04:02,300 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 09:32:07 (602.0M triples - 17.5K triples/sec)
INFO 2018-08-28 12:43:58,405 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 10:12:03 (603.0M triples - 16.4K triples/sec)
INFO 2018-08-28 13:30:12,661 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 10:58:17 (604.0M triples - 15.3K triples/sec)
INFO 2018-08-28 14:15:03,520 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 11:43:08 (605.0M triples - 14.3K triples/sec)
INFO 2018-08-28 14:48:40,217 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 12:16:45 (606.0M triples - 13.7K triples/sec)
INFO 2018-08-28 15:44:13,948 [Stardog.Executor-0] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 13:12:19 (607.0M triples - 12.8K triples/sec)
INFO 2018-08-28 17:06:10,606 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 14:34:15 (608.0M triples - 11.6K triples/sec)
INFO 2018-08-28 18:20:02,745 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 15:48:07 (609.0M triples - 10.7K triples/sec)

The computer is a 6-core laptop with 32Gb of DDR4 RAM, however the db is on the secondary HDD which is a spinning disk.

2

Hi, from the looks of your output here it seems that while the machine has 32G of RAM, only about half of that is being dedicated to the Stardog process. You could try setting STARDOG_SERVER_JAVA_ARGS to something like -Xms12g -Xmx12g -XX:MaxDirectMemorySize=16g when you start Stardog so more of the memory is allocated for use by the Stardog JVM. You should also try setting memory.mode=bulk_load in your stardog.properties file to make the most efficient use of that memory.

3

Hi Stephen,
After working for some time, it started analysing memory, and never again restarted parsing triples:

INFO 2018-08-28 18:20:02,745 [Stardog.Executor-2] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 15:48:07 (609.0M triples - 10.7K triples/sec)

INFO 2018-08-28 20:02:04,806 [Stardog.Executor-5] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 17:30:09 (610.0M triples - 9.7K triples/sec)

INFO 2018-08-28 21:08:39,168 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Memory usage 92% - initiating detailed analysis - will check again in 150s

INFO 2018-08-28 21:08:39,190 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Stardog JVM

RssAnon = 16G

RssFile = 9.8M

RssShmem = 0B

VmData = 16G

VmExe = 4.0K

VmHWM = 16G

VmLck = 0B

VmLib = 19M

VmPTE = 33M

VmPeak = 23G

VmPin = 0B

VmRSS = 16G

VmSize = 23G

VmStk = 136K

VmSwap = 0B

INFO 2018-08-28 21:08:39,333 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): 4 value file mappings occupy 0B of memory

INFO 2018-08-28 21:08:39,335 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Stardog Metrics

dbms.mem.direct.buffer.max = 7.5G

dbms.mem.direct.buffer.used = 7.7G

dbms.mem.direct.max = 8.0G

dbms.mem.direct.pool.max = 0B

dbms.mem.direct.pool.used = 0B

dbms.mem.direct.used = 7.7G

dbms.mem.heap.max = 7.4G

dbms.mem.heap.pool.max = 0B

dbms.mem.heap.pool.used = 0B

dbms.mem.heap.used = 2.6G

dbms.mem.mapped.max = 78M

dbms.mem.mapped.used = 16M

dbms.mem.system.rss = 16G

dbms.memory.direct.reserve =

dbms.memory.heap.reserve =

dbms.memory.managed.direct =

dbms.memory.managed.heap =

dbms.memory.management = NATIVE

dbms.memory.mode = BULK_LOAD

dbms.memory.monitor.interval = 600000

dbms.memory.monitor.report.threshold = 0.9

INFO 2018-08-28 21:08:39,336 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): System mem

Active = 28552589312

Active(anon) = 26969939968

Active(file) = 1582649344

AnonHugePages = 0

AnonPages = 28215037952

Bounce = 0

Buffers = 129515520

Cached = 3711053824

CmaFree = 0

CmaTotal = 0

CommitLimit = 18936037376

Committed_AS = 49939787776

DirectMap1G = 0

DirectMap2M = 31725715456

DirectMap4k = 2481446912

Dirty = 493125632

HardwareCorrupted = 0

Hugepagesize = 2097152

Inactive = 3512745984

Inactive(anon) = 2607915008

Inactive(file) = 904830976

KernelStack = 44744704

Mapped = 1021927424

MemAvailable = 2699071488

MemFree = 278822912

MemTotal = 33577119744

Mlocked = 1277952

NFS_Unstable = 0

PageTables = 240017408

SReclaimable = 414076928

SUnreclaim = 238104576

Shmem = 1353056256

ShmemHugePages = 0

ShmemPmdMapped = 0

Slab = 652181504

SwapCached = 21872640

SwapFree = 1324961792

SwapTotal = 2147479552

Unevictable = 1277952

VmallocChunk = 0

VmallocTotal = 35184372087808

VmallocUsed = 0

Writeback = 942080

WritebackTmp = 0

INFO 2018-08-28 21:08:39,357 [Stardog-Linux-Memory-Monitor] com.complexible.stardog.api.LinuxMemoryMonitor:output(169): Other processes

F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND

4 1000 2531 2476 20 0 212160 352 poll_s Ssl+ tty2 0:00 /usr/lib/gdm3/gdm-x-session --run-script env GNOME_SHELL_SESSION_MODE=ubuntu gnome-session --session=ubuntu

4

Hi Stephen,

After setting parameters as you said, it started to work nicely, but eventually the server crashed. I created a separated topic

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.