We are sending data to stardog in 300MB chunks. The result is tons of open file descriptors:
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 16918 root 565w REG 252,0 299999812 6164651 /tmp/ByteSourceOutputStream4510171724670963872.tmp (deleted)
java 16918 root 566w REG 252,0 299999537 6164601 /tmp/ByteSourceOutputStream7811405495696113442.tmp (deleted)
java 16918 root 567w REG 252,0 299999541 6164586 /tmp/ByteSourceOutputStream6258517087835173454.tmp (deleted)
java 16918 root 568w REG 252,0 299999734 6164681 /tmp/ByteSourceOutputStream4585720650961702379.tmp (deleted)
java 16918 root 569w REG 252,0 299999642 6164541 /tmp/ByteSourceOutputStream7635892884365308214.tmp (deleted)
java 16918 root 570w REG 252,0 299999551 6164722 /tmp/ByteSourceOutputStream1799261860245371326.tmp (deleted)
java 16918 root 571w REG 252,0 299999664 6164653 /tmp/ByteSourceOutputStream4510209312531060973.tmp (deleted)
java 16918 root 638w REG 252,0 299999685 6164682 /tmp/ByteSourceOutputStream3555910240997193441.tmp (deleted)
java 16918 root 645w REG 252,0 299999573 6164550 /tmp/ByteSourceOutputStream4507927273707053369.tmp (deleted)
16918 is stardog process id. By the end of a single workflow execution in our application (i.e. one ETL pipeline, which uses stardog), unclosed fds consume 80G of disk space (oops). Restart of stardog obviously helps, but that is not an acceptable solution.
Stardog log looks normal, here is a snip for one chunk:
INFO 2019-12-11 20:11:20,878 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 28% complete in 00:00:06 (440K triples - 73.0K triples/sec)
INFO 2019-12-11 20:11:22,237 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 32% complete in 00:00:07 (510K triples - 69.1K triples/sec)
INFO 2019-12-11 20:11:23,385 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 36% complete in 00:00:08 (570K triples - 66.8K triples/sec)
INFO 2019-12-11 20:11:24,509 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 40% complete in 00:00:09 (630K triples - 65.3K triples/sec)
INFO 2019-12-11 20:11:25,910 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 44% complete in 00:00:11 (700K triples - 63.3K triples/sec)
INFO 2019-12-11 20:11:27,354 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 48% complete in 00:00:12 (760K triples - 60.8K triples/sec)
INFO 2019-12-11 20:11:28,525 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 51% complete in 00:00:13 (810K triples - 59.2K triples/sec)
INFO 2019-12-11 20:11:29,715 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 55% complete in 00:00:14 (870K triples - 58.5K triples/sec)
INFO 2019-12-11 20:11:30,899 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 59% complete in 00:00:16 (930K triples - 58.0K triples/sec)
INFO 2019-12-11 20:11:32,044 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 63% complete in 00:00:17 (990K triples - 57.6K triples/sec)
INFO 2019-12-11 20:11:33,059 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 66% complete in 00:00:18 (1.0M triples - 57.1K triples/sec)
INFO 2019-12-11 20:11:34,247 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 70% complete in 00:00:19 (1.1M triples - 56.7K triples/sec)
INFO 2019-12-11 20:11:35,256 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 73% complete in 00:00:20 (1.2M triples - 56.4K triples/sec)
INFO 2019-12-11 20:11:36,319 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 76% complete in 00:00:21 (1.2M triples - 55.9K triples/sec)
INFO 2019-12-11 20:11:44,975 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 78% complete in 00:00:30 (1.2M triples - 40.8K triples/sec)
INFO 2019-12-11 20:11:46,283 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 81% complete in 00:00:31 (1.3M triples - 40.7K triples/sec)
INFO 2019-12-11 20:11:47,553 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 85% complete in 00:00:32 (1.3M triples - 41.0K triples/sec)
INFO 2019-12-11 20:11:48,772 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 89% complete in 00:00:33 (1.4M triples - 41.3K triples/sec)
INFO 2019-12-11 20:11:49,795 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 92% complete in 00:00:34 (1.5M triples - 41.5K triples/sec)
INFO 2019-12-11 20:11:51,003 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 96% complete in 00:00:36 (1.5M triples - 41.8K triples/sec)
INFO 2019-12-11 20:11:52,089 [Stardog.Executor-1879] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 99% complete in 00:00:37 (1.6M triples - 41.9K triples/sec)
INFO 2019-12-11 20:11:59,757 [stardog-user-14] com.complexible.stardog.index.Index:printInternal(314): Parsing triples: 100% complete in 00:00:44 (1.6M triples - 34.9K triples/sec)