# Setting for bulk load **URL:** \ **Category:** Support\ **Created:** [August 9, 2018, 11:04am UTC](https://community.stardog.com/t/setting-for-bulk-load/1203 "2018-08-09T11:04:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mirae](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@mirae](https://community.stardog.com/u/mirae)\ **Post date:** [August 9, 2018, 11:04am UTC](https://community.stardog.com/t/setting-for-bulk-load/1203/1 "2018-08-09T11:04:06Z")
Hello. I am trying to upload 5 billion triples using stardog. Here are some things I did to upload quickly. 1. The size of the uploaded file was very very big (200GB). So I split it into 300 files. 2. memory.mode = bulk\_load in properties file. 3. export STARDOG\_SERVER\_JAVA\_ARGS="-Xmx8g -Xms8g -XX:MaxDirectMemorySize=16g" 4. Index named-graph OFF in database setting But uploading speed is more and more slow. stardog.log ``` INFO 2018-08-09 15:36:31,874 [Stardog.Executor-64] com.complexible.stardog.index.Index:printInternal(314): Indexing triples: 98% complete in 00:55:28 (2.0K triples/sec) INFO 2018-08-09 15:39:47,770 [Stardog.Executor-64] com.complexible.stardog.index.Index:printInternal(314): Indexing triples: 99% complete in 00:58:44 (1.9K triples/sec) ``` This is my server status ``` Backup Storage Directory : .backup CPU Load : 0.46 % Connection Timeout : 10m Export Storage Directory : .exports Memory Direct : 8.4G (Max: 16G) Memory Direct Buffers : 8.0G (Max: 15G) Memory Direct Mapped : 416M (Max: 156M) Memory Heap : 6.1G (Max: 7.8G) Memory Mode : BULK_LOAD Platform Arch : amd64 Platform OS : Linux 4.15.0-29-generic, Java 1.8.0_112 Query All Graphs : false Query Timeout : 5m Security Disabled : false Stardog Home : /hadoop/hdfs/data1/stardog/stardog Stardog Version : 5.3.2 Strict Parsing : false Uptime : 1 day 4 hours 55 minutes 49 seconds Databases : +-----------+---------------+-------------+------------------------------------------------------+--------------------------------------------+ | | | | Transactions | Queries | +-----------+---------------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+ | Database | Size | Connections | Open | Total | Avg. Size | Avg. Time (s) | Rate/sec | Running | Total | Avg. Time (s) | Rate/sec | +-----------+---------------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+ | kngi | 406,256,240 | 0 | 0 | 0 | 0 | 0.000 | 0.000 | 0 | 0 | 0.000 | 0.000 | | kngi_v1 | 396,260,611 | 0 | 0 | 0 | 0 | 0.000 | 0.000 | 0 | 80 | 398.430 | 0.001 | | mirae | 396,260,611 | 0 | 0 | 0 | 0 | 0.000 | 0.000 | 0 | 0 | 0.000 | 0.000 | | test | 29 | 0 | 0 | 0 | 0 | 0.000 | 0.000 | 0 | 0 | 0.000 | 0.000 | | wikidata | 1,155,150,328 | 0 | 0 | 3 | 6.7M | 6,320.092 | 0.000 | 0 | 15 | 135.261 | 0.000 | +-----------+---------------+-------------+-------+-------+-----------+---------------+----------+---------+-------+---------------+----------+ ``` Is there any tips for upload speed ? Too slow to upload for now. ---
**Author:** ![stephen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/stephen/32/27_2.png) [@stephen](https://community.stardog.com/u/stephen)\ **Post date:** [August 9, 2018, 11:36am UTC](https://community.stardog.com/t/setting-for-bulk-load/1203/2 "2018-08-09T11:36:15Z")
Hi there, According to [our documentation](https://www.stardog.com/docs/#_loading_compressed_data), loading compressed data files is actually the preferred way when bulk loading, versus lots of little files. Obviously the more memory you are able to dedicate to Stardog the faster your load will be, but I would first try compressing the 200GB file and loading that directly ---
**Author:** ![mirae](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@mirae](https://community.stardog.com/u/mirae)\ **Post date:** [August 9, 2018, 11:53am UTC](https://community.stardog.com/t/setting-for-bulk-load/1203/3 "2018-08-09T11:53:13Z")
Hello. stephen. I tried to upload compressed data files before. But there was some problem. Maybe log was **No space left Error** Space on STARDOG\_HOME path is about 2.5TB. Do I need more available space? Thank you. ---
**Author:** ![stephen](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/stephen/32/27_2.png) [@stephen](https://community.stardog.com/u/stephen)\ **Post date:** [August 9, 2018, 12:09pm UTC](https://community.stardog.com/t/setting-for-bulk-load/1203/4 "2018-08-09T12:09:56Z")
That sounds like enough space, unless STARDOG\_HOME is on a different volume than the one on which you’re running the Stardog server. Could you get the exact error message from the log? ---
**Author:** ![zachary.whitley](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/zachary.whitley/32/94_2.png) [@zachary.whitley](https://community.stardog.com/u/zachary.whitley)\ **Post date:** [August 9, 2018, 12:55pm UTC](https://community.stardog.com/t/setting-for-bulk-load/1203/5 "2018-08-09T12:55:34Z")
Could it be running out of space on the tmp directory? I’m not sure if bulk load uses the tmp directory, the documentation only says that temp is used “…for many different operations”. When you say “upload” do you mean that your 200Gb file is on the same server you’re running Stardog and you’re just trying to load it or do you mean you’re trying to load it from a remote client with --copy-server-side? ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex030/uploads/stardog/original/2X/e/ed66b48a616f505106a4acde3c9cee7e6da9bc67.svg) [@system](https://community.stardog.com/u/system)\ **Post date:** [August 23, 2018, 12:57pm UTC](https://community.stardog.com/t/setting-for-bulk-load/1203/6 "2018-08-23T12:57:20Z")
Closed on August 23, 2018, 12:57pm UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.