# Missing data in repository after import?

**URL:** <https://community.stardog.com/t/missing-data-in-repository-after-import/2835>\
**Category:** Bug\
**Created:** [December 9, 2020, 4:05pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835 "2020-12-09T16:05:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![M.Lissandrini](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/m.lissandrini/32/626_2.png) [@M.Lissandrini](https://community.stardog.com/u/M.Lissandrini)\
**Post date:** [December 9, 2020, 4:05pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/1 "2020-12-09T16:05:25Z")

</div>

Hello,

I was importing data from Yago4 into stardog.  
I've imported the same data in both stardog and another triplestore and checked online.

Now it seems that stardog skipped (?) some triples. That is if I run this query on my stardog endpoint I get an empty resultset, while the results exists (you can check online also [1]).

```auto
PREFIX schema: <http://schema.org/>
SELECT * FROM <http://yago-knowledge.org> WHERE {

?v0 schema:worksFor <http://yago-knowledge.org/resource/Dominick_Daly> .
  ?v0 schema:birthDate ?v3 .

} LIMIT 10

```

The result should be:

```auto
http://yago-knowledge.org/resource/Robert_Dalrymple_Ross "1827"^^xsd:gYear

```

Note that this query returns other results indeed.

```auto
PREFIX schema: <http://schema.org/>
SELECT * FROM <http://yago-knowledge.org> WHERE {

  ?v0 schema:birthDate ?v3 .

} LIMIT 10

```

[1] [Sparql | Yago Project](https://yago-knowledge.org/sparql)

---

<div class="post-metadata">

**Author:** ![noahgorstein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/noahgorstein/32/584_2.png) [@noahgorstein](https://community.stardog.com/u/noahgorstein)\
**Post date:** [December 9, 2020, 4:55pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/2 "2020-12-09T16:55:49Z")

</div>

Hi Matteo,

Thanks for your report. Can you share or inspect your `stardog.log`? It sounds like Stardog encountered some parsing errors during the data load.

Best,  
Noah

---

<div class="post-metadata">

**Author:** ![M.Lissandrini](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/m.lissandrini/32/626_2.png) [@M.Lissandrini](https://community.stardog.com/u/M.Lissandrini)\
**Post date:** [December 9, 2020, 5:20pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/3 "2020-12-09T17:20:52Z")

</div>

Hi,  
I've checked the log, and there are a couple of

```auto
INFO 2020-08-31 09:04:10,854 [stardog-user-1] com.complexible.stardog.StardogKernel:write(77): /data/yago/import/part-yago-wd-facts-lite.01.nt: '0000' is not a valid value for datatype http://www.w3.org/2001/XMLSchema#gYear [L8885804]

```

Does this mean that it skips the entire file when it encounters an error?

---

<div class="post-metadata">

**Author:** ![noahgorstein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/noahgorstein/32/584_2.png) [@noahgorstein](https://community.stardog.com/u/noahgorstein)\
**Post date:** [December 9, 2020, 7:43pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/4 "2020-12-09T19:43:07Z")

</div>

Hi Matteo,

Stardog will stop adding data from the data file during `db create` when the parser reports an error. In this case, the parser detected an invalid value for the [XML Schema](http://www.w3.org/2001/XMLSchema#gYear) datatype:

```auto
/Users/noahgorstein/Downloads/yago-wd-facts.nt.gz: '0000' is not a valid value for datatype http://www.w3.org/2001/XMLSchema#gYear [L3511416]

```

You can disable this default parsing behavior and allow Stardog to ingest this data with invalid values by setting the database configuration option `strict.parsing=false` at database creation time like so:

```auto
stardog-admin db create -n yago3 -o strict.parsing=false -- ~/Downloads/yago-wd-facts.nt.gz                
Bulk loading data to new database yago3.
Loaded 22,219,358 triples to yago3 from 1 file(s) in 00:05:11.530 @ 71.3K triples/sec.
Successfully created database 'yago3'.

```

I was able to run your original query once I set `strict.parsing=false` (as seen above) and get back your expected result:

```auto
http://yago-knowledge.org/resource/Robert_Dalrymple_Ross "1827"^^xsd:gYear

```

Please note: `strict.parsing` is an _immutable_ database configuration option. Its value can not be changed after the database is created.

Let us know if that helps.

Cheers,  
Noah

---

<div class="post-metadata">

**Author:** ![M.Lissandrini](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/m.lissandrini/32/626_2.png) [@M.Lissandrini](https://community.stardog.com/u/M.Lissandrini)\
**Post date:** [December 10, 2020, 8:00am UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/5 "2020-12-10T08:00:20Z")

</div>

Hi Noah,  
you've been the most helpful, thanks a lot!

I've seen this policy in other cases/systems, and I am always confused.  
Do you have any pointer for me to understand why a system should "stop" loading and not report any error except in the log?

Was I supposed to know whether my data was loaded without inspecting the log line by line? Or did I miss something?

Thanks again!

---

<div class="post-metadata">

**Author:** ![noahgorstein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/noahgorstein/32/584_2.png) [@noahgorstein](https://community.stardog.com/u/noahgorstein)\
**Post date:** [December 10, 2020, 2:50pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/6 "2020-12-10T14:50:10Z")

</div>

Hi Matteo,

You're very welcome. In addition to the error being reported in the `stardog.log`, you should have had this parser error reported in your console. For example, when I first attempted to load the data (without modifying `strict.parsing`) I got the following error:

```auto
❯ stardog-admin db create -n yago2 ~/Downloads/yago-wd-facts.nt.gz
Bulk loading data to new database yago2.
Errors were encountered during loading:
/Users/noahgorstein/Downloads/yago-wd-facts.nt.gz: '0000' is not a valid value for datatype http://www.w3.org/2001/XMLSchema#gYear [L3511416]
Loaded 3,500,139 triples to yago2 from 1 file(s) in 00:00:55.446 @ 63.1K triples/sec.
Successfully created database 'yago2'.

```

I think we can do better documenting the behavior of this command on parser errors like this. I can see how it could be confusing.

Best,  
Noah

---

<div class="post-metadata">

**Author:** ![M.Lissandrini](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/m.lissandrini/32/626_2.png) [@M.Lissandrini](https://community.stardog.com/u/M.Lissandrini)\
**Post date:** [December 10, 2020, 3:09pm UTC](https://community.stardog.com/t/missing-data-in-repository-after-import/2835/7 "2020-12-10T15:09:41Z")

</div>

I see, I have a workflow where steps are automated, so maybe I've missed it and just read the last line.

thanks again