java.lang.NullPointerException with Elasticsearch mapping

Support
1

I want to create a Virtual Graph for Elasticsearch. I have followed all the relevant instructions for the mapping and properties files. Even renamed the Elasticsearch mapping type to _doc. I keep getting the same error message: "java.lang.NullPointerException: null". Can someone help me figure out why this is happening?

WARN 2020-02-12 13:30:29,141 [I/O dispatcher 1] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas/_mapping?include_type_name=true] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
WARN 2020-02-12 13:30:29,363 [I/O dispatcher 1] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas/pumf/_mapping?include_type_name=true] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
WARN 2020-02-12 13:30:29,431 [I/O dispatcher 1] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas/_mapping/pumf?include_type_name=true] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
ERROR 2020-02-12 13:30:29,433 [stardog-user-3] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:createVirtualGraph(354): Cannot initialize virtual graph nesstar_virtual_datasets_json
** java.lang.NullPointerException: null**
at com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchTable.getFieldType(ElasticsearchTable.java:288) ~[stardog-virtual-core-7.0.2.jar:?]

2

I think this must be an index that has an object datatype, which we don't currently support. A fix for the error message will be in the next release.

Could you share details on your use of Object data types? Are they multiple levels deep? Are there arrays? Do you use the "nested" types as well?

Thanks,
Paul Jackson

3

Here it is! Yes, I do have inner objects with "en" and "fr" fields to support bilingual texts. The index does have arrays as well. Will you have a fix for the support any time soon? Thanks.

{"daaas1":{"mappings":{"_doc":{"properties":{"abstract":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"citation":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"dateCreated":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"dateDocProd":{"properties":{"en":{"type":"date"},"fr":{"type":"date"}}},"dateStudyProd":{"properties":{"en":{"type":"date"},"fr":{"type":"date"}}},"dateStudyVersion":{"properties":{"en":{"type":"date"},"fr":{"type":"date"}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"keywords":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"sampling":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"sections":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"subjects":{"type":"object"},"survey":{"properties":{"title":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"title":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"universe":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"variables":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"weight":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr" :{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}}}

4

The error message will be fixed in 7.1.3 but proper support for Object type is still an open ticket (#8517). I'll let the owners of the roadmap know that you asked about it.

-Paul

5

@PaulJackson, thanks for the update. So, I simplified the index and was able to generate a virtual graph. However, I cannot query that graph. I get the following error:

QueryEval: com.complexible.stardog.plan.eval.ExecutionException: java.lang.NullPointerException

I decided to import the graph in the database and I get:

Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result

Any idea why?

6

Is there a full stack trace in the stardog.log file?

7

ERROR 2020-02-14 19:43:04,428 [stardog-user-1] com.complexible.stardog.protocols.http.server.StardogHttpServiceLoader:accept(231): An unexpected exception was handled by the server
com.stardog.stark.query.QueryExecutionFailure: com.complexible.stardog.plan.eval.ExecutionException: java.lang.NullPointerException
at com.complexible.stardog.query.DefaultQueryFactory$TupleQuery.execute(DefaultQueryFactory.java:204) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$TupleQuery.execute(DefaultQueryFactory.java:187) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.protocols.http.server.ProtocolUtils.executeReadQuery(ProtocolUtils.java:449) ~[stardog-protocols-http-server-7.0.2.jar:?]
at com.complexible.stardog.protocols.http.server.SPARQLProtocol.executeQuery(SPARQLProtocol.java:127) ~[stardog-protocols-http-server-7.0.2.jar:?]
at com.complexible.stardog.protocols.http.server.SPARQLProtocol.post(SPARQLProtocol.java:92) ~[stardog-protocols-http-server-7.0.2.jar:?]
at com.stardog.http.server.undertow.jaxrs.ExtractRoutes.lambda$handleIt$5(ExtractRoutes.java:192) ~[stardog-protocols-http-server-7.0.2.jar:?]
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [shiro-core-1.3.0.jar:1.3.0]
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [shiro-core-1.3.0.jar:1.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: com.complexible.stardog.plan.eval.ExecutionException: java.lang.NullPointerException
at com.complexible.stardog.plan.eval.ExecutablePlanFactory.lambda$createOptimized$1(ExecutablePlanFactory.java:123) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.IndexUtils.doInTx(IndexUtils.java:112) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.ExecutablePlanFactory.createOptimized(ExecutablePlanFactory.java:110) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.getExecutablePlan(QueryEngine.java:466) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.getExecutablePlan(QueryEngine.java:407) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeSelect(QueryEngine.java:288) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$TupleQuery.execute(DefaultQueryFactory.java:201) ~[stardog-7.0.2.jar:?]
... 10 more

8

WARN 2020-02-17 13:46:28,052 [I/O dispatcher 13] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas1/_mapping/_doc?include_type_name=true] returned 1 warnings: [29
9 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
WARN 2020-02-17 13:46:28,479 [I/O dispatcher 13] org.elasticsearch.client.RestClient:logResponse(65): request [POST http://localhost:9200/daaas1/_doc/_search?scroll=1m] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in search requests is deprecated."]
ERROR 2020-02-17 13:46:28,704 [stardog-user-3] com.complexible.stardog.db.DatabaseConnectionImpl:apply(764): There was an error adding data
com.complexible.stardog.plan.eval.operator.OperatorException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200
** OK} into class** com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:166) ~[stardog-virtual-core-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:262) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:253) ~[stardog-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.updateWithIteration(BaseIndexWriter.java:88) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.add(BaseIndexWriter.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.Indexes.add(Indexes.java:372) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.index.ConnectableIndexRWConnectionImpl$IndexResourceTransaction.apply(ConnectableIndexRWConnectionImpl.java:654) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.index.ConnectableIndexRWConnectionImpl.apply(ConnectableIndexRWConnectionImpl.java:377) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.ConnectableConnectionCollection.apply(ConnectableConnectionCollection.java:193) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.apply(DatabaseConnectionImpl.java:761) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.add(DatabaseConnectionImpl.java:791) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphsWhole(AbstractGraphUpdateOperator.java:376) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphs(AbstractGraphUpdateOperator.java:355) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AddOperatorImpl.computeNext(AddOperatorImpl.java:32) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AddOperatorImpl.computeNext(AddOperatorImpl.java:21) ~[stardog-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:147) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:134) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.base.CloseableIterator.size(CloseableIterator.java:200) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:76) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:147) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:134) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:26) ~[stardog-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:147) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:134) ~[stardog-utils-common-7.0.2.jar:?]
at com.google.common.collect.Iterators.size(Iterators.java:163) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:330) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:315) ~[stardog-7.0.2.jar:?]

9

There must be more Caused by clauses in this stack trace. I especially need the last one.

Thanks,
-Paul

10

Caused by: com.complexible.stardog.db.DatabaseException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.db.DatabaseConnectionImpl.throwDatabaseException(DatabaseConnectionImpl.java:755) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.add(DatabaseConnectionImpl.java:795) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphsWhole(AbstractGraphUpdateOperator.java:376) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphs(AbstractGraphUpdateOperator.java:355) ~[stardog-7.0.2.jar:?]
... 32 more
Caused by: com.complexible.stardog.db.DatabaseException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.db.DatabaseConnectionImpl.throwDatabaseException(DatabaseConnectionImpl.java:755) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.apply(DatabaseConnectionImpl.java:765) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.add(DatabaseConnectionImpl.java:791) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphsWhole(AbstractGraphUpdateOperator.java:376) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphs(AbstractGraphUpdateOperator.java:355) ~[stardog-7.0.2.jar:?]
... 32 more
Caused by: com.complexible.stardog.plan.eval.operator.OperatorException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:166) ~[stardog-virtual-core-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:262) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:253) ~[stardog-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.updateWithIteration(BaseIndexWriter.java:88) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.add(BaseIndexWriter.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.Indexes.add(Indexes.java:372) ~[stardog-7.0.2.jar:?]

11

There might be something wrong with this community web site that's causing larger message to be broken into multiple sections and/or lost. If that's the case, please accept my apology for that. Perhaps we can try putting the full exception into its own file and attaching it.

The last bit that came through was:

Caused by: com.complexible.stardog.plan.eval.operator.OperatorException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:166) ~[stardog-virtual-core-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:262) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:253) ~[stardog-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.updateWithIteration(BaseIndexWriter.java:88) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.add(BaseIndexWriter.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.Indexes.add(Indexes.java:372) ~[stardog-7.0.2.jar:?]

Looking at there source code and the error message there ought to be at least another section that has:

Caused by: xxxx
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:<some number between 147 and 162>

and probably a few more levels.

I appreciate your patience.

-Paul

12

Here it is:
stardog.log (225.5 KB)

13

That worked. I can see a couple different issues in that file. I'll investigate and get back to you.

Thanks,

-Paul

14

I see two issues in the log. The first is a NullPointerException having to do with unique.key.set fields that can't be found in the ES mapping. I am assuming this virtual graph must have been configured with a value set for the "unique.key.sets" property. We should give a better error message explaining which field could not be found and I've created an issue for that (#8535). This could also have something to do with the next issue.

The second issue has to do with the format of the ES response changing from ES version 6.x to 7.5. We currently support version 6. I've set up a ES 7.x instance here and reproduced your issue. I'll have to do some more work before I can say when we'll be able to support that.

-Paul

15

Thanks for you input! I downgraded to ES 6.7.1 and it now works.

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.