java.lang.NullPointerException with Elasticsearch mapping

I want to create a Virtual Graph for Elasticsearch. I have followed all the relevant instructions for the mapping and properties files. Even renamed the Elasticsearch mapping type to _doc. I keep getting the same error message: "java.lang.NullPointerException: null". Can someone help me figure out why this is happening?

WARN 2020-02-12 13:30:29,141 [I/O dispatcher 1] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas/_mapping?include_type_name=true] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
WARN 2020-02-12 13:30:29,363 [I/O dispatcher 1] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas/pumf/_mapping?include_type_name=true] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
WARN 2020-02-12 13:30:29,431 [I/O dispatcher 1] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas/_mapping/pumf?include_type_name=true] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
ERROR 2020-02-12 13:30:29,433 [stardog-user-3] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:createVirtualGraph(354): Cannot initialize virtual graph nesstar_virtual_datasets_json
** java.lang.NullPointerException: null**
at com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchTable.getFieldType(ElasticsearchTable.java:288) ~[stardog-virtual-core-7.0.2.jar:?]

I think this must be an index that has an object datatype, which we don't currently support. A fix for the error message will be in the next release.

Could you share details on your use of Object data types? Are they multiple levels deep? Are there arrays? Do you use the "nested" types as well?

Thanks,
Paul Jackson

Here it is! Yes, I do have inner objects with "en" and "fr" fields to support bilingual texts. The index does have arrays as well. Will you have a fix for the support any time soon? Thanks.

{"daaas1":{"mappings":{"_doc":{"properties":{"abstract":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"citation":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"dateCreated":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"dateDocProd":{"properties":{"en":{"type":"date"},"fr":{"type":"date"}}},"dateStudyProd":{"properties":{"en":{"type":"date"},"fr":{"type":"date"}}},"dateStudyVersion":{"properties":{"en":{"type":"date"},"fr":{"type":"date"}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"keywords":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"sampling":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"sections":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"subjects":{"type":"object"},"survey":{"properties":{"title":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"title":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"universe":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"variables":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"weight":{"properties":{"en":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fr" :{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}}}

The error message will be fixed in 7.1.3 but proper support for Object type is still an open ticket (#8517). I'll let the owners of the roadmap know that you asked about it.

-Paul

@PaulJackson, thanks for the update. So, I simplified the index and was able to generate a virtual graph. However, I cannot query that graph. I get the following error:

QueryEval: com.complexible.stardog.plan.eval.ExecutionException: java.lang.NullPointerException

I decided to import the graph in the database and I get:

Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result

Any idea why?

Is there a full stack trace in the stardog.log file?

ERROR 2020-02-14 19:43:04,428 [stardog-user-1] com.complexible.stardog.protocols.http.server.StardogHttpServiceLoader:accept(231): An unexpected exception was handled by the server
com.stardog.stark.query.QueryExecutionFailure: com.complexible.stardog.plan.eval.ExecutionException: java.lang.NullPointerException
at com.complexible.stardog.query.DefaultQueryFactory$TupleQuery.execute(DefaultQueryFactory.java:204) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$TupleQuery.execute(DefaultQueryFactory.java:187) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.protocols.http.server.ProtocolUtils.executeReadQuery(ProtocolUtils.java:449) ~[stardog-protocols-http-server-7.0.2.jar:?]
at com.complexible.stardog.protocols.http.server.SPARQLProtocol.executeQuery(SPARQLProtocol.java:127) ~[stardog-protocols-http-server-7.0.2.jar:?]
at com.complexible.stardog.protocols.http.server.SPARQLProtocol.post(SPARQLProtocol.java:92) ~[stardog-protocols-http-server-7.0.2.jar:?]
at com.stardog.http.server.undertow.jaxrs.ExtractRoutes.lambda$handleIt$5(ExtractRoutes.java:192) ~[stardog-protocols-http-server-7.0.2.jar:?]
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [shiro-core-1.3.0.jar:1.3.0]
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [shiro-core-1.3.0.jar:1.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: com.complexible.stardog.plan.eval.ExecutionException: java.lang.NullPointerException
at com.complexible.stardog.plan.eval.ExecutablePlanFactory.lambda$createOptimized$1(ExecutablePlanFactory.java:123) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.IndexUtils.doInTx(IndexUtils.java:112) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.ExecutablePlanFactory.createOptimized(ExecutablePlanFactory.java:110) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.getExecutablePlan(QueryEngine.java:466) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.getExecutablePlan(QueryEngine.java:407) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeSelect(QueryEngine.java:288) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$TupleQuery.execute(DefaultQueryFactory.java:201) ~[stardog-7.0.2.jar:?]
... 10 more

WARN 2020-02-17 13:46:28,052 [I/O dispatcher 13] org.elasticsearch.client.RestClient:logResponse(65): request [GET http://localhost:9200/daaas1/_mapping/_doc?include_type_name=true] returned 1 warnings: [29
9 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Using include_type_name in get mapping requests is deprecated. The parameter will be removed in the next major version."]
WARN 2020-02-17 13:46:28,479 [I/O dispatcher 13] org.elasticsearch.client.RestClient:logResponse(65): request [POST http://localhost:9200/daaas1/_doc/_search?scroll=1m] returned 1 warnings: [299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in search requests is deprecated."]
ERROR 2020-02-17 13:46:28,704 [stardog-user-3] com.complexible.stardog.db.DatabaseConnectionImpl:apply(764): There was an error adding data
com.complexible.stardog.plan.eval.operator.OperatorException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200
** OK} into class** com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:166) ~[stardog-virtual-core-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:262) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:253) ~[stardog-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.updateWithIteration(BaseIndexWriter.java:88) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.add(BaseIndexWriter.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.Indexes.add(Indexes.java:372) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.index.ConnectableIndexRWConnectionImpl$IndexResourceTransaction.apply(ConnectableIndexRWConnectionImpl.java:654) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.index.ConnectableIndexRWConnectionImpl.apply(ConnectableIndexRWConnectionImpl.java:377) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.ConnectableConnectionCollection.apply(ConnectableConnectionCollection.java:193) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.apply(DatabaseConnectionImpl.java:761) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.add(DatabaseConnectionImpl.java:791) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphsWhole(AbstractGraphUpdateOperator.java:376) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphs(AbstractGraphUpdateOperator.java:355) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AddOperatorImpl.computeNext(AddOperatorImpl.java:32) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AddOperatorImpl.computeNext(AddOperatorImpl.java:21) ~[stardog-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:147) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:134) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.base.CloseableIterator.size(CloseableIterator.java:200) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:76) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:147) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:134) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:26) ~[stardog-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:147) ~[stardog-utils-common-7.0.2.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:134) ~[stardog-utils-common-7.0.2.jar:?]
at com.google.common.collect.Iterators.size(Iterators.java:163) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:330) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:315) ~[stardog-7.0.2.jar:?]

There must be more Caused by clauses in this stack trace. I especially need the last one.

Thanks,
-Paul

Caused by: com.complexible.stardog.db.DatabaseException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.db.DatabaseConnectionImpl.throwDatabaseException(DatabaseConnectionImpl.java:755) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.add(DatabaseConnectionImpl.java:795) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphsWhole(AbstractGraphUpdateOperator.java:376) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphs(AbstractGraphUpdateOperator.java:355) ~[stardog-7.0.2.jar:?]
... 32 more
Caused by: com.complexible.stardog.db.DatabaseException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.db.DatabaseConnectionImpl.throwDatabaseException(DatabaseConnectionImpl.java:755) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.apply(DatabaseConnectionImpl.java:765) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.add(DatabaseConnectionImpl.java:791) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphsWhole(AbstractGraphUpdateOperator.java:376) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator.updateGraphs(AbstractGraphUpdateOperator.java:355) ~[stardog-7.0.2.jar:?]
... 32 more
Caused by: com.complexible.stardog.plan.eval.operator.OperatorException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:166) ~[stardog-virtual-core-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:262) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:253) ~[stardog-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.updateWithIteration(BaseIndexWriter.java:88) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.add(BaseIndexWriter.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.Indexes.add(Indexes.java:372) ~[stardog-7.0.2.jar:?]

There might be something wrong with this community web site that's causing larger message to be broken into multiple sections and/or lost. If that's the case, please accept my apology for that. Perhaps we can try putting the full exception into its own file and attaching it.

The last bit that came through was:

Caused by: com.complexible.stardog.plan.eval.operator.OperatorException: Couldn't parse HTTP response Response{requestLine=POST /daaas1/_doc/_search?scroll=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 200 OK} into class com.complexible.stardog.virtual.vega.elasticsearch.calcite.ElasticsearchJson$Result
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:166) ~[stardog-virtual-core-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:262) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.AbstractGraphUpdateOperator$QuadSourceContextOverrideStatementSource$1.computeNext(AbstractGraphUpdateOperator.java:253) ~[stardog-7.0.2.jar:?]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) ~[guava-26.0-jre.jar:?]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) ~[guava-26.0-jre.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.updateWithIteration(BaseIndexWriter.java:88) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.BaseIndexWriter.add(BaseIndexWriter.java:114) ~[stardog-7.0.2.jar:?]
at com.complexible.stardog.index.Indexes.add(Indexes.java:372) ~[stardog-7.0.2.jar:?]

Looking at there source code and the error message there ought to be at least another section that has:

Caused by: xxxx
at com.complexible.stardog.virtual.VirtualGraphIterator.computeNext(VirtualGraphIterator.java:<some number between 147 and 162>

and probably a few more levels.

I appreciate your patience.

-Paul