Stardog Transaction framework

manjusha · April 14, 2022, 5:25am

Hi,

We want to perform some operations when new data is inserted into Stardog DB (version 7.9.0) . We are trying Stardog Stardog transaction framework to achieve this. We are following the exapmle code - stardog-examples/examples/connectable at develop · stardog-union/stardog-examples · GitHub .

To make it compile successfully, we had to change the following method
@Override
public Set getQueryRewritings() {
// no query rewritings are performed
return ImmutableSet.of();
}
to
@Override
public Set getQueryRewritings() {
// no query rewritings are performed
return ImmutableSet.of();
}

(Just changed the methid signature) So is this code upto date and can be used with latest Stardog server version 7.9.0 ?

After successful compilation with above change, when deployed in stardog server, but our queries started failing with NullPointerException. If we remove this transaction listener, queries are executed successfully.

Transaction started for database mg-test
ERROR 2022-04-13 08:33:19,270 [stardog-user-2] com.complexible.stardog.QueryManager:exceptionTriggered(542): Error during execution of the query
SELECT ?cust ?custId ?contact WHERE { GRAPH <tenant1:xxx-profiles> {?cust a xxx:Customer ;
 xxx:id ?custId ; xxx:has_contact ?contact .
 ?contact ?p ?o .
 }
 VALUES ?o { "47337aac-ab1b-4fe7-8f13-a1445468f97b" }
 } 
with query plan
prefix : <http://api.stardog.com/>
prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>
prefix owl: <http://www.w3.org/2002/07/owl#>
prefix stardog: <tag:stardog:api:>

Projection(?cust, ?custId, ?contact)
`─ {
   `─ {
      `─ Scan[SPO](?cust, <http://ontologies.xyz.com/xxx/has_contact>, ?contact){<tenant1:xxx-profiles>}
      `─ Scan[SPO](?cust, <http://ontologies.xyz.com/xxx/id>, ?custId){<tenant1:xxx-profiles>}
      `─ Scan[SPO](?cust, rdf:type, <http://ontologies.xyz.com/xxx/Customer>){<tenant1:xxx-profiles>}
      `─ Scan[SPO](?contact, ?p, ?o){<tenant1:xxx-profiles>}
      }
      VALUES (?o) {
      `─ ( "47337aac-ab1b-4fe7-8f13-a1445468f97b" )
      }
   }
against database mg-test
java.lang.NullPointerException: null
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632) ~[?:?]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) ~[?:?]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543) ~[?:?]
	at com.complexible.stardog.db.ConnectableConnectionCollection.as(ConnectableConnectionCollection.java:110) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.db.DatabaseConnectionImpl.as(DatabaseConnectionImpl.java:764) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.db.DelegatingConnectableConnection.as(DelegatingConnectableConnection.java:61) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.db.ConnectableConnection.as(ConnectableConnection.java:54) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine$QueryTransactionalConnectableRWConnection.as(QueryEngine.java:805) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.cache.ReplaceCachedGraphs$Visitor.lambda$getReuseDecisionFunction$6(ReplaceCachedGraphs.java:270) ~[stardog-cache-core-7.9.0.jar:?]
	at com.complexible.stardog.plan.cache.ReuseDecisionFunctions$CombinedReuseDecisionFunction.lambda$decide$0(ReuseDecisionFunctions.java:175) ~[stardog-7.9.0.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:558) ~[?:?]
	at java.util.stream.ReferencePipeline.max(ReferencePipeline.java:594) ~[?:?]
	at com.complexible.stardog.plan.cache.ReuseDecisionFunctions$CombinedReuseDecisionFunction.decide(ReuseDecisionFunctions.java:176) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.cache.CachedPlan.isReusableFor(CachedPlan.java:55) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.cache.SimplePlanCache.lookup(SimplePlanCache.java:176) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.cache.SimplePlanCache.get(SimplePlanCache.java:100) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.ExecutablePlanFactory.lambda$createOptimized$2(ExecutablePlanFactory.java:153) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.index.IndexUtils.doInTx(IndexUtils.java:217) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.ExecutablePlanFactory.createOptimized(ExecutablePlanFactory.java:140) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.getExecutablePlan(QueryEngine.java:693) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executePlan(QueryEngine.java:419) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executePlan(QueryEngine.java:401) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeSelectPlan(QueryEngine.java:361) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeSelect(QueryEngine.java:344) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$SelectQuery.execute(DefaultQueryFactory.java:269) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$SelectQuery.execute(DefaultQueryFactory.java:255) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.query.DelegatingQuery.execute(DelegatingQuery.java:68) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPausingReadQuery.execute(DatabaseImpl.java:1779) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPausingReadQuery.execute(DatabaseImpl.java:1766) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.query.DelegatingQuery.execute(DelegatingQuery.java:60) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredReadQuery.execute(StardogKernel.java:4847) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredReadQuery.execute(StardogKernel.java:4834) ~[stardog-7.9.0.jar:?]
	at com.complexible.stardog.protocols.http.server.ProtocolUtils.executeReadQuery(ProtocolUtils.java:542) ~[stardog-protocols-http-server-7.9.0.jar:?]
	at com.complexible.stardog.protocols.http.server.ProtocolUtils.executeReadQuery(ProtocolUtils.java:515) ~[stardog-protocols-http-server-7.9.0.jar:?]
	at com.complexible.stardog.protocols.http.server.SPARQLProtocol.executeQuery(SPARQLProtocol.java:250) ~[stardog-protocols-http-server-7.9.0.jar:?]
	at com.complexible.stardog.protocols.http.server.SPARQLProtocol.getSparqlQuery(SPARQLProtocol.java:139) ~[stardog-protocols-http-server-7.9.0.jar:?]
	at com.stardog.http.server.undertow.jaxrs.ExtractRoutes.lambda$handleIt$5(ExtractRoutes.java:192) ~[stardog-protocols-http-server-7.9.0.jar:?]
	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [shiro-core-1.8.0.jar:1.8.0]
	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [shiro-core-1.8.0.jar:1.8.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Any idea why queries are failing and how to fix this issue.?

Is there any way to restrict invocation of this listener to a list of db?
Also where to find javadoc for com.complexible.stardog.db.ConnectableConnection and other classes used in this example?

Thanks!

zachary.whitley · April 14, 2022, 2:46pm

I'll let someone else respond to your other questions but the java docs are located directly at Stardog-8.2.0 API or you can also get to them from https://docs.stardog.com in the pull down menu under "Api Reference > Java API" or under the left hand navigation menu at the bottom under "Java API".

The Stardog-examples are a bit out of date as of 7.9.0

manjusha · April 18, 2022, 5:29am

Hi,
These java docs do not contain any of the classes used in this example. I searched for "ConnectableConnection" and did not find anything. Also the package com.complexible.stardog.db does not have ConnectableConnection class listed under it.

manjusha · April 21, 2022, 5:33am

any suggestions to fix this NullPointerException?

Also is there any other way to write transaction listeners in Stardog?

Thanks!

pavel · April 21, 2022, 8:43am

Can you share the full stack trace of the NPE (and ideally your extension's source code too)? It's indeed possible that we need to update the example.

Cheers,
Pavel

manjusha · April 21, 2022, 12:56pm

We are using the same example code, except changed the method signature as mentioned in first message to make it compile.
Also that's the complete stack trace logged in stardog.log file.

Thanks!

pavel · April 21, 2022, 1:28pm

Sorry, I somehow missed the stacktrace. That should be enough info. We're currently in the midst of prepping a release so many need some time to look into that, but we will.

Thanks,
Pavel

pavel · April 22, 2022, 8:18am

I just tweaked the ExampleConnectableConnection source code to make it compile with Stardog 7.9.0. I have also built the jar (using gradlew clean -x test build), copied it to server/dbms/ of my Stardog distro, and then tried to add data to a test database as follows:

pavel$ stardog query test "insert data { :a :p :b ; :q :c  }"
Transaction committed successfully in 00:00:00.216

I see the following in stardog.log inside the STARDOG_HOME dir:

Transaction started for database test
Database: test Change: Add Addition: true
<http://api.stardog.com/a> <http://api.stardog.com/p> <http://api.stardog.com/b> .
<http://api.stardog.com/a> <http://api.stardog.com/q> <http://api.stardog.com/c> .

Hope it helps,
Pavel

manjusha · April 25, 2022, 9:58am

Hi Pavel,

Thank you for fixing this example code. Now it is working properly.

We have few questions on the usage of this framework:

Right now, this listener is getting called for all the databases. Is there any way we can configure list of databases so that only when data is added or removed from only these databases, this listener should be called?
This example code has a class - ExampleConnectableOption. Can you please explain its usage and significance. The code comment says :

If the custom connectable requires some configuration parameters they can  be defined in this class, set by users using the regular admin functionality  and read by the connectable at runtime.

Can you please elaborate more and give some example?

Where can we find the java doc for all the classes used in this example. we did check official java API, but did not find any.

Thanks again for your help!

pavel · April 25, 2022, 10:14am

The listener is going to be called for transactions against all databases so you'd need to check inside the listener. I believe this part is an example of that: stardog-examples/ExampleConnectableConnection.java at develop · stardog-union/stardog-examples · GitHub

You can modify that to check the database name instead of the example.connectable.enabled option. It also shows how to use custom database options for the listener, example.connectable.enabled is such option. You can set it using the standard stardog-admin metadata set command and then read its value inside the listener. ExampleConnectableOption must define all such options.

We'll check issues with the javadoc and include the missing bits, thanks for that. Let us know if you have specific questions.

Best,
Pavel

manjusha · April 25, 2022, 2:33pm

Hi Pavel,

We tried setting "example.connectable.enabled" option to one of our databases using CLI. But it fails with error "Invalid option":

stardog-admin metadata set -o example.connectable.enabled=false -- my_db
Invalid option: example.connectable.enabled.

are we missing something?
-Thanks!

pavel · April 25, 2022, 2:51pm

Did you register the new config option like here: stardog-examples/ExampleModule.java at develop · stardog-union/stardog-examples · GitHub

It's supposed to be available after that, let us know if it doesn't

Cheers,
Pavel

manjusha · April 26, 2022, 7:19am

Yes, the new config option is registered as shown in the example. But still we get error.

pavel · April 27, 2022, 10:57am

Yeah, you're right. This is a client-side error, the CLI command does not recognise the option. Looks like the option needs to be registered on the client too and it will require changes to the example.

It does seem to work on the server though, e.g. I can set with curl (i.e. no validation checks on the client):

$ curl -d '{"example.connectable.enabled":"false"}' -H "Content-Type: application/json" -X POST http://localhost:5820/admin/databases/test/options
{ "message" : "The option(s) for the database 'test' were successfully set." }

Unfortunately you can't query it with metadata get for the same reason but I executed an update and checked that it's indeed unset. You can use curl for that too.

Sorry for the inconvenience, will look into that.,
Pavel

manjusha · April 29, 2022, 9:25am

Hi Pavel,

This workaround is working. We are able to set the options using HTTP API.

Please do let us know one you fix this in example.

Thanks!

manjusha · May 5, 2022, 4:37am

Hi Pavel,

did you get a chance to fix the example code to for configuration issue?

Thanks!

pavel · May 5, 2022, 6:56am

Not yet, unfortunately. I created PLAT-4289 ticket about it. We need to think if we drop the option validation on the client side at all (i.e. the CLI will send values as strings, like curl) or we have make the example more complicated and require you to register the option on the client side too. I am not sure I like the 2nd option (from the engineering perspective).

This is a product enhancement request now so if you'd like to have it prioritised, please contact us off the list.

Thanks,
Pavel

manjusha · May 5, 2022, 9:25am

Hi Pavel,

Thanks for creating ticket to track this issue. We will get back to you if we need this on priority.

The way , we can define configuration options for database, is it possible to define custom properties for the server? The transaction listener we are implementing needs some configuration properties. Can we define and register them the way it is done for "example.connectable.enabled"? We want to avoid standard JAVA properties approach (system property or environment variable or properties file) as that will require server restart if we want to change the property.

Also let us know how to set them. We believe these need to be set by using
stardog-admin property set command as there is no HTTP API to set properties.

Thanks!

pavel · May 5, 2022, 11:11am

Yeah, you should be able to do that by using server() instead of database() when defining the option in the same place. I might be wrong here (need to check) but it's possible that the CLI won't validate server options on the client when doing property set so it may work (as opposed to metadata set).

In fact CLI works via the HTTP API so it must be supported there as well: API Reference | ReDoc

Again, didn't have time to check now so please try and report if something is amiss,
Pavel

manjusha · May 5, 2022, 2:40pm

Hi Pavel,

Thanks for quick response

We were able to create new server properties and able to set using CLI.

Thanks!

Topic		Replies	Views
SPARQL query on virtual graph Support	7	747	August 10, 2018
NullPointerException during query evaluation Bug	6	587	August 1, 2018
Stardog not compliant with W3C standard HTTP interface? Support	3	351	March 19, 2020
Error in stardog log when running a query Bug	7	832	May 9, 2018
Multiple query in one request? Support	3	640	September 3, 2020

Stardog Transaction framework

Related topics