Databricks virtualized data query times questions

[hall00ween]

@PaulJackson's reply from another post is apropos for the ~50 second query shown above:

@PaulJackson

Hi Jamie,

That's the text mode I was talking about. Studio also has a graphical mode that can hide some important details. No worries.

What this plan is showing is Stardog has a fairly big plan that has a number of joins with virtual graph queries as arguments. For whatever reason the optimizer opted for HashJoins rather than ServiceJoins, no matter. The main point is that Stardog is executing a plan that includes some plan nodes that it is executing locally and many sub-plans that it is executing on remote database. For this reason, you can't compare the output of the queries that appear on Databricks to the overall SPARQL query. The Databricks queries are for different pieces of the larger plan. Their output is being further operated on on the Stardog side.

There's a couple things you can do at this point. It's not required, but you can encourage Stardog to push more of the plan down to the Databricks side by avoiding what we call virtual transparency, which is what's used when the dataset for the query is a wildcard, like stardog:context:virtual in this case. If you change the dataset from virtual to virtual://TJC-C360-Demo-Update-Azure-DB__data__Databricks-C360-Demo-Database, the only virtual graph that's being referenced in this query, you'll more likely get a single VirtualGraphSql plan node with all the SQL for the query in that one node.

That shouldn't change the output. You should expect to still get no results, but the SQL that shows up in the Databricks query history should also return no results.

From there, figuring out why you're getting no results is an iterative process of breaking your query down into smaller bits and building it back up iteratively and taking note at which point the results are dropped. From there we can usually figure out what's going on.

-Paul