# Stardog spark connector fails when executed as jar job on Azure Databricks

**URL:** <https://community.stardog.com/t/stardog-spark-connector-fails-when-executed-as-jar-job-on-azure-databricks/3407>\
**Category:** Support\
**Created:** [December 8, 2021, 11:59am UTC](https://community.stardog.com/t/stardog-spark-connector-fails-when-executed-as-jar-job-on-azure-databricks/3407 "2021-12-08T11:59:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![manjusha](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@manjusha](https://community.stardog.com/u/manjusha)\
**Post date:** [December 8, 2021, 11:59am UTC](https://community.stardog.com/t/stardog-spark-connector-fails-when-executed-as-jar-job-on-azure-databricks/3407/1 "2021-12-08T11:59:41Z")

</div>

Hi,

I am using Stardog spark connector (stardog-spark-connector-1.0.0.jar) for running graph analytics algorithms on Azure Databricks. I have followed instructions given on Stardog documentation [Setup and Usage | Stardog Documentation Latest](https://docs.stardog.com/graph-analytics/setup#databricks)

My Azure databricks configuration is : Driver: Standard\_DS3\_v2, Workers: Standard\_DS3\_v2, 2-8 workers, 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)

But my job fails with following error for all algorithms:

```auto
21/12/08 07:06:33 INFO SparkContext: Successfully stopped SparkContext
21/12/08 07:06:33 INFO ProgressReporter$: Removed result fetcher for 6455807286084721632_7483483563914453930_job-125519-run-11-action-3497022421842561
21/12/08 07:06:33 WARN ScalaDriverWrapper: Spark is detected to be down after running a command
21/12/08 07:06:33 WARN ScalaDriverWrapper: Fatal exception (spark down) in ReplId-5997a-285b9-755be-0
com.databricks.backend.common.rpc.SparkStoppedException: Spark down: 
	at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:530)
	at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:634)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:427)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:370)
	at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:221)
	at java.lang.Thread.run(Thread.java:748)

```

Also I noticed that though the job status is shown as failed in Databricks UI, when the job is completed, the analytics graph is created successfully in stardog DB. So it seems the algorithm is getting executed successfully, but the job is failing when the spark connector jar is trying to stop the Spark cluster. Due to this , my pipeline is failing. Is this a bug in spark connector? Is there any way to get around it?

Thanks!

---

<div class="post-metadata">

**Author:** ![evren](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.stardog.com/evren/32/1515_2.png) [@evren](https://community.stardog.com/u/evren)\
**Post date:** [December 9, 2021, 7:13pm UTC](https://community.stardog.com/t/stardog-spark-connector-fails-when-executed-as-jar-job-on-azure-databricks/3407/2 "2021-12-09T19:13:55Z")

</div>

This is an issue we were aware of and have already fixed. You can get the latest [connector](https://stardog-spark-connector.s3.amazonaws.com/stardog-spark-connector-1.0.1.jar) which should resolve this issue.

Best,  
Evren

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex030/uploads/stardog/original/2X/e/ed66b48a616f505106a4acde3c9cee7e6da9bc67.svg) [@system](https://community.stardog.com/u/system)\
**Post date:** [December 23, 2021, 7:14pm UTC](https://community.stardog.com/t/stardog-spark-connector-fails-when-executed-as-jar-job-on-azure-databricks/3407/3 "2021-12-23T19:14:35Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.