Stardog recovery

stephen · February 8, 2017, 2:03pm

Original mailing list request from s*@g***y.com

Dear support, how do i recover stardog? I have 10 databases that are not display at the interface anymore. The files are in the filesystem.

I tried the command 
$STARDOG/bin/stardog-admin db repair coredb

But I got the error:
Database 'coredb' does not exist.

Any command that can fix this?

Here the log trace:

.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 65% complete in 00:00:31 (486.7K triples/sec)
INFO  2017-02-08 10:51:06,579 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 67% complete in 00:00:32 (483.9K triples/sec)
INFO  2017-02-08 10:51:07,648 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 68% complete in 00:00:33 (475.5K triples/sec)
INFO  2017-02-08 10:51:14,314 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 70% complete in 00:00:40 (408.4K triples/sec)
INFO  2017-02-08 10:51:22,240 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 71% complete in 00:00:48 (346.1K triples/sec)
INFO  2017-02-08 10:51:31,387 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 72% complete in 00:00:57 (295.0K triples/sec)
INFO  2017-02-08 10:51:41,158 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 73% complete in 00:01:07 (255.6K triples/sec)
INFO  2017-02-08 10:51:53,333 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 74% complete in 00:01:19 (219.3K triples/sec)
INFO  2017-02-08 10:52:01,618 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 75% complete in 00:01:27 (201.2K triples/sec)
INFO  2017-02-08 10:52:12,955 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 76% complete in 00:01:38 (180.5K triples/sec)
INFO  2017-02-08 10:52:14,001 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 77% complete in 00:01:39 (181.0K triples/sec)
INFO  2017-02-08 10:52:15,480 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 79% complete in 00:01:41 (183.0K triples/sec)
INFO  2017-02-08 10:52:16,900 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 81% complete in 00:01:42 (185.0K triples/sec)
INFO  2017-02-08 10:52:31,460 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 83% complete in 00:01:57 (166.1K triples/sec)
INFO  2017-02-08 10:52:32,862 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 84% complete in 00:01:58 (166.1K triples/sec)
INFO  2017-02-08 10:52:34,046 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 85% complete in 00:01:59 (166.4K triples/sec)
INFO  2017-02-08 10:52:35,652 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 87% complete in 00:02:01 (168.1K triples/sec)
INFO  2017-02-08 10:52:36,805 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 88% complete in 00:02:02 (168.4K triples/sec)
INFO  2017-02-08 10:52:41,434 [Stardog.Executor-236] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 100% complete in 00:02:07 (184.4K triples/sec)
INFO  2017-02-08 10:52:41,435 [Stardog.Executor-236] com.complexible.stardog.StardogKernel:stop(326): 
INFO  2017-02-08 10:52:41,435 [Stardog.Executor-236] com.complexible.stardog.StardogKernel:stop(329): Indexing triples finished in 00:02:07.379
INFO  2017-02-08 10:53:06,830 [Stardog.Executor-271] com.complexible.stardog.StardogKernel:printInternal(314): Persisting dictionary: 100% complete in 00:00:25
INFO  2017-02-08 10:53:06,830 [Stardog.Executor-271] com.complexible.stardog.StardogKernel:stop(326): 
INFO  2017-02-08 10:53:06,830 [Stardog.Executor-271] com.complexible.stardog.StardogKernel:stop(329): Persisting dictionary finished in 00:00:25.385
ERROR 2017-02-08 10:53:06,832 [Stardog.Executor-236] com.complexible.stardog.protocols.server.SPECServerFunction:query(536): There was an error executing query: prefix gn: <http://sws.geonames.org/>
prefix geo: <http://www.opengis.net/ont/geosparql#>
prefix sf: <http://www.opengis.net/def/sf/>
prefix g: <http://geophy.com/>
prefix ws: <http://www.w3.org/2003/01/geo/wgs84_pos#>
insert {
 ?s geo:hasGeometry ?geometry .
 ?geometry a  sf:Point .
 ?geometry geo:asWKT ?point.

         } where {
 ?s ws:lat ?lat .
 ?s ws:long ?lon .

 bind(strdt(concat("Point(", str(?lon)," ", str(?lat),")"),geo:wktLiteral )  as ?point) .
 bind(URI(concat("http://geophy.data/lon-lat/", str(?lon),"-", str(?lat) )) as ?geometry) .
}
com.complexible.stardog.plan.eval.operator.OperatorException: Query execution cancelled: Execution time exceeded query timeout 300000
	at com.complexible.stardog.plan.eval.operator.impl.AbstractOperator.checkCanceled(AbstractOperator.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:129) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.3.jar:?]
	at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3765) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3752) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:311) ~[stardog-protocols-api-server-4.2.3.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
ERROR 2017-02-08 10:53:06,971 [Stardog.Executor-236] com.complexible.common.protocols.server.rpc.ServerHandler:exceptionCaught(401): exceptionCaughtServerHandler
com.complexible.stardog.plan.eval.operator.OperatorException: Query execution cancelled: Execution time exceeded query timeout 300000
	at com.complexible.stardog.plan.eval.operator.impl.AbstractOperator.checkCanceled(AbstractOperator.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:129) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.3.jar:?]
	at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3765) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3752) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:311) ~[stardog-protocols-api-server-4.2.3.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
ERROR 2017-02-08 10:53:06,972 [StardogServer.WorkerGroup-1] com.complexible.stardog.protocols.http.server.HttpMessageEncoder:createErrorResponse(293): The result encoder received an error message it could not encode, error was:
com.complexible.stardog.plan.eval.operator.OperatorException: Query execution cancelled: Execution time exceeded query timeout 300000
	at com.complexible.stardog.plan.eval.operator.impl.AbstractOperator.checkCanceled(AbstractOperator.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:129) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.3.jar:?]
	at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3765) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3752) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:311) ~[stardog-protocols-api-server-4.2.3.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
INFO  2017-02-08 11:15:01,214 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:15:01,237 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
Stardog home is currently locked, cannot start kernel.  Multiple instances of Stardog cannot use the same home directory, 
shutdown the current Stardog server before starting another.  If no other server is running and the Stardog home directory 
appears to be locked, you may remove the file '/data/stardog_data/system.lock' from your Stardog home directory to remove the lock.
INFO  2017-02-08 11:15:47,193 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:15:47,214 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
Stardog home is currently locked, cannot start kernel.  Multiple instances of Stardog cannot use the same home directory, 
shutdown the current Stardog server before starting another.  If no other server is running and the Stardog home directory 
appears to be locked, you may remove the file '/data/stardog_data/system.lock' from your Stardog home directory to remove the lock.
INFO  2017-02-08 11:16:04,653 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:16:04,668 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
INFO  2017-02-08 11:16:05,592 [main] com.complexible.stardog.StardogKernel:start(1871): Initializing Stardog

************************************************************
This copy of Stardog is licensed to Geophy Enterprise #1 (http://geophy.com), Geophy
This is a Enterprise license
This license will expire in 300 days on Tue Dec 05 13:38:59 CET 2017
************************************************************

                                                             :;   
                                      ;;                   `;`:   
  `'+',    ::                        `++                    `;:`  
 +###++,  ,#+                        `++                    .     
 ##+.,',  '#+                         ++                     +    
,##      ####++  ####+:   ##,++` .###+++   .####+    ####++++#    
`##+     ####+'  ##+#++   ###++``###'+++  `###'+++  ###`,++,:     
 ####+    ##+        ++.  ##:   ###  `++  ###  `++` ##`  ++:      
  ###++,  ##+        ++,  ##`   ##;  `++  ##:   ++; ##,  ++:      
    ;+++  ##+    ####++,  ##`   ##:  `++  ##:   ++' ;##'#++       
     ;++  ##+   ###  ++,  ##`   ##'  `++  ##;   ++:  ####+        
,.   +++  ##+   ##:  ++,  ##`   ###  `++  ###  .++  '#;           
,####++'  +##++ ###+#+++` ##`   :####+++  `####++'  ;####++`      
`####+;    ##++  ###+,++` ##`    ;###:++   `###+;   `###++++      
                                                    ##   `++      
                                                   .##   ;++      
                                                    #####++`      
                                                     `;;;.        
************************************************************


Stardog server 4.2.3 started on Wed Feb 08 11:16:06 CET 2017.

Stardog server is listening on all network interfaces.
HTTP server available at http://localhost:5820.

STARDOG_HOME=/data/stardog_data 

INFO  2017-02-08 11:16:06,008 [main] com.complexible.stardog.cli.impl.ServerStart:call(249): Memory options
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(250): Min Heap Size: 2.0G
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(251): Max Heap Size: 25G
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(252): Max Off-Heap : 12G
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(253): System Memory: 35G
INFO  2017-02-08 11:17:23,133 [Stardog.Executor-24] com.complexible.stardog.StardogKernel:shutdown(2119): Stardog server 4.2.3 shutdown on 2017-02-08T11:17:23.132+0100.


INFO  2017-02-08 11:17:56,219 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:17:56,237 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
INFO  2017-02-08 11:17:56,857 [main] com.complexible.stardog.StardogKernel:start(1871): Initializing Stardog

stephen · February 8, 2017, 2:18pm

Hi Samur,

Am I correct in assuming that the output of a stardog-admin db list command says there are no databases in the system? Or is it specifically the coredb database that you are being told doesn’t exist?

samur · February 9, 2017, 8:16am

HI Stephen, no database is displayed or listed.

All the databases get ignored at start-up. Do you suggest any fix to this?

stephen · February 9, 2017, 12:32pm

Samur,

There is a method that you can try to get the databases showing up again:

Create a new STARDOG_HOME directory
Start up the Stardog server pointing at that new home
Create an empty database (named the same) for each database you need
Stop the Stardog server
Move the contents of the database folders for each database from the old STARDOG_HOME to the new one
Restart the Stardog server and it should recognize and read the databases

samur · February 9, 2017, 12:53pm

Hi Stephen, Thank you. This is quite a trick.

If I have a production application and this problem happens again. Is this the only way to do it?

Could you give me some background to understand why this happened?

Best,

stephen · February 9, 2017, 12:58pm

Well with a production application it's probably a better idea to use the db backup/import functionality [1].
The only way I have seen for this to happen is for something to happen to the system db/folder. Usually I have deleted it for whatever reason, which the server doesn't take too kindly to. Not sure if in your case it was deleted (maybe somebody tried to delete system.lock and deleted system instead?), or if the system db was just corrupted in some other way (e.g., server killed during a write).

[1] 404 | Stardog Documentation Latest

samur · February 9, 2017, 1:44pm

Thank you Stephen. It may be that the system was deleted accidentally. Thank you for the tip.

stephen · February 23, 2017, 1:44pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to remove database completely Support	5	433	February 10, 2021
How to re-attach a Stardog database? Support	4	406	August 21, 2018
Restore an older backup in 7.6.5 Support	1	323	July 7, 2021
File not found exceptions when adding a db Support	3	535	May 16, 2018
Stardog Instance data loss Support	7	348	November 10, 2021

Stardog recovery

Related topics