Stardog recovery

Original mailing list request from s*@g***y.com

Dear support, how do i recover stardog? I have 10 databases that are not display at the interface anymore. The files are in the filesystem.

I tried the command 
$STARDOG/bin/stardog-admin db repair coredb

But I got the error:
Database 'coredb' does not exist.

Any command that can fix this?
Here the log trace:

.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 65% complete in 00:00:31 (486.7K triples/sec)
INFO  2017-02-08 10:51:06,579 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 67% complete in 00:00:32 (483.9K triples/sec)
INFO  2017-02-08 10:51:07,648 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 68% complete in 00:00:33 (475.5K triples/sec)
INFO  2017-02-08 10:51:14,314 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 70% complete in 00:00:40 (408.4K triples/sec)
INFO  2017-02-08 10:51:22,240 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 71% complete in 00:00:48 (346.1K triples/sec)
INFO  2017-02-08 10:51:31,387 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 72% complete in 00:00:57 (295.0K triples/sec)
INFO  2017-02-08 10:51:41,158 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 73% complete in 00:01:07 (255.6K triples/sec)
INFO  2017-02-08 10:51:53,333 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 74% complete in 00:01:19 (219.3K triples/sec)
INFO  2017-02-08 10:52:01,618 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 75% complete in 00:01:27 (201.2K triples/sec)
INFO  2017-02-08 10:52:12,955 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 76% complete in 00:01:38 (180.5K triples/sec)
INFO  2017-02-08 10:52:14,001 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 77% complete in 00:01:39 (181.0K triples/sec)
INFO  2017-02-08 10:52:15,480 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 79% complete in 00:01:41 (183.0K triples/sec)
INFO  2017-02-08 10:52:16,900 [Stardog.Executor-266] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 81% complete in 00:01:42 (185.0K triples/sec)
INFO  2017-02-08 10:52:31,460 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 83% complete in 00:01:57 (166.1K triples/sec)
INFO  2017-02-08 10:52:32,862 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 84% complete in 00:01:58 (166.1K triples/sec)
INFO  2017-02-08 10:52:34,046 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 85% complete in 00:01:59 (166.4K triples/sec)
INFO  2017-02-08 10:52:35,652 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 87% complete in 00:02:01 (168.1K triples/sec)
INFO  2017-02-08 10:52:36,805 [Stardog.Executor-264] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 88% complete in 00:02:02 (168.4K triples/sec)
INFO  2017-02-08 10:52:41,434 [Stardog.Executor-236] com.complexible.stardog.StardogKernel:printInternal(314): Indexing triples: 100% complete in 00:02:07 (184.4K triples/sec)
INFO  2017-02-08 10:52:41,435 [Stardog.Executor-236] com.complexible.stardog.StardogKernel:stop(326): 
INFO  2017-02-08 10:52:41,435 [Stardog.Executor-236] com.complexible.stardog.StardogKernel:stop(329): Indexing triples finished in 00:02:07.379
INFO  2017-02-08 10:53:06,830 [Stardog.Executor-271] com.complexible.stardog.StardogKernel:printInternal(314): Persisting dictionary: 100% complete in 00:00:25
INFO  2017-02-08 10:53:06,830 [Stardog.Executor-271] com.complexible.stardog.StardogKernel:stop(326): 
INFO  2017-02-08 10:53:06,830 [Stardog.Executor-271] com.complexible.stardog.StardogKernel:stop(329): Persisting dictionary finished in 00:00:25.385
ERROR 2017-02-08 10:53:06,832 [Stardog.Executor-236] com.complexible.stardog.protocols.server.SPECServerFunction:query(536): There was an error executing query: prefix gn: <http://sws.geonames.org/>
prefix geo: <http://www.opengis.net/ont/geosparql#>
prefix sf: <http://www.opengis.net/def/sf/>
prefix g: <http://geophy.com/>
prefix ws: <http://www.w3.org/2003/01/geo/wgs84_pos#>
insert {
 ?s geo:hasGeometry ?geometry .
 ?geometry a  sf:Point .
 ?geometry geo:asWKT ?point.

         } where {
 ?s ws:lat ?lat .
 ?s ws:long ?lon .

 bind(strdt(concat("Point(", str(?lon)," ", str(?lat),")"),geo:wktLiteral )  as ?point) .
 bind(URI(concat("http://geophy.data/lon-lat/", str(?lon),"-", str(?lat) )) as ?geometry) .
}
com.complexible.stardog.plan.eval.operator.OperatorException: Query execution cancelled: Execution time exceeded query timeout 300000
	at com.complexible.stardog.plan.eval.operator.impl.AbstractOperator.checkCanceled(AbstractOperator.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:129) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.3.jar:?]
	at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3765) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3752) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:311) ~[stardog-protocols-api-server-4.2.3.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
ERROR 2017-02-08 10:53:06,971 [Stardog.Executor-236] com.complexible.common.protocols.server.rpc.ServerHandler:exceptionCaught(401): exceptionCaughtServerHandler
com.complexible.stardog.plan.eval.operator.OperatorException: Query execution cancelled: Execution time exceeded query timeout 300000
	at com.complexible.stardog.plan.eval.operator.impl.AbstractOperator.checkCanceled(AbstractOperator.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:129) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.3.jar:?]
	at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3765) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3752) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:311) ~[stardog-protocols-api-server-4.2.3.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
ERROR 2017-02-08 10:53:06,972 [StardogServer.WorkerGroup-1] com.complexible.stardog.protocols.http.server.HttpMessageEncoder:createErrorResponse(293): The result encoder received an error message it could not encode, error was:
com.complexible.stardog.plan.eval.operator.OperatorException: Query execution cancelled: Execution time exceeded query timeout 300000
	at com.complexible.stardog.plan.eval.operator.impl.AbstractOperator.checkCanceled(AbstractOperator.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:129) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.3.jar:?]
	at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.3.jar:?]
	at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
	at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3765) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3752) ~[stardog-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.3.jar:?]
	at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:311) ~[stardog-protocols-api-server-4.2.3.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
INFO  2017-02-08 11:15:01,214 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:15:01,237 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
Stardog home is currently locked, cannot start kernel.  Multiple instances of Stardog cannot use the same home directory, 
shutdown the current Stardog server before starting another.  If no other server is running and the Stardog home directory 
appears to be locked, you may remove the file '/data/stardog_data/system.lock' from your Stardog home directory to remove the lock.
INFO  2017-02-08 11:15:47,193 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:15:47,214 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
Stardog home is currently locked, cannot start kernel.  Multiple instances of Stardog cannot use the same home directory, 
shutdown the current Stardog server before starting another.  If no other server is running and the Stardog home directory 
appears to be locked, you may remove the file '/data/stardog_data/system.lock' from your Stardog home directory to remove the lock.
INFO  2017-02-08 11:16:04,653 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:16:04,668 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
INFO  2017-02-08 11:16:05,592 [main] com.complexible.stardog.StardogKernel:start(1871): Initializing Stardog

************************************************************
This copy of Stardog is licensed to Geophy Enterprise #1 (http://geophy.com), Geophy
This is a Enterprise license
This license will expire in 300 days on Tue Dec 05 13:38:59 CET 2017
************************************************************

                                                             :;   
                                      ;;                   `;`:   
  `'+',    ::                        `++                    `;:`  
 +###++,  ,#+                        `++                    .     
 ##+.,',  '#+                         ++                     +    
,##      ####++  ####+:   ##,++` .###+++   .####+    ####++++#    
`##+     ####+'  ##+#++   ###++``###'+++  `###'+++  ###`,++,:     
 ####+    ##+        ++.  ##:   ###  `++  ###  `++` ##`  ++:      
  ###++,  ##+        ++,  ##`   ##;  `++  ##:   ++; ##,  ++:      
    ;+++  ##+    ####++,  ##`   ##:  `++  ##:   ++' ;##'#++       
     ;++  ##+   ###  ++,  ##`   ##'  `++  ##;   ++:  ####+        
,.   +++  ##+   ##:  ++,  ##`   ###  `++  ###  .++  '#;           
,####++'  +##++ ###+#+++` ##`   :####+++  `####++'  ;####++`      
`####+;    ##++  ###+,++` ##`    ;###:++   `###+;   `###++++      
                                                    ##   `++      
                                                   .##   ;++      
                                                    #####++`      
                                                     `;;;.        
************************************************************


Stardog server 4.2.3 started on Wed Feb 08 11:16:06 CET 2017.

Stardog server is listening on all network interfaces.
HTTP server available at http://localhost:5820.

STARDOG_HOME=/data/stardog_data 

INFO  2017-02-08 11:16:06,008 [main] com.complexible.stardog.cli.impl.ServerStart:call(249): Memory options
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(250): Min Heap Size: 2.0G
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(251): Max Heap Size: 25G
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(252): Max Off-Heap : 12G
INFO  2017-02-08 11:16:06,009 [main] com.complexible.stardog.cli.impl.ServerStart:call(253): System Memory: 35G
INFO  2017-02-08 11:17:23,133 [Stardog.Executor-24] com.complexible.stardog.StardogKernel:shutdown(2119): Stardog server 4.2.3 shutdown on 2017-02-08T11:17:23.132+0100.


INFO  2017-02-08 11:17:56,219 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(162): Initializing R2RML registry
INFO  2017-02-08 11:17:56,237 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(171): Loaded R2RML registry with 0 sources
INFO  2017-02-08 11:17:56,857 [main] com.complexible.stardog.StardogKernel:start(1871): Initializing Stardog

Hi Samur,

Am I correct in assuming that the output of a stardog-admin db list command says there are no databases in the system? Or is it specifically the coredb database that you are being told doesn’t exist?

HI Stephen, no database is displayed or listed.

All the databases get ignored at start-up. Do you suggest any fix to this?

Samur,

There is a method that you can try to get the databases showing up again:

  1. Create a new STARDOG_HOME directory
  2. Start up the Stardog server pointing at that new home
  3. Create an empty database (named the same) for each database you need
  4. Stop the Stardog server
  5. Move the contents of the database folders for each database from the old STARDOG_HOME to the new one
  6. Restart the Stardog server and it should recognize and read the databases
1 Like

Hi Stephen, Thank you. This is quite a trick.

If I have a production application and this problem happens again. Is this the only way to do it?

Could you give me some background to understand why this happened?

Best,

Well with a production application it’s probably a better idea to use the db backup/import functionality [1].
The only way I have seen for this to happen is for something to happen to the system db/folder. Usually I have deleted it for whatever reason, which the server doesn’t take too kindly to. Not sure if in your case it was deleted (maybe somebody tried to delete system.lock and deleted system instead?), or if the system db was just corrupted in some other way (e.g., server killed during a write).

[1] https://docs.stardog.com/man/db-backup.html

Thank you Stephen. It may be that the system was deleted accidentally. Thank you for the tip.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.