Maximum file length exceeded: 0B

Support
1

I get this error:
Failed to run query: com.complexible.stardog.plan.eval.operator.OperatorException: Uncaught error during query evaluation: IOException: Maximum file length exceeded: 0B

For this query:
PREFIX ProductRecommendation: tag:stardog:designer:ProductRecommendation:model:
PREFIX rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns#
SELECT DISTINCT ?product ?name ?imageURL ?price ?productAverageScore ?bassScore ?microphoneScore ?comfortScore ?soundQualityScore ?ANC_Score ?batteryLifeScore ?connectivityScore ?designScore ?durabilityScore ?featureScore ?highsScore ?isolationScore ?midrangeScore ?portabilityScore ?valueScore
WHERE {
VALUES ?productType { ProductRecommendation:Headphone }
VALUES ?connectivity { ProductRecommendation:WirelessConnectivity }
?product rdf:type ?productType.
?product rdf:type ?connectivity.
?product ProductRecommendation:productName ?name;
ProductRecommendation:productPriceMSRP ?price.
OPTIONAL {?product ProductRecommendation:has_ProductImage ?image.
?image ProductRecommendation:productImageURL ?imageURL.}
OPTIONAL {?product ProductRecommendation:has_productRating ?rating.
OPTIONAL {?rating ProductRecommendation:productAverageScore ?productAverageScore.}
OPTIONAL {?rating ProductRecommendation:bassScore ?bassScore.}
OPTIONAL {?rating ProductRecommendation:microphoneScore ?microphoneScore.}
OPTIONAL {?rating ProductRecommendation:comfortScore ?comfortScore.}
OPTIONAL {?rating ProductRecommendation:soundQualityScore ?soundQualityScore.}
OPTIONAL {?rating ProductRecommendation:ANC_Score ?ANC_Score.}
OPTIONAL {?rating ProductRecommendation:batteryLifeScore ?batteryLifeScore.}
OPTIONAL {?rating ProductRecommendation:connectivityScore ?connectivityScore.}
OPTIONAL {?rating ProductRecommendation:designScore ?designScore.}
OPTIONAL {?rating ProductRecommendation:durabilityScore ?durabilityScore.}
OPTIONAL {?rating ProductRecommendation:featureScore ?featureScore.}
OPTIONAL {?rating ProductRecommendation:highsScore ?highsScore.}
OPTIONAL {?rating ProductRecommendation:isolationScore ?isolationScore.}
OPTIONAL {?rating ProductRecommendation:midrangeScore ?midrangeScore.}
OPTIONAL {?rating ProductRecommendation:portabilityScore ?portabilityScore.}
OPTIONAL {?rating ProductRecommendation:valueScore ?valueScore.}
}
FILTER (BOUND(?imageURL) && BOUND(?productAverageScore))
}
ORDER BY DESC(?productAverageScore)
LIMIT 15

By removing one OPTIONAL column the results appear.
Before I didn't have this issue and added much more conditions.

How can I fix this?

2

The error indicates that Stardog has run out of memory while executing the query, tried to use disk to spill some intermediate query results, but spilling is disabled by setting spilling.max.file.length to 0. Thus the query terminates.

Is this Stardog Cloud or your local instance? If the latter, please indicate Stardog version.

In either case, there could be multiple reasons why the query has run out of memory: the data got larger, the query uses a less optimal query plan, etc. You can use the query profiler to see where the query consumes memory, or share it here. It works even when the query gets killed due to this error.

Best,
Pavel

3

Thanks Pavel for your quick reply!
It seems that this action takes a lot of memory:

+─ HashJoinOuter(?product) [#116], memory: {total=229M (99.5%); max=229M}, spilled to disk: {total=2.0K (100.0%); spills=1}, results: 0, wall time: 1042 ms (54.5%)

I attached the profiler file in this message.

I don't have a lot of products but I have several nodes with different attributes attached to them.
What would you suggest?
NONE-Untitled-2_sparql_profile.txt (17.1 KB)

Best,
Guillaume

4

Hi Guillaume,

OK, so I think you're running into the well-known problem with multi-valued properties in this part of the query (which is exactly the right handside of that hash join):

?product ProductRecommendation:has_productRating ?rating.
OPTIONAL {?rating ProductRecommendation:productAverageScore ?productAverageScore.}
OPTIONAL {?rating ProductRecommendation:bassScore ?bassScore.}
OPTIONAL {?rating ProductRecommendation:microphoneScore ?microphoneScore.}
OPTIONAL {?rating ProductRecommendation:comfortScore ?comfortScore.}
OPTIONAL {?rating ProductRecommendation:soundQualityScore ?soundQualityScore.}
OPTIONAL {?rating ProductRecommendation:ANC_Score ?ANC_Score.}
OPTIONAL {?rating ProductRecommendation:batteryLifeScore ?batteryLifeScore.}
OPTIONAL {?rating ProductRecommendation:connectivityScore ?connectivityScore.}
OPTIONAL {?rating ProductRecommendation:designScore ?designScore.}
OPTIONAL {?rating ProductRecommendation:durabilityScore ?durabilityScore.}
OPTIONAL {?rating ProductRecommendation:featureScore ?featureScore.}
OPTIONAL {?rating ProductRecommendation:highsScore ?highsScore.}
OPTIONAL {?rating ProductRecommendation:isolationScore ?isolationScore.}
OPTIONAL {?rating ProductRecommendation:midrangeScore ?midrangeScore.}
OPTIONAL {?rating ProductRecommendation:portabilityScore ?portabilityScore.}
OPTIONAL {?rating ProductRecommendation:valueScore ?valueScore.}

Indeed, there're only few hundred products but some (at least one) properties of "rating" nodes have multiple values. That leads to a combinatorial growth of score combinations, e.g. if you have :r :score1 ?s1 . :r :score2 ?s2 and the :r node has 2 :score1 values and 3 :score2 values, the pattern will generate 2*3 = 6 score combinations.

You can see that growth in the profiler output:

                              │  │  │  │  `─ MergeJoinOuter(?rating) [#121K], results: 1.6M, wall time: 430 ms (22.5%)
                              │  │  │  │     +─ MergeJoinOuter(?rating) [#84K], results: 784K, wall time: 210 ms (11.0%)
                              │  │  │  │     │  +─ MergeJoinOuter(?rating) [#58K], results: 397K, wall time: 104 ms (5.4%)
                              │  │  │  │     │  │  +─ MergeJoinOuter(?rating) [#43K], results: 209K, wall time: 54 ms (2.8%)
                              │  │  │  │     │  │  │  +─ MergeJoinOuter(?rating) [#32K], results: 107K, wall time: 27 ms (1.4%)
                              │  │  │  │     │  │  │  │  +─ MergeJoinOuter(?rating) [#22K], results: 54K, wall time: 13 ms (0.7%)
                              │  │  │  │     │  │  │  │  │  +─ MergeJoinOuter(?rating) [#15K], results: 27K, wall time: 6 ms (0.3%)
                              │  │  │  │     │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#11K], results: 14K, wall time: 4 ms (0.2%)
                              │  │  │  │     │  │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#8.7K], results: 8.7K, wall time: 2 ms (0.1%)
                              │  │  │  │     │  │  │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#7.8K], results: 6.6K, wall time: 2 ms (0.1%)
                              │  │  │  │     │  │  │  │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#5.8K], results: 3.4K, wall time: 1 ms (0.1%)
                              │  │  │  │     │  │  │  │  │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#4.2K], results: 1.9K, wall time: 1 ms (0.1%)
                              │  │  │  │     │  │  │  │  │  │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#3.5K], results: 1.5K, wall time: 0 ms (0.0%)
                              │  │  │  │     │  │  │  │  │  │  │  │  │  │  │  │  +─ MergeJoinOuter(?rating) [#3.4K], results: 826, wall time: 1 ms (0.1%)

it begins with 826 scores and the incrementally grows to 1.6M score combinations, and that's where the hash join finally runs out of memory, tries to spill to disk, and fails because that is disabled for the server.

We might be able to remedy this with query hints but maybe this is not the results that you want in the first place? Quite often people don't want value combinations, just all values, and can get that using UNIONs instead of joins.

Hope it makes sense,
Pavel