Riskscape slowing down at the end of a job

Community
1

Hi,
I am having an issue where riskscape will slow right down while processing the last couple of objects in a data set. The output speed will drop from ~250/s for the first 139983 objects down to 0.000/s for the last 10 or so. Riskscape will eventually finish if left to run but it only takes a few minutes to get to the point where it slows down and then an hour or more from there to finish.
I can reproduce this issue with this (and other) datasets on both Windows 11 and Linux

riskscape --pipeline-threads=8 model run Flood-Risk-Land -p fullTableOutput=Table_DVR_Land_Results_Coast_60000 -p inputLimit=10000 -p inputOffset=60000 -p annualisedOutput=DVR_Annualised_Land_Exposure_Impacts_Coast_60000 -p exposure=NZ_Properties -p hazardLayer=hazard_coastal -p lettable_unit_code=All --output=output/Land/DVRCoastCountry/60000 --replace
14:22:33.886 [main] WARN  nz.org.riskscape.engine.OsUtils - Changed console to use UTF-8 encoding (was chcp 850)
[WARNING] The 'beta' plugin is enabled. This contains experimental features that may significantly change or be deprecated in future releases.
14:22:39.711 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it
14:22:39.724 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it
14:22:39.724 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it

Progress:
  139983 total,     0.000/s avg: steps-sample_hazard_layer~>analysis-sink.in

CPU usage:

image
image649×641 30.9 KB

2

I tried with --pipeline-threads=1 incase it was some kind of deadlock but that also does not work

riskscape --pipeline-threads=1 model run Flood-Risk-Land -p fullTableOutput=Table_DVR_Land_Results_Coast_60000 -p inputLimit=10000 -p inputOffset=60000 -p annualisedOutput=DVR_Annualised_Land_Exposure_Impacts_Coast_60000 -p exposure=NZ_Properties -p hazardLayer=hazard_coastal -p lettable_unit_code=All --output=output/Land/DVRCoastCountry/60000 --replace
15:21:50.798 [main] WARN  nz.org.riskscape.engine.OsUtils - Changed console to use UTF-8 encoding (was chcp 850)
[WARNING] The 'beta' plugin is enabled. This contains experimental features that may significantly change or be deprecated in future releases.
15:21:55.738 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it
15:21:55.738 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it
15:21:55.738 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it

Progress:
    9016 total,     0.000/s avg: exposures.in
    9016 total,     0.000/s avg: exposures.out
    8159 total,     0.000/s avg: exposures_join_hazards.in
  114226 total,     0.000/s avg: exposures_join_hazards.out
    9646 total,     0.000/s avg: group_5-sink.in
  114143 total,     0.000/s avg: steps-sample_hazard_layer~>analysis-sink.in
    9646 total,     0.000/s avg: steps-select_1~>exposuresWithProject.in
    9646 total,     0.000/s avg: steps-select_1~>exposuresWithProject.out
3

I have narrowed it down to a single feature that is taking a long time to calculate

riskscape --pipeline-threads=8 model run Flood-Risk-Land -p fullTableOutput=Table_DVR_Land_Results_Coast_68000 -p inputLimit=1000 -p inputOffset=0 -p annualisedOutput=DVR_Annualised_Land_Exposure_Impacts_Coast_68000 -p exposure=68154 -p hazardLayer=hazard_coastal -p lettable_unit_code=All --output=output/Land/DVRCoastCountry/68154 --replace
11:42:20.516 [main] WARN  nz.org.riskscape.engine.OsUtils - Changed console to use UTF-8 encoding (was chcp 850)
[WARNING] The 'beta' plugin is enabled. This contains experimental features that may significantly change or be deprecated in future releases.
11:42:23.820 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it
11:42:23.820 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it
11:42:23.820 [Finalizer] ERROR org.geotools.jdbc - There's code using JDBC based datastore and not disposing them. This may lead to temporary loss of database connections. Please make sure all data access code calls DataStore.dispose() before freeing all references to it

Progress:
      13 total,     0.000/s avg: steps-sample_hazard_layer~>analysis-sink.in
4

In the end I filtered out any objects large than 50000000m2

input(relation: $exposure, name: 'exposure', offset: $inputOffset, limit: $inputLimit)
->select({*, measure(exposure) as area})
-> filter(area < 50000000) as exposures_input