Welcome, Guest
Username: Password: Remember me
Forum on HARMONIE Climate Branch

TOPIC: Trick to restart crashed Forecast.

Trick to restart crashed Forecast. 1 year 3 months ago #2180

Hi,
We're currently making early tests with cy43h (the pre-alpha branch) in climate mode as we'd like to run 3-5 year simulations to quantify any near surface biases which may appear due to new surface physics (diffusion soil, explicit snow...)

As the branch is still a bit unstable we sometimes get random crashes in Forecast, often during the lasts hours of a month, so I came up with a strategy to to re-launch the system close to the end of the month and save time.

What I do is "Harmonie start" with DTG let's say 720 (a few hours before the crash hour, 726). Set Forecast to "suspend". After Prep Ini surfex & First guess tasks have finished, manually change the links in the working directory to make them point to the last written file, like this:

fc_start -> /scratch/...../archive/2017/12/01/00/ICMSHHARM+0720
SURFXINI.fa -> /scratch/.../archive/2017/12/01/00/ICMSHHARM+0720.sfx
fc_start_sfx -> /scratch/.../20171231_00/SURFXINI.fa (no change)

Then resume the Forecast. So far the trick's worked, but I'm not sure, is that completely equivalent to a full Harmonie continue from the start of the month? I mean, apart from having a month spread over different folders...

Thanks in advance for any comment,
Samuel

Trick to restart crashed Forecast. 1 year 3 months ago #2185

  • Bert van Ulft
  • Bert van Ulft's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 98
  • Thank you received: 22
Hi Samuel,

I think that is indeed pretty much how we continue a run in blocks of a month. Just to be sure. you can perhaps check your method by comparing the results of a 2 day run, to a run where you restart after a day with your method. The results are probably not numerically identical, but the results should be very similar.
Do you have any clue why the model is crashing? In the end you don't want random crashes every now and then.

best wishes,

Bert

Trick to restart crashed Forecast. 1 year 3 months ago #2191

  • Ulf Andrae
  • Ulf Andrae's Avatar
  • OFFLINE
  • Administrator
  • Posts: 283
  • Thank you received: 30
Samuel,

Dumping the model state to a FA file and restart from that will not reproduce a continuous run for sure. If you have spurious crashes they might or might not happen again. In your case it seems like the crashes are robust enough to happen again...

The issue with restarts is not new and it's by construction climate runs that suffers most from this. Has anyone working in this area investigated the options for a reproducible restart further?

Ulf
Time to create page: 0.070 seconds