April 15:
1) failure of delivery for 00 UTC run. Last check at 1400 UTC.
2) failure of ussig ECSST and ice reported in HIRLAM forum
www.hirlam.org/index.php?option=com_kune...c=view&catid=8&id=70

Well the 00 UTC data finally appeared in the glameps.org server between 13:50 and 14:05 UTC, several hours late. Kai briefly checked at SMS logs and found no abnormal messages. So it is still something to check as to why the results arrives about 4 hours late?

Something to follow up next week.

Data are missing from about 23.4

Unfortunately it looks unlikely that we can improve the situation this week. Kai, who so far is the only one that manage the experimental suite, is currently on holiday and back next week (May 2).

It appears that the interruption was caused by delivery failure in ECMWF EPS products which in turn are used in construction of GLAMEPS initial and boundary perturbations.

Alex Decmyn tried yesterday to relauch aborted GLAMEPS run but failed to skip the first 3 days (23-25 April) as he originally intended. Thus the suite is now slowly recover. This morning the web products for April 23, 24 and 25 are online. Hopefully the real-time forecasts will be back by sometimes tomorrow.

Apparently we experienced two problems here. First there is a deficiency in the automatic launch script to resubmit ensemble generation suites after some time interval in case of EPS delivery delay/problems, so that manual intervention became necessary; Second the GLAMEPS production team needs basic instruction on how to properly relaunch a failed run in such relatively sinple situation, including the wish to skip several days. I trust both points are being addressed by Kai's work so that same shall not happen again when GLAMEPS_v1 is in place. WIth GLAMEPS_v1 we also need to construct the automatic backup scripts to replace missing HIRLAM/ALADIN ensemble members with the EPS members.

Just to report that the real time GLAMEPS forecasts have been back since yesterday afternoon, April 27.

Initially the recovery was a bit slow due to a very slow verification procedure launched once per finished day cycle. As this has been uncritical according to Carlos, Alex managed to turn this off, which speed up the catch-up runs.

Monday morning observations.

hirlam.org/portal/GLAMEPS/Products/Gl.PR...ex.html?choice_ind=0
- None of the +45,+48h plots are available.


hirlam.org/portal/GLAMEPS/Products/Gl.PR...ex.html?choice_ind=0
- Dependency on rank and hour could br sorted out by this change:
hirlam.org/trac/changeset/9283/trunk/har...util/monitor/WebgraF

GLAMEPS production is currently only to 42h. The menu is now adapted. Thanks for the hint.

GLAMEPS product has been missing for the last 3 cycles. The last available products were for 12 UTC on July 30. Courses unknown yet.

For completeness I confirm that the GLAMEPS has been back on track already a few hours after my initial report.

The first crash happened with cycle 12 UTC JUly 30 for HIRLAM- KFRK members, in which forecasts for all members got aborted after reading of input analysis. A simple restart for the same cycle yesterday made the run going though again. Presumably, the crash of the original ensemble run could be due to a temporal file access problem (with boundaries??).

The system used for the current real-time GLAMEPS production is GLAMEPS_v0. With the next upgrade, GLAMEPS_v1, the system scripts shall enable an automatic restart procedure to overcome similar problem.

Alex Deckmin and Carlos Santos are acknowledged to diagnose and solve the problem.

GLAMEPS webpage has not been updated since Aug 18. The operator group This email address is being protected from spambots. You need JavaScript enabled to view it. is now notified.

GLAMEPS webpage has not been updated since Aug 18. The operator group This email address is being protected from spambots. You need JavaScript enabled to view it. is now notified again.

The GLAMEPS suite is currently out of function. I'm trying to investigate the problem

The reason for the failure in the 2011081812 cycle is not really clear to me - the 2011081900 cycle was never started, it seems.

As of today the cycle date of the suite is not correct any more.
I have replayed the suit an reset the suite cycle date.

However, it seems there are still several problems:

1. Hirlam does not work
2. Hppv postPP aborts

So it's currently hard to say when the suite will be up and running again.

The problems in the GLAMEPS suite could only partly be resolved.

Some problems in Hppv postPP cannot be resolved currently, which means that the GLAMEPS production of graphic plots has been disabled, for the time being, in order to set the daily ensemble cycling into function again.

The GLAMEPS suite is now up and running again, and currently the system is under way to catch up from cycle 2011081900 to today. This process will take until tomorrow at least.

Please note:
As mentioned above the GLAMEPS suite will not produce graphics products currently, so the GLAMEPS web pages will remain empty.
However, the ensemble member data available from /ms_crit/glameps should be in place.

GLAMEPS products on glameps.org have not been updated since Nov 3rd. The operator group This email address is being protected from spambots. You need JavaScript enabled to view it. has been notified.