From the log files I can't see anything wrong prior to the sms-abort. I think the problem is of technical nature, somehow similar to the occasional stop of mini-SMS jobs experienced in several other platforms. Others may eventually be able to find out real reasons and cure behind these, but I'd like to exclude here possibilities related to orography, instability or other meteorological situations etc. as was suggested.

Since your current experiments with ANALYSIS=none are intermittent and short forecasts with little dependence between cycles, I assume it is not a problem for you to simply re-launch your cycles by 'Hirlam prod DTGEND=$final_cycle LL=6" whenever you see interruptions of similar type?

No problem!

Here are attached the log files that you asked!

Felix,
Could you please post your hirlam.log and mSMS.log from these aborted runs.
Xiaohua

Sorry but you can download the files and the logfiles from this site:

Files

Hi!

It's me again. I have some questions and I think that is the better place to discuss some problems that I had while I was running the model for long periods (Sept-Oct-Nov 2008 and Dec-Jan-Feb-Mar 2006).

The thing is that while I was running these periods suddenly stops (now I'm running the long period for 2006) at the beginning it stops in Postpp script after 2 forecast cycles (in this case with 20 nodes in Prog and Postpp scripts). Martynas recommend me that use less nodes for Postpp (now I'm using 6) and I did, but then it stops in the Prog script and I use less nodes in this (12 nodes). Even so it made more than 2 forecast cycles, sometimes this script Prog, stops and when I checked the log for these "stops" it doesn't point any specif thing like there are not the boundary conditions or the climate files (cause I have it and I check that). I talked this with Martynas, but we couldn't find any clue for this stops (there aren't any explosion in variables) So we have some ideas like the model doesn't fit in the system, or there are problem with my cluster system, or could it be the complex topography?

I attached the log files for the 20 nodes and for 12 stop for Prog script and my scripts folder on hl_home/exp folder just in case you want to check it. Any ideas or similar problems coul help to work better the model in our facilities!

Best regards!

Just saw the last two postings on this thread. I agree with Laura on the preference of using fc...+003 than fc...+000 for non-synoptic hours. In my view, if the purpose is to compare model to observations every three, or even every 1 hour, it is not a good idea, given the current situation, to perform 3 hourly or one hourly analysis.

Currently for the historical episode we are unable to find higher frequency lateral boundary data from ECMWF than 6-hourly analyses. Accordingly, you have received 4 observation/day data package centered around synoptic hour (00/06/12/18 UTC). Changing cycling interval will require quite some additional experiment runs with need for numerous modifications of scripts in order to fetch available observation and lateral boundary data.

On top of these, you are not likely to get better model data if that's what you are interested. I mean,e.g.
fc2006091503+000
will unlikely be of higher quality than
fc2006091500+003

In the above example, obtaining first model data requires quite some extra effort, but it may not be of better quality than the second one, which you already have now. This is due to typical shortcomings in assimilation technique and in moisture spin-up. Furthermore, if the quantity you want to compare is precipitation or other things that are closely related to moisture process, I'd actually recommend use of forecasts with longer forecast lead time than 6 h.