Welcome, Guest
Username: Password: Remember me
Forum on HARMONIE Climate Branch

TOPIC: Convert FA files to grib format

Convert FA files to grib format 1 year 3 months ago #2130

Hello,

I am trying to convert the outputs from Harmonie into grb. I use gl utility but unfortunately it only converts about 50 fields out of about 150.
I have looked into the trans_tab.h table and I've seen there aren't some variables so I suppose I have to include them and recompile before using 'gl' again. (The same as with nc_tab.h and nc format)

But also, there are some varibles included in trans_tab.h that are not converted into grib, particularly the accumulated fluxes (LEC, HC, ...). So, I guess the definitions of the parameters in the grib definition files (name.def, paramId.def, etc...) are not found or don't correspond to the values at trans_tab.h. The problem may be in the last number passed to the function trans_tab which is '4' in the case of accumulated variables.

Where does gl read these definitions from? Are they read from the local instalation of the machine or from some place of the Harmonie set-up?
I would like to include or modify the grib definitions to be able to convert all the fields into grib.

I have already made a try defining a ECCODES_DEFINITION_PATH environment variable and also a GRIB_DEFINITION_PATH including some definitions of my own but with no luck!

Thank you!

Best wishes,

Juan Carlos.

Convert FA files to grib format 1 year 3 months ago #2132

  • Bert van Ulft
  • Bert van Ulft's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 98
  • Thank you received: 22
Hi Juan Carlos,

you can list all fields in a FA file with something like:
gl -f ICMSHFULL+0180.sfx
For fields that are defined in trans_tab.h, grib parameters and a name or short description are shown. Fields that are not defined in trans_tab.h lack this output. For example:
SFX.LEC         > 001:121-  000-105@20170610_00:00+003h00m tri:004 000 Latent heat flux
SFX.LEIC        >                   20170610_00:00+003h00m         000
LEC is set in trans_tab.h, LEIC is not. If you want to add fields to trans_tab.h you indeed have to recompile gl.
As far as I know gl uses trans_tab.h to translate FA names to grib parameters, and nc_tab.h to translate to netcdf parameters and is not in anyway linked to ECCODES_DEFINITION_PATH and GRIB_DEFINITION_PATH, but I may be wrong.
I just converted LEC from a FA to a grib file using:
gl -c ICMSHFULL+0180.sfx -n LEC.nam -o LEC.grb
where test.nam is used to only read LEC:
&naminterp
  readkey%name = "SFX.LEC"
/
This worked fine, but it did this locally, as I don't have a compiled version of HCLIM at the ECMWF at the moment. Where did you try your conversion?
Note that accumulated fields are absent in the very first output file.

best wishes,

Bert
The following user(s) said Thank You: Juan Carlos Sanchez Perrino

Convert FA files to grib format 1 year 3 months ago #2135

  • Ulf Andrae
  • Ulf Andrae's Avatar
  • OFFLINE
  • Administrator
  • Posts: 283
  • Thank you received: 30
Juan Carlos,

As Bert says the FA/LFI to grib conversion is defined in inc/trans_tab.h. It means that you either have to edit for missing fields OR provide the translations by namelist using user_trans as described on the wiki

hirlam.org/trac/wiki/HarmonieSystemDocum...BFALFIfileconversion

The eccodes (or grib_api) tables are not used in gl which is still based on gribex. For gl_grib_api the tables are used to define the shortnames and the connection between the HARMONIE GRIB1 and GRIB2 namespace. The HARMONIE specific eccodes tables can be found under gl_grib_api/defninitions.

For cy43, currently being built in the HARMONIE context, the aim is to phase out gl and have only gl_grib_api based on eccodes providing the opporunity to write either GRIB1 or GRIB2 as long as appropriate tables are defined.

Ulf

Convert FA files to grib format 1 year 3 months ago #2140

Hello,

Thank you Bert and Ulf!

In trans_tab.h it is specified the FA variable name with the grib codification associated to it, for example:

trans_tab(i)=trans('LEC ',' ',' ',1,001,121,000,105,4,'Latent heat flux','')

And when the conversion to grib is done the variable is assigned with a shortName, units, etc... This information is not in trans_tab.h. It use to be in a directory called definitions/. In this case it seems to be in gl_grib_api/definitions as Ulf said.

For example, I have tested a namelist with 'user_trans' key and a codification taken of another variable (mean thermocline depth) which is in the grib tables but not in trans_tab.h for converting a FA variable and it was converted with the shortName = 'mthd'. It means that this has been read from some other place.

So, if I include a new variable in trans_tab.h, first I need to assign a set of numbers to it. Must I invent a set of numbers or must I read them in some place? I suppose that if it is a new variable I have to invent a new set but I guess there is a convention to follow. Where can I find this information?
And second, I have to add this 'new' set in the grib definition files. Are these files in the place that Ulf mentioned, gl_grib_api/definitions? If you look here there are two folders, one for grib1/ and another for grib2/ and below there are more with names eidb/ eswi/, have I to create new definition files one or modify some of them?

A particular codification would be for different patches. I have seen that for WG1, for example, the first patch seems to be defined as it appears with grib parameters when you use gl to see the content, like Bert says, but the patch two, X002WG1, has not grib parameters associatted:

X001WG1 > 001:086- 800-105@19900601_00:00+720h00m tri:000 000 Sfc liq. vol. wat. cont......
X002WG1 > 19900601_00:00+720h00m 000 ....

Thanks a lot!

Best wishes,

Juan Carlos.

Convert FA files to grib format 1 year 3 months ago #2142

  • Ulf Andrae
  • Ulf Andrae's Avatar
  • OFFLINE
  • Administrator
  • Posts: 283
  • Thank you received: 30
Juan Carlos,

For the surfex variables we are heavily violating table 001 and there are no shortname <-> indicatorOfParameter connections in our setup. If you check gl_grib_api/scr/harmonie_grib1_2_grib2.pm you will see all our connections specific for HARMONIE. But that's all for table 253. If you have a reasonably recent version of gl_grib_api you can use the flag '-igs' when listing/converting. This will imply no use of any shortname information.

There's no real documentation about all this details yet...

In harmonie-40h1.1.1 several new conversions, including X002WG1 have been included.

trans_tab( i ) = trans('X001WG1         ',' ',' ',1,001,086,t%lev,105,0,'Sfc liq. vol. wat. cont. patch','')
trans_tab( i ) = trans('X002WG1         ',' ',' ',1,001,086,t%lev,105,0,'Sfc liq. vol. wat. cont. patch','')


Ulf

Convert FA files to grib format 1 year 3 months ago #2144

Hi,

Thank you.

I have realised that when I see the content of a grib file after converting, I use 'grib_dump' command and it probably uses the definitions of the local machine to show the shortNames.

I have checked that the variables showed with gl, which have a grib parameter associated, are correctly converted into grib but I don't really know from where are taken the shortNames.

At least, it seems that I can use these parameters: centre, table, parameter, level, indicatorOfTypeOfLevel and TimeRangeIndicator to code the variables, using trans_tab.h.

One last question, if I use a namelist like this of the example:

&naminterp
user_trans%full_name ='CLSTEMPERATURE',
user_trans%tab = 253,
user_trans%par = 123,
user_trans%typ = 105,
user_trans%lev = 002,
/

Is it possible to specify the keys for specifying the centre and TimeRangeIndicator like in trans_tab.h? (I think they are the first and last of the 6 numbers used for coding the variables in trans_tab.h).
But I see in the definition of the TYPE trans in trans_tab.h that the type contains 10 variables:
full_name,
cpar,
ctyp,
flev,
tab,
par,
lev,
typ,
txt,
unit,
however, when a new variable is added there are 11 arguments in the trans(...) sentence (?)

Thank you very much!

Juan Carlos.

Convert FA files to grib format 1 year 3 months ago #2146

  • Ulf Andrae
  • Ulf Andrae's Avatar
  • OFFLINE
  • Administrator
  • Posts: 283
  • Thank you received: 30
You can add some more information to the user_trans ( i.e. type trans ):

hirlam.org/trac/browser/trunk/harmonie/u...z.f90?rev=17575#L160

Generating centre is given as outgeo%center to the namelist

hirlam.org/trac/browser/trunk/harmonie/u...ez.f90?rev=17575#L31

Ulf
The following user(s) said Thank You: Juan Carlos Sanchez Perrino

Convert FA files to grib format 1 year 3 months ago #2148

Hi,

Thank you for all the information! It has been very useful!

Best wishes,

Juan Carlos.

Convert FA files to grib format 1 year 3 months ago #2149

  • Ulf Andrae
  • Ulf Andrae's Avatar
  • OFFLINE
  • Administrator
  • Posts: 283
  • Thank you received: 30
Juan Carlos,

The documentation is now a bit less outdated:

hirlam.org/trac/wiki/HarmonieSystemDocum...n/PostPP/gl_grib_api


Ulf
The following user(s) said Thank You: Juan Carlos Sanchez Perrino

Convert FA files to grib format 1 year 3 months ago #2150

Thank you!! ;)
Time to create page: 0.095 seconds