Welcome, Guest
Username: Password: Remember me
Forum on HIRLAM/HARMONIE Observation Handling

TOPIC: MARS + additional observations, best way to do it?

MARS + additional observations, best way to do it? 3 years 6 months ago #1545

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Hi,
Just a quick question... I'm wondering what would be the best way to use additional conventional observations (on top of the observations fetched from MARS) when running Harmonie at ecgate? Also the additional observations are in BUFR format so maybe I can just cat them in Oulan to the big file I get from MARS? Or is there some more clever option?
Thanks for help :)
Tuuli

MARS + additional observations, best way to do it? 3 years 6 months ago #1546

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 195
  • Thank you received: 33
Hi Tuuli,

Adding a line in the Oulan script to cat your "extra" file to the ob retrieved by MARS is probably the easiest way to use your additional conventional observations.

Something like the following might work:
    #--- MARS queue
    if [ -f retrin ]; then
      mars retrin || { echo "MARS failed" ; exit 1 ; }
    else
      echo "No MARS request exists. Please select some observation types!"
    fi
    cat /ccapath/to/extrafile >> ${BUFRFILE}

MARS + additional observations, best way to do it? 3 years 3 months ago #1650

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Hi Eoin,

I finally come back to this now that my runs are technically working (thanks for that!)... So I now manage to cat my bufr file into the MARS bufr file in Oulan but when I look at the odb files from finished run I clearly only have the MARS observations read in and not my additional observations. This is a very abstract question since I don't really know at all what might be the problem... But I ask you anyway in case you happen to have some ideas.

My additional observations are soundings from a ship measuring campaign, they should be in the temp format. I'm wondering if the changing location or station code of a ship or something might blacklist the observations? I've been trying to look at the blacklists but didn't find the station code of my observations there... Do you know some other places where I could look at? Maybe MARS bufr behaves differently? Could I whitelist the observations somehow?

This is my cat command in Oulan, it looks ok, right?:
      cat /perm/ms/fi/fm2/oden_temps/$DATE/${DATE}_${oden_hour}/bufr309052.bfr >> ${BUFRFILE}

By the way, do you happen to know some easy way to see the observation locations inside the huge bufr file (e.g. "ob2014070618") without opening it with a bufr reader..? That's what I would also like to check...

Oh and in the MakeCycleInput log I have this kind of info:
Extraction des TEMPSHIP

 #---------------------------------------------#
 DOMAIN CHECK FOR TEMP
 MINLAT:    62.000000000000000      MAXLAT:    86.000000000000000
 MINLON:   -10.000000000000000      MAXLON:    104.00000000000000
 #---------------------------------------------#
 READING TEMP BUFR FILE FOR DATE:
 YEAR: 2014 - MONTH: 07 - DAY: 06 - HOUR: 12
 OPEN:temps

 --------------------------------------
 BUFR FILE: temps

 NUMBER OF TOTAL MESSAGES :            1
 NUMBER OF TOTAL TEMPSHIP :            0
 NUMBER OF VALID TEMPSHIP :            0
 --------------------------------------


 ++++++++++++++++++++++++++++++++++++++
    SKIPPED TEMP OBSERVATIONS REPORT


 ERROR IN BUS012   =            0
 NOT TEMP OBS      =            1
 ERROR IN BUFREX   =            0
 ERROR IN BUSEL    =            0
 DATE/TIME CHECK   =            0
 HEIGHT CHECK      =            0
 DOMAIN CHECK      =            0
 NO VALID LEVELS   =            0
 ---------------------------------
 TOTAL SKIPPED     =            1

 ++++++++++++++++++++++++++++++++++++++

Obviously I cannot be sure that this is exactly my added observation but I think so, maybe there is something wrong in the bufr file I add if it's first read in as a TEMPSHIP and then flagged as "NOT TEMP OBS"?

Thanks for help and any ideas you might have concerning this! And sorry for the super-long message!

Tuuli

MARS + additional observations, best way to do it? 3 years 3 months ago #1651

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 195
  • Thank you received: 33
Hi Tuuli,

At the moment Oulan is quite rigid in what flavours of BUFR it accepts. Your extra TEMPSHIP observation is in the "new" WMO format that includes displacement information. Oulan can read either the "new" BUFR or the "old" style BUFR, but not both.

You have a few options:
1. Throw away the extra displacement information and re-cast the BUFR in the old format

2. Use only new format BUFR by setting LNEWTEMPBUFR=1 in scr/include.ass

3. Be brave and set USEOBSOUL=0 in scr/include.ass to use BATOR to convert all your BUFR to ODB

4. Be stubborn and carry out some re-coding in oulan so that it can read both "old" and "new" BUFR.

Eoin
The following user(s) said Thank You: Tuuli Perttula

MARS + additional observations, best way to do it? 3 years 3 months ago #1652

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 195
  • Thank you received: 33
Regarding an easy way to view your BUFR:

I use metview. On ecgate you should use the following command:
metview -e BUFR /hpc/perm/ms/fi/fm2/oden_temps/$DATE/${DATE}_${oden_hour}/bufr309052.bfr
The following user(s) said Thank You: Tuuli Perttula

MARS + additional observations, best way to do it? 3 years 3 months ago #1653

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Thanks!

Good to know, now I'll only have to choose which path to follow... Probably I'll start with the easiest.

Also the metview line will surely be useful! I have only used that as a graphical tool in Meteo France and porting that to my computer was nearly impossible so I had already forgotten the whole thing.

Tuuli

MARS + additional observations, best way to do it? 3 years 3 months ago #1654

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Just one more question, so are the MARS temp bufrs in the old format i.e. if I use the LNEWTEMPBUFR=1 I'll only assimilate my added ship temps?

Tuuli

MARS + additional observations, best way to do it? 3 years 3 months ago #1655

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 195
  • Thank you received: 33
Hi Tuuli,

The Oulan script takes this into account but I am not sure what data were available in 2014.

Eoin

MARS + additional observations, best way to do it? 3 years 3 months ago #1656

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Hi again Eoin,

I have tried the options 2 and 3 and it doesn't seem to work either way... I was just wondering, I have the same observations also in temp messages, would I be able to use them as well? Maybe it would be easier than with bufr? Is it possible to just cat them in the OBSOUL file or should I pass them though some script first? I have a, b, c and d temp messages, do you have some idea which to use or all of them?

Thanks again for your help!

Tuuli

MARS + additional observations, best way to do it? 3 years 3 months ago #1657

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Hi once again,

It seems to work after all, I was looking at the wrong data and thought that the observations weren't used!

Tuuli

MARS + additional observations, best way to do it? 3 years 3 months ago #1658

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 195
  • Thank you received: 33
Hi Tuuli,

I am glad to hear that my Oulan claims are true. I was afraid I was going to have to review the code again.

If there are other radiosonde stations in your domain you should check the contents of your MARS BUFR before progressing too far with your experiments.

Eoin

MARS + additional observations, best way to do it? 3 years 3 months ago #1659

  • Tuuli Perttula
  • Tuuli Perttula's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 29
  • Thank you received: 1
Hi Eoin,

One more question (again...), if I would like to switch off assimilation of all the other ship observations and just use my added observations what would be the best option? I don't know the names of all the other ships so maybe blacklisting isn't an option but can I whitelist my added observations and blacklist all the others? I do want to use all the observations on land though.

Thanks for your help, again :)

Tuuli

MARS + additional observations, best way to do it? 3 years 3 months ago #1660

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 195
  • Thank you received: 33
Hi Tuuli,

perhaps the easiest way to switch all the other SHIP TEMP observations would be to add all known SHIP TEMP identifiers to nam/LISTE_NOIRE_DIAP as described here: https://hirlam.org/trac/wiki/HarmonieSystemDocumentation/UseofObservation

However, the easiest way is rarely the best way to achieve the desired result. The best way to "whitelist" your SHIP TEMP would be to edit src/blacklist/mf_blacklist.b with something like:
if (CODTYP = tempship) then
  if (STATID) notin ("YOURSHIP") then fail (CONSTANT); endif;
endif;
  1. Please test this carefully (as I have not!)
  2. Check your ODB to ensure your blacklisting is working as desired before proceeding with your experiment.

Eoin
Time to create page: 0.079 seconds