Welcome, Guest
Username: Password: Remember me
Forum header

TOPIC: Communication timeout in 4DVminim_loop2.1

Communication timeout in 4DVminim_loop2.1 8 years 8 months ago #182

Hi, all

Has anyone seen this kind of error message occuring in 4DVmini, second loop:
Communication subsystem error: Communication timeout

This is the second trial, in the first time I got segmentation fault and no other explanation. I use the Hirlam trunk from the date 20110428, without any modifications in the data assimilation. Observations are taken from MARS as by default.

I attach the whole log-file from this run.

I will try once again.

Kalle



RTM_REPC Accepted for PP: 0 ( 0.0%)
RTMIOSYS Reading RTMSYS-file:noaa15amsu-a_rtmsys.dat
RTMGETSYS No OBS error available in file for Sat:noaa15 Instr:amsu-a Surface:land
RTMGETSYS No OBS error available in file for Sat:noaa15 Instr:amsu-a Surface:other
RTMIOSYS Reading RTMSYS-file:noaa16amsu-a_rtmsys.dat
RTMGETSYS No OBS error available in file for Sat:noaa16 Instr:amsu-a Surface:land
RTMGETSYS No OBS error available in file for Sat:noaa16 Instr:amsu-a Surface:other
RTMIOSYS Reading RTMSYS-file:noaa16amsu-b_rtmsys.dat
RTMGETSYS No OBS error available in file for Sat:noaa16 Instr:amsu-b Surface:land
RTMGETSYS No OBS error available in file for Sat:noaa16 Instr:amsu-b Surface:sea ice
RTMGETSYS No OBS error available in file for Sat:noaa16 Instr:amsu-b Surface:other
RTMIOSYS Reading RTMSYS-file:noaa18amsu-a_rtmsys.dat
RTMGETSYS No OBS error available in file for Sat:noaa18 Instr:amsu-a Surface:land
RTMGETSYS No OBS error available in file for Sat:noaa18 Instr:amsu-a Surface:other
RTMIOSYS Reading RTMSYS-file:noaa18mhs_rtmsys.dat
RTMGETSYS No OBS error available in file for Sat:noaa18 Instr:mhs Surface:land
RTMGETSYS No OBS error available in file for Sat:noaa18 Instr:mhs Surface:sea ice
RTMGETSYS No OBS error available in file for Sat:noaa18 Instr:mhs Surface:other
ERROR: 0032-171 Communication subsystem error: Communication timeout has occurred. in MPI_Alltoallv, task 50
ERROR: 0031-250 task 50: Terminated
ERROR: 0031-250 task 42: Terminated
ERROR: 0031-250 task 61: Terminated
ERROR: 0031-250 task 18: Terminated
ERROR: 0031-250 task 11: Terminated
ERROR: 0031-250 task 12: Terminated
ERROR: 0031-250 task 14: Terminated

Re:Communication timeout in 4DVminim_loop2.1 8 years 8 months ago #183

I will attach the log file

Kalle

Re:Communication timeout in 4DVminim_loop2.1 8 years 8 months ago #184

Still trying to attach the log-file

Kalle

File Attachment:

File Name: 4DVminim_loop2.txt
File Size: 351892

Re:Communication timeout in 4DVminim_loop2.1 8 years 7 months ago #185

  • xiaohua yang
  • xiaohua yang's Avatar
It looks that the minimisation stops at the ATOVS data handling. What is untypical is that this occurs in the second loop. Maybe someone else (Frank?) can comment about this.

Re:Communication timeout in 4DVminim_loop2.1 8 years 7 months ago #233

  • Laura Rontu
  • Laura Rontu's Avatar
  • OFFLINE
  • Administrator
  • Finnish Meteorological Institute
  • Posts: 150
  • Thank you received: 8
Not sure this is the same issue, but to get 4DVAR minimisation working in a small domain, I need to use modifications in Env_expdesc. I have got them ready-made from somewhere (Sweden via Xiaohua?).


> SCATT=no # use ASCAT, F. Tvette option
151,152c148
< # ILRES=9,6 # resolution decrease for consecutive inner loops
< ILRES=3,3 # resolution decrease for consecutive inner loops
---
> ILRES=9,6 # resolution decrease for consecutive inner loops
160,161c156
< # SPNDTIME=1800,1200 # time step in spectral TLM and ADM
< SPNDTIME=1200,1200 # time step in spectral TLM and ADM
---
> SPNDTIME=1800,1200 # time step in spectral TLM and ADM
163,164c158
< # NQLIMIT=40,30 # iteration limits for consecutive innerloops
< NQLIMIT=30,30 # iteration limits for consecutive innerloops
---
> NQLIMIT=40,30 # iteration limits for consecutive innerloops
175,176c169
< VARQCSTART=10 # onset time for VarQC (iteration step)
< # VARQCSTART=12 # onset time for VarQC (iteration step)
---
> VARQCSTART=12 # onset time for VarQC (iteration step)

Re:Communication timeout in 4DVminim_loop2.1 8 years 7 months ago #235

I had also this message while fighting with ATOVS. If you have amsua on, switch them off. This solved my problem a few weeks ago. For me I could see from the output that it occurred during processing of satellite information and also in the second loop.

Kalle
Time to create page: 0.086 seconds