Automatic Data Conversion and Scouring

Whereas many WXP programs can access the raw ingested data directly, some of WXP's programs use converted files as input. These converted files can be generated by running programs such as sacvt, smcvt, uacvt and radcvt.  In some cases, these programs can be run manually prior to running the surface, upper air and radar programs but it is recommended to automate this task so the end user does not have to run these decoders.  Automating the task guarantees a current decoded file will be available but in some cases, it will not contain all the data since some data will come in between conversion times.  It is recommended to convert often.

A second task to automate is the generation of header files for non-WXP ingested data.   This is recommended for sites using the LDM ingestor.  This involves running the hdrparse and griblook programs.  For more information on header files, see the parsing data and header files section of the Users Guide.

The last task is to scour old data to prevent data and converted files from exhausting available disk space.  For more information on a script for scouring data, see the scouring old data section of the Users Guide.

Surface Data Conversion

METAR data is only valid over land surfaces but report once an hour.  METARs are also rather sparse outside of the US. Synoptic data gives better global coverage including ship and buoy reports to cover ocean areas but they only are reported reliably once every 6 hours.  To produce a surface plot with the best data coverage, it is necessary to use more than the standard METAR data.  Many ships and buoys report data every hour and thus could be added to the output converted file. Here is a script that does the additional conversion:

#! /bin/csh -f
#  sfcconvert: converts surface data and adds hourly ship and cman reports for
#     coverage over ocean areas.
setenv wxpdefault /home/wxp/etc
set path=( /home/wxp/bin $path )

/home/wxp/bin/sacvt -cu -pf=over -me=print -pa=update
/home/wxp/bin/smcvt -cu -dh=cu-15+15 -of=sfc_cvt -pa=update -pf=app \
   -me=print -ou=ship,cman,buoy,dribu
/home/wxp/bin/smcvt -cu -dh=cu-15+15 -if=cman_dat -pa=update -of=sfc_cvt \
   -pf=app -me=print 

Synoptic Data Conversion

Synoptic data normally reports once every 6 hours.  It provides the best global coverage but since it is somewhat different from METAR surface data, it is usually placed in a separate converted file.  Some sites such as Australia don't report on standard hours such as 0, 6, 12 and 18Z.  As a result, synoptic data is converted over a range of 3 hours surrounding the standard hours to give the best global coverage possible.   Here is a sample synoptic conversion script:

#! /bin/csh -f
#  synconvert: converts synoptic data over a 3 hour period to catch reports on
#     off hours.
setenv wxpdefault /home/wxp/etc
set path=( /home/wxp/bin $path )

/home/wxp/bin/smcvt -ho=-106 -cu -nh=-2 -dh=-6 -if=syn_dat -pf=over -me=none

Upper Air Data Conversion

Upper air data conversion is simpler than the surface and synoptic conversion.   For LDM sites, the conversion has to span two hours to get late reports put in the 01 and 13Z files.  Here is an upper air conversion script:

#! /bin/csh -f
#  uaconvert: converts upper air data over a 2 hour period to catch late reports
setenv wxpdefault /home/wxp/etc
set path=( /home/wxp/bin $path )

/home/wxp/bin/uacvt -pf=over -ba -me=out3 -ho=-12 -nh=1

MDR Radar Data Conversion

The MDR radar composite is created using the radcvt program.   This should be run once or twice an hour.

#! /bin/csh -f
#  radconvert: converts MDR radar data
setenv wxpdefault /home/wxp/etc
set path=( /home/wxp/bin $path )

/home/wxp/bin/radcvt -pf=over -ba

RCM Radar Data Conversion (NOAAPORT Only)

The RCM radar composite is created using the radcvt program.   This should be run at least twice an hour.

#! /bin/csh -f
#  rcmconvert: converts RCM radar data
setenv wxpdefault /home/wxp/etc
set path=( /home/wxp/bin $path )

/home/wxp/bin/radcvt -inp=rcm -pf=over -ba

UNIX Cron Setup

In order to automate this process, it is recommended that these conversion processes be scheduled to occur at regular intervals. For Unix systems, this is done through the cron program. In some cases, these scripts will need to be run manually.  Here is a sample cronfile:

# Cronfile for user wxp: /home/wxp/cronfile
 7,13,20,45   *     * * * /home/wxp/scripts/saconvert
30        2,8,14,20 * * * /home/wxp/scripts/synconvert
45,59         *     * * * /home/wxp/scripts/radconvert
0,30          *     * * * /home/wxp/scripts/rcmconvert
40  8,9,11,20,21,23 * * * /home/wxp/scripts/uaconvert
# hdrcreate used for LDM ingest
*             *     * * * /home/wxp/scripts/hdrcreate
# scour to clean up old data
35            *     * * * /home/wxp/scripts/scour_data

The hdrcreate script needs only to be run if the ingestor is not WXP.