Data Parsing

Parsers are different from decoders in that they only select from existing data.   A parse may reformat the data but no decoding is done.  A parse is used to search for a particular METAR report or a particular public forecast.  WXP provides several data parsing programs:

Text Data Parser

The parse program is a generic parser for specific text products.  This program will search through multiple raw data files for a product based on a product pattern matching scheme:

Second line parsing is also possible.  For many products, the second line of the product is the AWIPS header:

   ** FPUS1 KIND 022030 ***
   SFPIN

which is this case is "SFPIN". To parse for this, specify either "FPUS1_KIND" or "/SFPIN". 

To parse for more than one product, wildcards can be used:

   FPUS[145]_K(CHI|IND)

will parse various public forecasts (FPUS1, 4 and 5) from Chicago and Indianapolis. This program will also parse for specific station identifiers and forecast zones. A identifier is and character sequence that begins a line and is NOT the product header. The only exceptions to this are with:

• upper air data: uses the station identifier in the TTAA,TTBB, etc. reports
• frontal data: uses the valid time in the product
• forecast zones: uses the product zone specification.

The parse program is set up to parse for the first (resource parameter=first) or last occurrence (last) of requested data. By default, the program will display the full bulletin or for identifiers, the line which contains the identifier. If more data is needed, the program can list a set number of lines after the match (line=#), list up to the first blank line (blank) or dollar sign (dollar). Blank lines can be eliminated (noblank), headers can be eliminated (nohdr) or just the headers can be printed (hdr). The parse program will, if specified, save data to a data file with the file name consisting of the name of the earliest hourly data file plus the first three characters of the bulletin identifier as an extension.

Public Forecast Parser

This program will parse for state and metropolitan forecasts, state forecast discussions and coded city forecasts. This program will also parse for specific forecast zones by decoding the zone header in the state zone forecasts. This program provides a simpler interface into the generic parse program.

Surface Data Parsing

This sa_parse program will search for a specific list of stations in a ingested raw surface file or in a decoded surface data file. The input can be raw ingested data file in which the raw report (METAR/SAO) is displayed.  If the input is a converted file, the output is a tabular list of decoded information from each station. 

ID   TIME   T  TD  RH DIR SPD GST ALT SLP VIS CIL COV WX  MAX MIN  PR6 PR24 SC
KLNP 0240  52  43  71 150   4     002      10     CLR
K4BK 2347  69  52  54 310  10         114  30     CLR      72  59

The search list is based on the plot_domain, stat_prior, and identifier resources.  See the Station Sets section of the Users Guide for more information.

Upper Air Parsing

The ua_parse program is a way of printing out information for a particular upper air sounding.  The output includes a table of input data such as temperature, height, dewpoint and winds and a list of sounding parameters including stability indices and wind parameters.  The output is equivalent to the output of uacalplt but without the option to plot a sounding.

MOS Data Parsing

The fo_parse program reformats the MOS data into a simple table.  The input can be NGM, ETA MOS, upper air MOS from NGM and ETA and the MRF MOS parameters.

  NGM MOS data for IND
Initial time: 00Z 29 SEP 97

 HOUR   TIME   TEMP DEW  DIR SPD  CIL COV  VIS  WX  MX/MN POP12 SNOW
   6   29/ 6Z    59  53  260  11  250 BKN  10
  12   29/12Z    54  49  240  11  250 SCT  10
  18   29/18Z    70  48  250  22  250 SCT  10
  24   30/ 0Z    66  50  290  18  250 SCT  10         73    21
  30   30/ 6Z    57  50  280  12  250 BKN  10
  36   30/12Z    54  49  280  12  250 BKN  10         52    30
  42   30/18Z    65  51  300  15   20 BKN  10
  48    1/ 0Z    60  50  320  11  250 SCT  10         68     8
  54    1/ 6Z    54  50  310   5      CLR
  60    1/12Z    51  48  300   5      CLR             49     0       

GRIB File Parsing

The griblook program is useful for parsing HRS GRIB files for their contents. Since these data are in a binary format, it cannot by viewed with a file listing program. The griblook program will read every grid contained in the file and give a brief summary of the type of data saved in the grid. The program can parse for specific grids in the file.

  0: ETA analysis for 0000Z 29 SEP 97-100 mb Vertical velocity (Pa/s)
  1: ETA analysis for 0000Z 29 SEP 97-150 mb Vertical velocity (Pa/s)
  2: ETA analysis for 0000Z 29 SEP 97-200 mb Vertical velocity (Pa/s)
  3: ETA analysis for 0000Z 29 SEP 97-250 mb Vertical velocity (Pa/s)
  4: ETA analysis for 0000Z 29 SEP 97-300 mb Vertical velocity (Pa/s)  
...