Resource: prod_file

The product file is used by the prog:wmoingest program to select and process products from various NWS data feeds. The selection process uses a simple regular expression syntax (similar to selection by prog:wmoparse and the output of the selection process uses a similar mechanism to the file naming convention file.

This resource specifies the name of the bulletin file.  If a relative filename is used, the value of the file_path resource is prepended to the filename.  The default filename is "ingest.bul".  Often it is convenient to define a separate bulletin file for each data feed but this is not necessary.

Product File Contents

The product file consists of several lines, each defining a product to be selected.  There are three values listed on each line: the pattern to match, the action, and the output filename or command:

header [action] [command...]
header [action] [command...]

Here is a sample:

# Pattern        Action Filename          Header Filename
#
S[AP]             >>-15 %D/%h%m%d%y.sao
S[IMNS]           >>-05 %D/%h%m%d%y.syn
SD                >>+07 %D/%h%m%d%y.rad
...
F[^OT]            >>    @for_dat          @for_hdr
A                 >>    %D/%6h%m%d%y.sum  %D/hd%6h%m%d.sum
...
#
# Specific forecast products
#
FXUS01            >     %D/fore/48hr
...
FXUS06            >     %D/fore/10d_dsc
FPUS6_            >     %D/fore/30d
FPUS5_KIND        |     /usr/local/bin/parse - -ph=FPUS5_KIND -id=%%INZ029 -pa=dollar -of=%D/fore/laf_zone -me=none
FXUS07            >     %D/fore/30d_dsc
...
#
# HDS products
#
Y/M89             >>    %D/%12h%m%d%y.gbm
Y/M39G211         >>    %D/%12h%m%d%y.gbv
Y/M64G211         >>    %D/%12h%m%d%y.gbv

Pattern

Here is the simple pattern matching scheme:

Short patterns such as "F" are the equivalent of "F*".  It will match on any pattern starting with "F".

Some example header strings are:

In some cases, it might be necessary to use product contents in selecting a product.  The ingest program does allow you to use the second line of a product in selecting a product.  For example, AFOS PILs are often listed on the second line.  To select the zone forecasts for Oklahoma, you can use the following pattern:

   /ZFPOKC

This also allows you to save HDS products based on their content rather than worrying about the header which often times can be misleading.  The syntax for this is:

/[Xvvv][Xvvv][Xvvv]...

Where X is:
M for model number
G for grid number
L for level type
H for level value
T for forecast time
V for variable number

For example, to save the ETA model, you can use "/M89".  To save the initialized field of the NGM model, use "/M39/G211/T0".

Action

The actions resemble standard Unix redirection symbols:

By default, all products are stripped of control characters and generally cleaned up prior to writing to file.  There are a couple of exceptions.  First, if the product is a GRIB or BUFR product, it is assumed to contain binary data no stripping occurs.  Second, you can tell the ingest program to save the raw data by prepending a "R" to the action:

   R>>

When saving to the output filename, often you may want to offset the time slightly.  A good example of this is surface data where reports start streaming in up to 15 minutes before the top of the hour.  If you use the system clock to determine the filename, these early reports will get stuck in the previous hour's file.  A time offset in minutes can be appended to the action.  For example:

   >>-15   %D/%h%m%d%y.sao

will start saving data into the current hour's file starting 15 minutes prior to the hour.  It will stop saving data at 45 minutes past the hour.  In some cases as with model data, you may want this offset to be more than an hour.

Filename or Command

The last piece of information on each line is the filename or command (which depends on the action).  The filename of command is expanded from a set of wildcard characters.  These are the same as those used in the filename conventions with some additions.

Examples based on system time 1455Z Jan 12, 1997,
product header FPUS5 KIND 281512

Some of the above wildcards can be preceded with a number.  For dates, the number is a modifier which rounds down to the nearest value which is a multiple of that number.  For example, "%6h" would round down to the nearest 6 hour boundary.  For the previous example, it results in the value 12.  

For the product type and locale, this number is used in a substring operation.  The first digit of the number is the offset into the string and the second digit refers to the number of characters to use.  For example, "%12T" results in "FP".  To get "IND", use "%23L".

Filenames

In general, you will use a simplistic file naming convention for the filenames.  In the above example, "%h%m%d%y" is used with an extension.  This generally have to match a similar naming convention in the name convention file to be useful.  As a shortcut, you can specify a name convention tag.  For example, you could list "@sfc_dat" rather than the verbose string listed above.  Sometimes, you still might want to use wildcard setup in ingest to determine file names.

Header Files

Header files are provided by the ingestor to speed up the searching of data. Since most data is keyed on the product header, it is often easier to search through a list of headers rather than a file with headers and product contents.  The header file essentially is a list of the headers and the position in the full file that each header exists.  Here is a sample:

      0 FZAK65 PANC 052359 CCA
   1782 FPUS5 KABQ 052359
   9360 FZUS45 KFAR 052359
  11094 FXUS21 KLSE 060001
  13115 FXUS21 KLCH 060002
  14230 FQUS1 KELP 060001
  14934 FPCN11 CWVR 052355 PAA

The first column is the byte offset into the file, followed by the actual header.  

The ingestor can create these automatically if a header file naming convention is specified.  For the append, write and file actions, the header file is just separated from the regular filename by a white space character:

F[^OT]            >>    @for_dat          @for_hdr
A                 >>    %D/%6h%m%d%y.sum  %D/hd%6h%m%d.sum

Last updated August 20, 2011