WXP
File Reference

Product File (.prd)

The ingest programs uses a bulletin file to set up which products are to be selected from the data feed and which actions to perform on them. The bulletin filename is specified with the prod_file resource.

FORMAT

The bulletin file contains a list of headers, actions and commands to be performed:

header [action] [command/filename...] [header file]
header [action] [command/filename...] [header file]
...

The header can specify the exact header or a pattern to which headers can be matched. The headers listed in the file can use the following wildcard characters:

. or ? match a single character
- or * match any character
[letters] match a character from the set.
[^letters] match any character except those from the set
(str1|str2...) match strings
_ underscore matches a space
/data match extra information

Some example header strings are:

AB Anything that starts with AB
S[AP] SA or SP
(W|AC|RG) Starts with W or AC or RG
F[^O] Anything that starts with F, second character NOT O
FQUS1_KIND Full header specification with spaces as underscores
*_KIND Wildcard match on any product that ends with KIND

When the product is GRIB, the header is parsed for specific product parameters. This information can then be used to select the product. The syntax for this selection is:

/[Xvvv][Xvvv][Xvvv]...

Where X is:

The values for each parameter are listed in the WXP Product Description Appendix. Using the internal GRIB parameters is more reliable than selecting by the WMO header because more than one product may have the same header:

HVAC98 KWBC 070000 from Sea Wave model
HVAC99 KWBC 070000 from Aviation model

To separate the two products, use the model specifications: /M77 for the Aviation model and /M10 for the Sea Wave model.

Actions

The actions are:

>> append to file with header
append same as above
> write to file with header, previous content overwritten
write same as above
# write to file without header, previous contents overwritten
file same as above
| pipe product to listed command
pipe same as above
@ run command when product complete
run same as above

Also, the action can be prepended by a set of flags:

Command or Filename

The command is generally the file to place the output or the command to run with the pipe or run actions. The command can have several escape characters:

Examples based on system time 1455Z Jan 12, 1997,
product header FPUS5 KIND 281512

Wildcard Explanation Example
@tag Name convention tag  
%Y current system year 1997
%y current system year (last 2 digits) 97
%m current system month 01
%d current system day 12
%j current system Julian day 12
%h current system hour 14
%n current system minute 55
%pd product day 28
%ph product hour 15
%pn product minute 12
%T product type FPUS5
%t product type (lower case) fpus5
%L product locale KIND
%l product locale (lower case) kind
%D data_path resource  
%C con_path resource  
%R raw_path resource  
%G grid_path resource  
%W watch_path resource  
%I image_path resource  
%F file_path resource  

Some of the above wildcards can be preceded with a number.  For dates, the number is a modifier which rounds down to the nearest value which is a multiple of that number.  For example, "%6h" would round down to the nearest 6 hour boundary.  For the previous example, it results in the value 12.  

For the product type and locale, this number is used in a substring operation.  The first digit of the number is the offset into the string and the second digit refers to the number of characters to use.  For example, "%12T" results in "FP".  To get "IND", use "%23L".

Header Files

To aid in the parsing of products from the various feeds, a header file can be created by the ingest program. This essentially lists the header of each product in the file along with its byte offset into the file. Since most parsing is based on header, it is far easier to search the smaller header file than to parse through the much larger product file.

To produce these files automatically by the ingestor, add the file name convention to the end of the line in the bulletin file:

F[^O]          >>  %D/%y%m%d%6h_for.wmo  %D/%y%m%d%6h_for.hdr

The first name convention listed "%D/%y%m%d%6h_for.wmo" is the filename where the actual product is saved. The second name convention "%D/%y%m%d%6h_for.hdr" is where the header file information is saved. The syntax of the file is as follows:

offset header / extra
offset header / extra
....
where:

A sample from a forecast data header file:

     0 FPUS86 KPQR 282359 / OPUPDX
  3264 FPUS85 KGGW 290001 / OPUGGW
  3548 FPAK11 PAYA 282207 / &ZCZC JNULFPYAK
  4190 FPUS73 KFGF 282359 / NOWFAR

For more information on header files, see the section on header files.

PAN (Product Arrival Notices) Messages

To set up the WXP ingestor for PAN messages the following pieces of information must be added to the bulletin file. At some point in the file, a PAN configuration line must be added.

# PAN Setup
@PAN id=45 sock:steve:5566 sock:dev5:5000 pan.log

The "@PAN" is a keyword in the bulletin file for the PAN configuration line. The "id=45" specifies the NOAAPORT unique server ID which is broadcast as field 2 in the PAN message. The rest of the line lists destinations. The "sock" keyword specifies the PAN go over a UDP socket. The string "steve:5566" is the network name of the destination computer and the TCP/IP port number. If the sock keyword is omitted, the PAN is save to the listed filename such as "pan.log". Up to 10 destinations can be listed. Each destination is addressed starting with 0 and going to 9 in the order listed on the PAN line.

By default, no PAN messages are sent even if the PAN line is added to the bulletin file. To enable PAN messages, the "P" flag must be added to the action for each product being saved on the server. For example a product line would look like:

# Pattern Action Filename Header Filename

FT       >> %D/%y%m%d%h_term.wmo %D/%y%m%d%h_term.hdr

To enable this product type for PAN messages, add the "P" flag to the action.

FT      P>> %D/%y%m%d%h_term.wmo %D/%y%m%d%h_term.hdr

This will send a PAN message to all listed destinations whenever this products is received. If you don't want to send a PAN to all destinations, the destination IDs can be listed:

FT   P035>> %D/%y%m%d%h_term.wmo %D/%y%m%d%h_term.hdr

In this case, PAN messages will only be sent to the 0, 3 and 5th destinations.

EXAMPLES

# Pattern        Action Filename               Header Filename
#
S[AP]             >>-15 %D/%y%m%d%h_sao.wmo
S[IMNS]           >>-05 %D/%y%m%d%h_syn.wmo
SD                >>+07 %D/%y%m%d%h_rad.wmo
U[^AB]            >>-65 %D/%y%m%d%12h_upa.wmo 
ASUS1_            >>    %D/%y%m%d%3h_frt.wmo
WWUS40            >>    %D/%y%m%d%6h_wws.wmo
FO                >>    %D/%y%m%d%12h_mod.wmo %D/%y%m%d%12h_mod.hdr
A                 >>    %D/%y%m%d%6h_sum.wmo  %D/%y%m%d%6h_sum.hdr
C                 >>    %D/%y%m%d%6h_cli.wmo  %D/%y%m%d%6h_cli.hdr
W                 >>    %D/%y%m%d%6h_sev.wmo  %D/%y%m%d%6h_sev.hdr
#
# Specific forecast products
#
FXUS01            >     %D/fore/48hr
FXUS02            >     %D/fore/3-5d_Hem
FPUS5_KIND        |     /usr/local/bin/parse - -ph=FPUS5_KIND -id=%%INZ029 -pa=dollar -of=%D/fore/laf_zone -me=none
*_KIND            >>    %D/Indy/%m%d.dat
#
# HDS products
#
Y/M89             >>    %D/%y%m%d%12h_eta.grb %D/%y%m%d%12h_eta.hdr
Y/M39G211         >>    %D/%y%m%d%12h_ngm.grb %D/%y%m%d%12h_ngm.hdr
Y/M64G211         >>    %D/%y%m%d%12h_ngm.grb %D/%y%m%d%12h_ngm.hdr

A sample product file

SEE ALSO

Last updated May 10, 2020