Product File (.prd)
The ingest programs uses a bulletin file to set up which products are to be selected from the data feed and which actions to perform on them. The bulletin filename is specified with the prod_file resource.
FORMAT
The bulletin file contains a list of headers, actions and commands to be performed:
header [action] [command/filename...] [header file] header [action] [command/filename...] [header file] ...
The header can specify the exact header or a pattern to which headers can be matched. The headers listed in the file can use the following wildcard characters:
| . or ? | match a single character |
| - or * | match any character |
| [letters] | match a character from the set. |
| [^letters] | match any character except those from the set |
| (str1|str2...) | match strings |
| _ | underscore matches a space |
| /data | match extra information |
Some example header strings are:
| AB | Anything that starts with AB |
| S[AP] | SA or SP |
| (W|AC|RG) | Starts with W or AC or RG |
| F[^O] | Anything that starts with F, second character NOT O |
| FQUS1_KIND | Full header specification with spaces as underscores |
| *_KIND | Wildcard match on any product that ends with KIND |
When the product is GRIB, the header is parsed for specific product parameters. This
information can then be used to select the product. The syntax for this selection is:
/[Xvvv][Xvvv][Xvvv]...
Where X is:
M -- model number
G -- grid number
L -- level type
H -- level value
T -- forecast time
V -- variable number
vvv -- the value of the parameter
The values for each parameter are listed in the WXP Product Description Appendix. Using
the internal GRIB parameters is more reliable than selecting by the WMO header
because more than one product may have the same header:
HVAC98 KWBC 070000 from Sea Wave model
HVAC99 KWBC 070000 from Aviation model
To separate the two products, use the model specifications: /M77 for the Aviation
model and /M10 for the Sea Wave model.
Actions
The actions are:
| >> | append to file with header |
| append | same as above |
| > | write to file with header, previous content overwritten |
| write | same as above |
| # | write to file without header, previous contents overwritten |
| file | same as above |
| | | pipe product to listed command |
| pipe | same as above |
| @ | run command when product complete |
| run | same as above |
Also, the action can be prepended by a set of flags:
- R -- specifies to save the file as a raw file and not strip control characters.
- B -- specifies a product to be a binary product and not strip unprintable characters
- P -- specifies to send a PAN message at the completion of a product
Command or Filename
The command is generally the file to place the output or the command to run with the pipe or run actions. The command can have several escape characters:
Examples based on system time 1455Z Jan 12, 1997,
product header FPUS5 KIND 281512
| Wildcard | Explanation | Example |
|---|---|---|
| @tag | Name convention tag | |
| %Y | current system year | 1997 |
| %y | current system year (last 2 digits) | 97 |
| %m | current system month | 01 |
| %d | current system day | 12 |
| %j | current system Julian day | 12 |
| %h | current system hour | 14 |
| %n | current system minute | 55 |
| %pd | product day | 28 |
| %ph | product hour | 15 |
| %pn | product minute | 12 |
| %T | product type | FPUS5 |
| %t | product type (lower case) | fpus5 |
| %L | product locale | KIND |
| %l | product locale (lower case) | kind |
| %D | data_path resource | |
| %C | con_path resource | |
| %R | raw_path resource | |
| %G | grid_path resource | |
| %W | watch_path resource | |
| %I | image_path resource | |
| %F | file_path resource |
Some of the above wildcards can be preceded with a number. For dates, the number is a modifier which rounds down to the nearest value which is a multiple of that number. For example, "%6h" would round down to the nearest 6 hour boundary. For the previous example, it results in the value 12.
For the product type and locale, this number is used in a substring operation. The first digit of the number is the offset into the string and the second digit refers to the number of characters to use. For example, "%12T" results in "FP". To get "IND", use "%23L".
Header Files
To aid in the parsing of products from the various feeds, a header file can be created by the ingest program. This essentially lists the header of each product in the file along with its byte offset into the file. Since most parsing is based on header, it is far easier to search the smaller header file than to parse through the much larger product file.
To produce these files automatically by the ingestor, add the file name convention to the end of the line in the bulletin file:
F[^O] >> %D/%y%m%d%6h_for.wmo %D/%y%m%d%6h_for.hdr
The first name convention listed "%D/%y%m%d%6h_for.wmo" is the filename where the actual product is saved. The second name convention "%D/%y%m%d%6h_for.hdr" is where the header file information is saved. The syntax of the file is as follows:
offset header / extra
offset header / extra
....
where:
- offset -- is the byte offset into the file,
- header -- is the product header in its entirety is listed after the offset
- extra -- extra information about the product which is normally the AWIPS header
A sample from a forecast data header file:
0 FPUS86 KPQR 282359 / OPUPDX 3264 FPUS85 KGGW 290001 / OPUGGW 3548 FPAK11 PAYA 282207 / &ZCZC JNULFPYAK 4190 FPUS73 KFGF 282359 / NOWFAR
For more information on header files, see the section on header files.
PAN (Product Arrival Notices) Messages
To set up the WXP ingestor for PAN messages the following pieces of information must be added to the bulletin file. At some point in the file, a PAN configuration line must be added.
# PAN Setup @PAN id=45 sock:steve:5566 sock:dev5:5000 pan.log
The "@PAN" is a keyword in the bulletin file for the PAN configuration line. The "id=45" specifies the NOAAPORT unique server ID which is broadcast as field 2 in the PAN message. The rest of the line lists destinations. The "sock" keyword specifies the PAN go over a UDP socket. The string "steve:5566" is the network name of the destination computer and the TCP/IP port number. If the sock keyword is omitted, the PAN is save to the listed filename such as "pan.log". Up to 10 destinations can be listed. Each destination is addressed starting with 0 and going to 9 in the order listed on the PAN line.
By default, no PAN messages are sent even if the PAN line is added to the bulletin file. To enable PAN messages, the "P" flag must be added to the action for each product being saved on the server. For example a product line would look like:
# Pattern Action Filename Header Filename
FT >> %D/%y%m%d%h_term.wmo %D/%y%m%d%h_term.hdr
To enable this product type for PAN messages, add the "P" flag to the action.
FT P>> %D/%y%m%d%h_term.wmo %D/%y%m%d%h_term.hdr
This will send a PAN message to all listed destinations whenever this products is received. If you don't want to send a PAN to all destinations, the destination IDs can be listed:
FT P035>> %D/%y%m%d%h_term.wmo %D/%y%m%d%h_term.hdr
In this case, PAN messages will only be sent to the 0, 3 and 5th destinations.
EXAMPLES
# Pattern Action Filename Header Filename # S[AP] >>-15 %D/%y%m%d%h_sao.wmo S[IMNS] >>-05 %D/%y%m%d%h_syn.wmo SD >>+07 %D/%y%m%d%h_rad.wmo U[^AB] >>-65 %D/%y%m%d%12h_upa.wmo ASUS1_ >> %D/%y%m%d%3h_frt.wmo WWUS40 >> %D/%y%m%d%6h_wws.wmo FO >> %D/%y%m%d%12h_mod.wmo %D/%y%m%d%12h_mod.hdr A >> %D/%y%m%d%6h_sum.wmo %D/%y%m%d%6h_sum.hdr C >> %D/%y%m%d%6h_cli.wmo %D/%y%m%d%6h_cli.hdr W >> %D/%y%m%d%6h_sev.wmo %D/%y%m%d%6h_sev.hdr # # Specific forecast products # FXUS01 > %D/fore/48hr FXUS02 > %D/fore/3-5d_Hem FPUS5_KIND | /usr/local/bin/parse - -ph=FPUS5_KIND -id=%%INZ029 -pa=dollar -of=%D/fore/laf_zone -me=none *_KIND >> %D/Indy/%m%d.dat # # HDS products # Y/M89 >> %D/%y%m%d%12h_eta.grb %D/%y%m%d%12h_eta.hdr Y/M39G211 >> %D/%y%m%d%12h_ngm.grb %D/%y%m%d%12h_ngm.hdr Y/M64G211 >> %D/%y%m%d%12h_ngm.grb %D/%y%m%d%12h_ngm.hdr
SEE ALSO
Last updated May 10, 2020