Data Ingest: Binary Feeds

The WXP ingestor can read in and process data from the High-resolution Data Service (HDS). This data circuit provides gridded output fields that have been derived from the various types of numerical and analytical models. The Unidata McIDAS channel available to Unidata sites provides satellite imagery.  The NIDS data feed provides real-time Doppler radar images from each of the operational WSR88D sites.  WXP also can process WSI's NOWRad radar composite product.  The WXP ingestor cannot handle McIDAS, NIDS and NOWRad feeds.  They can be ingested through the LDM.

High-resolution Data Service (Alden format)

The HDS data circuit delivers numerical forecast model data in terms of grid fields that contain information for a specific variable and pressure level. These grids are delivered with a leading product header and a trailing checksum. The product header is a standard WMO header which are described in the Appendix (WMO Headers for Gridpoint Data. The product can be described as follows:

   [SOH][CR][CR][LF]
   seq[CR][CR][LF]
   header[CR][CR][LF]
   data..........
   [CR][CR][LF][ETX]
   CC

Where:

• seq -- A sequence number which is incremented one for each successive bulletin.
• header -- The WMO header information describing the type, origin and observation time of the data.
• CC  -- The 16 bit CCITT checksum.

To determine if a product is valid, pass all bytes of the product from the [SOH] to the last checksum byte to the CCITT checksum algorithm and if the results is 0, the product is correct, otherwise the product is corrupted.

The data are encoded into four different formats. The first format is a straight ASCII format for all textual administrative messages.

GRIB Format

The second format is the GRIB format (binary grid) for gridded data transmission. The GRIB format is a binary data format where all gridpoint data is packed into its most efficient form. The GRIB format is laid out into six sections:

1. Indicator block: the characters "GRIB" plus product size.
2. Product definition block: this block describes all data contained in the product
3. Grid definition block: this block describes in detail the layout of the grid (OPTIONAL)
4. Bit map block: this block is used to define where valid data is located is a sparse grid (OPTIONAL)
5. Binary data block: binary packed gridpoint data
6. End of record block: the characters "7777"

GRID Format

The third format is the GRID format. This is a ASCII equivalent to the GRIB format. Because this is an ASCII character stream, gridpoint and floating point resolution is usually restricted in order to maintain small product size. The GRID format is laid out into six sections:

1. Indicator block: the characters "GRID"
2. Product definition block: this block describes all data contained in the product
3. Grid definition block: this block describes in detail the layout of the grid (OPTIONAL)
4. Gridpoint data block: packed gridpoint data
5. Checksum information block (OPTIONAL)
6. End of record block: redundant identification plus the characters "777="

BUFR Format

The fourth format is BUFR format. This is a binary format for storing non-gridded data. The BUFR format is laid out into the following sections:

1. Indicator block: the characters "BUFR" plus product size.
2. Identification block: this describes the contents of the product
3. Data description block: this describes the type and format of data contained in the product
4. Data section: the packed data
5. End of record block: the characters "7777"

Unidata McIDAS data stream

This data stream is a composite stream developed to supply real-time data to McIDAS display systems. This data circuit does provide hourly navigated digital satellite imagery as well as surface, upper air, radar, model gridpoint and profiler data. As of now, WXP will only handle satellite data. In general there are 4 satellite image products:

1. Visible sector
2. Infrared sector
3. Water vapor
4. Floater sector of specified location and resolution

The visible, infrared and water vapor images are provided both  GOES east and west.

In addition to the satellite data, the broadcast also contains:

• Surface and upper air data
• Radar summary data
• Raw textual data (SA only)
• HDS gridpoint data
• Profiler data

Satellite Imagery Data

WXP supports McIDAS AREA files, GOES Tap (obsolete) and NOAAPORT satellite imagery.   WXP cannot ingest McIDAS AREA files (use LDM) and GOES Tap images.

NIDS (Nexrad Information Dissemination Service)

NIDS provides a feed of real-time Nexrad Doppler radar data.  The feed is broken down by Nexrad site with the user selecting a particular set of sites to receive.  Each site provides a set of products once every 5 minutes.  The products range from reflectivity and velocity at various elevations to composite reflectivity, precipitation total and wind data.  The data are broadcast in NIDS format.

NOWRad Data

WSI Corp provides a radar summary product with a resolution of 2 and 8 km.  This is a composite of the available NWS radar sites (WSR-57, 74 and 88D).  The products are available once 5 minutes for the 2 km image and every 15 minutes for the 8km image.  The NOWRad data are broadcast in a proprietary binary format.

Unisys Radar Mosaic Data

Unisys Corp provides a composite radar summary product with a resolution of 2, 4 and 8 km with a selection of base reflectivity, precipitation, precip tops and storm attributes.  This is a composite of the available NWS radar sites (WSR-57, 74 and 88D).  The products are available approximately once every 5 minutes.  

RRWDS Radar Data

As with GOES tap data, there is currently no mechanism within WXP to ingest this data but the rad program will display them. There are several mechanisms to ingest this data stream available through commercial vendors.

NLDN Lightning Data

SUNY Albany provides access to NLDN data to university sites. This data is individual strike data and comes in once every 6 minutes over the LDM.