Data Ingest: Scouring Old Data

Since the ingest program only saves data to disk, there needs to be a mechanism to clean up old data to prevent the disk from filling up.  WXP provides a simple scour program that will delete data based on a size or age limit.  The syntax of the program is:

scour /home/wxp/data msize=400

The first parameter is the directory to scour and the second parameter is the size in megabytes to scour to.  This will delete the oldest files in the directory until the directory size is 400 megabytes.

Data can be deleted based on age:

scour /home/wxp/convert age=10

This will remove all files that are greater than 10 days old.

Automated Scouring

To insure scour is run, it is recommended that it be installed into the cron list for the user that is running the ingestor (to prevent permission conflicts). Here is a sample scour script for normal WXP processing:

#! /bin/csh -f
#  scour_data:  removes old data files
/home/wxp/bin/scour /home/wxp/data    msize=500
/home/wxp/bin/scour /home/wxp/convert msize=25

Here is a sample scour script for NOAAPORT WXP processing:

#! /bin/csh -f
#  scour_data:  removes old data files
/home/wxp/bin/scour /home/wxp/data    msize=300
/home/wxp/bin/scour /home/wxp/text    msize=150
/home/wxp/bin/scour /home/wxp/model   msize=500
/home/wxp/bin/scour /home/wxp/convert msize=25
/home/wxp/bin/scour /home/wxp/sat     msize=500