Database

Build

The Basic Fusion program takes as an argument a text file containing all of the HDF files it needs to process. This text file can either be created manually, or it can be generated from a database query. The SQlite database is built using a Python script that parses an unordered list of all MOPITT, CERES, ASTER, MODIS and MISR files available in the file system. The general workflow is as such:

1. Find all Terra files that exist in the file system. This should be done using a script. Create a separate text file for each instrument file list.
2. Zip these files using gzip
3. Use python interpreter to call the fusionBuildDB script. Pass it the appropriate parameters. An example of this workflow can be seen in the metadata-input/build/generateDB.sh script. You can either simply use this script that performs all of these steps for you, or you can perform the steps manually. Note that the script can use different different versions of the file finder. The three different scripts available are genInputHpss, genInputPosix, and ROGER-findFiles.

The --discards, --anomalies, and --trace arguments to the Python script can be passed so that file parsing can be logged. The arguments will display discarded files, files that the script does not know how to handle (anomalies), and a verbose error log, respectively.

Queries

Once the database has been created, SQlite queries can be performed to gather a list of HDF files based on various parameters. A shell wrapper script has been provided for various SQlite queries under metadata-input/queries.bash. These queries can be called from the terminal or from another shell script.

Because the Fusion program is particularly picky about how the input files list is ordered, significant care has to be taken to order the files after they have been queried from the database. A shell script has been created that does the work of querying the database and then also ordering the files. This is located under metadata-input/genInput/genFusionInput.sh. This script makes it easy to generate a properly formatted input file for the Fusion program.

Please refer to the database README.md file located at: basicFusion/metadata-input/README.md.