The DB System
The DB system is the collective name for the programs and supporting structures used to access and manipulate data formatted in CPD2 style
Data Access
- data.avg - Average, fill, and/or rectangularize data.
- data.consolidate - Access data by individual variables.
- data.get - The primary interface used to access data.
Intensive Data Generation
- data.aggregate.humidograph - Generate humidograph f(RH) curve fits.
- data.sd.optical.calc - Calculate optical scattering/absorption from a size distribution.
Data Export
- data.aggregate.cpd1 - Generate cpd1 style records.
- data.aggregate.narsto - Generate NARSTO formatted data.
- data.aggregate.ames - Generate NASA-Ames formatted data.
- data.export - Reformat and rectangularize data for use in other programs.
- data.csvformat - Reformat data to be strict CSV (possibly with headers).
- data.demultiplex - Extract records from a multi-record multiplexed stream.
- data.multiplex - Multiplex data from multiple sources into a single stream.
Secondary Data Interfaces
- data.coverage - Generate a report with estimated data coverage.
- data.coverage.plots - Generate monthly coverage plots.
- data.flight - Get flight information (start end end).
Comment System
Comments are arbitrary strings associated with an archive, type, and a time range. Most types are allowed to overlap. Non-overlapping comments for when data is (re)processed and passed are automatically generated.
- data.comments.get - Retrieve comments for a station.
- data.comments.modify - Modify station comments.
Segmentation System
Segments are single line strings associated with a type and time range. Segments are non-overlapping.
- data.segmentation.avg - Generate averaged data based on segments.
- data.segmentation.get - Retrieve segments for a station.
- data.segmentation.modify - Modify station segments.
- data.segmentation.neph - Generate neph error segments.
Data Storage Interfaces
- data.archive.get - Get data from a DB archive.
- data.archive.put - Update data in a DB archive.
- data.newstation - Create required structures for a new station.
- data.newstation.sqldb - Create required tables for SQL data export.
Editing and Correction System
- data.edit.dilution - Generate corrected or apply dilution corrections to data.
- data.edit.get - Generate edited data.
- data.edit.mentor.edcum - Synchronize the edit database with the ed_cum file.
- data.edit.mentor.generate - Generate suggested mentor edits.
- data.edit.mentor.generate.nepherrors - Generate suggested mentor edits based on neph errors.
- data.edit.mentor.generate.spikefilter - Generate suggested mentor edits based on spike detection.
- data.edit.mentor.get - Get defined mentor edits.
- data.edit.mentor.modify - Modify mentor edits.
- data.edit.psap_recalculatetr - Recalculate transmittances at higher precision.
Correction Chain
These programs are normally called by data.edit.corr but they can function as part of an external pipeline, provided they are called with the correct arguments.
- data.edit.corr.cut_blank - Enforce data blanking after cut switches.
- data.edit.corr.field - Apply a general Perl expression to the data.
- data.edit.corr.neph_stp - Correct neph data to STP.
- data.edit.corr.neph_trunc - Apply the neph truncation correction.
- data.edit.corr.neph_wl - Correct neph data to alternate wavelengths.
- data.edit.corr.psap_bond - Apply the Bond correction to PSAP data.
- data.edit.corr.psap_cal - Apply a flow change or spot size correction to PSAP data.
- data.edit.corr.psap_oldflags - Handle flags for translated PSAP data.
- data.edit.corr.psap_weiss_undo - Undo the Weiss correction to PSAP data.
- data.edit.corr.scale - Apply a scale, offset, or polynomial to fields in data.
Internal
These programs are not usable outside of being called by data.edit.get.
- data.edit.corr - Apply the correction chain to the data stream.
- data.edit.cut_normalize - Ensure equal numbers of samples on both cut sizes.
- data.edit.mentor - Apply mentor edits to the data stream.
- data.edit.psap_cts - Apply CTS correction to PSAP data.
- data.edit.standard_corr - Apply a simple standard correction sequence.
System Utilities
- data.aggregate.allstations - Run aggregation jobs for all stations.
- data.aggregate.cpd1ftp - Update cpd1 style zip files on the FTP server.
- data.aggregate.nilu.nrt - Upload NRT data to the NILU archive.
- data.aggregate.plots.yearly - Generate/update yearly time base plots.
- data.aggregate.sqldb - Update data in an SQL database.
- data.aggregate.station - Run aggregation jobs for a given station
- data.aggregate.tostnnew - Generate data for stnnew to process.
- data.cache.clean - Clean the cache for a given station.
- data.cache.cleanall - Clean the caches for all stations.
- data.cache.invalidate - Invalidate cached data for a station.
- data.flight.process - Run flight processing for a station.
- data.legacy.cpd1 - Converter for CPD1 style data to CPD2 style.
- data.legacy.cpd1nk - Converter for CPD1 spancheck data to CPD2 style.
- data.legacy.frh - Converter for CPD1 f(RH) fit data to CPD2 style.
- data.legacy.get - Access and convert legacy data.
- data.legacy.stripchart - Converter for old stripchart transcriptions to CPD2 style.
- data.localdata.get - Access locally storage station data.
- data.process.allmail - Mail notices for all stations.
- data.process.allstations - Process new data for all stations.
- data.process.mail - Mail notices for a given station.
- data.process.mail.checkclean - Generate a report for out of data clean data.
- data.process.mail.cpdconfchanges - Generate a report for cpd.conf changes.
- data.process.mail.getcomments - Extract monitor comments into a report.
- data.process.mail.nepherrors - Process and report on neph error states.
- data.process.new.printspanchecks - Report the results of any spanchecks.
- data.process.station - Process new station data.
- data.rrneph.get - Get RR neph data corrected for zeros from the TSI neph.
- data.sd.smps.get - Convert raw SMPS data into CPD2 format.
- data.sd.sems.get - Convert raw SEMS data into CPD2 format.
- data.segmentation.allstations - Update segmentation for all stations.
- data.update.daily - Run required daily tasks on the DB system.
- data.wx.get - Wrapper for get_wx.
System synchronization
Found in $DB/bin/sync
- data.sync.lockserver - Remote lock server to coordinate locks for synchronization.
- data.sync.station - Synchronize a station to or from a remote DB instance.
Configuration Files
- corr.conf - Controls the correction chain programs for data.edit.corr.
- cts.conf - Controls the CTS PSAP correction algorithm for data.edit.psap_cts.
- standard_corr.conf - Controls the corrections applied by data.edit.standard_corr.
Other
- avg_int - Use PSAP/CLAP intensities to generate lower frequency absorption data.
- cnvt_Xme - Generate Xme MET data from me_ CPD1 files.
- cpx2 - CPX2 data viewer.
- generate_psap_spots - Generate PSAP spot segments from CPD1 ini archives.