data.consolidate

data.consolidate is an interface to request data based on variable names (or expressions to match variable names) instead of by complete records. The process of searching and combining records based on this is somewhat time intensive, so data.consolidate should only be used when normal records are not sufficient. Non-exact matching is particularly intensive, so it should be used with care.

Uses records.conf to determine what records might contain the requested variables.

Command Line Usage

data.consolidate [--source=archive] [--regex] [--noautoavg] [--record=a]
                 [station] start end variable [variable...]

Data is output to standard output.

Arguments

start and end

The time specifiers for the data to be retrieved. Start is inclusive while end is exclusive, so all data contained within the half open interval [start,end) will be returned. Any convertible time format is accepted.

station

The station identifier code. For example 'brw'. Case insensitive.

variable

Variable specification, each argument can contain multiple variables (separated by a comma, semicolon or space. Shell wildcards are allowed but it should be noted that using wildcards incurs a significant search overhead to start in most cases

--source=archive

Select the archive to retrieve data from, with the default being raw. May also be ”-” to read from standard input. In this mode all arguments are variable definitions. That is, the station, start and end are omitted.

--regex

Interpret variables are Perl regular expressions instead of simple wildcard searching. All variables are implicitly wrapped in '^variable$', so the expression must match the head and tail of the variable.

--noautoavg

Disable automatic matching of variables produced by data.avg, if not present then the pattern is applied to the base variable as well instead of only the literal names. Note that when using this it is necissary to specify patterns that resolve both the base records and to the produced records (for example “BsG0?_S11” with –regex).

--record=a

Set the output record code. Defaults to “a”.

Example Usage

Red and blue scattering

data.consolidate sgp 2008:10 2008:11 BsR_S11 BsB_S11

All edited scattering related data

data.consolidate --source=edited sgp 2008:10 2008:11 'Bs*'

Neph flags, scattering and absorption

data.consolidate --regex brw 2008:10 2008:11 'Bb?[ab][BRGSLCOFAKV]_[AS]11' F2_S11