data.export

data.export is used to reformat CPD2 style data into formats more easily understood by other programs. It takes its input from standard input and produces output on standard output based on various switches set on the command line.

Command Line Usage

data.export [--mode=excel|xl|csv|idl|r|R|archive|idllegacy|cpd2] 
            [--merge=none|last|interpolate]
            [--join=csv|space] 
            [--station=on|off] 
            [--date-epoch=on|off] [--date-excel=on|off] [--date-iso=on|off] 
            [--date-fyear=on|off] [--date-julian=on|off]
            [--date-yeardoy=on|off] [--date-doy=on|off]
            [--mvc-type=blank|mvc|na] [--mvc-flag=follow|end|off]
            [--flags-type=breakdown|0x|default] [--header-mark=...]
            [--header-names=on|off|-1] [--header-desc=on|off]
            [--header-namesstderr=on|off] [--header-wavelength=on|off]
            [--header-cpd2global=on|off] [--header-mvcs=on|off]
            [--format-default=off|<format>] [--format-<name>=<format>]

The mode argument is processed before any other arguments, so further arguments can be used to override the defaults set by the mode.

Arguments

--mode=excel|xl|csv|idl|idllegacy|r|R|archive|cpd2

Select mode. Setting a mode sets the defaults for all parameters relevant to that mode.

CSV

Export data in CSV format with year and DOY timestamps.

Normal usage:

... | data.export --mode=csv > data.csv
Excel, xl

Export data in CSV format for importing into Excel.

Normal usage:

... | data.export --mode=excel > data.csv
idl

Export data in space separated and converted format for reading with IDL. Also generates a header on standard error with the names and index (zero-based) of each column.

Normal usage:

... | data.export --mode=idl > data

Will generate a data file and a header on standard error with the column names as above.

r and R

Export data in CSV format for importing into R.

archive

Archive export mode.

idllegacy

IDL legacy export mode.

cpd2

Reformatted CPD2 style data. This mode does not accept any other format switches.

--merge=none|last|interpolate

Select merging mode for data inputs. Note that this has no effect when operating on already merged data (such as data generated from data.avg). The default is none.

--join=csv|space

Select joining mode. Joining determines how values are combined into record lines and how they are separated within those lines.

csv

Comma separated value join mode. If values contain spaces or any other condition that would require quoting, that value is quoted as per Text::CSV_XS.

space

Space separated values. No handling for values with embedded spaces; do not use on string data.

--station=on|off

Enable or disable the leading column with the station code.

--date-epoch=on|off

Enable or disable outputting epoch time (seconds since Jan 1 1970).

--date-excel=on|off

Enable or disable outputting Excel formatted time (yyyy-mm-dd hh:mm:ss).

--date-iso=on|off

Enable or disable outputting ISO formatted time (yyyy-mm-ddThh:mm:ssZ).

--date-fyear=on|off

Enable or disable outputting fractional year.

--date-julian=on|off

Enable or disable outputting fractional Julian day.

--date-yeardoy=on|off

Enable or disable outputting a year and fractional day of year.

--date-doy=on|off

Enable or disable outputting a fractional day of year.

--mvc-type=blank|mvc|na

Select format for undefined values.

--mvc-flag=follow|end|off

Set the mode for MVC flags for each column. The flag is either 0 (valid value) or 1 (invalid).

--flags-type=breakdown|0x|default

breakdown

Enable breakdowns of bitwise flags. When enabled instead of a single column with the bit set of flags N columns of values of either 0 or 1 will be generated for each bitwise flag. N is the number of bits that the bit set can contain (e.g. 16 for a four digit hexadecimal number).

0x

Insert '0x' before any hexadecimal numbers.

default

Do not reformat flags.

--header-mark=...

Set the header marker. This string is prepended to all header lines.

--header-names=on|off|-1

Enable or disable outputting a header line before the data begins with the names of each variable. The names do not contain spaces. If set to -1 then the names are output immediately before the actual data instead of at the start of the header lines.

--header-desc=on|off

Enable or disable outputting a header line with the field description of each variable before data. The field description may contain spaces.

--header-namesstderr=on|off

Enable or disable printing a header line to standard error with the names and column numbers of the data. The line is comma separated values of the form “NAME:INDEX”. Where the name is the name as –header-names above and the index is the zero-based index of that column.

--header-wavelength=on|off

Enable or disable printing a header line with the wavelength information for each variable.

--header-cpd2global=on|off

Enable or disable printing header lines for all CPD2 global headers.

--header-mvcs=on|off

Enable or disable printing a header line with the MVC for each variable.

--format-default=off|<format>

Set the default format for all value columns or disable default formatting. When a default format is set, all columns use that format unless overridden with –format-<name> (see below). The format is a CPD2 extended printf code.

--format-<name>=<format>

Override the format for a given variable (specified by <name>). The format is a CPD2 extended printf code.

Example:

  1. -format-BaG_A11=%07.2f

Example Usage

Default Excel

data.get sgp S11a 2008:10 2008:11 | data.export --mode=excel > data.csv

IDL with flags handling off

data.avg sgp S11a 2008:10 2008:11 | data.export --mode=idl --flags-type=default

Excel with station and fractional year

data.get sgp S11a 2008:10 2008:11 | data.export --mode=excel --station --date-fyear > data.csv
  

Archive output format

 data.get app S11a,A11a,N71a,X1a,X2a,XIe 2010:1 1d avgh | data.export --mode=archive > aerosol_main_app_2010_hr.csv