This component provides a simple mechanism to write output data in CSV-like formats. This can include "true" CSV data as well as variations like CPD2 style and tab separated forms.

Usage

da.export [switches…​] [[station] variables times [archive]|[file]]

Switches

--mode=TYPE

This is the base export mode. It defines the defaults for all other options. Those defaults can be overridden by setting the option directly. For some export modes (XML and raw) there are no other options so any specified are ignored. Those exports are direct translations of the input data stream.

Default: MS Excel compatible

The possible values are:

--mode=CPD1

CPD1 compatible (CSV, station, year and doy)

--mode=CSV

General CSV (CSV, year-doy, blank-mvc)

--mode=Excel

Same as --mode=xl

--mode=IDL

IDL compatible (space-separated, Julian day, standard mvc)

--mode=IDLLegacy

IDL legacy code compatible (station, year and doy)

--mode=JSON

JSON data (no other options applied)

--mode=R

R/S compatible (CSV, epoch time, "NA"-mvc)

--mode=S

Same as --mode=R

--mode=XML

XML data (no other options applied)

--mode=direct1

Direct base64 data in version 1 format (no other options applied)

--mode=raw

Raw binary data (no other options applied)

--mode=raw1

Raw binary data in version 1 format (no other options applied)

--mode=xl

MS Excel compatible (CSV, date-time, blank-mvc)

--bounds[=BOOLEAN]

If set then columns for containing the absolute minimum and maximum of the data in the average will be output.

--count[=BOOLEAN]

If enabled then columns with the number of observations for each column will be included.

Default: Enabled

--cover[=BOOLEAN]

If set then columns representing the fraction of data available in each average will be output.

--cut-split[=BOOLEAN]

If set then variables will be renamed/split based on their cut size even if they do not overlap. If set to false then automatic detection is disabled and data are never split. Disabling this can have unpredictable results if used with data that does overlap.

Default: Automatic

--cut-string[=BOOLEAN]

If set then the output includes an indicator containing a semicolon delimited list of cut sizes.

--end[=BOOLEAN]

If set then columns representing the value at the end of the average will be output.

--fill=INTERVAL

This is maximum interval to create filling lines at. For example, five minutes would create an output every five minutes during gaps, starting at the beginning of the gap. Setting it to undefined creates a single output for the entire gap. This is best coupled with averaging of the same interval for uniform data output.

Undefined intervals are accepted. The interval must be greater than zero in length.

--flags=TYPE

Sets the flags breakdown and output mode. This determines how flags values are written out.

The possible values are:

--flags=0x

Output flags out as hexadecimal numbers with a leading "0x".

--flags=breakdown

Output flags as a series of zero or one values corresponding to each possible flag.

--flags=default

Output flags out like a conventional value.

--flags=list

Output flags as a list of flag names in the field.

--format=STRING

If set then this is the format (like printf(2) or of the form "000.000") used to output all numeric values.

--header-cut[=BOOLEAN]

If set then a line with a cut size for the whole column is included in the header.

--header-description[=BOOLEAN]

If set then a line with field descriptions is included in the header.

--header-flags[=BOOLEAN]

If set then lines for each possible flag will be included in the header.

--header-mark=STRING

If set then this is added to the start of all header lines.

--header-mvcs[=BOOLEAN]

If set then a line with field missing value codes is included in the header.

--header-names[=BOOLEAN]

If set then a line with field names is included in the header.

--header-names-stderr[=BOOLEAN]

If set then a line with field names is output to standard error (regardless of invocation context).

--header-wavelength[=BOOLEAN]

If set then a line with field wavelengths (including possibly time dependent information) is included in the header.

--join=TYPE

This is the base field joining mode. The mode set here determines the defaults used to join together the output. Those options can be overridden individually as well.

The possible values are:

--join=csv

Standard comma separated values. Fields are joined together with a comma (ASCII 0x2C) and quoted with a double quote (ASCII 0x22).

--join=space

Space separated values. Fields are joined together with a space (ASCII 0x20) and quoted with double quotes (ASCII 0x22).

--join=tab

Tab separated values. Fields are joined together with a tab (ASCII 0x09).

--join-delimiter=STRING

If set then this overrides the string used to delimit the join between multiple fields.

--join-quote=STRING

If set then this overrides the string added to the start and end of a field if it needs to be quoted.

--join-quote-escape=STRING

If set then this overrides the string used to replace all instances of the quote string within a quoted field.

Default: The quote string repeated three times

--mean[=BOOLEAN]

If set then the primary data columns will contain the unweighted means when they are available instead of coverage weighted and/or difference measurements.

--mvc=TYPE

This is the MVC output style. Missing or undefined values are written in accordance with this style. Cannot be used with a direct specification of the MVC.

The possible values are:

--mvc=NA

Output the string "NA" for missing values.

--mvc=blank

Output an empty field for missing values.

--mvc=default

Use the default MVC for each value.

--mvc-all=STRING

If set then this is the MVC string used for all values. Cannot be used with a MVC mode specification.

--mvc-flag=TYPE

Sets the missing value flag mode. If enabled missing value flags are fields that are either zero or one to indicate if the value corresponding to the flag was defined in the line.

The possible values are:

--mvc-flag=disable

Disable MVC flags.

--mvc-flag=end

Output MVC flags at the end of the record in the same order as the prior fields in the record.

--mvc-flag=follow

Output MVC flags immediately following the value they correspond to.

--numeric-only[=BOOLEAN]

If set then only values with a numeric representation will be output, all others will be considered missing.

--paths[=BOOLEAN]

If set then all complex values have their component paths output separately, resulting in one column for each path.

--quantiles[=BOOLEAN]

If set then columns for the standard quantiles (0.00135, 0.00621, 0.02275, 0.05, 0.06681, 0.15866, 0.25, 0.30854, 0.5, 0.69146, 0.75, 0.84134, 0.93319, 0.95, 0.97725, 0.99379, and 0.99865) of the data in the average will be output.

--squash=INTERVAL

This sets a limit below which data are combined into a single record. That is, if two variables have values separated by less than this threshold, they will be combined into a single output record.

Undefined intervals are accepted. The interval must be greater than zero in length.

--station[=BOOLEAN]

If set then a station code is included in the output.

--stddev[=BOOLEAN]

If enabled then columns for the standard deviation of the averages will be output.

Default: Enabled

--time-doy[=BOOLEAN]

If set then the output includes the fractional day of year (with January 1 = 1.00000).

--time-epoch[=BOOLEAN]

If set then the output includes the number of seconds since 1970-01-01T00:00:00Z.

--time-excel[=BOOLEAN]

If set then the output includes the date and time in a format recognized by Excel.

--time-fractional-year[=BOOLEAN]

If set then the output includes the fractional year with enough precision to represent seconds.

--time-iso[=BOOLEAN]

If set then the output includes a date-time formatted in compliance with ISO 8601.

--time-julian-day[=BOOLEAN]

If set then the output includes the fractional Julian day. That is the fractional number of days since 12:00 January 1, 4713 BC.

--time-yeardoy[=BOOLEAN]

If set then the output includes the year and fractional day of year (with January 1 = 1.00000) as separate fields.

Arguments

If no bare word input specification is supplied then data are read from standard input.

station

This argument is used to specify the station used to look up variables that do not include a station as part of an archive read specification. The station is the three letter GAW station code of the location, such as BND. The argument accepts multiple stations specified as regular expressions and separated by : or ; or ,. For example BND,MLO and (BND|MLO) are two ways of selecting both the BND and MLO stations.

The special value allstations may also be specified to select all stations.

variables

This argument may be split into multiple actual program arguments. Each part consists of one or more variable specifications separated by , (commas). The station and archive used if none are explicitly given are defined in the other arguments or inferred from system defaults. For example simply BsB_S11 selects the blue scattering from the S11 instrument and returns all cut sizes and metadata. If instead ::BsB_S11:pm10 is used then only PM10 scattering is returned. That is, the selection specifies "any" station and archive (the defaults are set in the other arguments) then requires that it has the "PM10" flavor. A specification such as bnd:BsB_S11,thd:BsB_S11 allows for data selection from multiple stations.

This may also be used to select a "record" type alias such as "S11a". In this mode all the variables defined in the alias are returned.

Finally the special value everything can be used to select all available data for the given station(s) and archive(s).

times

This argument may be split into multiple actual program arguments. This full list of time arguments defines a time bounds list that sets the range of data queried. The final resulting data are clipped to the given time bounds but any values that intersect them are returned.

archive

This argument is used to specify the archive used to look up variables that do not include an archive as part of an archive read specification. The archive is the internal name, such as raw or clean_meta. The argument accepts multiple archives specified as regular expressions and separated by : or ; or ,. For example raw(_meta)? selects both the raw and raw_meta archives.

The special value allarchives may also be used to select all available archives.

file

This argument is used to specify the the file to read data from. If it is present and exists then data is read from the given file name instead of from standard input. Alternatively - (a single dash) may be used to explicitly specify standard input.

Examples

Default MS Excel formatted
da.export bnd S11a 2015-05-01 2015-05-03

Or:

da.export input_file.c3d

Or:

da.get bnd S11a 2015-05-01 2015-05-03 | da.export
CSV with year and DOY
da.export --mode=CSV bnd S11a 2015-05-01 2015-05-03
Excel with station and fractional year
da.export --mode=xl --station --time-fractional-year bnd S11a 2015-05-01 2015-05-03