This component provides a simple mechanism to write output data in CSV-like formats. This can include "true" CSV data as well as variations like CPD2 style and tab separated forms.
Usage
da.export [switches…] [[station] variables times [archive]|[file]]
Switches
- --mode=TYPE
-
This is the base export mode. It defines the defaults for all other options. Those defaults can be overridden by setting the option directly. For some export modes (XML and raw) there are no other options so any specified are ignored. Those exports are direct translations of the input data stream.
Default: MS Excel compatible
The possible values are:
- --mode=CPD1
-
CPD1 compatible (CSV, station, year and doy)
- --mode=CSV
-
General CSV (CSV, year-doy, blank-mvc)
- --mode=Excel
-
Same as --mode=xl
- --mode=IDL
-
IDL compatible (space-separated, Julian day, standard mvc)
- --mode=IDLLegacy
-
IDL legacy code compatible (station, year and doy)
- --mode=JSON
-
JSON data (no other options applied)
- --mode=R
-
R/S compatible (CSV, epoch time, "NA"-mvc)
- --mode=S
-
Same as --mode=R
- --mode=XML
-
XML data (no other options applied)
- --mode=direct1
-
Direct base64 data in version 1 format (no other options applied)
- --mode=raw
-
Raw binary data (no other options applied)
- --mode=raw1
-
Raw binary data in version 1 format (no other options applied)
- --mode=xl
-
MS Excel compatible (CSV, date-time, blank-mvc)
- --bounds[=BOOLEAN]
-
If set then columns for containing the absolute minimum and maximum of the data in the average will be output.
- --count[=BOOLEAN]
-
If enabled then columns with the number of observations for each column will be included.
Default: Enabled
- --cover[=BOOLEAN]
-
If set then columns representing the fraction of data available in each average will be output.
- --cut-split[=BOOLEAN]
-
If set then variables will be renamed/split based on their cut size even if they do not overlap. If set to false then automatic detection is disabled and data are never split. Disabling this can have unpredictable results if used with data that does overlap.
Default: Automatic
- --cut-string[=BOOLEAN]
-
If set then the output includes an indicator containing a semicolon delimited list of cut sizes.
- --end[=BOOLEAN]
-
If set then columns representing the value at the end of the average will be output.
- --fill=INTERVAL
-
This is maximum interval to create filling lines at. For example, five minutes would create an output every five minutes during gaps, starting at the beginning of the gap. Setting it to undefined creates a single output for the entire gap. This is best coupled with averaging of the same interval for uniform data output.
Undefined intervals are accepted. The interval must be greater than zero in length.
- --flags=TYPE
-
Sets the flags breakdown and output mode. This determines how flags values are written out.
The possible values are:
- --flags=0x
-
Output flags out as hexadecimal numbers with a leading "0x".
- --flags=breakdown
-
Output flags as a series of zero or one values corresponding to each possible flag.
- --flags=default
-
Output flags out like a conventional value.
- --flags=list
-
Output flags as a list of flag names in the field.
- --format=STRING
-
If set then this is the format (like printf(2) or of the form "000.000") used to output all numeric values.
- --header-cut[=BOOLEAN]
-
If set then a line with a cut size for the whole column is included in the header.
- --header-description[=BOOLEAN]
-
If set then a line with field descriptions is included in the header.
- --header-flags[=BOOLEAN]
-
If set then lines for each possible flag will be included in the header.
- --header-mark=STRING
-
If set then this is added to the start of all header lines.
- --header-mvcs[=BOOLEAN]
-
If set then a line with field missing value codes is included in the header.
- --header-names[=BOOLEAN]
-
If set then a line with field names is included in the header.
- --header-names-stderr[=BOOLEAN]
-
If set then a line with field names is output to standard error (regardless of invocation context).
- --header-wavelength[=BOOLEAN]
-
If set then a line with field wavelengths (including possibly time dependent information) is included in the header.
- --join=TYPE
-
This is the base field joining mode. The mode set here determines the defaults used to join together the output. Those options can be overridden individually as well.
The possible values are:
- --join=csv
-
Standard comma separated values. Fields are joined together with a comma (ASCII 0x2C) and quoted with a double quote (ASCII 0x22).
- --join=space
-
Space separated values. Fields are joined together with a space (ASCII 0x20) and quoted with double quotes (ASCII 0x22).
- --join=tab
-
Tab separated values. Fields are joined together with a tab (ASCII 0x09).
- --join-delimiter=STRING
-
If set then this overrides the string used to delimit the join between multiple fields.
- --join-quote=STRING
-
If set then this overrides the string added to the start and end of a field if it needs to be quoted.
- --join-quote-escape=STRING
-
If set then this overrides the string used to replace all instances of the quote string within a quoted field.
Default: The quote string repeated three times
- --mean[=BOOLEAN]
-
If set then the primary data columns will contain the unweighted means when they are available instead of coverage weighted and/or difference measurements.
- --mvc=TYPE
-
This is the MVC output style. Missing or undefined values are written in accordance with this style. Cannot be used with a direct specification of the MVC.
The possible values are:
- --mvc=NA
-
Output the string "NA" for missing values.
- --mvc=blank
-
Output an empty field for missing values.
- --mvc=default
-
Use the default MVC for each value.
- --mvc-all=STRING
-
If set then this is the MVC string used for all values. Cannot be used with a MVC mode specification.
- --mvc-flag=TYPE
-
Sets the missing value flag mode. If enabled missing value flags are fields that are either zero or one to indicate if the value corresponding to the flag was defined in the line.
The possible values are:
- --mvc-flag=disable
-
Disable MVC flags.
- --mvc-flag=end
-
Output MVC flags at the end of the record in the same order as the prior fields in the record.
- --mvc-flag=follow
-
Output MVC flags immediately following the value they correspond to.
- --numeric-only[=BOOLEAN]
-
If set then only values with a numeric representation will be output, all others will be considered missing.
- --paths[=BOOLEAN]
-
If set then all complex values have their component paths output separately, resulting in one column for each path.
- --quantiles[=BOOLEAN]
-
If set then columns for the standard quantiles (0.00135, 0.00621, 0.02275, 0.05, 0.06681, 0.15866, 0.25, 0.30854, 0.5, 0.69146, 0.75, 0.84134, 0.93319, 0.95, 0.97725, 0.99379, and 0.99865) of the data in the average will be output.
- --squash=INTERVAL
-
This sets a limit below which data are combined into a single record. That is, if two variables have values separated by less than this threshold, they will be combined into a single output record.
Undefined intervals are accepted. The interval must be greater than zero in length.
- --station[=BOOLEAN]
-
If set then a station code is included in the output.
- --stddev[=BOOLEAN]
-
If enabled then columns for the standard deviation of the averages will be output.
Default: Enabled
- --time-doy[=BOOLEAN]
-
If set then the output includes the fractional day of year (with January 1 = 1.00000).
- --time-epoch[=BOOLEAN]
-
If set then the output includes the number of seconds since 1970-01-01T00:00:00Z.
- --time-excel[=BOOLEAN]
-
If set then the output includes the date and time in a format recognized by Excel.
- --time-fractional-year[=BOOLEAN]
-
If set then the output includes the fractional year with enough precision to represent seconds.
- --time-iso[=BOOLEAN]
-
If set then the output includes a date-time formatted in compliance with ISO 8601.
- --time-julian-day[=BOOLEAN]
-
If set then the output includes the fractional Julian day. That is the fractional number of days since 12:00 January 1, 4713 BC.
- --time-yeardoy[=BOOLEAN]
-
If set then the output includes the year and fractional day of year (with January 1 = 1.00000) as separate fields.
Arguments
If no bare word input specification is supplied then data are read from standard input.
- station
-
This argument is used to specify the station used to look up variables that do not include a station as part of an archive read specification. The station is the three letter GAW station code of the location, such as
BND
. The argument accepts multiple stations specified as regular expressions and separated by:
or;
or,
. For exampleBND,MLO
and(BND|MLO)
are two ways of selecting both the BND and MLO stations.The special value
allstations
may also be specified to select all stations. - variables
-
This argument may be split into multiple actual program arguments. Each part consists of one or more variable specifications separated by
,
(commas). The station and archive used if none are explicitly given are defined in the other arguments or inferred from system defaults. For example simplyBsB_S11
selects the blue scattering from the S11 instrument and returns all cut sizes and metadata. If instead::BsB_S11:pm10
is used then only PM10 scattering is returned. That is, the selection specifies "any" station and archive (the defaults are set in the other arguments) then requires that it has the "PM10" flavor. A specification such asbnd:BsB_S11,thd:BsB_S11
allows for data selection from multiple stations.This may also be used to select a "record" type alias such as "S11a". In this mode all the variables defined in the alias are returned.
Finally the special value
everything
can be used to select all available data for the given station(s) and archive(s). - times
-
This argument may be split into multiple actual program arguments. This full list of time arguments defines a time bounds list that sets the range of data queried. The final resulting data are clipped to the given time bounds but any values that intersect them are returned.
- archive
-
This argument is used to specify the archive used to look up variables that do not include an archive as part of an archive read specification. The archive is the internal name, such as
raw
orclean_meta
. The argument accepts multiple archives specified as regular expressions and separated by:
or;
or,
. For exampleraw(_meta)?
selects both theraw
andraw_meta
archives.The special value
allarchives
may also be used to select all available archives. - file
-
This argument is used to specify the the file to read data from. If it is present and exists then data is read from the given file name instead of from standard input. Alternatively
-
(a single dash) may be used to explicitly specify standard input.
Examples
- Default MS Excel formatted
-
da.export bnd S11a 2015-05-01 2015-05-03
Or:
da.export input_file.c3d
Or:
da.get bnd S11a 2015-05-01 2015-05-03 | da.export
- CSV with year and DOY
-
da.export --mode=CSV bnd S11a 2015-05-01 2015-05-03
- Excel with station and fractional year
-
da.export --mode=xl --station --time-fractional-year bnd S11a 2015-05-01 2015-05-03