[Esip-documentation] on documenting instrument and platform metadata
Jim Biard
jbiard at cicsnc.org
Thu Mar 19 13:46:07 EDT 2015
All,
I have to miss the meeting today as well. John, the three arenas
definitely overlap, but in my thinking a given metadatum is often more
strongly associated with one arena in comparison with the others. As an
example, in my mind the units for a variable are strongly associated
with use, only weakly associated with discovery, and almost irrelevant
to provenance. As another example, I think CF focuses almost entirely on
use, with a node to provenance and a shrug towards discovery.
Grace and peace,
Jim
On 3/19/15 12:58 PM, John Graybeal wrote:
> I have always seen the Venn diagram of these (provenance, use,
> discovery) as overlapping. Interesting to pursue further.
>
> If there is a call today I'm afraid i will have to miss it.
>
> John
>
> On Mar 19, 2015, at 9:00 AM, Ge Peng - NOAA Affiliate via
> Esip-documentation <esip-documentation at lists.esipfed.org
> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>
>> Actually, I think that Jim’s suggestion of developing ACUP - the
>> Attribute Conventions for Usage and Provenance is a great idea.
>> Hopefully, it will be picked up by someone and developed by the group
>> in not so distant future.
>>
>>
>> --- Peng
>>
>>
>> On Thu, Mar 19, 2015 at 11:25 AM, Jim Biard via Esip-documentation
>> <esip-documentation at lists.esipfed.org
>> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>>
>> Hi.
>>
>> I think that Nan's last comment is well worth considering in
>> relation to ACDD. Some metadata are focused more on provenance,
>> and some are focused on discovery. The primary purpose of ACDD is
>> to capture discovery metadata, not provenance metadata. That
>> doesn't mean that provenance metadata is unimportant - it can, in
>> fact, be critical to proper use, but that doesn't mean that ACDD
>> needs to address it.
>>
>> Perhaps someone should start developing ACUP - the Attribute
>> Conventions for Usage and Provenance. (grin)
>>
>> Grace and peace,
>>
>> Jim
>>
>>
>> On 3/17/15 11:30 AM, Nan Galbraith via Esip-documentation wrote:
>>> Hi Aaron -
>>>
>>> Having been involved in the oceansites specification on this, I
>>> agree that
>>> no one solution always works. I adopted (and try to follow, and
>>> promote)
>>> the NODC templates as closely as possible. Their development of
>>> the ancillary
>>> variable 'instrument' was brilliant, in my opinion.
>>>
>>> My datasets range from met files where all the data is from a single
>>> logger but each of 6 or 8 variables might need its sensor
>>> described at
>>> the variable level, to depth merged temperature files where
>>> there's only
>>> one data variable but 30 instruments (and 4 or 5 different
>>> models). In the met
>>> files, variable attributes would work, but I use 'instrument'
>>> ancillary variables
>>> so I can link data variables recorded by a single sensor module
>>> (e.g. Inst_HRH
>>> is linked to data variables HRH and ATMP).
>>>
>>> The depth merged files show the real beauty of the instrument
>>> variable, though;
>>> since it can have a depth dimension, you can describe the
>>> different instruments
>>> deployed at different depths. There's no good way to do that
>>> using attributes, as
>>> far as I've seen - lists break down, especially when data is
>>> sub-sampled i.e. in thredds.
>>>
>>> And, although much of this isn't yet at a stage where software
>>> can use it, any and all
>>> information that you have about the sensors and platform should
>>> still be recorded
>>> wherever it seems most appropriate to you, since it might
>>> otherwise be lost. Re-formatting
>>> this kind of info (if and when a useful standard emerges) will
>>> be much less effort
>>> than trying to track down something that was not recorded
>>> because there didn't
>>> seem like a standard way to encode it.
>>>
>>> If ACDD can't find and understand instrument variables yet,
>>> maybe that's something
>>> that could be looked at in another rev. Or, maybe instruments
>>> aren't a common
>>> discovery concept - I really don't know.
>>>
>>> Cheers -
>>> Nan
>>>
>>>
>>> On 3/16/15 6:41 PM, Aaron Sweeney via Esip-documentation wrote:
>>>> Thanks very much for the guidance, John. I think I may stick
>>>> with the NODC or OceanSites platform and instrument variables
>>>> to define the particular instances.
>>>>
>>>> That being said, for my two use cases, I think I can still
>>>> capture the type of platform and instrument in the ACDD global
>>>> attributes. This amounts to:
>>>>
>>>> :platform = "DART"
>>>> :platform_vocabulary = "NASA/GCMD Platform Keywords.
>>>> Version 8.0"
>>>> :instrument = "BOTTOM PRESSURE GAUGES"
>>>> :instrument_vocabulary = "NASA/GCMD Instrument Keywords.
>>>> Version 8.0"
>>>>
>>>> and
>>>>
>>>> :platform = "COASTAL STATIONS"
>>>> :platform_vocabulary = "NASA/GCMD Platform Keywords.
>>>> Version 8.0"
>>>> :instrument = "TIDE GAUGES"
>>>> :instrument_vocabulary = "NASA/GCMD Instrument Keywords.
>>>> Version 8.0"
>>>>
>>>> Cordially,
>>>> Aaron
>>>>
>>>> On 03/16/2015 02:43 PM, John Graybeal wrote:
>>>>> What great questions! The following thoughts are my own, and
>>>>> are unrelated to my previous ACDD activities.
>>>>>
>>>>> Clearly some data files will have data from multiple platforms
>>>>> and instruments; whereas others will get all their variables
>>>>> from a single platform/instrument; and some will have data so
>>>>> heavily processed as to be barely traceable to the source
>>>>> platforms/instruments. Each of the specifications is oriented
>>>>> toward somewhat different scenarios, and none of them are yet
>>>>> great at representing all the scenarios.
>>>>>
>>>>> Let's consider ACDD first. Note that this specification
>>>>> provides _recommendations_ for attributes, not _requirements_;
>>>>> and the platform and instrument attributes are only
>>>>> 'Suggested'. These give a broad-brush categorization of the
>>>>> platform(s) or instrument(s) that collected a data set; if it
>>>>> seems reasonable to specify that information for the whole
>>>>> data set, use these attributes to do so. (If it isn't
>>>>> useful/applicable, then don't worry about these attributes.)
>>>>> The most precise way to give that info is to use a controlled
>>>>> vocabulary (see
>>>>> https://marinemetadata.org/references?filter0=**ALL**&filter1%5B%5D=129&filter1%5B%5D=141
>>>>> <https://marinemetadata.org/references?filter0=**ALL**&filter1[]=129&filter1[]=141> for
>>>>> a partial list of vocabularies), and typically I'd expect a
>>>>> type vocabulary here (with e.g. "CTD", "Mooring") not a
>>>>> specific instance.
>>>>>
>>>>> If you *are* following ACDD, I don't recommend putting the
>>>>> names of variables containing information on platforms and
>>>>> instruments in the global attributes 'platform' and
>>>>> 'instrument'. Without ACDD describing that as a possibility,
>>>>> it's highly unlikely any software will know how to treat that,
>>>>> and some people will get confused too. The existence of the
>>>>> variables will be pretty self-explanatory, so just leave out
>>>>> those ACDD attributes. (I see this conflicts with NODC and
>>>>> OceanSITES recommendations, to my surprise; perhaps others
>>>>> will offer another view here.)
>>>>>
>>>>> Whether to put the information in a global instrument
>>>>> variable, or attributes of each data variable, or both, seems
>>>>> to me a matter of circumstances and taste. Let's stipulate the
>>>>> best solution is human-recognizable, computer-parseable,
>>>>> extensible, and unambiguous. Thus, global attributes are
>>>>> inadequate. Assuming a sophisticated data system 5 years from
>>>>> now would look in both variables and data attributes, then the
>>>>> model of 'one variable per instrument or platform', and
>>>>> association from the data variables to those variables or
>>>>> platforms seems best.
>>>>>
>>>>> But given that the examples in each of the different profiles
>>>>> still seems inconsistent, likely you should follow the
>>>>> guidance most close to your work. If you aren't particularly
>>>>> close to any of them, I'd do it a bit differently still. But
>>>>> I'll hold that thought unless you want to extend the
>>>>> discussion in that direction.
>>>>>
>>>>> John
>>>>>
>>>>>
>>>>> On Mar 16, 2015, at 11:56, Aaron Sweeney via
>>>>> Esip-documentation <esip-documentation at lists.esipfed.org
>>>>> <mailto:esip-documentation at lists.esipfed.org>> wrote:
>>>>>
>>>>>> Hi, folks,
>>>>>>
>>>>>> I’m struggling with how to capture platform and
>>>>>> instrument information in the appropriate places in netCDF
>>>>>> files. The NODC templates suggest creating separate
>>>>>> variables for platforms and instruments, with multiple
>>>>>> variable attributes (make, model, serial number, precision,
>>>>>> accuracy, etc.), and associating these platform and
>>>>>> instrument variables with physical variables via an
>>>>>> “ancillary_variable” variable attribute attached to the
>>>>>> physical variable. Along these lines, SeaDataNet suggests
>>>>>> including variable attributes for instrument
>>>>>> (sdn_instrument_urn and sdn_instrument_name), and OceanSites
>>>>>> suggests either the use of a separate instrument variable
>>>>>> (with make, model, serial number attributes) or capturing
>>>>>> make, model and serial number as variable attributes
>>>>>> contained directly within the relevant physical variable.
>>>>>> But ACDD-1.3 only has single /global/ attributes for platform
>>>>>> and instrument (and vocabularies). The NODC templates suggest
>>>>>> placing the names of the variables containing information on
>>>>>> platforms and instruments in the global attributes:
>>>>>> instrument and platform. I like NODC’s and OceanSites’
>>>>>> “associative” approach (i.e. “this instrument goes with this
>>>>>> physical variable”), but the use of variable names in global
>>>>>> attributes conflicts with ACDD-1.3.
>>>>>>
>>>>>> It seems that in order to comply with ACDD-1.3, I need
>>>>>> to capture instruments and platforms at a global level
>>>>>> (dropping their association with physical measurement
>>>>>> variables), but in order to comply with OceanSites and
>>>>>> others, I need to capture instrument and platform information
>>>>>> in separate variables and associate these with physical
>>>>>> measurement variables via an ancillary_variable attribute.
>>>>>>
>>>>>> In order to comply with both, in the interest of
>>>>>> enabling interoperability, I seem to need to repeat
>>>>>> instrument and platform metadata in two different places.
>>>>>>
>>>>>> Any thoughts or guidance?
>>>>>>
>>>>>> References:
>>>>>> NODC netCDF timeseries template:
>>>>>> http://www.nodc.noaa.gov/data/formats/netcdf/v1.1/timeSeriesOrthogonal.cdl
>>>>>> OceanSITES Data Format Reference Manual (See Appendix 2 for
>>>>>> sensor variables):
>>>>>> http://www.oceansites.org/docs/oceansites_data_format_reference_manual.pdf
>>>>>> SeaDataNet Data Transports Manual (See Section 4.2.2
>>>>>> Co-ordinate Variables):
>>>>>> http://www.seadatanet.org/content/download/16251/106283/file/SDN2_D85_WP8_Datafile_formats.pdf
>>>>>>
>>>>>> Thanks,
>>>>>> Aaron
>>>
>>> --
>>> *******************************************************
>>> * Nan Galbraith Information Systems Specialist *
>>> * Upper Ocean Processes Group Mail Stop 29 *
>>> * Woods Hole Oceanographic Institution *
>>> * Woods Hole, MA 02543(508) 289-2444 <tel:%28508%29%20289-2444> *
>>> *******************************************************
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Esip-documentation mailing list
>>> Esip-documentation at lists.esipfed.org <mailto:Esip-documentation at lists.esipfed.org>
>>> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
>>
>> --
>> <hjdfgbcg.png> <http://www.cicsnc.org/> Visit us on
>> Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC
>> <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA's National Climatic Data Center <http://ncdc.noaa.gov/>
>> 151 Patton Ave, Asheville, NC 28801
>> e: jbiard at cicsnc.org <mailto:jbiard at cicsnc.org>
>> o: +1 828 271 4900 <tel:%2B1%20828%20271%204900>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Esip-documentation mailing list
>> Esip-documentation at lists.esipfed.org
>> <mailto:Esip-documentation at lists.esipfed.org>
>> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
>>
>>
>>
>>
>> --
>> *Ge Peng, Ph.D*
>> *Research Scholar*
>> Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
>> North Carolina State University <http://ncsu.edu/>
>> NOAA's National Climatic Data Center <http://ncdc.noaa.gov/>
>> 151 Patton Ave, Asheville, NC 28801
>> ge.peng at noaa.gov <mailto:ge.peng at noaa.gov>
>> o: +1 828 257 3009
>> f: +1 828 257 3002
>>
>> Following CICS-NC on Facebook <http://www.facebook.com/cicsnc>
>>
>>
>>
>>
>> _______________________________________________
>> Esip-documentation mailing list
>> Esip-documentation at lists.esipfed.org
>> <mailto:Esip-documentation at lists.esipfed.org>
>> http://lists.deltaforce.net/mailman/listinfo/esip-documentation
--
CICS-NC <http://www.cicsnc.org/> Visit us on
Facebook <http://www.facebook.com/cicsnc> *Jim Biard*
*Research Scholar*
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA's National Climatic Data Center <http://ncdc.noaa.gov/>
151 Patton Ave, Asheville, NC 28801
e: jbiard at cicsnc.org
o: +1 828 271 4900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.deltaforce.net/pipermail/esip-documentation/attachments/20150319/ec3c66d5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: aagcejaf.png
Type: image/png
Size: 11847 bytes
Desc: not available
URL: <http://lists.deltaforce.net/pipermail/esip-documentation/attachments/20150319/ec3c66d5/attachment-0001.png>
More information about the Esip-documentation
mailing list