[Esip-documentation] information on related files

Mon Sep 24 12:37:22 EDT 2018

Additional background for the roundtable.

Sean

From: Esip-documentation <esip-documentation-bounces at lists.esipfed.org> on behalf of Jim Biard via Esip-documentation <esip-documentation at lists.esipfed.org>
Reply-To: Jim Biard <jbiard at cicsnc.org>
Date: Thursday, September 20, 2018 at 6:28 AM
To: "esip-documentation at lists.esipfed.org" <esip-documentation at lists.esipfed.org>
Subject: Re: [Esip-documentation] information on related files

Hi.

THREDDS aggregation won't be of much use will it? Nan is saying that this data may well not be netCDF format.

Jim

On 9/19/18 5:38 PM, Armstrong, Edward M (398G) via Esip-documentation wrote:

Hi Nan:

I reviewed the findings of the 2015 NASA ESDSWG group on Virtual Collections and they were a set of use cases (including similar to yours) with some capabilities for solutions including THREDDS aggregations and event-based  search capability for datasets (and implicitly granules for those datasets). So your proposal on virtual THREDDS aggregations seems a good path forward.  But it would likely need manual curation to maintain as new granules are created.

On 9/19/18, 2:09 PM, "Armstrong, Edward M (398G)" <Edward.M.Armstrong at jpl.nasa.gov><mailto:Edward.M.Armstrong at jpl.nasa.gov> wrote:

    Hi Nan:

    I didn't think about THREDDS  virtual aggregation but that might be an approach that works.....

    A few years ago there was a NASA ESDSWG on this subject (virtual collections).  I'll look back and see if there are any recommendations that could help. I believe this type of THREDDS aggregation was one of them.

    On 9/19/18, 12:47 PM, "Esip-documentation on behalf of Nan Galbraith via Esip-documentation" <esip-documentation-bounces at lists.esipfed.org on behalf of esip-documentation at lists.esipfed.org><mailto:esip-documentation-bounces at lists.esipfed.orgonbehalfofesip-documentation@lists.esipfed.org> wrote:

        Thanks Ed, Dave, and Aleksandar.

        These are all possibilities, although I was hoping there would be an

        attribute

        in some convention, somewhere, that was more specific. Both 'references',

        and 'metadata_link' seem to be intended to provide information about the

        NetCDF

        file in which they're found, and 'external_variables' is meant to list

        'variables which

        are named by attributes in the file'.

        What we were hoping to do was point to external data that was collected

        at the same

        place and time as the data in the file, but which is not IN the file for

        various reasons.

        This might be because it's not time series data, or because it's from

        another domain

        where NetCDF is unknown.

        Making this information machine-readable would require it to be pretty

        specific, and

        might even require more than one attribute - one giving  the location,

        one describing

        the remote content, and maybe one describing how the data sets are related.

        Maybe the terminology that THREDDS uses would be a good place to start.

        I think

        I've got some THREDDS documentation ... somewhere around here. Also,

        NcML has the

        concept of Aggregation (to create "virtual" NetCDF datasets, according

        to the NcML page

        at unidata), but I'm not sure how far that's gone since I last looked at it.

        Thanks again -

        Nan

        On 9/19/18 11:48 AM, Aleksandar Jelenak via Esip-documentation wrote:

        > Hello!

        >

        > There's also this:

        > http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#external-variables

        >

        > Apparently this approach is chosen for the CMIP6 data and is lax on

        > the specifics by design. The context where to find files with those

        > referenced external netCDF variables is to be provided in some

        > unspecified manner.

        >

        > It is possible to include links to related files on THREDDS dataset

        > pages but this is outside of file. The metadata_link keeps the

        > connection in the file but requires that the application code handle

        > ISO XML.

        >

        > -Aleksandar

        >

        >> On Sep 19, 2018, at 11:32 AM, Armstrong, Edward M (398G) via

        >> Esip-documentation <esip-documentation at lists.esipfed.org <mailto:esip-documentation at lists.esipfed.org>

        ><mailto:esip-documentation at lists.esipfed.org>> <mailto:esip-documentation at lists.esipfed.org><mailto:esip-documentation at lists.esipfed.org>> wrote:

        >>

        >> Hi:

        >> I would recommend the ACDD attribute “references”:

        >> references

        >>

        >> Published or web-based references that describe the data or methods

        >> used to produce it. Recommend URIs (such as a URL or DOI) for papers

        >> or other references. This attribute is defined in the CF conventions.

        >>

        >> In CF global attributes there is also a “comment” attribute where

        >> such info could be placed in a generic sense.

        >> I have often thought of this use case too and its an area where ACDD

        >> or CF could be improved.

        >> The metadata_link could point to an ISO 19115 record where multiple

        >> citations could point to other datasets cleanly.

        >> *From:*Esip-documentation on behalf of David Neufeld

        >> *Reply-To:*David Neufeld <david.neufeld at noaa.gov <mailto:david.neufeld at noaa.gov>

        ><mailto:david.neufeld at noaa.gov>> <mailto:david.neufeld at noaa.gov><mailto:david.neufeld at noaa.gov>>

        >> *Date:*Wednesday, September 19, 2018 at 8:18 AM

        >> *To:*Nan Galbraith <ngalbraith at whoi.edu<mailto:ngalbraith at whoi.edu> <mailto:ngalbraith at whoi.edu><mailto:ngalbraith at whoi.edu>>

        >> *Subject:*Re: [Esip-documentation] [thredds] information on related files

        >> Hi Nan,

        >> I'd recommend using a 'metadata_link' attribute.

        >> http://wiki.esipfed.org/index.php/Attribute_Convention_for_Data_Discovery_1-3#Additional_Metadata:_metadata_link_attribute

        >> -Dave

        >> On Wed, Sep 19, 2018 at 8:56 AM, Nan Galbraith <ngalbraith at whoi.edu <mailto:ngalbraith at whoi.edu>

        ><mailto:ngalbraith at whoi.edu>> <mailto:ngalbraith at whoi.edu><mailto:ngalbraith at whoi.edu>> wrote:

        >>> Hi ESIP, NetCDF, and THREDDS folks -

        >>>

        >>> First, I'm sorry for the cross posting. I have a hard time figuring

        >>> out which NetCDF list does what; there are 'best practices' threads

        >>> on several lists, interspersed with install and development threads.

        >>>

        >>> The OceanSITES project would like to find the most useful way to

        >>> provide a link within a NetCDF file to an external dataset.

        >>>

        >>> > Another piece I am missing ... is an optional field, that link to a

        >>> > data DOI. Why? Here's an example: We operate an OceanSITES site off

        >>> > the Cape Verdes and the sediment trap data is ... not included in an

        >>> > netcdf file but processed and stored elsewhere

        >>> >https://doi.pangaea.de/10.1594/PANGAEA.863969

        >>>

        >>> Does THREDDS have a standard way to provide links to related or

        >>> co-located data? Is this something we discussed as part of ESIP's

        >>> ACDD recommendations? I've been searching through my store

        >>> of old messages, and haven't come up with anything.

        >>>

        >>> Thanks very much for any advice on this!

        >>>

        >>> - Nan Galbraith

        >>>

        >>>

        >>>

        --

        *******************************************************

        * Nan Galbraith        Information Systems Specialist *

        * Upper Ocean Processes Group            Mail Stop 29 *

        * Woods Hole Oceanographic Institution                *

        * Woods Hole, MA 02543                 (508) 289-2444 *

        *******************************************************

        _______________________________________________

        Esip-documentation mailing list

        Esip-documentation at lists.esipfed.org<mailto:Esip-documentation at lists.esipfed.org>

        https://lists.esipfed.org/mailman/listinfo/esip-documentation

_______________________________________________

Esip-documentation mailing list

Esip-documentation at lists.esipfed.org<mailto:Esip-documentation at lists.esipfed.org>

https://lists.esipfed.org/mailman/listinfo/esip-documentation

--
[CICS-NC]<http://www.cicsnc.org/>Visit us on
Facebook <http://www.facebook.com/cicsnc>

Jim Biard
Research Scholar
Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/>
North Carolina State University <http://ncsu.edu/>
NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/>
formerly NOAA’s National Climatic Data Center
151 Patton Ave, Asheville, NC 28801
e: jbiard at cicsnc.org<mailto:jbiard at cicsnc.org>
o: +1 828 271 4900

Connect with us on Facebook for climate<https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics<https://www.facebook.com/NOAANCEIoceangeo> information, and follow us on Twitter at @NOAANCEIclimate<https://twitter.com/NOAANCEIclimate> and @NOAANCEIocngeo<https://twitter.com/NOAANCEIocngeo>.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.esipfed.org/pipermail/esip-documentation/attachments/20180924/20486e45/attachment-0001.html>