[Esip-documentation] Let's get rid of spatial and temporal bounds in ACDD

Thu Mar 20 17:45:47 EDT 2014

Trying to move forward on an area I think there's rough agreement -- it will be a good thing if software that creates data also creates good metadata -- I've worked with Ted to start a wiki page to list those services. In the short term this provides some additional information to consider the status and possible ways forward; it may prove useful in the longer term, for the community to understand what to expect with the different utilities.

The page is at http://wiki.esipfed.org/index.php/NetCDF_Utilities_Metadata_Handling, and as you can see it is mostly not filled out. Those who find it helpful are welcome to add information to the document, or make other constructive changes as you see fit. Thanks!

John

On Mar 20, 2014, at 09:24, Steve Hankin <steven.c.hankin at noaa.gov> wrote:

> Hi Ken, et. al.,
> 
> I am grateful for your suggestion that the topic be raised and discussed.  Now, can that discussion be broadened to include alternative strategies by which you might achieve your goals (and hopefully avoid the down sides)?
> 
> Consider the suggestion that Dave Neufeld just offered:  "server technologies [should] implement a caching/refresh strategy (Hyrax/THREDDS) for geospatial and temporal bounds".  This is a recognition that the static geo-positioning attributes are unreliable and redundant (can readily be recomputed).  Why not take this insight to the next step:  recommend that servers synthesize the global bounds.  Between ncISO, ncdump, and the major Web servers you'll be close to a clean sweep of the use cases where the global attributes contribute significant value.  (They make negligible contributions to users of desktop analysis and visualization tools.)
> 
> There is of course a down side to this approach:  what is the leverage to force these software changes to be made?  The current strategy offers more leverage, because it breaks CF software, making the software look bad and the Web practitioners look sloppy.  (I've already railed against that.)  Following this strategy will require getting some major players on board through persuasion.
> 
>     - Steve
> 
> =====================================================
> 
> On 3/20/2014 4:17 AM, Kenneth S. Casey - NOAA Federal wrote:
>> Hi All - 
>> 
>> Perhaps this group could lay out a simple "proposal" of sorts… that could be discussed and refined in this thread, and agreed to at ESIP Rocky Mountain High this summer if not sooner.  Perhaps that proposal would look something like:
>> 
>> "Dear Software Providers:  Please do the right thing with global attributes, and properly update spatial and temporal bounding attributes when you modify a netCDF file and either re-write or create a new one.  While you are at it, add some info to the history attribute too like you are supposed to.  In the meantime, dear community, be wary of global attributes that relate to coordinate variables.. trust the coordinate variables and if you notice a discrepancy with their corresponding global attributes SCREAM VERY LOUDLY at the provider of the software which generated that netCDF file."
>> 
>> Specific actions could then be requested of the big players to make the appropriate updates to their code. 
>> 
>> I think we need global attributes in general, even ones relating to coordinate variables.  Everything said here about coordinate attributes actually applies more generally…  many, many, of the global attributes can and should be updated depending on the provenance of the file and who did what to it.  The only difference is that the attributes relating to coordinate variables can actually be tested against the data.
>> 
>> I'd add one other point… while computationally doing a max/min on the coordinate variables is not too terrible, much of the time (esp. with netCDF-3) you have to decompress the entire file first, and that is computationally terrible for large numbers of large files that are externally compressed (like we have with GHRSST, for example… loving that GHRSST Data Specification v2 now uses netCDF-4 with internal compression!).
>> 
>> Ken
>> 
>> 
>> 
>> On Mar 19, 2014, at 11:12 PM, "Signell, Richard" <rsignell at usgs.gov> wrote:
>> 
>>> Gang,
>>> I understand the importance having the bounds information in metadata
>>> -- in fact we start our workflows by querying catalog services which
>>> uses bounding box information contained in the ISO metadata.  But this
>>> ISO metadata was calculated by ncISO by reading the CF coordinate
>>> variables via OPeNDAP, and the metadata points to the OPeNDAP service
>>> endpoint, so I know that the bounds data is correct.
>>> 
>>> It would seem that NASA, OCEANSITES, and others could use this
>>> approach as well, which would yield the same functionality as reading
>>> metadata from the actual dataset, but without the drawbacks.
>>> 
>>> Having read all the arguments so far,  I'm going to continue
>>> recommending that people not write these bounds attributes into their
>>> datasets, because I remain convinced they do our community more harm
>>> than good.  But I'll explain to them the arguments for and against.
>>> 
>>> -Rich
>>> 
>>> On Wed, Mar 19, 2014 at 5:43 PM, Armstrong, Edward M (398M)
>>> <Edward.M.Armstrong at jpl.nasa.gov> wrote:
>>>> Hello,
>>>> 
>>>> Just to continue this thread and the way a popular tool works......I checked the
>>>> output of LAS and it does not update any attributes...just inherits what it
>>>> natively subsetted.  Its includes this global attribute:
>>>> 
>>>> :FERRET_comment = "File written via LAS. Attributes are inherited from
>>>> originating dataset";
>>>> 
>>>> 
>>>> 
>>>> On Mar 18, 2014, at 10:42 AM, Ted Habermann <thabermann at hdfgroup.org> wrote:
>>>> 
>>>> All,
>>>> 
>>>> Just wanted to point out that CF is really use metadata used by tools that
>>>> are actually reading the data. ACDD are discovery conventions originally
>>>> motivated by the lack of discovery information included in CF. The opinion
>>>> of the CF mailing list is, therefore, not relevant in this discussion.
>>>> 
>>>> I agree with Ken and Ed... There are two sets of metadata because they serve
>>>> two communities. We need to do our best to make sure they are both correct.
>>>> 
>>>> Seems like the netCDF download service in THREDDS is a good place to
>>>> start... Does anyone know how it behaves with global attributes? What about
>>>> LAS?
>>>> 
>>>> Ted
>>>> 
>>>> On Mar 18, 2014, at 10:52 AM, Armstrong, Edward M (398M)
>>>> <Edward.M.Armstrong at jpl.nasa.gov> wrote:
>>>> 
>>>> Hi Steve,
>>>> 
>>>> I do agree with Ken and Ted on this issue....also from experience working with
>>>> granules in a large data center.
>>>> 
>>>> I think the crux of the counter argument is that from the perspective of the
>>>> data producer we want them to add more metadata, not less to the native
>>>> granule.  And many people do use these global bounds in some context, mostly
>>>> just looking at the granules through a browser like ncdump and Panoply.
>>>> Space and time bounds is just a natural thing most people look at in a
>>>> granule when they first acquire it.
>>>> 
>>>> The key, then is to recognize if the granule has been altered by a tool, to
>>>> treat the bounds with caution.  Nan had a great idea that tools should point
>>>> back to the original unaltered granule.  I think this is first requirement
>>>> for any tool that does modification to the granule via subsetting or
>>>> similar.
>>>> 
>>>> And the next step is to encourage the updating of global bounds by any tool
>>>> or aggregation operations. I do agree that is a challenge.
>>>> 
>>>> 
>>>> On Mar 18, 2014, at 8:51 AM, Steve Hankin <steven.c.hankin at noaa.gov> wrote:
>>>> 
>>>> Hi Ken,
>>>> 
>>>> I think you are actually agreeing with me, rather than Ted.  The text that I
>>>> proposed did not say geo-bounds attributes should be forbidden.  It said
>>>> that they should be used with caution -- only in situations where they are
>>>> unlikely to lead to corruption.
>>>> 
>>>> It is recommended that 1) the use of these global attributes be restricted
>>>> to files whose contents are known to be completely stable -- i.e. files very
>>>> unlikely to be aggregated into larger collections;  and 2) as a matter of
>>>> best practice, software reading CF files should ignore these global
>>>> attributes; instead it should compute the geo-spatial bounds by scanning the
>>>> coordinate ranges found within the CF dataset, itself.
>>>> 
>>>> These words can certainly be adjusted and improved.  (The discussion that I
>>>> am still hoping will happen!)  The intent is to apply common sense,
>>>> practical thinking so that our interoperability frameworks works in
>>>> practice.
>>>> 
>>>> You have argued that at NODC you *need* these geo-bounds attribute in order
>>>> to avoid impossibly large processing burdens of examining the data, itself.
>>>> What would your strategy be if  the attributes proved not to be trustworthy?
>>>> Perhaps within the community of satellite swath folks you can agree that you
>>>> will all maintain these attributes faithfully.  Great.  Do so.  That is in
>>>> the spirit of the discussion that we should be having.  But should you speak
>>>> for the modeling community, too?  (I presume the NODC granules must be swath
>>>> data, because if they were grids, then the processing required to determine
>>>> the bounds from the data is negligible.)
>>>> 
>>>> Lets look at the context of this discussion.  What we are seeing here is a
>>>> collision between the priorities of two communities involved in developing
>>>> standards. Our emails here are largely confined to one of the communities.
>>>> Envision taking this discussion topic to the CF email list.  I predict what
>>>> you would see is that the potential of  redundant, static global attributes
>>>> to create corruption to the datasets would fly up as a giant red flag.  It
>>>> is not only aggregation operations that will lead to corruption, it is also
>>>> subsetting operations -- the most routine of all netCDF operations, and
>>>> performed by generic utilities that are unaware of CF or ACDD.  These are
>>>> glaring, unsolved problems in the use of these geo-bounds.  There is such a
>>>> strong case to be more nuanced when standardizing attributes that we can
>>>> plainly see are going to lead to many corrupted datasets.  What is the
>>>> counter-argument for ignoring these self-evident problems?
>>>> 
>>>>    - Steve
>>>> 
>>>> ===========================================
>>>> 
>>>> 
>>>> On 3/18/2014 3:48 AM, Kenneth S. Casey - NOAA Federal wrote:
>>>> 
>>>> Hi All -
>>>> 
>>>> I've been silent but following this thread carefully.  Time to jump in now.
>>>> 
>>>> I concur with Ted's statements below.  I would characterize his responses as
>>>> a good example of combating the "Tyranny of the Or". It doesn't have to be
>>>> one solution or the other.  For us at NODC, processing literally tens and
>>>> maybe hundreds of millions of netCDF granules, having easily and quickly
>>>> read global attributes is not a convenience. It is a practical necessity.
>>>> We also like the idea of encouraging softwarians to write better software.
>>>> Corrupting attributes through negligence and inaction is not acceptable.
>>>> Fix it. If they can't, we can encourage our users to stop using that
>>>> software.
>>>> 
>>>> I love the idea of building the congruence checker into the ACDD rubric and
>>>> catalog cleaner, and I think an "ncdump -bounds" option, where the result is
>>>> calculated from the actual bounds, is terrific too.  These kinds of
>>>> additions to exiting tools would help encourage better practices and would
>>>> give us some simple tools to improve our management of netCDF data.
>>>> 
>>>> Ken
>>>> 
>>>> 
> 
> _______________________________________________
> Esip-documentation mailing list
> Esip-documentation at lists.esipfed.org
> http://www.lists.esipfed.org/mailman/listinfo/esip-documentation

John Graybeal
jbgraybeal at mindspring.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.esipfed.org/pipermail/esip-documentation/attachments/20140320/5aa69096/attachment-0001.html>