[Esip-documentation] updates to ACDD

Wed Oct 13 17:53:19 EDT 2021

Matthew,

Thanks for this, it simplifies the answers a lot!

Short answer is that (as I understand it) the ACDD conventions are 'managed' by the ESIP Documentation Cluster, and to start the process of adding an attribute to those conventions you would make the request to that cluster. It's my hope that the cluster would relatively quickly form a team and process to support the request. They* would not necessarily have to discuss any other change, and other than formalizing what process they want to follow, it could be quite quick to decide. (Posting updated documents might take longer!)  It could be a valuable exercise for the Documentation Cluster and community, in my humble opinion.

When you say "an attribute explicitly for people identifiers", you are right that ACDD allowed this use but purposefully left it ambiguous in the creator_url. At some point, if you want to make ACDD less ambiguous, the amount of duplicate content starts going up because of backward-compatibility requirements, and maybe you want a new path that's incompatible with all the existing metadata. (That's what we ran into last time.)

You can certainly add a contributor_url, though nothing preclused using a URL for contributor_name. But I agree a URL would be good. (Or maybe these days, an IRI. And be sure you allow multiples! and roles for each! oops, going beyond my mandate :->)

Finally, totally agree the metadata profile doesn't go outside of the ACDD conventions, it's fine really. I just want to encourage individual users to get that uniquely identifiable recognition too!

Very clarified, and appreciate closing the loop on this question. Someone from the Documentation Cluster may wish to comment from this point. 

John

* I'm a member of the Documentation Cluster but have not often had time to participate, alas. 

> On Oct 13, 2021, at 6:06 AM, Mathew Biddle - NOAA Affiliate <mathew.biddle at noaa.gov> wrote:
> 
> Bob, John, and Chris,
> 
> I just joined this list yesterday, so I'm just seeing Bob's response now. I wholeheartedly agree with all of your comments, thank you for walking through your logic on updating existing attributes/definitions. I think there might have been some mischaracterization of what it is Chris was asking for so I'd like to take a step back. 
> 
> The problem: the Animal Telemetry Network (ATN) is looking to collect persistent identifiers for people (preferably using ORCiD, but other options are available). ATN would like to include those identifiers in the netCDF metadata at the appropriate location. So, we did a quick search through ACDD and didn't see an attribute explicitly for people identifiers. So, my first response is that this might be something worthwhile to add to the upstream conventions, how might we do that?
> 
> Where we stand: After discussing with John on the #marinedata slack channel (thanks for monitoring that channel BTW!) I was reminded of the ACDD 1.3 creator_url <https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3#creator_url> attribute, which will suit the ATNs purpose. See that discussion in this ticket <https://github.com/ioos/ioos-atn-data/issues/24#issuecomment-937836068>. However, it would also be beneficial if we could supplement the ACDD 1.3 recommended attributes with a contributor_url attribute which doesn't exist now. This would be an addition to the existing attributes, not a change to definitions or attribute names. So, the question becomes, how do we contribute/start a conversation on adding a new attribute to the ACDD conventions? Is this even possible?
> 
> As for the IOOS Metadata Profile v1.2 description for what goes in the creator_url, I'll discuss it with the team. From how I read it, it's not going outside of the ACDD 1.3 conventions. It's providing more explicit guidance as to how the IOOS community should use it (maybe the creator_type should always be 'institution' for the IOOS profile to make that connection more clear). 
> 
> I hope this clarifies things.
> 
> Thanks everyone for your valuable input.
> 
> Matt
> 
> On Tue, Oct 12, 2021 at 11:49 PM Work Sonic via Esip-documentation <esip-documentation at lists.esipfed.org <mailto:esip-documentation at lists.esipfed.org>> wrote:
> Well said, I agree with all of your statements here. (Well, except for the part where you said "And they are forgetting about all the consumers of data files who have to deal with different versions of a standard that have different attribute names and definitions." I don't think any of us forgot about those people—many of us were those people who had to deal with existing data sets—but we had varying opinions about whether breaking changes were a good idea. In the end, the group decided that for that version, we would not make any breaking changes.)
> 
> I expect Chris has a pretty clear picture at this point! :-)
> 
> Thanks Bob for the input.
> 
> John
> 
>> On Oct 12, 2021, at 6:38 AM, Bob Simons - NOAA Federal <bob.simons at noaa.gov <mailto:bob.simons at noaa.gov>> wrote:
>> 
>> Regarding "IOOS had recommended creator_url be ...":
>> IOOS can recommend whatever they want, but it doesn't change the ACDD 1.3 definition of "creator_url" which is 
>> "The URL of the person (or other creator type specified by the creator_type attribute) principally responsible for creating this data." [emphasis added]
>> and which seems to be directly at odds with the IOOS recommendation.
>> 
>> Regarding "at least one major user would not accept any changes to existing ACDD attributes that would invalidate any use that followed a previous version" and the desire of some people to make major changes to ACDD:
>> The person resisting changes to existing attribute names and definitions was me. I think the reasons for that should be obvious:
>> This community of NOAA (especially NCEI with its archive), NASA, and hundreds of other groups, has 100's of thousands of datasets and 100's of millions of files using the ACDD terms as defined in the various versions 1.0 - 1.3 of the ACDD standard. If you change one of the attribute names or definitions:
>> At best you introduce a complication (a file's reader has to be aware of the difference between ACDD versions and interpret the attribute differently according to the stated ACDD version). That means crosswalks to other metadata formats need to be more sophisticated in order to deal with the differences between ACDD versions. That might not be too bad for the changes in one version of ACDD, but what about after 5,6,7 versions of ACDD? 
>> You introduce uncertainty: Did the file's creator properly understand the differences between how the attribute was different versions of ACDD? This makes the crosswalks much harder to write.
>> You can see both of those problems already (although with minor consequences) with the pointless change in the spelling of the "acknowledgment" (the US spelling of the word) in pre-1.3 versions of ACDD to "acknowledgement" (the British spelling) in ACDD 1.3. It is now common to see files claiming to be ACDD 1.3 compliant with "acknowledgment" (incorrect) and others using "acknowledgement" (correct).  
>> It should be obvious that if a definition changed (instead of the spelling of the attribute name), it would be a far more complex situation where the reader must then guess what the file creator intended.
>> 
>> This situation highlights something else: there is a huge difference between the perspective of a person creating a new data file for a new dataset, and a person reading files from various sources. A person creating a new data file for a new dataset has no prior constraints. All they want to do is express the metadata content into the file using the standard. But everyone and every project has different needs. For them, it's easy to get frustrated with a standard because it doesn't fit their idea of what the perfect metadata standard would be. Given a blank slate and working alone, everyone would create a different standard. The hard part of making the standard (and it was hard) is that we had to reconcile all these different ideas about what the perfect metadata standard would be. So "perfect" gets thrown out (since it is impossible) and "acceptable compromise" is the best we can hope for. (Yes, as my wife says, "compromise is when nobody is happy".) People making ACDD 1.3 had very diverse ideas about what topics should be addressed, what the attribute names should be, and especially what the definitions should be. Unfortunately, some people naturally retain this idea that we should revamp the standard, but they are forgetting about all the other people who would revamp the standard in a different way. And they are forgetting about all the consumers of data files who have to deal with different versions of a standard that have different attribute names and definitions
>> 
>> As a great example of not changing attribute names or definitions but just adding new attribute names and definitions, look at CF, which has been quite stable through 9(?) versions. The result is that a writer of a file with CF metadata can reliably write attributes to the file as they have been for years (although periodically adding new attributes to their vocabulary), and a reader of a file with CF metadata can pretty reliably ignore the stated CF version and just see what attributes are present. If a given attribute is present, it's definition is known. Thank goodness!
>> 
>> Ethan Davis had the remarkable luxury of having a blank slate and (I think) of working alone when he created ACDD 1.0. But now, forever more, new versions of ACDD will be painfully hashed out by numerous people working toward an acceptable compromise. On behalf of other consumers of data files and on behalf of software developers who write software that processes data files, please, please, please, let's keep existing attribute names and definitions stable. If you want to add new attribute names and definitions to address new concepts, go for it. 
>> 
>> Here's a compromise (but where everyone is happy) for all of you who really want to make massive changes to ACDD (essentially starting from a blank slate): go for it! Create your own metadata standard (just as Ethan did with ACDD 1.0), but just give it a different name, not "ACDD". After all, that is what your new metadata standard is: a new metadata standard. If it is a great standard, as ACDD 1.0 was, it will fill a niche and be widely adopted and possibly supplant ACDD (if it addresses the same issues). But effectively killing off the current ACDD 1.x by labelling your new and very different standard ACDD 2.0, is wrong unless everyone in the ESIP Documentation Cluster agrees that it is time to kill off ACDD 1.x and go down that different route (and you probably won't get my vote).  I like ACDD (1.0 to 1.3) and its stability is incredibly valuable to me. I have a lot invested in ACDD 1.3.. I know I'm not alone -- think of NASA and NCEI with millions of archived files with ACDD 1.x metadata and all the software that writes and reads these files.
>> 
>> Best wishes.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Mon, Oct 11, 2021 at 4:33 PM John Graybeal via Esip-documentation <esip-documentation at lists.esipfed.org <mailto:esip-documentation at lists.esipfed.org>> wrote:
>> Hi Chris,
>> 
>> I believe there have been 3, maybe 4 threads in the past 2-3 years about updating ACDD. I wouldn't say any of them were as action-oriented as yours—sometimes interest in a particular attribute, other times general interest in whether it's being maintained. None has gone so far as to say "I want to open a new round of discussion for ACDD." The list archive may have some details. 
>> 
>> I note that nothing *precludes* your using those identifiers for the people or organizations in the contributor attributes, all those identifiers can be named via URLs, which is consistent with the ACDD spec. What are you trying to do exactly that isn't already possible?
>> 
>> Within the past week, there was a question in the #marinedata Slack channel of the ESIP workspace about ORCiDs in netCDF, followed by a long discussion about the ACDD 1.3 creator_url. In the course of that discussion it was mentioned that IOOS had recommended creator_url be
>>> The URL of the institution that collected the data. Note that this should always reference an institution URL, and not a personal URL, even if creator_type=person.
>> 
>> I wasn't there so I can't fairly assess that guidance, but it is sufficiently, umm, unexpected that it'd be nice to get your needs met by ACDD directly.
>> 
>> That said, two considerations about ACDD: (1) At the last update round, at least one major user would not accept any changes to existing ACDD attributes that would invalidate any use that followed a previous version. So our ability to update certain fields the way many members wanted to was effectively blocked. (2) With ESIP's Documentation Cluster(Committee?) as the current 'standards body' for ACDD, you'd be starting down a path that has not been travelled yet, to my knowledge.
>> 
>> I hope that is the right level of information to share in response to your query! I think it would be great for ACDD to get another round, especially if it was clear that a break from the past was necessary to improve metadata quality from what the current standard can support. Obviously that would open up quite a number of questions that just might go beyond your own interest. ;-)
>> 
>> John
>> 
>>> On Oct 11, 2021, at 1:02 PM, Chris Turner via Esip-documentation <esip-documentation at lists.esipfed.org <mailto:esip-documentation at lists.esipfed.org>> wrote:
>>> 
>>> Hello all,
>>> 
>>> I'm curious about any movement or interest to update the ACDD. I know that v1.3 is 6 years old, and the ESIP wiki makes it looks like there hasn't been interest or discussion in this since 2017. Is there still any intent to develop v2.0?
>>> 
>>> My sudden interest in this comes from the need to include identifiers (ORCID, ResearchID, AuthorID, etc) for the person listed in the creator attributes and the people listed in the contributor attributes. I'd like to do this in a community-vetted way, but but if there isn't an active community working on ACDD anymore, we can look at including these attributes in one of the netCDF community profiles - probably the the IOOS metadata profile <https://ioos.github.io/ioos-metadata/ioos-metadata-profile-v1-2.html>. 
>>> 
>>> Thanks for whatever you can tell me.
>>> 
>>> - Chris
>>> 
>>> 
>>> -- 
>>> Chris Turner
>>> Data Librarian | Axiom Data Science
>>> chris at axiomdatascience.com <mailto:chris at axiomalaska.com>
>>> _______________________________________________
>>> Esip-documentation mailing list
>>> To start a new topic: Esip-documentation at lists.esipfed.org <mailto:Esip-documentation at lists.esipfed.org>
>>> To unsubscribe and manage prefs: https://lists.esipfed.org/mailman/listinfo/esip-documentation <https://lists.esipfed.org/mailman/listinfo/esip-documentation>
>> 
>> ----------------------
>> John Graybeal
>> Administrator—ESIP Community Ontology Repository
>> jbgraybeal at sonic.net <mailto:jbgraybeal at sonic.net>
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Esip-documentation mailing list
>> To start a new topic: Esip-documentation at lists.esipfed.org <mailto:Esip-documentation at lists.esipfed.org>
>> To unsubscribe and manage prefs: https://lists.esipfed.org/mailman/listinfo/esip-documentation <https://lists.esipfed.org/mailman/listinfo/esip-documentation>
> 
> 
> _______________________________________________
> Esip-documentation mailing list
> To start a new topic: Esip-documentation at lists.esipfed.org <mailto:Esip-documentation at lists.esipfed.org>
> To unsubscribe and manage prefs: https://lists.esipfed.org/mailman/listinfo/esip-documentation <https://lists.esipfed.org/mailman/listinfo/esip-documentation>
> 
> 
> -- 
> Mathew Biddle, Data Management Analyst
> NOAA/NOS
> US Integrated Ocean Observing System Office
> 1315 East-West Highway
> Silver Spring MD 20910
> 
> ORCiD: 0000-0003-4897-1669 <https://orcid.org/0000-0003-4897-1669>
> Contractor, Integrated Systems Solutions
> http://www.ioos.noaa.gov/ <http://www.ioos.noaa.gov/>
----------------------
John Graybeal
Administrator—ESIP Community Ontology Repository
jbgraybeal at sonic.net

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.esipfed.org/pipermail/esip-documentation/attachments/20211013/bc0d568e/attachment-0001.htm>