[ESIP-AQ] CyAir Best Practices - need your review!

Thu May 24 13:36:07 EDT 2012

Dear all,

    very nice! Unfortunately, I will not be able to join the telecon tonight (European time zone) as I am in a meeting in Davos. Therefore, a few review comments in written form:

1)      Either the title should be changed to "... the North American air quality community", or you should make an effort to include the best practices from European initiatives, notably what is done at EEA, and in the MACC and PASODOBLE projects. I do think that this document could be very helpful for the European programs as well, but readers may be "confused" about the scope of the present document, unless this is clearly stated up front.

2)      Section 3.2.1, example 3: we should be able to provide you with a CF-compliant example for a "ragged" station output - we developed a (FORTRAN) tool for sampling gridded model output at specified station locations, and we use this CF-1.6 standard to save the output. If you are interested in this, I can find such a file when I am back in Jülich and, for example, upload it on the ESIP wiki.

3)      3.2.1/3.2.2: I suggest to include the "ICARTT" data format as a de-facto standard for aircraft data (e.g. from NOAA). Tom Ryerson (NOAA Boulder) should be able to help with details and should in fact be involved in this best practices document. [on a side: it would be good if we can extend the WUSTL/JUELICH server to read ICARTT as well]

4)      Another, more general and technical aspect with respect to formats: some data sets can become very large (e.g. high-resolution satellite or model data), so it is important to exchange these data efficiently (for example you will not want to transfer such files as ascii or xml structures) - related (not sure where it fits and not yet implemented): in order not to keep users of interoperable services waiting too long (or at least let them know how long they should expect to wait), we will need a way to estimate the size of a request before processing it. This may fit into 5.2.5 also. Furthermore (this may seem counter-intuitive at first), one should consider "batch mode" options in interoperable applications. I have seen those on some sites, where you go through the selection process, then press a button and you will receive email with a link from where you can get your data (in the interoperable world, this shouldn't be a simple download link, but a link to the web application with the view of the data you selected)

5)      Section 4.2.1: may be worth mentioning that one should not shy away from proposing new names if there is a need for this (even if the process can at times be somewhat tedious) - also, concerning CF, it should be clear that a file can be CF compliant even if it contains variables for which no standard_name exists. The best practice would be to provide these data anyhow, but at the same time initiate the standard_name discussion - then add the standard_name attribute when the name (or its modified version) gets adopted.

6)      Also 4.2.1 - if you like you can also list the Juelich CF compliance checker as an alternative tool: http://repositories.icg.kfa-juelich.de/hg/CFchecker/

7)      At the end of 4.2.1: definitively worth adding a link to EEA's INSPIRE stuff. Backgroun don INSPIRE at http://www.eea.europa.eu/about-us/what/information-sharing-1/inspire-directive, AQ portal at http://www.eionet.europa.eu/aqportal/ - here in particular links at the bottom pointing to "code lists for testing"

8)      Section 5.2.1: maybe worth noting that the choice of protocol should not necessarily be taken by the provider, but rather by the user. This means, if the server(s) support more than one protocol, it can attract more users from different communities with different experiences and backgrounds in data handling. (this is then addressed in 5.2.3, I see)

9)      5.2.2: the community WCS server could soon migrate from sourceforge to a redmine system we have just installed in Jülich. This system offers much additional flexibility for community development, bug and feature tracking, wiki documentation, etc. We shall discuss this with Rudy and Kari very soon and hope to have it in place before the best practices are actually published.

10)   The examples in 5.2.2 implicitly make a distinction between servers and clients which is not mentioned in the text before. For example, the Jülich web interface listed here is the application - the server (listed in 5.2.1) is the server. Perhaps the client-server architecture needs a brief description in the intro, and sections 5.2.1 and 5.2.2 should contain subsections on server(s) and client(s). The same principles apply, but at least the examples differ.

11)   5.2.1/5.2.2: it could be added that, if an open source software doesn't satisfy your needs immediately, one should contact the developers and see if the missing features can be implemented within reasonable time, before deciding not to use this software.

12)   5.2.3: rather than "identify use cases" I would say "define the work flow and product/service specifications". Use case could have a flavor of a one-time demonstration?

13)   5.2.5: user notification is needed not only for decommissioning or migration, but also for upgrading of services (we are just going through this with the MACC server)

14)   6.2.1: "metadata can be provided in a separate file" - I would argue that there should still be at least some very basic metadata included in the file itself, in case the URL referred to in the file header is down for whatever reason. At least a title and contact (institution and/or person) plus a version/date label should be there.

15)   6.2.3 (just a comment): I don't think that the issues of granularity and traceability are handled well enough in the current standards. This will be a major discussion point for the Dublin workshop (see http://wiki.esipfed.org/index.php/GEO_AQ_CoP_Workshops).

16)   6.2.5: I would disagree with the statement that "metadata Is not intended for experienced users" - some aspects of metadata, such as provenance, processing history, calibration information, ... are specifically valuable for experienced users. Likewise the search and discovery metadata are useful for everyone. This should rather read "Some descriptive metadata is specifically intended for new users of the data who are not familiar with this type of data, or the measurement platform or model from which the data are generated."

17)    6.2.5: (implicitly contained in the recommendation of metadata standards): use different sections (e.g. in XML metadata files) for different aspects of metadata; distinguish between information about the instrument/model/measurement technique, the data provider or distributor, the dataset version and history, and other aspects.

18)   6.2.6: should link to the "long-term planning" section - if metadata are stored, for example, in a web folder, make sure this can be maintained over long periods (if the server migrates, create an alias, ...); likewise: if you deliver metadata to a catalogue, plan the metadata update process and make sure to inform the catalogue about changes in your metadata.

PS: would you mind if I share the section on the data quality labeling with a colleague from the GAW science advisory group with whom I am currently preparing a "measurement guidelines" document on tropospheric ozone?

Best regards,

Martin

Von: esip-aqcluster-bounces at lists.esipfed.org [mailto:esip-aqcluster-bounces at lists.esipfed.org] Im Auftrag von Lough, Glynis C
Gesendet: Mittwoch, 23. Mai 2012 17:24
An: esip-aqcluster at rtpnet.org
Betreff: [ESIP-AQ] CyAir Best Practices - need your review!

Dear ESIP -

The CyAir team needs the community's help with review of the Best Practices for Interoperability of Air Quality Data.  We will be holding a series of online meetings to get community input.  The first call will be TOMORROW, Thursday 24 May, at 1pm EDT.  Everyone can review the Best Practices document at: http://cyair.net/sites/default/files/CyAir_Best_Practices_April_30_2012.pdf

Please join us!

Thursday 24 May, 1pm EDT: (1) Introduction and Purpose, and (2) Overarching Best Practices

Thursday 7 June, 1pm EDT: (3) Data Format Standards, (4) Naming Conventions, and (5) Web Services

Thursday 28 June, 1pm EDT: (6) Metadata and (7) Data Publication and Discovery

To join the web portion of the meeting:
https://esipfed.webex.com/mw0306ld/mywebex/default.do?siteurl=esipfed&service=1

1. Click join next to the meeting name, CyAir Best Practice Telecon(Note: If the meeting has not yet started, join will not be visible, and it will say 'Display Info'. Wait a minute and refresh screen)
2. Enter your name and email address
3. Password, enter the access code: 23127112

After you've joined the web portion, click on the audio on the screen and connect via your computer or call in:
1-877-668-4493
Attendee Access code: 23127112#

Thank you -
Glynis
and the CyAir Best Practices team: Steve Ludewig, Tim Dye, Stefan Falke, Uma Shankar, Terry Keating

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Kennen Sie schon unseren neuen Film? http://www.fz-juelich.de/film
Kennen Sie schon unsere app? http://www.fz-juelich.de/app
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.lists.esipfed.org/pipermail/esip-aqcluster/attachments/20120524/9f06b7cf/attachment-0001.html>