[Esip-discovery] time for new challenges?

[Lynnes, Christopher S. (GSFC-6102)]

On Dec 19, 2012, at 11:57 AM, Gallagher James wrote:

> 
> On Dec 19, 2012, at 6:58 AM, Lynnes, Christopher S. (GSFC-6102) wrote:
> 
>> Here is one not obviously related to OpenSearch...Faisal Hossain (this year's winner of the Falkenberg!) wrote an editorial in BAMS lamenting the difficulty of discovering useful web applications (e.g., Giovanni) via major search engines by the applications data content:  http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-12-00035.1
>> 
>> Hook and I have talked with Microsoft, and indirectly with Google, and there appears to be no quick and easy silver bullet.
> 
> When you talked to folks about this, did they draw comparisons with other aspects of their search tools that index stuff other than the 'simple' web pages? For example, google scholar indexes a fair number of papers that are behind query interfaces. 
> 
> We (OPeNDAP) experimented with different ways to leverage Google to find our servers, with varying levels of success. Since most DAP servers have some .html pages, they can be crawled by Google (and by other, less well-behaved bots). We've embedded various UUIDs in Hyrax's top-level pages and can search for (crawlable) servers that way. However, because most DAP servers also return dataset metadata that's automatically generated on pages that can be reached by a crawler, bots are often blocked. This is a consequence of exposing large numbers of datasets (e.g., files) that are expensive to access (e.g., because the files are compressed as opposed to having the data they contain compressed). If only Google crawled a site with 10^7 files and decompressed each one, maybe nobody would mind, but when 50 bots do it, people get irked ;-) There are lots of crawling/indexing bots out there… 
> 
> I mentioned Google scholar and it's (apparent) indexing of a number of systems that (probably) implement different query interfaces because I wonder if we could use something like the opensearch, THREDDS catalog and WCS GetCapabilities responses to make web data access points findable. The crawler(s) would have to be smart enough to not descend into the holdings too deeply and we might have to modify those catalogs so they provide better information for crawlers.

At least in our case, it's not a matter of not indexing it at all; it just comes in rather low on the results sort. 
--
Dr. Christopher Lynnes     NASA/GSFC, Code 610.2    phone: 301-614-5185