[Esip-discovery] Kickstarting a geoscience search evaluation effort in discovery cluster

[Mcgibbney, Lewis J (398M)]

Hi Folks,
Is there any interest in starting an effort which will explore and ultimately define a process for evaluating geoscience information retrieval systems?

The following provides a detailed description of the problem space…

Evaluation is a central aspect in the research and development of information retrieval systems. In academia the quantitative evaluation of such systems is mostly known under the term Cranfield paradigm. This research method has been established for more than 25 years in international evaluation campaigns such as the Text Retrieval Conference (TREC) or the Conference and Labs of the Evaluation Forum (CLEF). Meanwhile industrial research has taken a completely different approach. Many companies are able to access a large number of users and their interactions, which can be recorded and evaluated. These infrastructures allow alternative evaluations like large-scale A/B experiments or other online methods. In the last years different approaches to go beyond TREC-style evaluations emerged to close the gap and to bring together academic and industrial evaluation.

        The WG would want to generate a report on novel evaluation efforts, like:
        - Living labs
        - Evaluation as a service
        - Large-scale A/B tests
        - Interactive retrieval evaluation
        - Session-based evaluation
        - User-centered evaluation
        - Counterfactual evaluation
        - Novel evaluations in application domains such as cultural heritage, digital libraries, social media, expert search, health information, etc.
        - Other evaluations that go beyond TREC


I think this could be the ticket which would bring the discovery cluster back to life.
Any takers?
Thanks

Dr. Lewis John McGibbney Ph.D., B.Sc.(Hons)
Data Scientist III
Computer Science for Data Intensive Applications Group (398M)
Instrument Software and Science Data Systems Section (398)
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel:  (+1) (818)-393-7402
Cell: (+1) (626)-487-3476
Fax:  (+1) (818)-393-1190
Email: lewis.j.mcgibbney at jpl.nasa.gov<mailto:lewis.j.mcgibbney at jpl.nasa.gov>
ORCID: orcid.org/0000-0003-2185-928X

           [signature_1706758482]

 Dare Mighty Things
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.esipfed.org/pipermail/esip-discovery/attachments/20190423/6310b77f/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3432 bytes
Desc: image001.png
URL: <http://lists.esipfed.org/pipermail/esip-discovery/attachments/20190423/6310b77f/attachment.png>