From lewis.j.mcgibbney at jpl.nasa.gov Tue Apr 23 17:16:38 2019 From: lewis.j.mcgibbney at jpl.nasa.gov (Mcgibbney, Lewis J (398M)) Date: Tue, 23 Apr 2019 21:16:38 +0000 Subject: [Esip-discovery] Kickstarting a geoscience search evaluation effort in discovery cluster Message-ID: Hi Folks, Is there any interest in starting an effort which will explore and ultimately define a process for evaluating geoscience information retrieval systems? The following provides a detailed description of the problem space? Evaluation is a central aspect in the research and development of information retrieval systems. In academia the quantitative evaluation of such systems is mostly known under the term Cranfield paradigm. This research method has been established for more than 25 years in international evaluation campaigns such as the Text Retrieval Conference (TREC) or the Conference and Labs of the Evaluation Forum (CLEF). Meanwhile industrial research has taken a completely different approach. Many companies are able to access a large number of users and their interactions, which can be recorded and evaluated. These infrastructures allow alternative evaluations like large-scale A/B experiments or other online methods. In the last years different approaches to go beyond TREC-style evaluations emerged to close the gap and to bring together academic and industrial evaluation. The WG would want to generate a report on novel evaluation efforts, like: - Living labs - Evaluation as a service - Large-scale A/B tests - Interactive retrieval evaluation - Session-based evaluation - User-centered evaluation - Counterfactual evaluation - Novel evaluations in application domains such as cultural heritage, digital libraries, social media, expert search, health information, etc. - Other evaluations that go beyond TREC I think this could be the ticket which would bring the discovery cluster back to life. Any takers? Thanks Dr. Lewis John McGibbney Ph.D., B.Sc.(Hons) Data Scientist III Computer Science for Data Intensive Applications Group (398M) Instrument Software and Science Data Systems Section (398) Jet Propulsion Laboratory California Institute of Technology 4800 Oak Grove Drive Pasadena, California 91109-8099 Mail Stop : 158-256C Tel: (+1) (818)-393-7402 Cell: (+1) (626)-487-3476 Fax: (+1) (818)-393-1190 Email: lewis.j.mcgibbney at jpl.nasa.gov ORCID: orcid.org/0000-0003-2185-928X [signature_1706758482] Dare Mighty Things -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 3432 bytes Desc: image001.png URL: