[ESIP-all] Looking for experts on GeoTIFF, MODIS & Landsat, plus a place to stash a few TB of images.

Thu Aug 15 22:55:08 EDT 2013

On Aug 15, 2013, at 11:19 AM, Peter Cornillon wrote:

> Rather than plowing through terabytes of data, it might be useful to find samples under very clear conditions that contain the known location of the Schooner SV Nina from a prior cruise to see if it can be seen when you know exactly where to look. This would serve two purposes: (1) it would help determine in which types of data, if any, can you see the boat when you know exactly where to look, and (2) it might help develop an automated algorithm(s) to look for it in those data, again, if any, the boat can be seen.

I've recommended #1 to them ... see if they can verify it was at the place it was reporting before they lost contact & see if they can find it in subsequent passes so they can reduce the search area.  The search & rescue groups are used to flying sorties to look for planes, not using satellite data, so this is a bit of new territory for them.

As for #2, my understanding for the need to distribute the data back out is that there's a few groups in the solar physics community who do what we call 'feature finding'* ... they have routines that comb through images looking for different types of structures in solar images.  I assumed they'd have to re-train them to find boats, but if it works, we can process the images faster than by human eye.  (although, they're still working on that one, too ... the images at tomnod will get updated once the new images are processed)

I'm hoping that the image analysis folks can make something that we can put into AWS or Google Cloud or similar so that should something similar happen again, we just send the data there, pay them for the CPU time.  (it was the download fees that are the killer in the end, but they don't charge when you're sending the data to their compute services)

... and if nothing else, I'm sure there's a few papers that can be written on their work.  If other people are interested in trying, contact Barbara Thompson, as she's the one who's coordinating that effort:

	barbara.j.thompson at gmail.com

There's probably good research to be done in UI work, too ... what can we do to improve the human turk type systems so they can find things faster?  Are large windows (where you look at more per step) more efficient than smaller windows to look at?  How much overlap should you use between steps to reduce the chance of missing things at the edge?  Should you start people off w/ some training and make sure that you give them a boat to find early on, so you make sure that they're not really bad at the task?  Are the little tiny examples of what people as effective as showing the target in context?

-Joe

* They're actually having their annual meeting on 'solar image processing' (which also covers some other stuff) this week : http://www.sipwork.org/

> On Aug 14, 2013, at 11:56 PM, Joe Hourcle <oneiros at grace.nascom.nasa.gov> wrote:
> 
>> 
>> It seems that everyone was so helpful in getting the requested
>> images to help search for the Schooner SV Niña, that there's now
>> issues with actually going through all of the data.
>> 
>> (see Barbara's message, below.)
>> 
>> Thanks in advance for any help.  And Barbara and I might be slow
>> to respond tomorrow, as we're both going to be pretty well tied
>> up 'til about 2 pm. 
>> 
>> -Joe
>> 
>> ps. Yes, that really is my e-mail address.  You can check whois
>>   if you don't believe me.
>> 
>> pps. If someone offers you tickets to a Weird Al concert, I
>>    highly recommend taking them up on it.
>> 
>> 
>> 
>> 
>> Begin forwarded message:
>> 
>>> From: Barbara Thompson <barbara.j.thompson at gmail.com>
>>> Date: August 14, 2013 11:05:21 PM EDT
>>> To: Joe Hourclé <oneiros at annoying.org>
>>> Subject: Favor?
>>> 
>>> Hi Joe,
>>> 
>>> Would you mind forwarding the following request to the lists that you
>>> originally sent the messages to?  You got such a huge response.  Sorry to
>>> ask, but I think it'll be quicker if it comes from you because it'll go
>>> straight to the people.  If there's any way I can pay back the favor (a
>>> stash of Weird Al Yankovic tickets??) I'll be glad to do so.
>>> 
>>> Thanks!
>>> 
>>> Barbara
>>> ********
>>> 
>>> (Please note that this effort is entirely staffed by volunteers.  Those of
>>> you who know me should be aware that I am contacting you as a private
>>> citizen - this effort is not associated with my professional activities.
>>> - Barbara)
>>> 
>>> First and foremost, I thank all of you for your advice regarding the search
>>> for the Schooner SV Niña.   More stories are surfacing of people who were
>>> lost at sea for months, and it is very encouraging.  Aerial searches are
>>> continuing, and we are receiving support from Earth-observing satellites to
>>> provide imagery from the Tasman Sea.   Tim Paynter, one of our team
>>> members, has made some videos to familiarize people with the Niña and her
>>> crew, and why we find this effort so worthwhile:
>>> http://www.youtube.com/watch?v=4XmK0Rs4nSE
>>> http://www.youtube.com/watch?v=iLDn7fNGvWw
>>> (see his Youtube Channel for even more)
>>> 
>>> We have received many messages with excellent suggestions, and we are
>>> attempting to follow up on all of the information we've received.
>>> Unfortunately, there have been hundreds of emails, so I apologize if I
>>> haven't gotten back to you quickly.
>>> 
>>> I am writing you because we are hoping to attract help on three fronts:
>>> 
>>> 1) Tomnod.com has launched a search for the Niña (
>>> http://www.tomnod.com/nod/challenge/ninarescue2) and we welcome all
>>> volunteers.  However, we are about to receive a huge amount of raw data
>>> (several Terabytes) from DigitalGlobe, and we are trying to determine how
>>> to make this data available.  All of our funds are volunteer funds, and the
>>> sites for which we've gotten price estimates charge per GB of download -
>>> which can add up to quite a bit of money.   We have a test data set (1.5
>>> GB) already if you want to take a look.  The results of this effort will
>>> guide aerial searches and help determine where satellite images should be
>>> obtained.  Any advice on how to serve this much data would be greatly
>>> appreciated - we only need it for a couple of weeks!
>>> 
>>> 2) We need assistance in analyzing the data.  The images are in GeoTIFFs,
>>> and none of our team has experience with that format (I study solar imagery
>>> - looking in the wrong direction!).  We are working on attracting feature
>>> recognition and information processing experts to analyze the data, but
>>> some need us to export the data to a format that's more familiar to them.
>>> We welcome help from who has experience with GeoTIFFs and image processing.
>>> 
>>> 3) We've had several people point out that although many Earth Science
>>> imagers have too low of resolution to find a 70-foot vessel, the wake of
>>> the ship can extend for some distance and may be able to be seen in 250m
>>> resolution data.  Any information about the history of the ship's course
>>> could be useful helping drift modelers determine the most likely location
>>> of the Niña.  However, no one on our team has experience with MODIS or
>>> Landsat imagery.  We welcome assistance from anyone who may be able to help
>>> with a search of relevant Earth Science data sets.
>>> 
>>> Again, we are extremely grateful for all of the wonderful advice and
>>> encouraging messages.  However, our small team is unable to take advantage
>>> of all of the possible opportunities.  If you would like to assist in this
>>> effort, please let me know.  We are grateful for any time you can spare.
>>> 
>>> Best wishes,
>>> 
>>> Barbara Thompson
>>> on behalf of the Schooner SV Niña search team (updates at http://evxx.com)
>> 
>> _______________________________________________
>> ESIP-all mailing list
>> ESIP-all at lists.esipfed.org
>> http://www.lists.esipfed.org/mailman/listinfo/esip-all
> 
> --
> Peter Cornillon
>  215 South Ferry Road                                     Telephone: (401) 874-6283
>   Graduate School of Oceanography                          Fax: (401) 874-6283
>    University of Rhode Island                                 Internet: pcornillon at gso.uri.edu
>     Narragansett, RI 02882   USA
> 
> 