AI-enabled scientific discovery in natural world imagery

Natural world images collected by communities of enthusiast volunteers provide a vast and largely uncurated source of data. For instance, iNaturalist has over 180 million images tagged with species labels, already contributing immensely to research such as biodiversity monitoring and having been cited in over 4,000 scientific papers. Yet, these images are also known to contain a wealth of "secondary data" captured unintentionally or otherwise included in images and not properly reflected in image labels. Although this data contains crucial insights into interactions, animal social behavior, morphology, habitat, co-occurrence, and many more questions, the costly, time-consuming, or expert-dependent analysis needed to extract such information prevents breakthroughs. Advances in deep learning methods for language and computer vision have the potential to enable the efficient and automated processing techniques needed to unlock the "hidden treasure" in such datasets– being able to directly search large image collections for these concepts would enable richer analyses that span beyond species identification. We introduce INQUIRE, a new benchmark for expert-level text-to-image retrieval on natural world images. INQUIRE includes hundreds of scientifically motivated retrieval tasks, each composed of a text query paired with a set of all relevant image matches that we have comprehensively labeled over a new iNaturalist subset comprising 5 million natural world images from 10,000 species classes (iNat2024). The tasks collected and labeled for INQUIRE come from discussions and interviews with a range of experts including ecologists, biologists, ornithologists, entomologists, and oceanographers. Our benchmark provides a rigorous evaluation that challenges models to demonstrate advanced knowledge and visual reasoning. We perform a comprehensive evaluation of state-of-the-art multimodal models. Our evaluations show that INQUIRE poses a significant challenge which necessitates the development of the next-generation image retrieval models able to accelerate and automate scientific recovery within large collections. A key finding from our experiments is that reranking, a technique that has typically been used for text retrieval, offers a significant avenue of improvement for image retrieval.