Finally: An App That Can Identify the Animal You Saw on Your Hike

The legendary naturalist John Muir once wrote: “Whenever I met a new plant, I would sit down beside it for a minute or a day, to make its acquaintance, hear what it had to tell.” The first step to making an acquaintance is to get a name—and naming nature is not easy. This weekend, while walking through Great Falls Park, a butterfly landed on my friend’s leg. It was large, with yellow and black wings—clearly a swallowtail, but what species? That same day, a large black insect landed on a flower in front of me, and I snapped a portrait of it before it flew off. It was a dragonfly, but what kind of dragonfly?

Many of our experiences of nature take this form. You see something, but you don’t know what it is. You are surrounded by life, but much of it is anonymous. “People don’t identify as a naturalist but if you ask them if they’ve ever been outside, seen something, and wondered what it is, they’ll say: Oh yeah, sure,” says Scott Loarie from the California Academy of Sciences.

Loarie and his team have developed an app that can help. Known as iNaturalist, it began as a crowdsourced community, where people can upload photos of animals and plants for other users to identify. But a month ago, the team updated the app so that an artificial intelligence now identifies what you’re looking at. In some cases, it’ll nail a particular species—it correctly pegged the dragonfly I spotted as a slaty skimmer (Libellula incesta). For the butterfly, it was less certain. “We’re pretty sure this is in the genus Papilio,” it offered, before listing ten possible species.

“Our ecosystem is just unravelling in front of our eyes, and the pace of environmental change can be really overwhelming,” says Loarie. “But in our handbags, there’s another thing that has had the same pace of unbelievable change—the cellphone.” He hopes that the latter can help with the former by acting as a pocket naturalist, a cross between Shazam and an old-fashioned field guide.

The iNaturalist site began in 2008 as the master’s project of three students, and has since blossomed into a thriving community of around 150,000 people. Together, they’ve captured around 5.3 million photos representing 117,000 species. By labeling these images and tagging where they were taken, the site’s users are conducting an inadvertent census of the world’s animals. And sometimes, they make surprising discoveries.

In 2011, Luis Mazariegos, a retired Colombian businessman, uploaded a picture of a striking red-and-black frog, found on the patch of rainforest land that he had recently bought. Frog expert Ted Kahn realized that it was a completely new species, and the duo published a paper describing the amphibian a few years later. In 2014, a wildlife photographer named Scott Trageser uploaded a photo of a snail that he had taken in Vietnam. Twenty months later, mollusc expert Junn Kitt Foon identified the animal as Myxostoma petiverianum—a species that James Cook’s crew had discovered in the 1700s, but that no one had photographed before.

“It’s a rare win-win,” says Loarie. “We’re engaging people but also producing this stream of high-quality data for science. And we’re sitting on the biggest pile of accurately labeled images for living things that’s out there.” But iNaturalist could become a victim of its own exponential success. Around 20,000 new photos are uploaded every day, threatening to overwhelm the community of expert identifiers. Already, it takes an average of 18 days to get an identification.

Loarie and his colleagues realized that the only way of avoiding an inevitable backlog of unidentified critters was to train a computer in the art of taxonomy. They could feed a neural network—a computer system modeled on the brain—with images from the iNaturalist collection, and allow it to learn the distinctive features of each species. “The expectation, even a year ago, was that this stuff was light years away and unrealistic,” says Alex Shepard. But now, this kind of machine learning is increasingly powerful and user-friendly. Computers learned to program prosthetic arms, reverse-engineer smells, identify galaxies, or devise funny new names for colors.

Artificial intelligence is only as intelligent as the data you use to train it. Shepard only used “research-grade” photos that have been vetted by the iNaturalist community, and he only trained his neural network on the 13,730 species that were represented by at least 20 such photos. Using these photos, and after training himself using online tutorials, Shepard built a “training wheels” prototype that was good enough to identify visually distinctive things like monkeyflowers—and to impress his bosses at the California Academy of Sciences.

The proper version, released on June 29, is surprisingly good. It has learned to recognize several species from unusual angles—like the head-on slaty skimmer dragonfly that I asked it to identify. It can even cope with species that come in various patterns. “We spent a lot of time on ladybirds,” Shepard says. “Asian ladybirds come with a lot of different characteristics—you might see one that’s mostly black with red spots, and another that’s red with black spots. But even the early versions of our system could understand that.” (The app, however, seems to struggle with human children, who have variously been billed as northern leopard frogs and ringneck snakes.)

Identification apps aren’t new but almost all are restricted to specific groups of organisms, like birds or plants. A recently announced one, which claims to use AI to “identify any mushroom instantly with just a pic,” was swiftly derided by experts for being “potentially deadly.” Given how poisonous some mushrooms can be, a wrong ID from a blundering AI could be catastrophic.

Loarie’s team has tried to circumvent these risks by designing the app to be almost self-conscious about its own limitations. Rather than providing firm identifications, it instead gives “suggestions” or “recommendations.” For each photo, it offers ten possible species; so far, one of those ends up being right 78 percent of the time. It also gives one overarching suggestion, which varies in detail depending on how confident it is. When I showed it the crisp photo of the slaty skimmer, it assertively guessed at the species. When I challenged it with a blurry photo of a frog, it suggested that the animal was a frog, but didn’t venture further.

So, iNaturalist isn’t quite a biological version of Shazam—the app that identifies songs. It’s more like autocomplete, which offers increasingly accurate suggestions depending on the information you provide. “We want something that’s always accurate even if it’s not precise,” says Loarie.

Karen James, a biologist who has worked on citizen science projects, praises the app but notes that it’s not a “panacea for identification.” Since it relies entirely on photos, “the organism has to be big enough and its diagnostic characters have to be visible, which rules out large swaths of the tree of life.” It is also limited by what its users photograph. For that reason, it works better for North American animals than South American ones, for example, and for mammals and birds than nematode worms or nudibranch slugs.

Still, the app will only improve as it gorges on more data. Every couple of hours, another species crosses the magic threshold of 20 research-grade photos, allowing the computer to learn its features. And at a recent computer vision conference, the team ran a competition, sponsored by Google, to improve their AI.

Eventually, Loarie hopes that iNaturalist will be useful to other communities too, such as border agents who open suitcases full of smuggled animals, or biologists analyzing images captured by camera-traps. But James hopes that before this happens, the app’s results are independently verified. So far, “its accuracy is measured by comparing the computer-vision identifications against the very crowdsourced identifications that are used to train the computer. There should be ways of checking those,” such as by analyzing the DNA of samples that are then run through the app, or relying on trained taxonomists.

It all comes back to people in the end. If the app is successful, it’s only because it learned from the thousands of identifications that iNaturalist’s bustling community have contributed. They are still involved in checking the computer’s answers. When the app suggested that the butterfly I saw was a swallowtail, the community quickly confirmed that it was specifically the eastern tiger swallowtail (Papilio glaucus).