How a Guy AI Tracked a Chinese Spy Balloon Across the US

At the beginning of this month Entrepreneur Cory Jaskolski takes out his pen and imagines what a spy balloon launched by an American jet would look like from space. The balloon then fed a pattern and “gob” of recent satellite images of the area taken into the algorithms, which were synthesized by image and video recognition startups, and maintained.

Within two minutes, the algorithms found a 200-foot-long balloon off the coast of South Carolina, he said. “I couldn’t believe it,” Jaskolski said. When his wife eagerly showed him the result, she could not help it. But he estimated the height of the balloon in the image to be around 57,000 feet – the same height as the balloon was seen by a US spy plane – and images posted on social media 20 minutes before the image was taken appeared to confirm its existence. I found it.

Jaskolski is mining wind models and social media observations to feed his software, called RAIC (Rapid Automatic Image Classification), along with new satellite data from his company’s Planet Labs. The tool is designed to search large image collections for objects of interest using a single example image.

“We drew a big arc in time and space and started looking for that,” says Jaskolski. Once the balloon is found, the synthetic software can be trained on a real balloon image to further guide the search.

Over the next several days, Jaskolski put the RAIC to work. The company has since compiled six sightings of the balloon (five confirmed, one still under investigation) on satellite images and used wind data to estimate how it would move between these points. “We can draw a 1-kilometer-wide track across the United States and just follow the balloon,” he says. “We have a track from where it came in from Canada all the way to South Carolina, where it appeared, with six points on that arc.”

Jaskolski’s stratospheric scavenger hunt may be powered by smart software, but it also requires human expertise. The first picture of the craft looks like a Technicolor snowman – red, green and blue circles. The goal is to mimic the way satellites generally capture different wavelengths of light, creating different views of objects using different sensors that aren’t always synchronized in time. And it throws false positives.

But the ability to map a spy balloon’s path with such clarity could be a game-changer for national security, says Arthur Holland Michel, a senior fellow at the Carnegie Council and author of a book on drones and espionage. “The combination of AI and satellite imagery is undoubtedly a very powerful technology for surveillance and intelligence and counter-intelligence,” he said.

Holland Michel also points out that satellite imagery and AI have their limitations. The method by which the synthetic balloon was detected for the first time – using a picture – can lead to false positive results if the object of interest is more complex or less publicly reported, such as a tank. “Things often seem a little strange and unusual from above,” he says.

“There’s undoubted potential there, but it’s easy to think that this combination of satellite and AI is all-seeing that’s going to nullify everything,” said Holland-Michel. It’s useful in some cases, like the bladder, he says, but not in all cases.

That’s something Jaskolski acknowledges—but he also takes the project as an example of how human intelligence and noise can be augmented by AI. “This human-machine collaboration is my idea of ​​how AI works today,” he says. “And that’s certainly how we build our product.” The device is currently used for humanitarian purposes, including by the United Nations World Food Program to reach flood victims.

Just because Jaskolski was able to track it across the United States didn’t end his pursuit of the balloon. The process is “resource-intensive,” he said, because the software isn’t perfect and displays many views that must be tampered with by humans. “But we still want to continue the monitoring,” he said. “Even if we don’t go back to China, we feel like we’ve at least solved the technical problem. We’d be crazy not to try.”