Jun 9, 2021 7:00 AM

Watch a Drone Swarm Fly Through a Fake Forest Without Crashing

Each copter doesn’t just track where the others are. It constantly predicts where they’ll go.

Enrica Soria needed soft trees. The mathematical engineer and robotics PhD student from the Swiss Federal Institute of Technology Lausanne, or EPFL, had already built a computer model to simulate the trajectories of five autonomous quadcopters flying through a dense forest without hitting anything. But an errant copter wouldn’t survive a tête-à-tête with a physical tree.

So Soria built a fake forest the size of a bedroom. Motion-capture cameras lined a rail hanging above the space to track the movement of the quadcopters. And for “trees,” Soria settled on a grid of eight green collapsible kids’ play tunnels from Ikea, made of a soft fabric. “Even if the drones crash into them,” Soria recalls thinking, “they won't break.”

She built the soft playground for the drones to safely test a new form of autonomous control: programming drones to adjust their trajectory based on how they expect their neighbors to move—rather than relying on an omniscient computer to direct them. An autonomous swarm is generally risky—the robots could smash into unforeseen obstacles, such as trees or curious birds, or each other. And a collision could have a ripple effect that derails the whole flock.

But public and private interest in controlling “swarms” of drones (like Soria’s fake forest fliers) is growing. Designing a reliable control system has promise for real-world missions in which a swarm has to fly together, such as search and rescue efforts in forests or coordinated deliveries in cities. Some swarms are currently controlled by a central computer or person on the ground, like flying light shows that replace fireworks. The ag-tech company Rantizo earned approval last year to fly three drones over farms for its crop-spraying services, and those take direction from a pilot on the ground too. But large swarms, like the ones researchers want to use for monitoring air quality or other data collection, would benefit from more fully autonomous controls.

Autonomous swarms are usually controlled reactively, meaning based on their current distance from stuff they shouldn’t hit. If drones drift too far from each other, they'll pull in closer; if they approach an obstacle, they'll slow down and distance themselves.

This error correction makes sense. ("Hey drones, don't hit stuff.") But the time it takes to recognize, compute, and make those adjustments slows down the whole group. Soria's system avoids slowdowns with better planning. Her autopilot algorithm is based on what she calls "predictive control"—the drones communicate with each other and interpret real-time motion-capture data to predict where other nearby drones will move. Then they adjust themselves accordingly.

Once Soria sent the drones flying through her fabric forest, she soon confirmed that the softness of the obstacles didn’t really matter: The drones didn’t crash. The five quadcopters sprang up into randomized starting positions, coasted through the fake forest, and landed safely. “They are able to see ahead in time,” says Soria. “They can foresee a future slowdown of their neighbors and reduce the negative effect of this on the flight in real time.”

Based on the computer simulation and the fake-forest demonstration, Soria’s team showed that their drones zipped through the obstacles 57 percent faster than state-of-the-art “reactive” controls that don’t involve prediction. The results appeared in the journal Nature Machine Intelligence in May.

Although Soria’s drones rely on a computer on the ground to perform the many necessary calculations, her system imitates how drones would communicate with each other if the computation were entirely distributed. “If you want to fully deploy these things, we should really cut the need for communication with a central hub or computer,” says ‪Amir Barati Farimani, a mechanical engineering professor at Carnegie Mellon who is not affiliated with the study. “This is one step toward that goal.”

Photograph: Alain Herzog/2021 EPFL

A lot of inspiration for the science of simultaneously controlling multiple drones comes from gorgeously synchronized behavior in nature: flocks of birds, schools of fish, and swarms of bees. But bee swarms navigate unexpected obstacles better than drone swarms, and, Soria says, “biologists say that there's no central computer.” No one bird or fish or bee directs movement for the rest. Instead, each animal computes its own trajectory based on its neighbors’ flight. They avoid each other, as well as surprise interlopers. The wondrous synchrony of animal collective behavior reportedly relies on predictive computations. Our brains are also thought to operate by constantly comparing reality to predictions.

Soria's team at EPFL didn't invent the idea of predictive control for drones. Scientists have modeled it to navigate obstacle-free areas and systems for two vehicles traveling on predefined trajectories. But it’s not the norm, she says, because predictive control relies on a flood of real-time calculations that can max out whatever computational power fits on small drones, which weigh 10 times less than a smartphone.

Predictive control is all about finding the optimal answer to a problem with a ton of variables—like inter-drone distance and speed—that should all hover near desired values. To simulate predictive control, Soria programmed math equations representing the most important constraints. Drones shouldn’t slam into each other, so her model limits how close they can fly to another. Drones shouldn’t try to soar through an obstacle, so her model can keep a list of “no-fly zones” logged in the back of its mind. At the same time, each drone should reach and maintain a preferred speed toward its goal. So Soria programmed each drone’s autopilot to imagine a best trajectory based on its current state and these constraints. Importantly, each drone also imagines this trajectory for its nearest neighbors, based on its knowledge of their position and motion.

It’s like a couple tennis pros working out the best way to slam the ball back. “They are not only reacting to where the ball is at a given time,” Soria says. “They are also planning what's going to happen next, for instance based on the direction they see that the opponent is moving.”

The math, of course, gets messy. One drone’s trajectory influences the rest, and vice versa—a type of system referred to as being “nonlinear.” Solving the tangled web of nonlinearity is a slog. But reality is itself nonlinear. That makes Soria's computationally expensive approach worth it.

Soria’s team tested the new approach against a state-of-the-art reactive model on a simulation with five drones and eight obstacles, and confirmed their hunch. In one scenario, reactive swarms finished their mission in 34.1 seconds—the predictive one finished in 21.5.

Next came the real demonstration. Soria’s team gathered small Crazyflie quadcopters used by researchers. Each one was tiny enough to fit in the palm of her hand and weighed less than a golf ball, but carried an accelerometer, a gyroscope, a pressure sensor, a radio transmitter, and small motion-capture balls, spaced a couple of inches apart and between the four blades. Readings from the sensors and the room’s motion-capture camera, which tracked the balls, flowed to a computer running each drone’s model as a ground control station. (The small drones can’t carry the hardware needed to run predictive control computations onboard.)

Soria placed the drones on the floor in a “start” region near the first tree-like obstacles. As she launched the experiment, five drones sprang up and quickly moved to random positions in the 3D space above the takeoff area. Then the copters started moving. They slipped through the air, between the soft green obstacles, over, under, and around each other, and toward the finish line where they landed with a gentle bounce. No collisions. Just smooth uneventful swarming made possible by a barrage of mathematical computations updating in real time.

Video: Jamani Caillet/2021 EPFL

“The results of the NMPC [nonlinear model predictive control] model are quite promising,” writes Gábor Vásárhelyi, a roboticist at Eötvös Loránd University in Budapest, Hungary, in an email to WIRED. (Vásárhelyi’s team created the reactive model Soria used, but he was not involved in the work.)

However, Vásárhelyi notes, the study doesn’t address a crucial barrier to implementing predictive control: the computation requires a central computer. Outsourcing controls over long distances could leave the entire swarm susceptible to communication delays or errors. Simpler decentralized control systems may not find the best possible flight trajectory, but “they can run on very small onboard devices (such as mosquitoes, lady bugs or small drones) and scale much, much better with swarm size,” he writes. Artificial—and natural—drone swarms can’t have bulky onboard computers.

“It is a bit of a question of quality or quantity,” Vásárhelyi continues. “However, nature kind of has it both.”

“That's where I say ‘Yes, I can,’” says Dan Bliss, a systems engineer at Arizona State University. Bliss, who is not involved with Soria’s team, leads a Darpa project to make mobile processing more efficient for drones and consumer tech. Even small drones are expected to become more computationally powerful with time. “I take a couple-hundred-watt computer problem and try to put it on a processor that consumes 1 watt,” he says. Bliss adds that creating an autonomous drone swarm isn’t just a control problem, it’s also a sensing problem. Onboard tools that map the surrounding world, such as computer vision, require a lot of processing power.

Lately, Soria’s team has been working on distributing the intelligence among the drones to accommodate larger swarms, and to handle dynamic obstacles. Prediction-minded drone swarms are, like burrito-delivery drones, many years away. But that’s not never. Roboticists can see them in their future—and, most likely, in their neighbor’s too.