Are brain waves the next key to physical AI?

The frontier of physical artificial intelligence is a game of Jenga in a warehouse in San Leandro, California.

He seized that warehouse EncordA company that builds data tools used to train AI models. Andrew Ceja is a pilot — the company’s term for robot trainers — and he carefully pulls wooden blocks from a falling tower while wearing a headset with a camera that tracks what he sees. This alone is common enough for collecting robot training data, but this headset includes sensors that measure the block tower’s brain waves as it carefully dismantles it.

Encord is one of a small but growing number of startups that are making the case that the next real limitation to humanoid and warehouse robots will not be model architecture, but rather the sheer dearth of real-world physical training data. Instead of helping robotics companies manage the data they have, Encord is building a business around producing the data they don’t.

Produced by the brainwave headset that Ceja wears Zander Laboratoriesa German neuroscience startup that bets on measuring brain activity to infer mental states like error, intention, and surprise can create a more useful data set for training models. Encord’s work with Zander is currently a trial; The goal, Encord says, is to create an initial brainwave-tagged data set, run it through customers’ robotics models, and assess whether it actually improves performance before deciding to scale up.

Lucas Gehrke, a Zander neuroscientist who supervised the work, says that the amount of brain activity used at any given point during a given task helps modelers understand when to use their most effortful models.

According to Vineeth Velmurugan, Encord’s head of robotic learning, this is the “bleeding edge” of efforts to address the robotics data bottleneck. Velmurugan, a veteran of OpenAI’s robotics lab and warehouse automation firm Berkshire Gray, has joined Encord to build the company’s internal data generation team.

Encord was created to help companies building machine-vision software annotate data and evaluate models. As his clients — Velmurugan says he works with many leading robotics firms, but is not authorized to name them — began applying end-to-end learning to robot manipulation tasks, managers realized they needed to produce the training data themselves, rather than simply manage it. “The data just isn’t there,” Velmurugan said.

The bet that generative AI can do for bots what it did for chatbots continues to run into the same wall. Self-driving car companies are collecting physical world data themselves, but it’s hard to measure. Training from video can work, but it lacks the fidelity of real-world data. Velmurugan says that crunching a data set five times the size of YouTube’s video corpus would require a dataset—a scale that helps explain why data generation itself has become a business rather than just a research problem.

Satisfy your egocentric information needs

Companies building robot brains now turn to two primary sources: “egocentric” video collected by camera-wearing workers, often augmented with additional camera angles and other metrics, and data from robots controlled remotely. Encord does both, pulling in egocentric data from several factories around the world and using the San Leandro facility to collect datasets around specific skills for experimentation or fine-tuning with new techniques like brainwaves.

When TechCrunch visited, the pilots were using a pair of robotic arms, one controlled directly by a human operator and arms that mimic his movements, to generate data on tasks like pouring coffee from a pot into cups (very messy) and stacking poker chips. “Every humanoid company has asked us for these pieces,” Velmurugan said.

On the storage shelves were cardboard boxes of fake flowers in vases, books, plastic vegetables, cat litter boxes and scoops, bags and wires, a trade stock for making manipulatives for household chores.

At one of these stations, another pilot, Sofia Infante, maneuvers robotic arms to connect and disconnect ethernet cables behind a server—something data center operators would love to automate if only robots could control them with the necessary precision. Fidgeting behind the controls, I could see why it’s still out of reach: Flippers are much less flexible than human fingers and lack the degrees of freedom we normally take for granted in our arms.

Another new data method developed by Encord uses an array of sensors attached to the forearm to detect electrical signals in the muscles. Video of human hands manipulating objects typically doesn’t capture the entire hand, but Velmurugan hopes to use arm sensors to create a 3D representation of where the hand is at any given time and create a more robust understanding for the models.

To help LLM-based models understand what’s going on, Encord’s datasets are annotated with physical representations of what each video contains—”the right hand tightens the bolt.” Velmurugan estimates that such dense annotation is 100 times more valuable for teaching specific tasks and only 20 times more expensive to produce on paper.

But “20 times as much” is still real money, and that’s the fascinating aspect: scraping text off the web, LLM makers building their own models by borrowing from Stack Overflow and the rest of the web, doesn’t cost frontier labs anything. It is impossible to generate physical training data, and this is the limit of the physical-AI-as-LLM comparison. Such data must not only be collected, but produced, and this changes the economics of building models.

Velmurugan says that progress is being made—with Encord’s visibility into applications across industry, he can understand what works and what doesn’t for startup and frontier labs to improve their physical AI models. That vantage point — sitting among multiple robotics companies at once — is also part of Encord’s pitch. It can determine which data technique is gaining industry-wide traction before any customer.

That will keep the dozen or so pilots at Encord’s facility busy. Both Infante and Ceja are part of a growing workforce developing the building blocks for neural networks; they worked at Scale, another AI data annotation firm, before joining Encord.

Ceja worked for a waste management company, where his interest in technology found him responsible for keeping a robotic waste sorting machine in good working order. Now, as the Jenga tower flies, she says she enjoys solving training tasks for the robots — “It’s something new every day!”