The biggest announcement at Meta’s Connect event was its long-promised AR glasses, Orion. As expected, the prototype, which will reportedly cost around $10,000 apiece, won’t be available to the public anytime soon.
Meanwhile, Meta gave a sneak peek at new holographic avatars that will let you talk to lifelike holograms in augmented reality. The holograms are Codec Avatars, a technology Meta has been developing for several years. Mark Zuckerberg teased a version of this when he took part in a podcast interview on “The Metaverse” last year.
The technology might be closer than we think: Following the Connect keynote, I spoke with Mark Rabkin, Meta’s vice president of Horizon OS and Quest, who shared more about Meta’s codec avatars and why they’ll be coming to the company’s VR headsets in the future.
“In general, pretty much anything you can do on Orion, you can also do on Quest,” Rabkin says. Creating codec avatars in particular has become much easier: Whereas previously they required advanced camera scans, Rabkin explains that most internal avatars are now created with phone scans.
“The process of generating stylized avatars for VR is roughly the same in many ways, but the training set is different and the amount of computation required is different,” Rabkin explains. “For stylized avatars, the model has to be trained on a lot of stylized avatars and how they look and move. You need to have a lot of training data on what people perceive as similar to photos of themselves and what they perceive as moving well.”
“With Codec Avatars, the process is the same: you collect a huge amount of data. You collect data from very high-quality, high-performance camera scans. What people are actually creating are phone scans, so you collect data from phone scans and build on them until you get a good model. One of the challenges in both problems is making it fast enough and computationally inexpensive so that it can be used by millions of people.”
Rabikin said he expects that eventually these avatars will be playable in virtual reality on the company’s headsets. Currently, the Quest 3 and 3S lack the necessary sensors, such as eye tracking, needed for photorealistic avatars. But he said that could change with the next generation of VR headsets. “Hopefully, I think that’s probably possible with the next generation (of headsets).”
