Session Description: As augmented, virtual, and mixed realities begin to push the boundaries of the physical, we are finally seeing practical applications of AR and VR across a spectrum of industries. What will these new “realities” mean for business, society, and our sense of where and even who we are?
The full transcript is also available here as a downloadable PDF.
Kirkpatrick: I want to bring up Ina Fried, who’s a great friend of Techonomy and one of the great technology journalists of the world. She’s the chief technology correspondent and editor at AXIOS, which itself is a pretty extraordinary new institution in media.
Fried: Thank you, everyone. I’m really excited to talk about the next topic, which is Extending Reality: VR and AR, and welcome Marc and Tony. We focus a lot on where things are going, and that’s where I want to use most of our time, but I think that it’s really important to look at where we are today in VR and AR, and how we got here. I think there are some lessons to be learned.
I think this isn’t new to most of the people in the audience who’ve been through a few of these technology curves. We get super excited about something, a ton of money pours in, and then there’s this moment, [it’s been] called the trough of disillusionment, and I think we’d all agree that we’re there with parts of VR and AR. Let’s talk a little about where we are right now and how we got here.
Tony, Unity was born as a gaming engine, but it’s also really at the center of this because if you want to develop for VR and AR you need a world in which to do it, and you guys provide that world. Talk about where we are right now and how we ended up here.
Parisi: It was about three years ago when there was an explosion on the scene with the Oculus Rift, with Facebook spending billions of dollars to purchase this VR headset, and it shocked everyone. That excitement was around this consumer experience that would take you somewhere. And what we discovered is that people didn’t want to just get taken somewhere, they wanted to use VR to do a lot of different things.
There are several other manufacturers of virtual reality hardware—this kind of thing that obscures your whole face and vision—and in parallel with that, unbeknownst to a lot of folks, Microsoft was taking Kinect technology and working on this thing called the HoloLens, which gave you an augmented reality headset experience so that you could see the world around you.
What’s happened over the last couple of years is lots of industrial and non-gaming consumer entertainment and brand creative use cases have emerged, and it appears that the world truly is excited about this technology. And we’ve hit this place where, because of the bulky nature of a VR headset and because of the social aspects of wearing those, there’s been a bit of a stall in the marketplace where a lot of folks are expecting meteoric growth—as we do, we get ahead of ourselves and our expectations here—and that’s self-corrected. There’s less froth in the venture community to fund VR startups, and so on, and yet at the same time there are industrial use cases happening, which we can enumerate in a minute. So it’s proceeding at pace, but it’s gone from the front page news to becoming a technology that needs to find its way through the marketplace.
At the same time, phone-based augmented reality has burst on to the scene this year with Apple announcing ARKit and Google announcing ARCore, and the promise there is that we’re going to reach about a half a billion people based on the devices that they already have in their pockets or the next versions of phones from either Apple or Pixel, Google and Android phone makers. That’s interesting, it’s a very lightweight experience—you’re taking your phone, you’re holding your phone up; you’re seeing magical stuff through a magic window that is your phone. One of your hands is occupied doing this—my arm is already tired, by the way—and the other is doing that, so it’s not the same as HoloLens and all that magical stuff that we expect where we’re masters of reality manipulating things in mid-air; it’s not there yet. But it’s something that’s going to give us magical content right in front of us with a device in our pocket.
Fried: You don’t have to pay anything more—
Parisi: You don’t have to pay anything extra; you don’t put anything on your face. There’s nothing heating up on your face, and it doesn’t challenge the computer in the same ways that a super graphically intense VR thing does, like a mobile VR like Gear VR. So there’s a lot of goodness to it; it’s not by anyone’s estimation—I’m not speaking for Apple or Google on this—but I don’t believe anybody who’s into this believes that this is the end state. We believe this is the next step toward people understanding immersion, along with 360 videos, which we’re all fairly familiar with, which you can view on a flat screen, do this with your phone, or get in a headset to see.
We’re in this place right now where the high-end VR is working its way through industrial use cases, which I’m sure Marc will talk about, and the stuff in your pocket with augmented reality is going to be the consumer’s first taste and it’s going to be a place where developers get to go work for a while and presumably make some money. We’re seeing a bit of a shift in the developer community toward AR as well, but it’s all 3-D immersive content, which—me being at Unity—we love that. We make the 3-D engine, we make the platform, people buy our development software, and on mobile they also monetize through ads, so this is a fertile ground for us and we’re seeing it power industries across the board. So it’s super exciting.
Fried: From my sense, two things are happening: one, with the consumer for the moment you’ve still got a gaming community putting on headsets, but the mass market is shifting to this phone AR and that where the “ready for primetime” is. The headsets are there, some people put them on, the optics have to get better, the use cases have to get more compelling, but one of the really interesting things is what’s going on in the enterprise.
People don’t hear about it as much, and my guess is that it’s still small-scale types of things, but you have a couple of really interesting factors going on. You have the fact that companies, there’s big money in it so if they can get a little bit more productivity, increase safety—that sort of thing—that will pay for expensive gear. You’re not hanging around with your friends, so you’ll put up with a little more awkwardness. Marc, talk about what you’re seeing at Accenture, both in the labs and in the business.
Carrel-Billiard: I think Tony was absolutely spot on in describing the landscape today, which is really moving in different directions. [At] Accenture, we’re really a big consulting company in the enterprise business, and there’s two numbers that are really striking if you look at the last IDC survey related to the AR/VR market. Today, the estimate is about $40 billion dollars, that’s the market of AR/VR. And if you look at it, out of those $40 billion dollars, 70 percent is related to consumer business. And that 70 percent is not about service, it’s about devices that people buy, mainly in gaming. But what’s really interesting is that in three years’ time, by 2020, it’s estimated that the business is going to be $143 billion dollars. That’s starting to be interesting, because not only will the number be bigger, but the ratio is going to change completely. Instead of 30 percent for enterprise and 70 percent for consumer, it’s going to be completely the other way around. It will be 70 percent enterprise, 30 percent consumer. And there will be a lot of services developed. So, as you said, it’s all about use cases. It’s all about efficiencies; it’s all about different things we’re doing.
I wear two hats. I drive our global business around AR/VR—in fact we don’t even call it AR/VR to be honest, because we believe that it’s going to be beyond AR/VR. I think we should call that “extended reality” because we want to combine different sensors and different technology and capabilities that’s going to make you transported to a completely immersive experience. It’s not just the visualization, it’s also the audio feedback, it’s also the olfactory sense, combining that together, the cumulative effect is going to make something really magic, as Tony mentioned.
Going back to enterprise, that’s basically one part of my business. The other part of my business is to run all of the R&D globally for Accenture. What’s interesting here is to look at the use cases that are going to be prominent for our clients. There are a couple examples of things I want to share with you. Typical things that we’ve been doing for aeronautics, for example, for the aerospace industry. We’ve been releasing a project with Airbus on their 330 manufacturing platforms. What they do is here is that the one thing we can do to help the workers to be more efficient in the way that they bolt seats in the plane. For all of us it seems pretty simple: we get a bunch of bolts, we have an instruction guide, and off we go, we just have to bolt them. Well, it’s a bit more complex than that, because there are a bunch of holes in this whole plane, and very often what we figure out is that they start with the first seat, and they shift for one hole and the whole thing has to be done again. So what we’ve done here is used Vuzix, this display system, to provide augmented reality and tell them exactly where the hole is that they need to bolt the seat on to.
Fried: And the benefit here is obviously—
Carrel-Billiard: 500 percent increase in efficiency. This is it.
Fried: Also, you have this thing where people put up with a lot more bulkier thing. One of the issues with the technology that powers particularly AR, ODG and others have these glasses and they’re nice enough, but they’re not the kind of thing that you nor I would want to spend four or five hours with our friends wearing, and they cost more than we’d want to spend.
Carrel-Billiard: Yeah, and that’s it. I think you’re right. We’re not yet there, where we’re going to be able to deploy all of these devices at scale. Why? Because they’re, as you said, too heavy, the battery life is not good enough, and they don’t have all of the certifications. They can be IP6, but they need to be IP7, they need to be “x-compliant” for explosive atmosphere environments and all those different things you need, for example, in an oil and gas refinery. It’s not there yet, but it’s coming. And in fact, if you look there are three stages: there’s the consumer device that we play with and that can work in a confined environment; there’s the kind of “bridge” device—we had some people here who were showing these kinds of devices yesterday, Realware for example—it’s a device you can wear on your helmet that has IP6 certification and then you can start developing use cases there; and then in a year or two years, you’ll have a really rugged device, things that are going to last for 12 hours on battery and you can drop them, they don’t break.
Fried: So that piece, to me, as someone who’s covered tech for a while, that’s the piece that I’m pretty convinced gets solved. It’s a thing over time. It’s units, it’s Moore’s law, cost come down, and size comes down.
Carrel-Billiard: Yes.
Fried: On the content side, what are we going to do with this side? Will it just be the typical thing where on the consumer side, the Foxs and Disneys will come back when the numbers are there? Is there a catalyst in there? What drives things back to consumer and the masses? Or on the enterprise side, what takes this from niche use cases of fixing an airplane where the value is huge to the average productivity worker? I’ve seen demos where I’m typing on Microsoft Word and instead of using a monitor, I’ve got a headset because I can use six windows at once. What drives both sets of adoption?
Parisi: On the consumer side, the Foxs of the world are not just sitting around; they’re not just “wait and see.” Some of the big game studios are definitely “wait and see” and haven’t committed a ton of resources. But when you look at the entertainment industry more broadly, and Hollywood, this idea that you could create cinematic experiences that take people into a magical, artificial world, there might be an extension of a brand IP you already have in the theatre is attractive. It’s not exactly like the movie business is lighting up the world these days, so they’re also looking at this as potentially a new business model. They’re keeping a really close watch on it and they all have innovation labs, Fox Next, Viacom Next; there’s a lot of these out there in the entertainment world that are trying to push the envelope on the medium a little bit to understand what content creation means here. They don’t want to wait around and get caught by the time these devices hit mass scale.
That being said, other than a few on Steam for the HTC Vive for example, where you have a few hundred thousand of these headsets, there have been a few titles that have made a few million dollars, so there’s some small independent developers that have done pretty well on that. There were some early signs of success, but no one has had this breakout killer app on the consumer side yet.
While this is all going on, though, folks like Google aren’t standing still, both from their mobile side and even doing things for the Vive and these desktop platforms. They’re thinking about this as education, as tourism. They brought Google Earth into VR, they’re thinking about a program where they’re taking you places around the world and that can be a combo of video and synthetic media. When you look at a company like Google, they’re really thinking of this as the future consumer interface to teach people, to take them places, and they’re making some experiments.
Nothing has really broken through yet, but we’ve got the shared belief that something will. We’ve got to enjoy of these waves of technology adoptions in the past to know that we couldn’t have predicted the killer app going in, and so some of this is just a matter of “time will tell” and staying in the game.
Fried: In terms of what you’re seeing, Marc, do you see businesses that are thinking of this from a massive adoption standpoint? Do they see a date at which a lot of their workers are doing it, or is it niche as far as the eye can see?
Carrel-Billiard: That’s a really good question. I think for now it’s really niche but if we want to go to massive adoption there’s a couple of things we need. We discussed about use cases, and it’s not because the devices are not yet ready that we cannot start working on those use cases. The second thing that’s going to be very important is that it’s all about business case at that end. If there’s no business case to support your use case, then why would you want to deploy that at scale? One thing that’s going to be very important is the business case, and that supports the use case, and that’s something that we’re spending a lot of time with our clients to understand. So that’s where you need to be with the client, understand how he works, and basically the daily life of those workers, and how you can help them out.
And the last one is how you’re going to finance all of this investment. You have all of those devices—they’re pretty expensive—and, of course, if you order many the price will go down, but it’s still expensive infrastructure and in a very brownfield environment, sometimes our clients are not ready to invest so much. So that third thing is how you’re going to finance it, so all of the engineering with that, and then we can think about different ways. For example, you could pay for it and you have a subscription model where you amortize the upfront investment. There’s many ways to think about it.
If we combine all these things together, and you prove that it’s technically wise, I think you have something good.
Fried: And where does collaboration fit in? I want to get back to the consumer side, but business is often where the dollars are. The fact that most companies are spread out, they have employees in lots of different areas. There does seem to be something compelling about this idea that we could be hosting this panel and you couldn’t make it because of something, and somehow your hologram is onstage—is that a compelling business use?
Carrel-Billiard: It is.
Fried: How far is it technically—
Carrel-Billiard: I think it is. I think the next revolution is going to be the shrinking of distance. People will be on the stage with you, and it’s just bigger and better than Telepresence. I drive all the R&D globally at Accenture, and one thing we have are labs and smaller labs called “nanolabs” everywhere in the world, and I’m using those nanolabs as teletransportation pods. I have my clients come to visit us, but they don’t have access to the lab, so we’re working on a teleportation platform. Right now, we’re working with virtual reality and avatar, and I think the next road map will be holotransportation. Microsoft is testing that in their research center as well, and this is something that we could do. You can 3-D scan someone, and in almost real-time you reproduce that somewhere else, this is something that works well.
Parisi: I don’t know if you folks have seen, but Microsoft just announced these mixed-reality capture studios. They’ve had one in Seattle for years under wraps, they just opened one in San Francisco and in London, and the captures they’re doing with this are incredible. They capture things at very high-resolution point clouds, and then they can turn them into reasonable resolution 3-D meshs and texture maps you can bring into a Unity application.
Recently, I don’t know if you heard about this, there was a music video project that was just done this way using Unity. It’s both a music video and a VR piece interactively where Billy Corgan, who is the lead singer for the Smashing Pumpkins and is now doing a solo career, was captured playing at the piano this wonderful acoustic piece which is the first single off his new record. The folks from Viacom Next built the content and Danny Bittman, the premier Tilt Brush artist in the world made the most amazing environment in Tilt Brush. All at the center of it is this captured Billy Corgan that’s pretty good when you consider where we’re at with this technology. It’s the best thing I’ve ever seen out there.
And when I saw that—I’ve seen a few instances of that over the last year from that capture studio—I’ve started to think, given how small they can make it, that we’re probably not too far off from having the kind of holographic capture you’re talking about, where that could be virtual Marc over here. We’ve got a way to go before we can project like this in midair, but the idea that you could have headsets on and have that kind of experience where you’ve got a collaborative social thing going on for customer support, for teleconferencing, for entertainment, for social media purposes—I don’t think it’s that far off.
Fried: I do want to get back to where you guys see the future in just one second, but I want to talk about one of the present constraints. We’ve talked about technology and we’ve talked about use case. One of the others, which you’re probably keenly familiar with, Tony is the fact that we have a lot of people that know web development. We have a lot of people that know coding. To do VR or AR, or gaming, you need a specific set of 3-D skills and it’s a finite set of people. Are we seeing the market choose its bets—a bunch of them went it VR and now some of them are going into AR—and how big of a constraint is that, the number of people that know how to use your tools, for example?
Parisi: That number is getting bigger every day. I’ve been in this field for 20 years and it was horrible 20 years ago, it was really only a few hundred thousand people who knew how to do anything like this. Unity has a million monthly active developers. We have far more than that who have registered lifetime, and when you look at the number of folks who can use a 3-D modeling package coming from the visual effects industry or from architecture, it’s well into the millions at this point. And I believe that’s just going to be growing. We’re seeing kids picking up Unity in middle school now, so I believe that is a skill that people are going to pick up and develop. It’s not like you can just snap your fingers and instantly have a z-axis and you’re a 3-D designer, you do need to skill up for that. But I don’t think it’s as small as people might think it is, and so as this wave happens, just like 20 years ago nobody was a web developer, now anyone can be a web developer. As this expands I think we’re going to see more and more people pick up 3-D development skills and the tools like Unity will get even easier. It’s still a programmer’s tool right now, but we’re spending a lot of time with other industries like auto, the building and architecture industries, brand creative, and these other places that need a little bit different workflow and easier tools, so I think the tools will also level up to let other people get into the game.
Fried: We’ve talked about holograms [and] telepresence, we’ve talked about moving into this world where workers in a particular skill have glasses to help them do their job. Talk about some of the things that you think are five to ten years off that we aren’t thinking about. What are the use cases that aren’t ripped straight from science fiction?
Carrel-Billiard: One thing that the lab has been doing is looking at the automotive industry. Look at how they’ve been designing new models using clay to design the shape of the car. Obviously, they moved to CAD/CAM environments like Dassault Systèmes and so forth. But I think what we need to see now, and it goes back to the discussion about what type of skills we need, is that we’re going to see talent migrating there and tools that are going to start merging. We’ve going to see the CAD/CAM world where people are designing things in 2D and eventually they will transport this into a virtual reality world that will be shareable by everyone. So using HoloLens glasses or any type of mixed reality glass , people will be able to share what has been designed and be able to work it out from all over the world. That’s going to be really, really interesting. And there are companies that are really bridging those two worlds—the world of CAD/CAM, the world of virtual reality, mixed reality—and it’s been acquired by Microsoft recently because that’s exactly what they’re looking at, and we’re going to see more and more.
I want to go back to the discussion about the talent. We see a lot of people learning Unity platforms. The reason why people are pushing into the 3-D is what I call IT consumerization. A lot of things are being pushed to the people so that they learn by themselves. Look at Microsoft: The last two updates of Microsoft are all about 3-D; it’s all about VR, AR, mixed reality. Creator update, full update. Paint, all 3-D. Portal to access desktop, Visual Studio has Unity plugged in.
Parisi: PowerPoint has 3-D built in.
Carrel-Billiard: That’s right. People are curious, they’re going to start playing. Kids are playing with that at school and everything. So, we’re shifting, really, and I’m kind of joking with our leadership at Accenture because 30 years ago, what were we doing? We were digitalizing 2D documents using optical character recognition into PDF documents, and today we need to do that for three dimensions. We need to take every trillion paths out there, and 3-D scan them so that we build a hologram that you can bring to your mixed reality to do exactly what I said between the CAD/CAM and the gaming environment.
Fried: Tony, when you look out a little bit, what do you think is possible that most people haven’t gotten their heads around?
Parisi: It’s hard to imagine these without getting into science fiction, but I’m going to try.
Fried: You can.
Parisi: Let’s just pick up on the auto example that Marc just brought up. In that world of the future—the self-driving car that we’ve all been talking about yesterday on the panel here—that self-driving car will be designed completely in VR. The factory it’s built in will be completely designed in VR, the assembly procedures for putting in together—humans will still be involved in that, it won’t all be robots—will be in some kind of VR simulation and training because it’s going to increase productivity and improve safety dramatically. The self-driving [vehicle] won’t be put on the road until they’ve been simulated billions and billions of times with headless computers doing 3-D calculations. The network of these self-driving cars is going to be monitored by operators wearing AR headsets to see the vastness of the information there. My kid is going to get into one of these to go to his college, and what’s inside it—and they talked about this yesterday, actually—the cabin of that car is an entertainment center or it’s a game console or it’s a web browser or whatever you want to call it.
Fried: So, wait, you’re not actually going to drop off your kid at college? They’re going to get taken in a self-driving car and you’re going to have to have that goodbye moment, you’re going to be crying from a VR headset?
[LAUGHTER]
Carrel-Billiard: Eventually, yes.
Parisi: Yeah. But they’re going to get in there and this cabin is going to be this holographic cabin that’s going to be full of entertainment. They’re going to check their social feed, they’re going to go on Twitch and check V-Sports; they’re going to get a call from their Mom interrupting them and reminding them to eat, and she’s going to be a hologram in there. [LAUGHS].
And everything I just said, it’s not that science fiction-y because it’s all happening. Unity is having conversations—and I’m sure Accenture is too—with the automakers where they’re doing all this already. Marc already talked about this. They’re designing the cars and everything from that to the entertainment system, which initially is going to be little 2D screens in the cabin, and you can imagine it won’t be long before that’s holographic entertainment as well.
Carrel-Billiard: I want to go back to what Tony just said, it’s interesting to see this cockpit of the future in this car because with those driverless cars you need to find something to do.
[LAUGHTER]
Carrel-Billiard: And eventually this car will be turned into ‘les salon’, where you can have your hairdo. So this is how it’s going to work: You drive your car, and you use artificial intelligence to talk to your virtual agent to say ‘I’d like to have a new hairdo.” And what it will do is transform completely the car and the look and feel, and you’ll be sitting in this seat. At the same time, they will connect to the ‘Uber of barbers’ and the guy will stop by in your car, do the haircut, and then move away. What’s going to be very interesting is that today, we have the technology ready in terms of platform already, we just have to connect the dots together to invent a new world, and this is coming.
Fried: I have a couple more questions, but I do want to leave a little time for audience questions. If there are people out there, please wait for a mike, because then we can record your thoughts and trap your brainwaves for your future hologram.
[LAUGHTER]
Fried: Anyone with questions Right in front.
Audience Member 1: How does Unity think about voice design and integrating voice interfaces into the design and development processes for VR?
Parisi: We’re super excited about it. We don’t really have anybody in-house working on designs like this—we do have what we call Unity Labs, which is a group who does work on some design projects. Currently we’re more focused on the mixed-reality aspects of it, which includes gesture as well as which kind of headsets you’d be using to do everything from ‘make this table come to life’ to ‘make a business card turn into a live interface starting with a physical business card’. We do assume there will be some voice activation around that, but we have not plumed the depths of that design.
The way the company works is that we’re very much an open ecosystem, so we just assume someone’s going to come along with some voice plugins that it’s really going to drill in and specialize on that. So that’s not an area that we’ve done some deep work on yet.
Audience Member 2: You’ve talked about how the resolution isn’t quite there with VR glasses yet. When do you predict that we’ll have things that are the weight of normal glasses like the one Ina’s wearing that can actually do AR and VR in a resolution that looks like the real world?
Fried: These actually are, I’ve been watching Avatar the whole time.
[LAUGHTER]
Fried: Seriously, that really is the consumer question: When can I put on glasses that can do AR that look like this?
Carrel-Billiard: My guess is going to be definitely less than five years’ time. We see that already, there are a lot of labs and companies that are working on this. I think in less than five years we’ll see some real, tangible results of using these systems. Again, it’s all based on certification, but it’s coming very fast. I think one thing that we haven’t mentioned, also, is that we’ve been talking all about efficiency, productivity, a new way of doing things, like instant gratification, you could be transported anywhere in the world.
But we have not addressed the VR for good—how virtual reality is going to help to bring more inclusivity into what we’re doing. This is very important. I wanted to give you an example of what we did recently with the National Theatre in London for people that cannot hear, cannot listen to a play. They wear these very lightweight AR glasses and get the captions displayed as they’re watching the play. How cool is that? I think that’s what we need to see also. There’s going to be a move there for inclusivity, through VR for good, that we’re going to see new devices that are going to be developed like this: lightweight, better battery life, and different things.
Fried: And when you talk about what’s really been compelling so far in VR, to me if I were to list off the few most compelling examples in VR, they’ve been things that put me in someone else’s shoes. Clouds Over Sidra that took you into a Syrian refugee camp—
Carrel-Billiard: Yeah, empathy—
Fried: Or going places I physically couldn’t go. Maybe I can’t go to Costa Rica because I can’t afford it, or I don’t have the time; other people might have physical limitations. How important is it as a tool for inclusivity and for equality?
Carrel-Billiard: For us within Accenture, we’re about 420,000 people so we recruit about 60,000 people per year. So we’ve got to make sure that when we recruit those people, we bring the experience of what it is to work at Accenture. We’ve developed a VR application that mimics what it’s like on your first day, how stressed you might be, explaining this story. Empathy—how you manage empathy. We did another one, which is really cool. It’s all about gender swap. So we take people, and then we work with MENSA Lab, which is a company in France, and we’ve developed a simulation of how it is to have a meeting with, for example, five ladies and you’re the single male, or vice versa. And then we quiz you, and look at how you reply to questions and how you behave. Based on that we can help you out to improve in your management style.
Parisi: I’ve seen a couple great examples of this kind of gender swap or empathy exercises that are intended to educate around sexual harassment, sexual assault. I just became an advisor to a project like that. I think the potential is incredible. A lot of that is video based right now, I think we’ll see some combination of video and CG as this evolves but the storytelling in video is little bit more cost effective and it’s the thing that a lot of documentarians and educators know a little better, but I think there’s super high potential.
Fried: I’d love to keep talking about this, particularly in how one could change genders in VR, that might be interesting. It’s been a pleasure. We’re going to revisit this in a couple years, we’re going to have you back onstage and we’re going to have your avatar, Marc.
Carrel-Billiard: That’s right.
Fried: Excellent. Join me in thanking Marc and Tony.
