Colleges and universities, often at the foreground of providing creative incubation for the research and development of new technologies, are beginning to explore the boundaries of virtual reality (VR). A few of them have funded incubators with agendas that differ in both practice and theory. Graduates and undergraduates are creating prototypes that include everything from documentaries to mapping systems, and off-campus collaborators—be they media giants or government agencies—are exploring a myriad of possibilities with a technology that's still in its infancy.
"I'd say when you ask where we are now, I liken it to the end of the 19th century in the early days of cinema," says David Nelson, special project manager of Mark Bolas' seminal VR facility, MxR Lab, at University of Southern California's Institute for Creative Technologies, and producer and director of the documentaries Naked States and Positively Naked. "We're Thomas Edison shooting films for the kinetoscope, or we're the Lumières shooting a train pulling into a station right now. There's the element of novelty. People are experiencing it for the first time."
Nelson's assertion holds true especially when looking at The New York Times' development of iOS and Android VR apps and the company's recent distribution of Google Cardboard to print subscribers. While the content and hardware is very simple, it's astonishingly effective and there's excitement generated every time new subject matter is made available.
MIT's Open Documentary Lab is possibly the best known incubator in the US. It was founded on the idea that media is fundamentally changing in terms of how it's created, consumed and processed. Lab participants push the boundaries of documentary while grappling with the sea change in technology, and what it means for both the maker and the audience when viewed from a historical and cultural perspective.
Offering courses, workshops, a fellows program, public lectures and conferences, the Open Documentary Lab incubates experimental projects and develops tools and resources, while partnering with artists, journalists, technologists and media-makers. The lab collaborates with the Sundance Film Institute and the Tribeca Film Institute, as well as the National Film Board of Canada, to help build the field of interactive documentary as it advances technologically. Its director, Sarah Wolozin, remarks that VR is its latest moment. "Our lab has been around for four years," she explains, "and VR has been the one that has caught the attention of the culture much more so than online has with base documentaries. I think there's much more of an awareness now that we really are changing the way we tell stories, and we need to be thinking about media and documentary on other platforms."
Wolozin acknowledges the tension as doc-makers struggle with what VR can and cannot do and points out other users at MIT who employ it differently, such as gamers and architecture students: "You know, with documentaries, how it's the show versus the tell? Well, in VR now it's mostly show. And so we're still figuring out how to tell stories with virtual reality." The Lab encourages dialogue with those makers in other departments who contribute to VR's language and development in ways that could benefit the nature of storytelling as a whole.
If MIT's incubator is the open-source, academic guide for digital documentary storytelling as the technology shifts, USC's MxR Lab is tactical in its exploration of technologies, as they're designed to improve the fluency of human and computer interactions in the creation of synthetic experiences. As part of the Institute for Creative Technologies (ICT), the Lab works with the Interactive Media Division at the School of Cinematic Arts and the Viterbi School of Engineering (although all of the USC schools partner in its research).
Primarily engaged in research and development, some of its outside collaborators include large entertainment, animation, visual effects and motion capture companies, as well as the US military and healthcare and cultural organizations. They often work with hardware and software developers as well. Like Wolozin, Nelson is looking at VR's capabilities and its lack of a true filmic language, noting that an inability to edit without disorienting the viewer is a significant drawback. The lab is actively researching the developing language as it relates to elements of storytelling in fiction and nonfiction. They have a studio on campus at the School of Cinematic Arts, and lab participants rapidly prototype virtual and immersive experiences or scenarios to test what works and what doesn't to glean what aspects of the language stick.
What's most interesting to Nelson is that despite its current limitations, VR is still a facilitator for documentary storytelling. "I think the documentary form is working quite well in VR now," he explains. "Maybe better than fiction because the rules of fiction are maybe a bit more formal than the rules of nonfiction."
Nelson cites VR godmother Nonny de la Pena's 2012 work Hunger in Los Angeles, which was commissioned by the Institute for Creative Technologies, as an early example of the successful use of VR documentary storytelling. De la Pena had been a Graduate Fellow at ICT and is currently a Research Fellow at USC's Annenberg School for Communication and Journalism.
A chief area of interest at the lab is the kinds of tools consumers will favor when they engage with VR programming. While we're already seeing downloadable mobile content with Samsung Gear and Google Cardboard, which is low-threshold work, the more tethered platforms—Oculus Rift and the HTC Vive—engender higher bandwidth and richer, more complex content.
"You get a higher resolution and more interaction," Nelson notes. "There's just more computer processing power behind the machine that's tethered to a head-mounted display." Currently, most of the advances he's seeing for tethered platforms are in game development, while the more interesting experimental content is still on the mobile platforms, but that may be more a question of access and affordability than anything else.
Documentaries will only benefit from the intensifying development of VR's hardware and software components, and Nelson is already seeing its confluence with genres and best practices in the creation of immersive projects. "We've already had a little bit of a conversation about the platforms that people are going to use, mobile or tethered or full-body tracking or just head tracking," he explains. "That's the hardware. Then there are software platforms. They are building it like a game engine, like Nonny de la Pena, or they're shooting live action. Those are two different approaches that are being developed. That's an area of our research."
If MIT's Open Doc is theory, USC's VR/MxR Labs practicum, then Media Design Practices (MDP)—a graduate program at the Art Center College of Design that is an interdisciplinary design curriculum grounded in media and technology—is the future emanating from your smartphone, or anywhere else you could imagine having digital access.
Lab students conjure experiential, speculative prototypes at all scales. A documentary story may be a piece of a puzzle, found on an immersive project that relates to the temporal experience of riding in a car or floating above Mars. The collaboration may be with Jaguar or JPL.
Anne Burdick, chair of MDP, says the lab is somewhat at the outer edges and very much in discovery mode. Participants in the lab want to ask what the product is and what its implications, opportunities and cultural applications are. Most importantly, they need to determine what the role of design is when it's brought to life.
"We're definitely not looking at purely immersive narrative," Burdick maintains. "We're actually interested in looking at the way that AR and VR are integrated into all kinds of communication systems that exist and will persist."
Regarding MDP's intensive research orientation, Burdick says they encourage their students to get their hands on beta technologies to see what's possible. Their relationships with technology companies sometimes begin in the earliest formative stages of new tools, or they might be further along and in the fine tuning of consumer experience stages.
Last year, the lab collaborated with Intel Corporation and Jaguar Land Rover for a project that's instructive to immersive storytelling technologies. Students studied self-driving or semi-autonomous cars, and considered the idea that if someone isn't driving or if their relationship to the road is mediated through a technological interface, it would change the way they think about driving and what they're doing inside the car, which led to all kinds of possibilities.
According to Burdick, a lot of mixed reality, hybridized and mediated co-presence experiences were designed and speculated about, including the idea of friends in other cars visiting via hologram. Ultimately, it was about the way we change our relationship to a car—much like we'd change our relationship to the way we perceive a documentary piece once we're immersed in it.
Burdick points out that Intel is designing its computer chips 10 years out. While the company is interested in what kind of computational capacity a chip has to have, its questions are being driven by technological capability, and should resonate with VR story-makers. They are considering the capacity for live simulation. Intel is concerned about monitoring multiple data streams simultaneously. What will users do with these new kinds of experiential opportunities? Is this something we're going to be thinking about every day? It all becomes useful information for the future.
Elisabeth Greenbaum Kasson is a Los Angeles based writer and editor with an abiding interest in cross-platform storytelling and how it intersects with technology, business and culture.