Note: Augmented reality (AR) generally refers to digital content overlaid on the physical world. Mixed reality (MR) represents a more comprehensive integration of virtual and physical, in which virtual objects map onto, and are responsive to, features of the physical environment. Meanwhile, the term XR, or extended reality, is gaining traction as an umbrella term for a wide range of experiences that combine physical and virtual worlds, including AR, MRand VR. (X acts as a variable that can stand for any letter.) These related terms and concepts remain fluid; while this article focuses on AR, many of its observations apply equally to MR and XR.
Documentary has long helped to pioneer new media technologies, from the "actualities" filmed on the earliest movie cameras to today's groundbreaking nonfiction virtual reality. Documentarians have been some of the earliest adapters to everything from sync sound to mobile storytelling. The emerging and rapidly evolving fields of augmented reality (AR) and mixed reality (MR) present abundant new possibilities for the documentary field.
Three innovators who champion interdisciplinary and inclusive approaches for emerging media—Gabo Arora, VR creator and director of Johns Hopkins University’s new Immersive Storytelling and Emerging Technologies Program; Ingrid Kopp, curator for Tribeca Storyscapes and co-director of Electric South; and Kamal Sinclair, director of the Sundance Institute's New Frontier Labs Program and author of Making a New Reality—shared their insights on AR's documentary potential, as well as their advice for traditional media creators interested in exploring this new medium.
Arora, Kopp and Sinclair agree that AR presents many challenges. The available technology changes constantly, with new hardware and software announced daily. It's hard to keep up with these developments, as well as their potential impact on budgets and logistical decisions. It' also daunting to purchase expensive equipment when it may quickly become obsolete. "If you invest in a camera or even headsets, chances are, six months down the line it's going be out of date," Kopp notes. "And I do think it's literally six months. Not even a year."
Obtaining funding for AR projects can also be difficult. While independent documentary has a long-established support infrastructure—residencies, labs, major funders—this sort of backing for AR and other new media is still in its infancy. "There's definitely money out there," Kopp notes. "But a lot of it is very obtuse—it's not clear where the money is." Not only are potential funders scattered across different industries, but they are also understandably uncertain about emerging media's creative potential and often limited audience reach.
The terminology and very definition of these media are also in flux. AR is beginning to converge with VR; new headsets can switch between, and combine, the two modes. AR is also gradually being superseded by MR, both as a term and a conceptual framework. Arora has shifted away from the term AR, saying that "augmented just feels a little bit too passive" to encompass the interactive capabilities of MR systems like Hololens and Magic Leap. He uses the increasingly popular term XR to describe his studio, but also predicts that in general, "There's a lot of nomenclature that's going to feel clunky" in the near future.
Despite these challenges, creative AR is growing. Most applications of AR thus far have focused on industrial and commercial uses like military training, manufacturing and gaming. However, the technology is becoming more accessible. There are an increasing number of what Kopp calls "plug-and-play solutions"—software development kits like ARKit and ARCore, 3D sensing tools like DepthKit—that allow creators to produce AR projects without having to code from scratch, purchase expensive equipment or invent new workflows.
There is also a burgeoning community for creative and documentary AR, which has helped spread awareness of the medium’s potential, build support networks and generate opportunities for collaboration. Artistic AR projects can be found at pioneering festival programs like Sundance New Frontier and Tribeca Storyscapes; university research centers like MIT Open Documentary Lab; and incubators at arts and technology institutions like NEW INC at the New Museum.
These developments are heartening because in many ways AR is particularly well suited for documentary. One of its primary features is the ability to combine virtual content with physical environments. This allows creators to overlay information on structures and spaces, and explore new modes of interactivity, learning and storytelling. "There are some really interesting things you can do with space and place and documentary and storytelling when you start to think about AR," Kopp maintains. By placing virtual layers—whether information, ornamentation or provocation—on physical spaces, creators "can have people question their realities a little bit differently."
Documentary, in turn, can help illuminate AR's potential and shepherd its evolution. Current mainstream industry discourse focuses on a utopian vision of universal smartglasses deployed in pervasive, global AR environments. This conception of AR elides many practical, creative and ethical concerns, and desperately calls for alternative viewpoints and critical intervention. "If you're somebody that falls into some sort of traditional discipline, know that you have a lot to contribute to this space," Sinclair says.
Documentary filmmakers and other traditional media creators offer expertise on shaping narratives, partnering with communities, and dealing with sensitive subject matter, among other things. AR and other new media are highly collaborative endeavors, and Arora encourages people not to feel intimidated because they lack technical backgrounds. He points to innovative VR projects by artist Rachel Rossin and dancer Lily Baldwin, and adds, "Whenever I've asked for help from technical people, they're incredibly excited to work together because they need those other human experiences. They need the diversity of experience in order to make something that can really shine."
Documentary in particular also has a long tradition of discourse around ethical issues like privacy, consent and access. AR and related technologies present serious ethical concerns, including biometric surveillance, exclusionary algorithms, and a lack of diversity in terms of who is able to experience and create these new media. "Whether it's going be used for good or evil is up to us and how we design the value systems and infrastructure around it," Sinclair asserts. In this nascent moment, while the contours of emerging media remain malleable, it's vital for diverse creators to participate in shaping AR’s future as a technology, medium and platform.
Strategies for Documentary AR
For creators who are new to emerging media, Kopp suggests, "Always look at as many projects as you can, and try to talk to those creators about the process." She notes that it's important to be realistic and specific about a project’s budget, goals and intended audience. AR is expensive to produce and difficult to connect to audiences, and being aware of these challenges from the start can mitigate potential problems. Kopp adds that smaller projects—even just taking a few days to play around with a concept—are an easy way to experiment and learn.
Sinclair urges creators to consider how projects might intersect with, and perhaps reinforce, structural inequality and exclusion—like "who gets to go to certain neighborhoods." For example, avid Pokémon Go users sometimes played late into the night, traversing city streets and parks. But for young men of color, who are highly policed in public spaces, this sort of gameplay increased their exposure to potential profiling and violence. Since AR inherently deals with how people move around physical spaces, creators have a responsibility to consider spatial power dynamics.
On a broader level, AR as a concept encompasses many different forms of augmenting the physical environment—not only screen-based digital overlays, but also mixed-media installation, projection, the Internet of things, audio and haptics. Just as VR projects can take inspiration from gaming and immersive theater, AR experiences can draw on existing forms of spatial narrative and documentary praxis, including everything from LARPing to community murals. "Everyone immediately thinks of headsets, and they don’t think about some of the simpler ways that you can layer on top of reality, which is a shame," Kopp asserts. By exploring a wider range of AR, we open up space for more people to participate, and for more creative possibilities. The following approaches offer a starting point.
The Power of Place
Many AR experiences do not take advantage of location—3D graphics simply float in front of users, whether they are in a conference center, forest or city square. However, Sinclair says that for now, the most compelling AR projects tend to be place-based and geolocative. This type of AR can "show the invisible relationships of a community, and uncover the histories of space in powerful ways, to help inform where we're going in the future."
One such project is the upcoming mobile AR app Freedom Fighter, created by Taura Musgrove and John Hopkins University in collaboration with Arora as executive producer. At historical locations around Baltimore, users encounter a 3D model of civil rights leader Dr. Lillie May Carroll Jackson, who narrates each site. The project aims to highlight Baltimore’s history of civil rights activism, and how this history is embedded in the city’s streets. "AR has the potential to unpeel and unravel layers of existence," says Arora. He notes that while location-based projects are limited in their audience reach and present many technical challenges, they can also inspire more meaningful relationships to place and community.
Audio AR
AR does not have to be visual: We encounter non-visual forms of augmentation every day, including audio crosswalk signals and voice- and motion-activated smart devices. One of the most accessible modes of AR, for both creators and users, is audio AR: audio that is triggered based on a user's location. With fewer hardware and software requirements, and a long tradition of documentary sound production to draw on, creative audio AR is flourishing. One popular format is the mobile audio walking tour, exploring topics from literary history to local food. Audio AR is also being used for gaming, immersive theater and accessibility for the visually impaired.
Audio AR has a number of advantages over visual AR. While AR headsets tend to be bulky and awkward (most are also tethered to a computer), audio AR requires only a smartphone and headphones. And while visual AR can distract from its surroundings with overly elaborate interfaces and graphics, audio AR interfaces are generally minimal. In some ways, audio AR currently allows for a more seamless and immersive overlapping of virtual and physical.
Community-based Practices
Community-based practices like oral history and collective storytelling are a powerful model for AR, because community memories are already present as invisible layers on the physical landscape. Emerging media offer creators and communities new ways to explore these complex spatial histories. The 96 Acres Project, a series of community art projects at Chicago’s Cook County Jail, brings together activists, youth, jail administrators and formerly and currently incarcerated people to investigate the jail’s impact on the surrounding neighborhood. For one project, videos created by local artists and incarcerated men were projected on the jail wall, sharing unheard stories and reclaiming the space for the community.
The AR app Wikiupedia (a wickiup is a traditional Indigenous dwelling) also takes inspiration from community-based practices. Seeking to preserve cultural heritage and increase awareness of Indigenous histories, it overlays crowdsourced Indigenous stories onto several Canadian cities. Collaborative projects like these allow communities to reclaim and reimagine spaces and spatial narratives.
AR as Protest
Some of the most striking AR projects thus far have used the technology as a means of protest. For example, AR Occupy Wall Street allowed people to virtually protest on Wall Street after police prevented activists from physically demonstrating there. Crowdsourced 3D art was assembled into an AR layer blanketing Wall Street, in which large crowds raised protest signs and the New York Stock Exchange became an enormous slot machine. Projection also enables high-visibility protest that is relatively low-cost and low-risk; activist groups like The Illuminator and Chinatown Art Brigade project enormous protest slogans on buildings. Virtually annotating space in this way can call attention to injustice—and resistance—embedded in the physical environment.
AR projects can also critique the medium itself. Snapchat recently launched an AR art initiative with a digital Jeff Koons sculpture in Central Park; overnight, an identical virtual sculpture appeared in the same location, except this version was tagged with graffiti. This symbolic protest interrogates corporate control and commodification of virtual public space.
New Approaches to Archival
AR also presents new ways for documentary creators to engage with archival material. Rich historic archives can be brought out into public spaces, recontextualized and shared with new audiences. The mobile AR app Chicago 00 overlays photographs from the Chicago History Museum’s archives onto their corresponding present-day locations, creating striking side-by-side comparisons. (There are now quite a few of these historical image comparison apps, however, so creators should consider what new features they can bring to this format.)
Meanwhile, participatory AR projects generate new, crowdsourced archives. Invisible Monument is an audio AR app that allows users to record demonstrations onsite, creating a geolocated archive of participants' experiences. This approach democratizes the notion of archival by allowing anyone to contribute, and by emphasizing the value of quotidian and personal experiences.
Participatory Storytelling
Many of the projects discussed above are notable for their participatory approaches to storytelling: They invite users to craft their own stories and collectively shape an evolving experience. Participatory storytelling not only allows more—and more diverse—voices to be heard, but also reflects a media landscape in which audiences are increasingly creating and publishing their own content. While many existing AR projects are individual and passive, AR has the potential to add new layers of social interaction and collaborative creativity to physical spaces.
Participatory projects push creators to reexamine conventional notions of authorship and narrative—critical reflection that is essential for emerging media. AR and participatory projects also share an inherent unpredictability, due to the changing conditions of the physical environment and the shifting inclinations of users; participatory AR invites creative experimentation with this dynamism and porousness.
AR can serve documentary in many ways: helping to narrativize, contextualize and critique; facilitating collaboration, discussion and exploration; and changing how we understand history, place and each other. In turn, documentary can help reveal and expand AR’s potential, and investigate its pitfalls. In this nascent stage, the possibilities for AR are endless—and the stakes are high. It is crucial that we seize the present opportunity to gather together diverse voices in critically, creatively and boldly imagining the future of AR.
Sue Ding is a documentary filmmaker and emerging media consultant based in Los Angeles. She has written about AR and VR for Indiewire, Immerse and MIT Open Documentary Lab.