Experiential Learning with AR In the Carnegie Natural History Museum
User Experience Design
Cinema 4D / Sketchfab
Aero/ Reality Composer
In 2029, you have been tasked to improve the user experience of the Carnegie Natural History Museum with interactive environments that are:
Today, many museums are facing the challenge of transitioning from rigid institutions to experiential and flexible spaces. This is driven by such factors as expanding collections, increased competition for visitors, and visitor expectations for greater engagement. Museums are turning to virtual reality, apps, and interactive experiences to keep tech-savvy visitors engaged.
How can visitors engage more than their sense of sight?
How can immersive environment foster more engagement?
How can a designed system adapt to changing behavior and environment?
Aurum is an augmented reality experience for the Carnegie Natural History Museum that leverages smart-glass technology, immersive storytelling, and participatory experiences.
Exhibit-Level User Journey
3 major part of the Experiential Learning
01 Arrive at the Museum
Upon arrival, visitors receive their ticket , smart AR glasses, and a palm stamp. Smart glasses reveal AR content and the palm stamp serves as an image anchor for an AR virtual object.
02 Aurum Onboarding
During onboarding, visitors are introduced to Aurum and different gestures that will help them navigate through AR content.
Gesture of Pinch
Gesture of open palm
Gesture of toggle
03 Exhibit & Virtual Object Intro
As visitors explore the museum, floor makers indicate the start of AR-enabled exhibits. At the start of the Mineral Hall, visitors are introduced to the narrative of the exhibit (conflict minerals) and can choose a virtual mineral to guide their experience.
04 Learn Natural Science
Visitors can explore mineral geology, atomic structure, and properties. Properties include fracture, cleavage, radioactivity, fluorescence, and hardness related to the specimen that is in your palm.
05 Explore Human Impact
After learning about mineral science, visitors can explore human impact stories in a more immersive experience. Stories follow miners, supply chains, or global efforts around conflict minerals.
As visitors navigate through the space, they can discover their virtual object’s physical form and continue exploring more AR-enabled content.
06 Reflect on Experience
At the end of the exhibit, visitors can engage in a participatory experience by sharing or listening to the reflections of others.
07 Continue Discovering
Visitors can navigate through other AR-enabled exhibits by encountering more personalized contents.
System Controls1: Gestures
Aurum uses hand-tracking as an input modality. Visitors can navigate digital content using six gestures. We've learned a lot from the Oculus Design Principles, and landed in the following gestures in the design
System Controls2: Proximities
leveraging locational based anchor, we speculate that the Navigating system react to human proximity to the information, to prevent information overload, at the same time signify content curation.
The Aurum logo was inspired by mineral geometry and Da Vinci's Vitruvian Man. The Vitruvian Man was created during a time when people shifted their focus from God to humanity. We now hope to shift focus to humanity's relationship with the Earth.
Our research began at the Carnegie Natural History Museum. Luckily, we had the opportunity to speak with Rebecca Shreckengast, Director of Exhibition Experience. She gave us some valuable insights into the current museum experience from a curatorial and exhibition design perspective.
Q & A with Becca
8 Key Takeaways
Several key takeaways are generalized form this visit:
Museum's current mission is to find inspiration in the collections and advocate for a sustainable future.
The future of museum is bringing the audience a personalized, curated experience.
Narrations should be created in a way the audiences feel personally connected.
Complexity Doesn't Work
Complex design systems dont work well. (eg. prescribed wayfinding systems are usually not successful)
Successful experiences are participatory, allowing museumgoers to express themselves and trigger conversations. They are "sticky"
Deep time story-telling
Deep time is an important concept, but it's difficult to teach and engage the audience.
40% of the visitors come in groups. How can we design to trigger communication and interaction between them?
The museum owns 12 million specimins, only 3% of them are visible.
Identifying Opportunity - Hillman Hall of Mineral and Gems
When touring the museum, the Hillman Hall of Minerals and Gems caught our eyes. This exhibition artfully displays more than 1,300 specimens from all over the world that come in a large range of striking colors, fascinating forms, and dramatic shapes. While, after a 10-minute tour in the hall, we still remember very little about mineralogy and scientific information, because of the overwhelming volume of information and visual stimulations. We also talked to a few visitors to help us understand how they felt about the current experience.
Beautiful yet overwhelming
The Participatory Museum
When thinking about increasing engagement with museums, we read through Nina Simons' The Participatory Museum. This book discusses how cultural institutions can become more dynamic, relevant, essential places by creating a participatory museum's experience.
In order to understand the technology, we would be designing with, we took a look at the current tools of mixed reality and virtual reality tools and experiences, including Oculus Quest and Hololens headsets and various mobile AR prototyping tools such as Reality Composer, and Adobe Aero. Experiences include Teamlab's Crystal Universe, Smithsonian Museum of Natural History's AR project Skin and Bone, Cooper Hewitt Museum's project Pen.
01. Artifacts are missing context.
Minerals are decontextualized. The environments, ecosystems, and human relationships within which they are situated are not well represented.
02. Passive engagement.
This space requires little participation and engagement on the part of the viewer. Visitors only look at objects, as opposed to interacting with them.
03. Information is inaccessible.
The number of objects and accompanying visual content is a lot to digest, and can easily lead to cognitive overload and fatigue. Further, the content within the Mineral Hall is filled with scientific jargon that visitors may be unfamiliar with,
04. A larger narrative is absent.
A few successful experiences with the museum have strong, clear narratives structured around them. How can we create a stronger connection between viewer, artifact, and space by connecting to a bigger picture narrative? How can the space be more relevant to visitors?
How might we design for an AR museum experience to tell stories of Anthropocene, trigger communication and enable experiential learning?
Before diving into ideation, we discussed a larger system framework. Basked on our experience navigating through the mineral hall, we wanted three guiding phases for the experience:
Visitors will be introduced to the exhibit with relevant background knowledge
Visitors will experience guided wayfinding and exploration through space.
Visitors can engage in a participatory activity at the end of their experience
01. Head-Mounted Display
We speculate in the year 2029, AR glasses can be lighter, at the same time with 5G technology available, wearing AR glasses in the museum can be a seamless, immersive, and multi-sensory experience.
02.Marker-based and Markerless AR
We'd like to leverage two different ways of showing AR content to present content flexibly.
03.Cloud Anchors & Collective Sharing
Visitors can join their friends and interact with the same AR content.
We then started generating rough concept ideas around our principles, framework and technology.
Narrowing down the topic
We landed on the idea of a personalized AR object. At the start of each exhibit, visitors will be given a virtual object specific to that exhibit. They would use the object as an anchor to unlock more immersive learning experiences. This experience would leverage mark-based AR recognition on the visitor's hand.
AR & VR Interaction Paradigms
We looked into existing interaction paradigms for AR and VR. eg. Oculus' hand-taking design guidelines and LeapMotion and Hololens gesture-based apps. This helped us to think about the form of proximity, gesture controls, feedback, and signifiers.
3D AR Objects & Gesture Controls
We started exploring our AR palm object through physical prototyping. This helped us quickly understand how our object might interact within a physical environment. We also started thinking about specific hand gestures creating and sourcing OBJ files and paring them with live footage.
testing the gesture in the physical museum environment
testing the dimension of the digital content using analog
Prototyping scale gesture
Prototyping rotate gesture
cinema 4d content creation
pairing cinema 3d with live footage in after effects
Info browsing: Proximity aware & Haptic Feedback
We've incorporated the learnings from AR VR paradigm and designed the initiation of information as the gesture of pinch rather than an air-tap
pinch the button, rather than air-tap
proximity of content
reality composer prototype
Designing the User Interface in AR
Drawing inspiration from existing AR/VR platforms, we started by hand sketching our UI and detailed interaction flows for both primary and micro-interactions, We moved into Figma to continue exploring higher fidelity mockups on static museum images. This helped us get a better sense of space and proximity. After that, we did a final round of design variations in Cinema 4D, tested them in Adobe aero, and did the final prototype in after-effects.
designing user interface in sketch
3D UI variations
testing in Adobe Aero
Final Prototyp in After effect
We wanted to end each exhibit's journey with a participatory activity. Allowing museum visitors to reflect on their experience. After various iterations, feedback, and discussions, we landed on the simple yet powerful one, wishing and giving back the mineral. We decided to make this experience voice-based for engagement, ease of use and consistency.
concept sketch of the wishing bubble concept
designing in Cinema 4D
some other ideas
This project was challenging, but a lot of fun. We had to consider 3D digital objects and how these contents would interact with visitors and within the museum's environment. Research, trial, error, and critiques helped us figure out different ways to prototype our ideas. Along the way, we learned about paradigms, affordances, and signifiers for emerging AR and VR technology that we didn't notice before.
Digital prototyping for AR is a meticulous process that required workflow between various tools. Through the six weeks, we tried out over 20 different tools:
Cinema 3D SketchUp, Assets on SketchFab
Fast AR Prototyping
Aero, Reality Composer, Unity 3D
Oculus Quest, Hololens, LeapMotion
Figma, XD, Illustrator
After Effects, Lightroom, Audition, Mocha in after effect, Gopro
© 2020 Yiwei Huang. All rights reserved.