Experiential Learning with AR In the Carnegie Natural History Museum


Amrita Khoshoo
Hannah Koenig
Michelle Cedeño
Yiwei Huang


Fall 2019
6 weeks

My Role

Design Research
Technology Research

User Experience Design
Interaction Design
3D Prototyping


Cinema 4D / Sketchfab
After Effect
Illustrator/ Figma
Aero/ Reality Composer

How might AR enhance experiential learning in a physical museum?

Project Objective

In 2029, you have been tasked to improve the user experience of the Carnegie Natural History Museum with interactive environments that are: 

Today, many museums are facing the challenge of transitioning from rigid institutions to experiential and flexible spaces. This is driven by such factors as expanding collections, increased competition for visitors, and visitor expectations for greater engagement. Museums are turning to virtual reality, apps, and interactive experiences to keep tech-savvy visitors engaged.


How can visitors engage more than their sense of sight?


How can immersive environment foster more engagement?


How can a designed system adapt to changing behavior and environment?

The Outcome

Aurum is an augmented reality experience for the Carnegie Natural History Museum that leverages smart-glass technology, immersive storytelling, and participatory experiences.

Exhibit-Level User Journey


3 major part of the Experiential Learning

01 Arrive at the Museum

Upon arrival, visitors receive their ticket , smart AR glasses, and a palm stamp. Smart glasses reveal AR content and the palm stamp serves as an image anchor for an AR virtual object.


02 Aurum Onboarding 

During onboarding, visitors are introduced to Aurum and different gestures that will help them navigate through AR content.


Gesture of Pinch


Gesture of open palm


Gesture of toggle

03 Exhibit & Virtual Object Intro

As visitors explore the museum, floor makers indicate the start of AR-enabled exhibits. At the start of the Mineral Hall, visitors are introduced to the narrative of the exhibit (conflict minerals) and can choose a virtual mineral to guide their experience.

04 Learn Natural Science

Visitors can explore mineral geology, atomic structure, and properties. Properties include fracture, cleavage, radioactivity, fluorescence, and hardness related to the specimen that is in your palm.

05 Explore Human Impact

After learning about mineral science, visitors can explore human impact stories in a more immersive experience. Stories follow miners, supply chains, or global efforts around conflict minerals.

As visitors navigate through the space, they can discover their virtual object’s physical form and continue exploring more AR-enabled content. 

06 Reflect on Experience

At the end of the exhibit, visitors can engage in a participatory experience by sharing or listening to the reflections of others.


07 Continue Discovering

Visitors can navigate through other AR-enabled exhibits by encountering more personalized contents.

System Controls1: Gestures

Aurum uses hand-tracking as an input modality. Visitors can navigate digital content using six gestures. We've learned a lot from the Oculus Design Principles, and landed in the following gestures in the design

System Controls2: Proximities

leveraging locational based anchor, we speculate that the Navigating system react to human proximity to the information, to prevent information overload, at the same time signify content curation.


Visual System

The Aurum logo was inspired by mineral geometry and Da Vinci's Vitruvian Man. The Vitruvian Man was created during a time when people shifted their focus from God to humanity. We now hope to shift focus to humanity's relationship with the Earth.


The Interface

Exploratory Research

On-site Tour

Our research began at the Carnegie Natural History Museum. Luckily, we had the opportunity to speak with Rebecca Shreckengast, Director of Exhibition Experience. She gave us some valuable insights into the current museum experience from a curatorial and exhibition design perspective.


Q & A with Becca

8 Key Takeaways

Several key takeaways are generalized form this visit:


Environmental Advocacy

Museum's current mission is to find inspiration in the collections and advocate for a sustainable future.



The future of museum is bringing the audience a personalized, curated experience.



Narrations should be created in a way the audiences feel personally connected.


Complexity Doesn't Work

Complex design systems dont work well. (eg. prescribed wayfinding systems are usually not successful)



Successful experiences are participatory, allowing museumgoers to express themselves and trigger conversations. They are "sticky"


Deep time story-telling

Deep time is an important concept, but it's difficult to teach and engage the audience.



40% of the visitors come in groups. How can we design to  trigger communication and interaction between them?



The museum owns 12 million specimins, only 3% of them are visible.

Identifying Opportunity - Hillman Hall of Mineral and Gems

When touring the museum, the Hillman Hall of Minerals and Gems caught our eyes. This exhibition artfully displays more than 1,300 specimens from all over the world that come in a large range of striking colors, fascinating forms, and dramatic shapes. While, after a 10-minute tour in the hall, we still remember very little about mineralogy and scientific information, because of the overwhelming volume of information and visual stimulations. We also talked to a few visitors to help us understand how they felt about the current experience.


Beautiful yet overwhelming

The Participatory Museum

When thinking about increasing engagement with museums, we read through Nina Simons' The Participatory Museum. This book discusses how cultural institutions can become more dynamic, relevant, essential places by creating a participatory museum's experience.

Investigating Technology

In order to understand the technology, we would be designing with, we took a look at the current tools of mixed reality and virtual reality tools and experiences, including Oculus Quest and Hololens headsets and various mobile AR prototyping tools such as Reality Composer, and Adobe AeroExperiences include Teamlab's Crystal Universe, Smithsonian Museum of Natural History's AR project Skin and Bone, Cooper Hewitt Museum's project Pen.


Research Synthesis

01. Artifacts are missing context.

Minerals are decontextualized. The environments, ecosystems, and human relationships within which they are situated are not well represented.

02. Passive engagement.

This space requires little participation and engagement on the part of the viewer. Visitors only look at objects, as opposed to interacting with them.

03. Information is inaccessible.

The number of objects and accompanying visual content is a lot to digest, and can easily lead to cognitive overload and fatigue. Further, the content within the Mineral Hall is filled with scientific jargon that visitors may be unfamiliar with,

04. A larger narrative is absent.

A few successful experiences with the museum have strong, clear narratives structured around them. How can we create a stronger connection between viewer, artifact, and space by connecting to a bigger picture narrative? How can the space be more relevant to visitors?

How might we design for an AR museum experience to tell stories of Anthropocene, trigger communication and enable experiential learning?

Concept Generation

System Framework

Before diving into ideation, we discussed a larger system framework. Basked on our experience navigating through the mineral hall, we wanted three guiding phases for the experience: 


01. Orientation

Visitors will be introduced to the exhibit with relevant background knowledge


Visitors will experience guided wayfinding and exploration through space.


Visitors can engage in a participatory activity at the end of their experience


01. Head-Mounted Display

We speculate in the year 2029, AR glasses can be lighter, at the same time with 5G technology available, wearing AR glasses in the museum can be a seamless, immersive, and multi-sensory experience.


02.Marker-based and Markerless AR

We'd like to leverage two different ways of showing AR content to present content flexibly.

03.Cloud Anchors & Collective Sharing

Visitors can join their friends and interact with the same AR content.


We then started generating rough concept ideas around our principles, framework and technology.


Concept Refinement

Narrowing down the topic

We landed on the idea of a personalized AR object. At the start of each exhibit, visitors will be given a virtual object specific to that exhibit. They would use the object as an anchor to unlock more immersive learning experiences. This experience would leverage mark-based AR recognition on the visitor's hand.


Design & Prototyping

AR & VR Interaction Paradigms

We looked into existing interaction paradigms for AR and VR. eg. Oculus' hand-taking design guidelines and LeapMotion and Hololens gesture-based apps. This helped us to think about the form of proximity, gesture controls, feedback, and signifiers.



3D AR Objects & Gesture Controls

We started exploring our AR palm object through physical prototyping. This helped us quickly understand how our object might interact within a physical environment. We also started thinking about specific hand gestures creating and sourcing OBJ files and paring them with live footage.


testing the gesture in the physical museum environment


testing the dimension of the digital content using analog

Prototyping scale gesture

Prototyping rotate gesture


cinema 4d content creation


pairing cinema 3d with live footage in after effects

Info browsing: Proximity aware & Haptic Feedback

We've incorporated the learnings from AR VR paradigm and designed the initiation of information as the gesture of pinch rather than an air-tap


pinch the button, rather than air-tap


proximity of content

reality composer prototype

Designing the User Interface in AR

Drawing inspiration from existing AR/VR platforms, we started by hand sketching our UI and detailed interaction flows for both primary and micro-interactions, We moved into Figma to continue exploring higher fidelity mockups on static museum images. This helped us get a better sense of space and proximity. After that, we did a final round of design variations in Cinema 4D, tested them in Adobe aero, and did the final prototype in after-effects.


designing user interface in sketch


testing dimensions 


3D UI variations

testing in Adobe Aero 


Final Prototyp in After effect 

Participatory Experience

We wanted to end each exhibit's journey with a participatory activity. Allowing museum visitors to reflect on their experience. After various iterations, feedback, and discussions, we landed on the simple yet powerful one, wishing and giving back the mineral. We decided to make this experience voice-based for engagement, ease of use and consistency.


concept sketch of the wishing bubble concept


designing in Cinema 4D


some other ideas


This project was challenging, but a lot of fun. We had to consider 3D digital objects and how these contents would interact with visitors and within the museum's environment. Research, trial, error, and critiques helped us figure out different ways to prototype our ideas. Along the way, we learned about paradigms, affordances, and signifiers for emerging AR and VR technology that we didn't notice before.

Digital prototyping for AR is a meticulous process that required workflow between various tools. Through the six weeks, we tried out over 20 different tools:


3D Modeling

Cinema 3D SketchUp, Assets on SketchFab


Fast AR Prototyping

Aero, Reality Composer, Unity 3D



Oculus Quest, Hololens, LeapMotion



Figma, XD, Illustrator


Video production

After Effects, Lightroom, Audition, Mocha in after effect, Gopro

© 2020 Yiwei Huang. All rights reserved.