In my undergraduate design course that examines the future of interaction design in mixed reality, our group was tasked with exploring how everyday interactions with mobile devices can be transformed into magical/delightful extended reality (XR) experiences.

Inspired by Gen-Z culture and emojis, we designed a gestural menu in VR that allows people to create, store, and access fun visual expressions and objects, and use them to communicate with others in virtual realty.

My role mainly involved using Unity to prototype and design VR interactions using VR toolkits, scripts, and particle animations. I also contributed heavily to storytelling and video prototyping.

Ariel Chiang, Daniel Kim, Alvin Jeong, Jasper Xie, Andrew Tang

Unity 3D, Oculus

Through our initial exploration of communication in XR environments, we chose to investigate how typing in VR could benefit from different physical gestures. After trying VR typing ourselves, we learned that aiming and clicking on individual letters felt tedious and slow. When we began brainstorming keyboard and hardware redesigns to solve this problem, our mentors advised us to refocus our efforts on spatial interactions in XR instead.

We brainstormed different types of text and digital communication, categorized them, and created different types of scenarios for them. We landed on three general categories that piqued our interest:

Brainstorming reasons why someone would use emojis in XR over smartphones revealed two interesting insights:

We designed an emoji communication tool where people can feel the magic of sending and receiving exciting emoji effects in XR and communicated our vision of a use case through a storyboard with the following story elements:

Users of our tool need to bring up emojis in XR just as fast as on a keyboard, and through sketching and brainstorming sessions we based our emoji system off of video game emote systems. We were specifically inspired by online multiplayer video games because they usually have quick ways to communicate so that players can express ideas and play at the same time.

Specifically, we referred to League of Legends’ (LoL) emote wheel where players could create a wheel of five emotes that could be taken into any LoL match.  Borrowing this concept, we imagined users in XR could load their own expressions into a plate that could be taken into any online XR space or game.

We designed the system to have preset emojis to pick from, but we thought the ability to import custom emojis and expressions could allow for more personalized expression.

We came up with three constraints for both preset and imported objects to provide a consistent experience between the two. At this point, we decided to put more emphasis on designing the emojis' visual effects as they are sent to another person.

We compiled a list of interactions that we liked and I prototyped them in Unity for feasibility. If they worked, we would show our class and professor to gauge excitement and then fully prototype the most exciting interactions in Unity.

We made a video prototype, which I co-directed and filmed in a commercial-style, that focused on four cases that encompassed our entire design. The other members of my team created and supplied 3D models of emojis in Blender and handed them off to me to build in Unity.

The four use cases we decided on were:

We chose these four because they represented many of the purposes in which someone would use XR emojis. Whether its importing your own objects for self-expression, messing with friends, or throwing a virtual party, we believed these four use cases covered these interactions.

The final design was more magical than we had expected. Other members of our class found it extremely intuitive to grab and throw emojis. Our professor was excited at the potential to throw emojis back and forth with their friends and stated how this felt like the next step for visual communication in XR.

In our early exploration phase, our professor had suggested object alternatives to emojis, as he neither related to emojis nor thought of them as dynamic. Given these concerns, I would like to explore other abstract visual elements as forms of communications in the future.

If we had more time, we would have tried to develop more effects and gestures for our expressions to create more fun ways to activate emoji effects. We currently only utilize the throwing motion to trigger our expressions, but we had floated ideas that involved more dynamic movements.

Through this project, I learned a lot about VR prototyping and design and hope to pivot into designing more in the VR space. Learning how to utilize virtual space was a huge challenge but an important revelation about how VR designers see the world. Our professor said a quote that still sticks with me: "Designers should be designing for the future, so why are we still designing for phones?"