Why is this work relevant for Innovation?

In contrast to the superficial and inaccurate two-dimensional effects produced by Snapchat or FaceApp for lighthearted selfie-gratification, the Beyond Time experience generates individually accurate aging simulations that are built upon actual data from a century of research into the science of aging. Built in partnership with Japan’s leading experts in senescence and simulation technology, our project represents a world’s-first technology that creates an immersive 3D age simulation experience in real-time.

Background

Beyond Time was influenced by generative adversarial networks and other recent advances in machine learning, as well as applications of the technology to produce creative work like in The Next Rembrandt. Though this project, we sought to apply machine learning to the creation of an emotional experience that transcends time in the real world. Since the processes and algorithms of popular mobile apps are proprietary, we wrote and developed original software for the express purpose of creating our scientifically-accurate age simulation experience, taking care to work closely with Shiseido Research and legal teams to ensure that non-anonymous data was not retained in our systems for the protection of our participants’ data rights.

Describe the idea

The mind affects the body as much as the other way around. As a leader in the beauty category, Shiseido recognizes the importance of mental wellness and positivity as essential keys to physical beauty. Beyond Time is an innovative interactive installation designed to make it possible for individuals to speak with digitally aged or de-aged versions of friends, family, and loved ones. In doing so, we sparked emotional conversations that transcended time, led people to discover beautiful things about growing old, and helped them think differently about the effects of physical aging.

What were the key dates in the development process?

2018 Jan—Mar: Technical research / Definition Apr—Aug: Concepting / Aging system development / Draft copy and UX development June—Aug: Installation design August: First user tests Sep—Nov: Installation design Finalization Sep—Dec: UX, Copy, Design and Aging system update / Data processing system development / MVP development Nov—Dec: Data collection December: 2nd user tests 2019 January: Data collection Feb—Mar: Software development finalization Feb. -Mar: Final installation construction April: Quality assurance testing April 13 2019: Grand Opening at Shiseido Global Innovation Center

Describe the innovation/technology

Beyond Time is three high-spec computers, multiple Kinect sensors, IoT connected buttons, multiple 4k cameras, plus synchronized LEDs, directional microphones, variable-opacity walls for privacy, and more than fifteen thousand lines of code across six different programming languages. We combined these components into a bespoke technology solution based on cutting edge face capture and aging research. We built a fast, easy-to-setup and reliable three dimensional system to generate one of the largest face modeling databases in the world. We then created the first real-time, fully three-dimensional age and age-reversing simulation engine that detects motion, recognizes individual faces, generates their 3D face models, aging textures and even a dynamic wrinkles layer, and then synchronizes the experience to take participants on a journey through.

Describe the expectations/outcome

In the beginning, we were tasked with “creating something that will attract visitors to Shiseido’s new Global Innovation Center.” By the end, we engaged over 3,100 people in an emotional experience that continues to be fully-booked today. We developed and implemented groundbreaking 3D facial data capture and skin color recognition solutions. We generated one of the largest 3D facial databases in the world. We created a new platform for open conversations for changing perceptions on aging. And most importantly, we helped the iconic Shiseido brand make a cultural statement that beauty goes beyond young and perfect skin, and that growing old is a beautiful process.