Background

Drawing inspiration from the acclaimed television series "The Mandalorian," the use of in-camera VFX in overseas film production has become a hot topic in recent years, and we, AOI Pro., were looking for an opportunity to explore the possibility of utilizing this state-of-the-art technology in Japan. Thus, we teamed up with Hibino Corporation, which possesses high-definition LED displays, and our group company TREE Digital Studio, which specializes in post-production, VFX production, and has a wealth of camera and lighting equipment, to launch this project and pioneer the virtual production business in Japan. At the same time, we asked SDR, a music label that we have a close relation to, to collaborate on this project, which led to the production of a music video for Vaundy, one of the most captivating young artists in Japan.

Describe the creative idea

Together with the director, we devised a performance that extensively used in-camera VFX technology. The VFX environment that was created in Unreal Engine was projected on the LED panels in real time. The camera movement/lens was synchronized with the movement/depth of the VFX environment. For the train scene, we placed hanging straps in front of the LED panels and were able to blend them together with the VFX in the background, creating the illusion that they exist in the same space. Furthermore, for the platform, it appears as though the camera is orbiting around the hero and dancers. We achieved this by placing the hero and dancers on a rotating platform, simply moving the camera back and forth on a dolly. For the fence sequence, we were able to place a real fence in the foreground, which is difficult to pull off in post with a green screen.

Describe the strategy

The success of this music video was due to meticulous scheduling and the execution of multiple test shoots. In-camera VFX was a first for the entire crew, so we needed time to learn from the ground up, including production. For this reason we conducted three separate test shoots, repeating the process from start to finish. With each test, we verified and discovered new obstacles, modified the storyboard and VFX, and resolved them by the next test. Through slowly but steadily solving each issue, we were able to realize unprecedented shooting methods and visual expression, which led to the improvement of the overall quality of our film.

Describe the execution

We kicked off the project in mid-March with a meeting between the director, artist, and production. Immediately after, we shared the project outline with Hibino and TREE as well, and later showed the crew the actual LED panels. Since this was the first time for the crew to use in-camera VFX, we conducted three test shoots. With each test, new discoveries and difficulties became apparent, and we revised our plan daily. Based on the knowledge gained from the tests and advice from each department, we set aside a total of four days for our shoot window. The first day was spent setting up the LED panels, creating the set, and calibrating the camera lens. The second and third days were spent for prelight, and from the latter half of the third day we started rolling, and completed the shoot in just 2 days, regardless of the quantity of shots.

Describe the outcome

After finishing the shoot and post production, the music video was released on the official YouTube channel of Vaundy on October 19th, 2021. Behind-the-scenes footage was also released on the same day, showing how the in-camera VFX was utilized and revealing the stories behind the production of each department that was involved. Quickly becoming a hot topic, it got featured in various media such as Commercial Photo (a Japanese commercial magazine), PRONEWS (a Japanese web magazine for film production), CG world (A Japanese magazine focusing on CG topics) and Little Black Book. As of January 5th 2022, the music video has been viewed more than 3 million times.