We introduce Snapmoji, a system that can instantly generate animatable dual-stylized avatars.
Our dual stylization process re-imagines avatars in various artistic styles, enabling users to visualize themselves in diverse scenarios and create personalized stories.
Our approach also enables 3D stylized gaussian avatars generation and expression animation.
Snapmoji accomplishes the selfie-to-avatar conversion in just 0.9 seconds, and offers real-time functionality for mobile applications.
Dual-Stylized Avatars in 3D
Video 1a:
Dual-stylized avatars and their normals rendered with various facial expressions. Facial
expressions were captured from an iPhone with ARKit blendshapes. By choosing to animate the
avatars implicitly with blendshape weights, we can animate expressions which generalize to a
wide range of dual-stylized avatars.
Video 1b:
Dual-stylized avatars rendered from front and back. Although our LGM model takes only one image
as input, the model is still able to reconstruct a full 3D avatar.
AR Puppeting
Videos 2 & 3:
We combine 3DMM and blendshape features to animate our avatars. Alpha-compositing
the avatars with the original video then allows us to puppet the avatars in AR.
More Dual-Stylized Avatar Puppeting
Videos 4 & 5:
We demonstrate more examples of dual-stylized avatar puppeting.
Web Rendering Application
Video 6:
Our avatars, represented by Gaussian Splats, are small enough to
be rendered in a web browser at 90-100 FPS on a laptop, or 30-40 FPS on a phone. We can
control the
avatar's expression with a face tracker, then render them in 3D.
The face tracking is done by Mediapipe in WebAssembly, while the avatar is animated
using
the gsplat.js library. All computation is done on the client.
Mobile Rendering Application
Video 7:
Because our AR demo can be run in a browser, it is inherently cross platform. On a
iPhone 13
Pro, the avatars are rendered at 30-40 FPS. Note that the true FPS is higher than what
is
shown in the video due to screen recording.
Ablation Study on 3DMM Tracking
Video 8:
This video demonstrates the effects of using 3DMM features in conjunction with FACS
blendshape weights.
The combination enhances the expressiveness and fidelity of avatar animation,
accommodating
both realistic and stylized facial expressions.
Avatar Generation UI
Video 9:
A screen capture of the Snapmoji avatar generator interface. The interface includes
controls to
balance identity preservation and style in the generated avatars. Due to the system's
speed,
a
user
can
rapidly prototype different avatar designs to their liking. The avatar generator is
built
with
Gradio.
Blendshape Editor UI
Video 10:
A screen capture of the Snapmoji blendshape editor. The editor allows one to preview an
Snapmoji avatar in 3D, and modify facial expressions with blendshapes. Blendshape
weights
give
precise control over regions in the face. Linear combinations of different blendshapes
can
represent a variety of expressions. The editor is built with Viser.
