Digital Fitness Friends
Your digital gym buddy
Project Description
Digital Fitness friends is an early-access Meta Quest fitness app that utilizes the best and greatest of VR, AI and body tracking technologies to help people reach their fitness goals with their own personal trainer or gym buddy. As a previous gym-goer with a busy lifestyle, I created it to add motivation and variation for working out in a home-based setting.
To spice things up and keep things fresh, the app aims to gamify the fitness process; introducing new environments, characters and activities to help players hit their fitness goals. Features allow them to track their performance through advanced sensors and AI models on the headset, combined with optional fitness accessories for more granular feedback. The initial version of the app supports taking players through a vertical slice of functionality; demonstrating the correct form when doing a short workout routine. It tracks and assesses their performance and lets them review how it went with a virtual personal trainer.
Your helpful, virtual personal trainers
The main virtual P.T, Samantha, utilizes a mix of scripted and dynamic content, providing consistent and helpful information, while also allowing for free-form conversations to review progress and help players make their workout easier, harder (see the demo video below) or explore fitness topics like exercise techniques, nutrition and helps with goal setting. While she isn't perfect, she's always learning and player can bookmark her tips for review with your regular real-life personal trainer!
Her co-worker, Samuel will handle other topics in the future, like identifying gym equipment you have in your home to give you tailored advice and ideas on how to make the most out of what you've got. The two Sams are the ultimate team to help unlock your fitness journey!
The Technology
The software stack that drives the conversation system is a culmination of several years of part-time R&D into production-quality tools, animation systems, synthetic voice and speech recognition. All these have finally become responsive, cost-effective and scalable by recent global advancements in large-language models and generative AI. The underlying approaches are transferrable to other mediums and could be implemented into AR glasses with an AI voice assistant or as a smartphone app.
The app is built using Unity 6, cloud-agnostic microservices and leverages the OpenAI API, with a custom 3D model of the personal trainer, brought to life by a custom animation system that allows for dynamic and responsive animations to be played back in real-time. The app is designed to be used in a home gym environment using low-cost equipment or accessories, with a single user and a single personal trainer, but could be expanded to support other AI characters in the future.
Features
-
Load and create dynamic workouts - Edit workouts using JSON to change the app content without rebuilding. Allows generative AI tools like ChatGPT to create workout routines with the guidance of a real personal trainer, based on a library of exercises pre-loaded into the app.
-
Fitness minigames - Try boxing or snowboarding to boost your energy and challenge your coordination and balance.
-
Review your workout - Chat with the personal trainer and get advice on how to make your workout easier or harder.
-
Dynamic Voiceovers - Instead of pre-recorded dialog, the app generates and caches audio in realtime to reduce API calls and ongoing costs. Already this makes updating and tweaking content easy, and made better with plans for Unity Editor authoring tools to automatically pre-cache the audio during build time.
Generative Tools
This project was an exploration of how generative AI tools can speed up concepting and production workflows. A lot of the outputs took weeks of work down to a handful of days to do and allowed more time to be spent crafting the overall aesthetic and experience of the app, giving more high quality options to choose from in terms of content. Due to current limitations, most of the image and code outputs needed further refinement, including refining and adapting the code for production, redrawing the logo wordmark by using correctly licenced fonts so a brand identity could be created that would work for both web and print.
Initial 3D-renders of character concepts were created using generative AI, based on detailed outputs from LLM prompts and then headshots were brought into Reallusion Character Creator for sculpting onto a character rig. The final output draws from a rich animation library, including various fitness poses and animations to help teach players the correct form for their workouts. Because the character is human scale, some liberties were taken to ensure the animations looked correct and could be followed by the user.
Cutting-edge Tech
The app used the following generative AI tools, APIs and models in its creation:
- FLUX.1 [dev] / FLUX.1 [schnell] - Character concept renders and headshot generation and logo concepts
- Suno - For video trailers and in-game background music
- Llama 3.1 & 3.3 - Character system prompt generation and exploratory work on character appearance, personality, interests and backstory
- Google Gemini - Some scripting and game functions
- OpenAI API - Synthetic voice, speech transcription and character conversational model
