Digital Humans Transform Industries
The talk frames the problem of making computers interact with people in humanlike, low-latency ways across industries (customer service, healthcare, advertising, design). The presented approach is a stacked set of AI capabilities — multilingual speech recognition and synthesis, LLMs for conversational intent, generative animation tied to a 3D face mesh, and neural rendering (real-time path-traced subsurface scattering) — packaged as deployable microservices (NIMs) in Nvidia Ace and served on a global accelerated network. Evidence includes application examples: phone-scan-based AI interior design that returns multiple photorealistic options, a conversational healthcare agent safely answering a drug-allergy question, and EMA, a Japanese virtual model/brand ambassador. Outcome: a production-oriented microservice architecture that enables real-time digital humans with regional low-latency delivery (100+ regions), though it requires orchestration across multiple specialized models and heavy real-time rendering compute.
Key technical points
- Primary capability stack: multilingual speech recognition and synthesis + LLMs for understanding and generating conversation.
- Generative animation layer: a separate generative AI dynamically animates a lifelike 3D mesh of a face from audio/intent.
- Neural appearance models: real-time path-traced subsurface scattering to simulate skin light penetration and scattering for soft, translucent skin appearance.
- Nvidia Ace packaging: digital human technologies are provided as easy-to-deploy, fully optimized microservices (referred to in the talk as NIMs).
- Named NIMs and roles (as stated): 'Neotron slm and llm Nims' to understand intent and orchestrate other models; 'Reva speech Nims' for interactive speech and translation; 'audio to face and gesture nyss' for facial and body animation.
- Rendering infrastructure: Omniverse RTX with DLSS is used for neural rendering of skin and hair.
- Inputs and outputs: inputs include phone scans of environments (e.g., kitchens) and audio; outputs include photorealistic interior design suggestions, sourced materials/furniture options, conversational responses, and synchronized facial/body animation.
- Example applications: AI interior designers generating multiple photorealistic options from phone scans; AI customer service agents for more engaging/personalized interactions; digital healthcare workers providing timely personalized care (demonstrated with a medication/allergy exchange); AI brand ambassadors/virtual models (EMA).
- Microservice integration: Ace NIMs can be integrated into existing frameworks, game engines, and digital human experiences, implying modular runtime components.
- Deployment and latency: Ace NIMs run on Nvidia GDN (a global network of Nvidia-accelerated infrastructure) that delivers low-latency digital human processing to over 100 regions.
- End-to-end orchestration requirement: multiple specialized models (speech, LLMs, generative animation, neural rendering) must be orchestrated to produce synchronized, real-time digital humans.
Adopt a microservice architecture (Ace NIMs) to decouple speech, intent, animation, and rendering components for easier integration into existing engines and frameworks. Route sensory inputs explicitly: phone-captured 3D/photogrammetry data for design generation; audio streams for speech recognition/synthesis and audio-to-face animation pipelines. Use LLM/SLM NIMs as an orchestration/intent layer to sequence downstream models (speech, translation, animation) rather than tightly coupling logic into a single model. Leverage Reva-like speech NIMs for interactive speech and translation to support multilingual user interaction flows. Implement an audio→animation module (audio-to-face/gesture NIMs) to generate synchronized facial/body motion driven by speech. Plan rendering pipeline around Omniverse RTX + DLSS for neural rendering, and include support for real-time subsurface scattering for convincing skin appearance — budget for high GPU compute. Deploy on regionally distributed accelerated infrastructure (Nvidia GDN) to meet low-latency requirements; design for multi-region routing and failover. Prepare orchestration and timing controls to synchronize conversational output, animation frames, and neural-rendered frames for real-time responsiveness. Instrument pipelines for per-component performance (latency of speech-to-text, LLM response time, animation synthesis time, render frame time) to identify bottlenecks in end-to-end interactions. Integrate NIMs as modular services to allow iterative updates to individual components (speech model, LLM, animator, renderer) without full system redeploys.
EMA is cited as Japan's first virtual model — a concrete local use case demonstrating adoption in Japanese advertising/branding. Multilingual speech recognition and Reva speech NIMs with translation support are directly relevant for Japanese-language interactions and multi-language deployments in Japan. Nvidia GDN's coverage of over 100 regions supports low-latency hosting for Japan-based users and services; plan region-aware deployment targeting local edge nodes. Phone-scan-driven AI interior design and brand-ambassador use cases map to consumer and retail markets in Japan where photorealistic suggestions and virtual talent (EMA) can be applied.