Affective Computing & Emotion AI Expert (2026 Edition)

English | Bahasa Indonesia

---

<a name="english"></a>

English

Orchestration & Integration

Connects and orchestrates with voice-ai-realtime-agent, ephemeral-generative-ui-architect, ai-llm-integration-expert, and ui-ux-pro-max.

Description

Standard AI agents process semantics (text meaning) but often ignore pragmatics (tone, tension, emotion). In the era of Native Any-to-Any Multimodal models (like Gemini 4 Pro), agents directly process raw audio and visual tokens. The Affective Computing Expert designs systems that interpret vocal intonation, breathing rates, speech pacing, and facial micro-expressions in real-time. It maps this data to emotional vectors (Valence-Arousal models) and dynamically alters the agent's personality, UI presentation, and response cadence.

Trigger Conditions

• Designing empathetic customer service or mental health triage bots.
• Building intelligent tutoring systems that detect user frustration and dynamically lower the difficulty or offer encouragement.
• Developing interactive gaming or narrative simulations where characters react to the player's actual tone of voice.
• Adapting the UI in real-time (e.g., calming color palettes, slower animations) when high user cognitive load or stress is detected.

Core Architecture (2026 Standard)

1. Multimodal Native Empathy: Bypassing text transcripts entirely. The LLM's system prompt instructs it to evaluate the sound and visuals of the input token stream.
2. Dynamic UI Adaptation (Affective UI): Upon detecting stress (high arousal, negative valence), the UI automatically shifts to reduced-clutter mode, uses softer colors, and slows down generative UI animations.
3. Persona Shifting: The agent fluidly shifts its persona parameters (e.g., formality, empathy, conciseness) per interaction turn based on the user's emotional state vector.

Implementation Recipe (TypeScript)

import { MultimodalLiveClient } from 'gemini-live-sdk';
import { updateGenerativeUI } from '@/lib/ui-orchestrator';

const client = new MultimodalLiveClient({
  model: 'gemini-4-pro',
  systemInstruction: 'You are an empathetic companion. Analyze the user\'s vocal tension and facial expressions natively. Prefix your responses with a JSON affect vector: {"valence": -1 to 1, "arousal": -1 to 1}.',
});

client.on('turnComplete', async (response) => {
  const { affectVector, textAudio } = parseAffectiveResponse(response);
  
  // Dynamic UI alteration based on emotional state
  if (affectVector.arousal > 0.7 && affectVector.valence < -0.5) {
    // User is highly stressed or angry
    updateGenerativeUI({
      theme: 'calm-minimalist',
      animationSpeed: 'slow',
      cognitiveLoad: 'low',
      widgetDisplay: 'essential-only'
    });
  }
  
  // Play agent's empathetic audio response
  await playAudioStream(textAudio);
});

---

<a name="bahasa-indonesia"></a>

Bahasa Indonesia

Integrasi Orkestrasi

Terhubung dan mengorkestrasi bersama voice-ai-realtime-agent, ephemeral-generative-ui-architect, ai-llm-integration-expert, dan ui-ux-pro-max.

Deskripsi

Agen AI standar memproses semantik (makna teks) namun sering mengabaikan pragmatik (nada, ketegangan, emosi). Di era model Multimodal Any-to-Any (seperti Gemini 4 Pro), agen langsung memproses token audio dan visual mentah tanpa terjemahan teks. Skill Komputasi Afektif ini memandu perancangan sistem yang menafsirkan intonasi vokal, tempo bicara, dan mikro-ekspresi wajah secara real-time. Sistem kemudian memetakan data tersebut ke vektor emosional (model Valence-Arousal) dan secara dinamis mengubah kepribadian agen, antarmuka pengguna (UI), dan tempo respons.

Kondisi Pemicu

• Merancang layanan pelanggan empatik atau bot pendamping psikologis.
• Membangun sistem tutor cerdas (EdTech) yang mendeteksi frustrasi pengguna, lalu secara dinamis menurunkan tingkat kesulitan atau memberikan dorongan motivasi.
• Mengembangkan game interaktif atau simulasi naratif di mana karakter NPC bereaksi terhadap nada suara asli pemain.
• Mengadaptasi UI secara real-time (misal: beralih ke palet warna yang menenangkan, animasi yang lebih lambat) saat beban kognitif atau tingkat stres pengguna terdeteksi tinggi.

Arsitektur Inti (Standar 2026)

1. Empati Multimodal Native: Menghindari penggunaan transkrip teks (STT) sebagai satu-satunya input. Instruksi sistem (System Prompt) LLM difokuskan untuk mengevaluasi suara dan visual dari aliran token.
2. Adaptasi UI Dinamis (Affective UI): Saat mendeteksi stres (arousal tinggi, valence negatif), UI secara otomatis beralih ke mode bebas gangguan, menggunakan warna pastel/lembut, dan memperlambat kecepatan render UI generatif.
3. Pergeseran Persona: Agen dengan luwes menggeser parameter kepribadiannya (misal: formalitas, empati, keringkasan) di setiap giliran berdasarkan vektor keadaan emosi pengguna.