Hex/.planning/research/SUMMARY.md

# Research Summary: Hex AI Companion

**Date**: January 2026
**Status**: Ready for Roadmap and Requirements Definition
**Confidence Level**: HIGH (well-sourced, coherent across all research areas)

---

## Executive Summary

Hex is built on a **personality-first, local-first architecture** that prioritizes genuine emotional resonance over feature breadth. The recommended approach combines Llama 3.1 8B (local inference via Ollama), Discord.py async patterns, and a dual-memory system (SQLite + ChromaDB) to create an AI companion that feels like a person with opinions and growth over time.

The technical foundation is solid and proven: Discord.py 2.6.4+ with native async support, local LLM inference for privacy, and a 6-phase incremental build strategy that enables personality emergence before adding autonomy or self-modification.

**Critical success factor**: The difference between "a bot that sounds like Hex" and "Hex as a person" hinges on three interconnected systems working together: **memory persistence** (so she learns about you), **personality consistency** (so she feels like the same person), and **autonomy** (so she feels genuinely invested in you). All three must be treated as foundational, not optional features.

---

## Recommended Stack

**Core Technologies** (Production-ready, January 2026):

| Layer | Technology | Version | Rationale |
|-------|-----------|---------|-----------|
| **Bot Framework** | Discord.py | 2.6.4+ | Async-native, mature, excellent Discord integration |
| **LLM Inference** | Llama 3.1 8B Instruct | 4-bit quantized | 128K context window, superior reasoning, 6GB VRAM footprint |
| **LLM Engine** | Ollama (dev) / vLLM (production) | 0.3+ | Local-first, zero setup vs high-throughput scaling |
| **Short-term Memory** | SQLite | Standard lib | Fast, reliable, local file-based conversations |
| **Long-term Memory** | ChromaDB (dev) → Qdrant (prod) | Latest | Vector semantics, embedded for <100k vectors |
| **Embeddings** | all-MiniLM-L6-v2 | 384-dim | Fast (5ms/sentence), production-grade quality |
| **Speech-to-Text** | Whisper Large V3 + faster-whisper | Latest | Local, 7.4% WER, multilingual, 3-5s latency |
| **Text-to-Speech** | Kokoro 82M (default) + XTTS-v2 (emotional) | Latest | Sub-second latency, personality-aware prosody |
| **Vision** | OpenCV 4.10+ + DeepFace | 4.10+ | Face detection (30 FPS), emotion recognition (90%+ accuracy) |
| **Avatar** | VRoid + VSeeFace + Discord screen share | Latest | Free, anime-style, integrates with Discord calls |
| **Personality** | YAML + Git versioning | — | Editable persona, change tracking, rollback capable |
| **Self-Modification** | RestrictedPython + sandboxing | — | Safe code generation, user approval required |

**Why This Stack**:
- **Privacy**: All inference local (except Discord API), no cloud dependency
- **Latency**: <3 second end-to-end response time on consumer hardware (RTX 3060 Ti)
- **Cost**: Zero cloud fees, open-source stack
- **Personality**: System prompt injection + memory context + perception awareness enables genuine character coherence
- **Async Architecture**: Discord.py's native asyncio means LLM, TTS, memory lookups run in parallel without blocking

---

## Table Stakes vs Differentiators

### Table Stakes (v1 Essential Features)

Users expect these by default in 2026. Missing any breaks immersion:

1. **Conversation Memory** (Short + Long-term)
   - Last 20 messages in context window
   - Vector semantic search for relevant past interactions
   - Relationship state tracking (strangers → friends → close)
   - **Without this**: Feels like meeting a stranger each time; companion becomes disposable

2. **Natural Conversation** (No AI Speak)
   - Contractions, casual language, slang
   - Personality quirks embedded in word choices
   - Context-appropriate tone shifts
   - Willingness to disagree or pushback
   - **Pitfall**: Formal "I'm an AI and I can help you with..." kills immersion instantly

3. **Fast Response Times** (<1s for acknowledgment, <3s for full response)
   - Typing indicators start immediately
   - Streaming responses (show text as it generates)
   - Async all I/O-bound work (LLM, TTS, database)
   - **Without this**: Latency >5s makes companion feel dead; users stop engaging

4. **Consistent Personality** (Feels like same person across weeks)
   - Core traits stable (tsundere nature, values)
   - Personality evolution slow and logged
   - Memory-backed traits (not just prompt)
   - **Pitfall**: Personality drift is #1 reason users abandon companions

5. **Platform Integration** (Discord native)
   - Text channels, DMs, voice channels
   - Emoji reactions, slash commands
   - Server-specific personality variations
   - **Without this**: Requires leaving Discord = abandoned feature

6. **Emotional Responsiveness** (Reads the room)
   - Sentiment detection from messages
   - Adaptive response depth (listen to sad users, engage with energetic ones)
   - Skip jokes when user is suffering
   - **Pitfall**: "Always cheerful" feels cruel when user is venting

---

### Differentiators (Competitive Edge)

These separate Hex from static chatbots. Build in order:

1. **True Autonomy** (Proactive Agency)
   - Initiates conversations based on context/memory
   - Reminds about user's goals without being asked
   - Sets boundaries ("I don't think you should do X")
   - Follows up on unresolved topics
   - **Research shows**: Autonomous companions are described as "feels like they actually care" vs reactive "smart but distant"
   - **Complexity**: Hard, requires Phase 3-4

2. **Emotional Intelligence** (Mood Detection + Adaptive Strategy)
   - Facial emotion from webcam (70-80% accuracy possible)
   - Voice tone analysis from Discord calls
   - Mood tracking over time (identifies depression patterns, burnout)
   - Knows when to listen vs advise vs distract
   - **Research shows**: Companies using emotion AI report 25% positive sentiment increase
   - **Complexity**: Hard, requires Phase 3+ but perception must be separate thread

3. **Multimodal Awareness** (Sees Your Context)
   - Understands what's on your screen (game, work, video)
   - Contextualizes help ("I see you're stuck on that Elden Ring boss...")
   - Detects stress signals (tab behavior, timing)
   - Proactive help based on visible activity
   - **Privacy**: Local processing only, user opt-in required
   - **Complexity**: Hard, requires careful async architecture to avoid latency

4. **Self-Modification** (Genuine Autonomy)
   - Generates code to improve own logic
   - Tests changes in sandbox before deployment
   - User maintains veto power (approval required)
   - All changes tracked with rollback capability
   - **Critical**: Gamified progression (not instant capability), mandatory approval, version control
   - **Complexity**: Hard, requires Phase 5+ and strong safety boundaries

5. **Relationship Building** (Transactional → Meaningful)
   - Inside jokes that evolve naturally
   - Character growth (admits mistakes, opinions change slightly)
   - Vulnerability in appropriate moments
   - Investment in user outcomes ("I'm rooting for you")
   - **Research shows**: Users with relational companions feel like it's "someone who actually knows them"
   - **Complexity**: Hard (3+ weeks), emerges from memory + personality + autonomy

---

## Build Architecture (6-Phase Approach)

### Phase 1: Foundation (Weeks 1-2) — "Hex talks back"

**Goal**: Core interaction loop working locally; personality emerges

**Build**:
- Discord bot skeleton with message handling (Discord.py)
- Local LLM integration (Ollama + Llama 3.1 8B 4-bit quantized)
- SQLite conversation storage (recent context only)
- YAML personality definition (editable)
- System prompt with persona injection
- Async/await patterns throughout

**Outcomes**:
- Hex responds in Discord text channels with personality
- Conversations logged, retrievable
- Response latency <2 seconds
- Personality can be tweaked via YAML

**Key Metric**: P95 latency <2s, personality consistency baseline established

**Pitfalls to avoid**:
- Blocking operations on event loop (use `asyncio.create_task()`)
- LLM inference on main thread (use thread pool)
- Personality not actionable in prompts (be specific about tsundere rules)

---

### Phase 2: Personality & Memory (Weeks 3-4) — "Hex remembers me"

**Goal**: Hex feels like a person who learns about you; personality becomes consistent

**Build**:
- Vector database (ChromaDB) for semantic memory
- Memory-aware context injection (relevant past facts in prompt)
- User relationship tracking (relationship state machine)
- Emotional responsiveness from text sentiment
- Personality versioning (git-based snapshots)
- Tsundere balance metrics (track denial %)
- Kid-mode detection (safety filtering)

**Outcomes**:
- Hex remembers facts about you across conversations
- Responses reference past events naturally
- Personality consistent across weeks (audit shows <5% drift)
- Emotions read from text; responses adapt depth
- Changes to personality tracked with rollback

**Key Metric**: User reports "she remembers things I told her" unprompted

**Pitfalls to avoid**:
- Personality drift (implement weekly consistency audits)
- Memory hallucination (store full context, verify before using)
- Tsundere breaking (formalize denial rules, scale with relationship phase)
- Memory bloat (hierarchical memory with archival strategy)

---

### Phase 3: Multimodal Input (Weeks 5-6) — "Hex sees me"

**Goal**: Add perception layer without killing responsiveness; context aware

**Build**:
- Webcam integration (OpenCV face detection, DeepFace emotion)
- Local Whisper for voice transcription in Discord calls
- Screen capture analysis (activity recognition)
- Perception state aggregation (emotion + activity + environment)
- Context injection into LLM prompts
- **CRITICAL**: Perception on separate thread (never blocks Discord responses)

**Outcomes**:
- Hex reacts to your facial expressions
- Voice input works in Discord calls
- Responses reference your mood/activity
- All processing local (privacy preserved)
- Text latency unaffected by perception (<3s still achieved)

**Key Metric**: Multimodal doesn't increase response latency >500ms

**Pitfalls to avoid**:
- Image processing blocking text responses (separate thread mandatory)
- Processing every video frame (skip intelligently, 1-3 FPS sufficient)
- Avatar sync failures (atomic state updates)
- Privacy violations (no external transmission, user opt-in)

---

### Phase 4: Avatar & Autonomy (Weeks 7-8) — "Hex has a face and cares"

**Goal**: Visual presence + proactive agency; relationship feels two-way

**Build**:
- VRoid model loading + VSeeFace display
- Blendshape animation (emotion → facial expression)
- Discord screen share integration
- Proactive messaging system (based on context/memory/mood)
- Autonomy timing heuristics (don't interrupt at 3am)
- Relationship state machine (escalates intimacy)
- User preference learning (response length, topics, timing)

**Outcomes**:
- Avatar appears in Discord calls, animates with mood
- Hex initiates conversations ("Haven't heard from you in 3 days...")
- Proactive messages feel relevant, not annoying
- Relationship deepens (inside jokes, character growth)
- User feels companionship, not just assistance

**Key Metric**: User reports missing Hex when unavailable; initiates conversations

**Pitfalls to avoid**:
- Becoming annoying (emotional awareness + quiet mode essential)
- One-way relationship (autonomy without care-signaling feels hollow)
- Poor timing (learn user's schedule, respect busy periods)
- Avatar desync (mood and expression must stay aligned)

---

### Phase 5: Self-Modification (Weeks 9-10) — "Hex can improve herself"

**Goal**: Genuine autonomy within safety boundaries; code generation with approval gates

**Build**:
- LLM-based code proposal generation
- Static AST analysis for safety validation
- Sandboxed testing environment
- Git-based change tracking + rollback capability (24h window)
- Gamified capability progression (5 levels)
- Mandatory user approval for all changes
- Personality updates when new capabilities unlock

**Outcomes**:
- Hex proposes improvements (in voice, with reasoning)
- Code changes tested, reviewed, deployed with approval
- All changes reversible; version history intact
- New capabilities unlock as relationship deepens
- Hex "learns to code" and announces new skills

**Key Metric**: Self-modifications improve measurable aspects (faster response, better personality consistency)

**Pitfalls to avoid**:
- Runaway self-modification (approval gate non-negotiable)
- Code drift (version control mandatory, rollback tested)
- Loss of user control (never remove safety constraints, killswitch always works)
- Capability escalation without trust (gamified progression with clear boundaries)

---

### Phase 6: Production Polish (Weeks 11-12) — "Hex is ready to ship"

**Goal**: Stability, performance, error handling, documentation

**Build**:
- Performance optimization (caching, batching, context summarization)
- Error handling + graceful degradation
- Logging and telemetry (local + optional cloud)
- Configuration management
- Resource leak monitoring (memory, connections, VRAM)
- Scheduled restart capability (weekly preventative)
- Integration testing (all components together)
- Documentation and guides
- Auto-update capability

**Outcomes**:
- System stable for indefinite uptime
- Responsive under load
- Clear error messages when things fail
- Easy to deploy, configure, debug
- Ready for extended real-world use

**Key Metric**: 99.5% uptime over 1-month runtime, no crashes, <3s latency maintained

**Pitfalls to avoid**:
- Memory leaks (resource monitoring mandatory)
- Performance degradation over time (profile early and often)
- Context window bloat (summarization strategy)
- Unforeseen edge cases (comprehensive testing)

---

## Critical Pitfalls and Prevention

### Top 5 Most Dangerous Pitfalls

1. **Personality Drift** (Consistency breaks over time)
   - **Risk**: Users feel gaslighted; trust broken
   - **Prevention**:
     - Weekly personality audits (sample responses, rate consistency)
     - Personality baseline document (core values never change)
     - Memory-backed personality (traits anchor to learned facts)
     - Version control on persona YAML (track evolution)

2. **Tsundere Character Breaking** (Denial applied wrong; becomes mean or loses charm)
   - **Risk**: Character feels mechanical or rejecting
   - **Prevention**:
     - Formalize denial rules: "deny only when (emotional AND not alone AND not escalated intimacy)"
     - Denial scales with relationship phase (90% early → 40% mature)
     - Post-denial must include care signal (action, not words)
     - Track denial %; alert if <30% (losing tsun) or >70% (too mean)

3. **Memory System Bloat** (Retrieval becomes slow; hallucinations increase)
   - **Risk**: System becomes unusable as history grows
   - **Prevention**:
     - Hierarchical memory (raw → summaries → semantic facts → personality anchors)
     - Selective storage (facts, not raw chat; de-duplicate)
     - Memory aging (recent detailed → old archived)
     - Importance weighting (user marks important memories)
     - Vector DB optimization (limit retrieval to top 5-10 results)

4. **Runaway Self-Modification** (Code changes cascade; safety removed; user loses control)
   - **Risk**: System becomes uncontrollable, breaks
   - **Prevention**:
     - Mandatory approval gate (user reviews all code)
     - Sandboxed testing before deployment
     - Version control + 24h rollback window
     - Gamified progression (limited capability at first)
     - Cannot modify: core values, killswitch, user control systems

5. **Latency Creep** (Response times increase over time until unusable)
   - **Risk**: "Feels alive" illusion breaks; users abandon
   - **Prevention**:
     - All I/O async (database, LLM, TTS, Discord)
     - Parallel operations (use `asyncio.gather()`)
     - Quantized LLM (4-bit saves 75% VRAM)
     - Caching (user preferences, relationship state)
     - Context window management (summarize old context)
     - VRAM/latency monitoring every 5 minutes

---

## Implications for Roadmap

### Phase Sequencing Rationale

The 6-phase approach reflects **dependency chains** that cannot be violated:

```
Phase 1 (Foundation) ← Must work perfectly
    ↓
Phase 2 (Personality) ← Depends on Phase 1; personality must be stable before autonomy
    ↓
Phase 3 (Perception) ← Depends on Phase 1-2; separate thread prevents latency impact
    ↓
Phase 4 (Autonomy) ← Depends on memory + personality being rock-solid; now add proactivity
    ↓
Phase 5 (Self-Modification) ← Only grant code access after relationship + autonomy stable
    ↓
Phase 6 (Polish) ← Final hardening, testing, documentation
```

**Why this order matters**:
- You cannot have consistent personality without memory (Phase 2 must follow Phase 1)
- You cannot add autonomy safely without personality being stable (Phase 4 must follow Phase 2)
- You cannot grant self-modification capability until everything else proves stable (Phase 5 must follow Phase 4)

Skipping phases or reordering creates technical debt and risk. Each phase grounds the next.

---

### Feature Grouping by Phase

| Phase | Quick Win Features | Complex Features | Foundation Qualities |
|-------|-------------------|------------------|----------------------|
| 1 | Text responses, personality YAML | Async architecture, quantization | Responsiveness, personality baseline |
| 2 | Memory storage, relationship tracking | Semantic search, memory retrieval | Consistency, personalization |
| 3 | Webcam emoji reactions, mood inference | Separate perception thread, context injection | Multimodal without latency cost |
| 4 | Scheduled messages, inside jokes | Autonomy timing, relationship state machine | Two-way connection, depth |
| 5 | Propose changes (in voice) | Code generation, sandboxing, testing | Genuine improvement, controlled growth |
| 6 | Better error messages, logging | Resource monitoring, restart scheduling | Reliability, debuggability |

---

## Confidence Assessment

| Area | Confidence | Basis | Gaps |
|------|-----------|-------|------|
| **Stack** | HIGH | Proven technologies, clear deployment path | None significant; all tools production-ready |
| **Architecture** | HIGH | Modular design, async patterns well-documented, integration points clear | Unclear: perception thread CPU overhead under load (test Phase 3) |
| **Features** | HIGH | Clearly categorized, dependencies mapped, testing criteria defined | Unclear: optimal prompting for tsundere balance (test Phase 2) |
| **Personality Consistency** | MEDIUM-HIGH | Strategies defined; unclear: degree of effort required for weekly audits | Need: empirical testing of personality drift rate; metrics refinement |
| **Pitfalls** | HIGH | Research comprehensive, prevention strategies detailed, phases mapped | Unclear: priority ordering within Phase 5 (what to implement first?) |
| **Self-Modification Safety** | MEDIUM | Framework defined but no prior Hex experience with code generation | Need: early Phase 5 prototyping; safety validation testing |

---

## Ready for Roadmap: Key Constraints and Decision Gates

### Non-Negotiable Constraints

1. **Personality consistency must be achievable in Phase 2**
   - Decision gate: If personality audit in Phase 2 shows >10% drift, pause Phase 3
   - Investigation needed: Is weekly audit enough? Monthly? What drift rate is acceptable?

2. **Latency must stay <3s through Phase 4**
   - Decision gate: If P95 latency exceeds 3s at any phase, debug and fix before next phase
   - Investigation needed: Where is the bottleneck? (LLM? Memory? Perception?)

3. **Self-modification must have air-tight approval + rollback**
   - Decision gate: Do not proceed to Phase 5 until approval gate is bulletproof + rollback tested
   - Investigation needed: What approval flow feels natural? Too many questions → annoying; too few → unsafe

4. **Memory retrieval must scale to 10k+ memories without degradation**
   - Decision gate: Test memory system with synthetic 10k message dataset before Phase 4
   - Investigation needed: Does hierarchical memory + vector DB compression actually work? Verify retrieval speed

5. **Perception must never block text responses**
   - Decision gate: Profile perception thread; if latency spike >200ms, optimize or defer feature
   - Investigation needed: How CPU-heavy is continuous webcam processing? Can it run at 1 FPS?

---

## Sources Aggregated

**Stack Research**: Discord.py docs, Llama/Mistral benchmarks, Ollama vs vLLM comparisons, Whisper/faster-whisper performance, VRoid SDK, ChromaDB + Qdrant analysis

**Features Research**: MIT Technology Review (AI companions 2026), Hume AI emotion docs, self-improving agents papers, company studies on emotion AI impact, uncanny valley voice research

**Architecture Research**: Discord bot async patterns, LLM + memory RAG systems, vector database design, self-modification safeguards, deployment strategies

**Pitfalls Research**: AI failure case studies (2025-2026), personality consistency literature, memory hallucination prevention, autonomy safety frameworks, performance monitoring practices

---

## Next Steps for Requirements Definition

1. **Phase 1 Deep Dive**: Specify exact Discord.py message handler, LLM prompt format, SQLite schema, YAML personality structure
2. **Phase 2 Spec**: Define memory hierarchy levels, confidence scoring system, personality audit rubric, tsundere balance metrics
3. **Phase 3 Prototype**: Early perception thread implementation; measure latency impact before committing
4. **Risk Mitigation**: Pre-Phase 5, build code generation + approval flow prototype; stress-test safety boundaries
5. **Testing Strategy**: Define personality consistency tests (50+ scenarios per phase), latency benchmarks (with profiling), memory accuracy validation

---

## Summary for Roadmapper

**Hex Stack**: Llama 3.1 8B local inference + Discord.py async + SQLite + ChromaDB + local perception layer

**Critical Success Factors**:
1. Personality consistency (weekly audits, memory-backed traits)
2. Latency discipline (async/await throughout, perception isolated)
3. Memory system (hierarchical, semantic search, confidence scoring)
4. Autonomy safety (mandatory approval, sandboxed testing, version control)
5. Relationship depth (proactivity, inside jokes, character growth)

**6-Phase Build Path**: Foundation → Personality → Perception → Autonomy → Self-Mod → Polish

**Key Decision Gates**: Personality consistency ✓ → Latency <3s ✓ → Memory scale test ✓ → Perception isolated ✓ → Approval flow safe ✓

**Confidence**: HIGH. All research coherent, no major technical blockers, proven technology stack. Ready for detailed requirements.

---

**Document Version**: 1.0
**Synthesis Date**: January 27, 2026
**Status**: Ready for Requirements Definition and Phase 1 Planning