# VRM Rendering Research Findings

## Date: 2025-09-30

## Problem
Need to render VRM 3D model in a transparent desktop window using Python.

## VRM Format
- VRM is based on glTF 2.0 with VRM-specific extensions
- Uses MToon shader for toon/cel-shaded rendering
- Contains blend shapes for facial expressions
- Contains bone data for animations

## Python VRM/glTF Libraries

### pygltflib (RECOMMENDED)
- **Pros**: Can parse VRM files, read extension data, well-maintained
- **Cons**: Parsing only, doesn't handle rendering
- **Use**: Load VRM data, extract meshes/materials/animations

### Alternatives
- `gltflib`: Fork of pygltflib with divergent features
- `pyrender`: Can render glTF but **doesn't support MToon shader properly** (produces holes)

## Rendering Solutions

### ✅ CHOSEN: PyQt6 + QOpenGLWidget
**Best fit for desktop widget**

**Pros:**
- Native transparent window support (`WA_TranslucentBackground`)
- Always-on-top windows (`WindowStaysOnTopHint`)
- Frameless windows (`FramelessWindowHint`)
- Direct OpenGL rendering via QOpenGLWidget
- Excellent for desktop widgets/mascots
- Can implement custom MToon shader in GLSL

**Implementation:**
```python
class VRMWidget(QOpenGLWidget):
    def initializeGL(self):
        # Setup OpenGL state, load shaders
    def resizeGL(self, w, h):
        # Handle window resize
    def paintGL(self):
        # Render VRM model
```

**Window Setup:**
```python
window.setAttribute(Qt.WA_TranslucentBackground)
window.setWindowFlags(Qt.FramelessWindowHint | Qt.WindowStaysOnTopHint)
```

### Alternative: Panda3D
**Not chosen due to:**
- Limited transparent window support
- Requires embedding in another GUI framework anyway
- More complexity for our use case

### Alternative: Blender Python API
**Not chosen due to:**
- Too heavyweight for a desktop widget
- Not designed for real-time desktop applications

## MToon Shader Implementation
Since Python libraries don't support MToon, we need to:
1. Parse VRM using pygltflib to get material properties
2. Implement custom GLSL MToon shader in OpenGL
3. Apply shader during rendering

MToon shader resources:
- Reference: https://github.com/Santarh/MToon
- Can port GLSL version to PyQt's OpenGL context

## Recommended Tech Stack

### Core
- **PyQt6**: Window management, transparency, always-on-top
- **QOpenGLWidget**: OpenGL rendering context
- **pygltflib**: VRM/glTF parsing
- **PyOpenGL**: OpenGL bindings for Python

### Audio
- **pygame**: Sound effects (squeaks, etc.)
- **pyttsx3** or **Coqui TTS**: Text-to-speech
- **whisper.cpp Python bindings**: Speech-to-text

### LLM
- **llama-cpp-python**: Local LLM inference
- **ollama-python**: Alternative if using Ollama

### Discord
- **discord.py**: Discord bot

### Packaging
- **PyInstaller** or **Nuitka**: .exe compilation

## Next Steps
1. Install PyQt6, PyOpenGL, pygltflib
2. Create basic transparent window
3. Load VRM using pygltflib
4. Implement basic OpenGL renderer
5. Port MToon shader to GLSL
6. Add interactivity (drag, click)