Rosie/RESEARCH_FINDINGS.md

VRM Rendering Research Findings

Date: 2025-09-30

Problem

Need to render VRM 3D model in a transparent desktop window using Python.

VRM Format

• VRM is based on glTF 2.0 with VRM-specific extensions
• Uses MToon shader for toon/cel-shaded rendering
• Contains blend shapes for facial expressions
• Contains bone data for animations

Python VRM/glTF Libraries

pygltflib (RECOMMENDED)

• Pros: Can parse VRM files, read extension data, well-maintained
• Cons: Parsing only, doesn't handle rendering
• Use: Load VRM data, extract meshes/materials/animations

Alternatives

• gltflib: Fork of pygltflib with divergent features
• pyrender: Can render glTF but doesn't support MToon shader properly (produces holes)

Rendering Solutions

✅ CHOSEN: PyQt6 + QOpenGLWidget

Best fit for desktop widget

Pros:

• Native transparent window support (WA_TranslucentBackground)
• Always-on-top windows (WindowStaysOnTopHint)
• Frameless windows (FramelessWindowHint)
• Direct OpenGL rendering via QOpenGLWidget
• Excellent for desktop widgets/mascots
• Can implement custom MToon shader in GLSL

Implementation:

class VRMWidget(QOpenGLWidget):
    def initializeGL(self):
        # Setup OpenGL state, load shaders
    def resizeGL(self, w, h):
        # Handle window resize
    def paintGL(self):
        # Render VRM model

Window Setup:

window.setAttribute(Qt.WA_TranslucentBackground)
window.setWindowFlags(Qt.FramelessWindowHint | Qt.WindowStaysOnTopHint)

Alternative: Panda3D

Not chosen due to:

• Limited transparent window support
• Requires embedding in another GUI framework anyway
• More complexity for our use case

Alternative: Blender Python API

Not chosen due to:

• Too heavyweight for a desktop widget
• Not designed for real-time desktop applications

MToon Shader Implementation

Since Python libraries don't support MToon, we need to:

1. Parse VRM using pygltflib to get material properties
2. Implement custom GLSL MToon shader in OpenGL
3. Apply shader during rendering

MToon shader resources:

• Reference: https://github.com/Santarh/MToon
• Can port GLSL version to PyQt's OpenGL context

Recommended Tech Stack

Core

• PyQt6: Window management, transparency, always-on-top
• QOpenGLWidget: OpenGL rendering context
• pygltflib: VRM/glTF parsing
• PyOpenGL: OpenGL bindings for Python

Audio

• pygame: Sound effects (squeaks, etc.)
• pyttsx3 or Coqui TTS: Text-to-speech
• whisper.cpp Python bindings: Speech-to-text

LLM

• llama-cpp-python: Local LLM inference
• ollama-python: Alternative if using Ollama

Discord

• discord.py: Discord bot

Packaging

• PyInstaller or Nuitka: .exe compilation

Next Steps

1. Install PyQt6, PyOpenGL, pygltflib
2. Create basic transparent window
3. Load VRM using pygltflib
4. Implement basic OpenGL renderer
5. Port MToon shader to GLSL
6. Add interactivity (drag, click)