Dani
a657979bfd
- Project structure with modular architecture - State management system (emotion states, conversation history) - Transparent, draggable PyQt6 window - OpenGL rendering widget with placeholder cube - Discord bot framework (commands, event handlers) - Complete documentation (README, project plan, research findings) - Environment configuration template - Dependencies defined in requirements.txt MVP features working: - Transparent window appears at bottom-right - Window can be dragged around - Placeholder 3D cube renders and rotates - Emotion state changes on interaction - Event-driven state management Next steps: VRM model loading and rendering
106 lines
2.9 KiB
Markdown
106 lines
2.9 KiB
Markdown
# VRM Rendering Research Findings
|
|
|
|
## Date: 2025-09-30
|
|
|
|
## Problem
|
|
Need to render VRM 3D model in a transparent desktop window using Python.
|
|
|
|
## VRM Format
|
|
- VRM is based on glTF 2.0 with VRM-specific extensions
|
|
- Uses MToon shader for toon/cel-shaded rendering
|
|
- Contains blend shapes for facial expressions
|
|
- Contains bone data for animations
|
|
|
|
## Python VRM/glTF Libraries
|
|
|
|
### pygltflib (RECOMMENDED)
|
|
- **Pros**: Can parse VRM files, read extension data, well-maintained
|
|
- **Cons**: Parsing only, doesn't handle rendering
|
|
- **Use**: Load VRM data, extract meshes/materials/animations
|
|
|
|
### Alternatives
|
|
- `gltflib`: Fork of pygltflib with divergent features
|
|
- `pyrender`: Can render glTF but **doesn't support MToon shader properly** (produces holes)
|
|
|
|
## Rendering Solutions
|
|
|
|
### ✅ CHOSEN: PyQt6 + QOpenGLWidget
|
|
**Best fit for desktop widget**
|
|
|
|
**Pros:**
|
|
- Native transparent window support (`WA_TranslucentBackground`)
|
|
- Always-on-top windows (`WindowStaysOnTopHint`)
|
|
- Frameless windows (`FramelessWindowHint`)
|
|
- Direct OpenGL rendering via QOpenGLWidget
|
|
- Excellent for desktop widgets/mascots
|
|
- Can implement custom MToon shader in GLSL
|
|
|
|
**Implementation:**
|
|
```python
|
|
class VRMWidget(QOpenGLWidget):
|
|
def initializeGL(self):
|
|
# Setup OpenGL state, load shaders
|
|
def resizeGL(self, w, h):
|
|
# Handle window resize
|
|
def paintGL(self):
|
|
# Render VRM model
|
|
```
|
|
|
|
**Window Setup:**
|
|
```python
|
|
window.setAttribute(Qt.WA_TranslucentBackground)
|
|
window.setWindowFlags(Qt.FramelessWindowHint | Qt.WindowStaysOnTopHint)
|
|
```
|
|
|
|
### Alternative: Panda3D
|
|
**Not chosen due to:**
|
|
- Limited transparent window support
|
|
- Requires embedding in another GUI framework anyway
|
|
- More complexity for our use case
|
|
|
|
### Alternative: Blender Python API
|
|
**Not chosen due to:**
|
|
- Too heavyweight for a desktop widget
|
|
- Not designed for real-time desktop applications
|
|
|
|
## MToon Shader Implementation
|
|
Since Python libraries don't support MToon, we need to:
|
|
1. Parse VRM using pygltflib to get material properties
|
|
2. Implement custom GLSL MToon shader in OpenGL
|
|
3. Apply shader during rendering
|
|
|
|
MToon shader resources:
|
|
- Reference: https://github.com/Santarh/MToon
|
|
- Can port GLSL version to PyQt's OpenGL context
|
|
|
|
## Recommended Tech Stack
|
|
|
|
### Core
|
|
- **PyQt6**: Window management, transparency, always-on-top
|
|
- **QOpenGLWidget**: OpenGL rendering context
|
|
- **pygltflib**: VRM/glTF parsing
|
|
- **PyOpenGL**: OpenGL bindings for Python
|
|
|
|
### Audio
|
|
- **pygame**: Sound effects (squeaks, etc.)
|
|
- **pyttsx3** or **Coqui TTS**: Text-to-speech
|
|
- **whisper.cpp Python bindings**: Speech-to-text
|
|
|
|
### LLM
|
|
- **llama-cpp-python**: Local LLM inference
|
|
- **ollama-python**: Alternative if using Ollama
|
|
|
|
### Discord
|
|
- **discord.py**: Discord bot
|
|
|
|
### Packaging
|
|
- **PyInstaller** or **Nuitka**: .exe compilation
|
|
|
|
## Next Steps
|
|
1. Install PyQt6, PyOpenGL, pygltflib
|
|
2. Create basic transparent window
|
|
3. Load VRM using pygltflib
|
|
4. Implement basic OpenGL renderer
|
|
5. Port MToon shader to GLSL
|
|
6. Add interactivity (drag, click)
|