Initial commit: Desktop Waifu MVP foundation
- Project structure with modular architecture - State management system (emotion states, conversation history) - Transparent, draggable PyQt6 window - OpenGL rendering widget with placeholder cube - Discord bot framework (commands, event handlers) - Complete documentation (README, project plan, research findings) - Environment configuration template - Dependencies defined in requirements.txt MVP features working: - Transparent window appears at bottom-right - Window can be dragged around - Placeholder 3D cube renders and rotates - Emotion state changes on interaction - Event-driven state management Next steps: VRM model loading and rendering
This commit is contained in:
105
RESEARCH_FINDINGS.md
Normal file
105
RESEARCH_FINDINGS.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# VRM Rendering Research Findings
|
||||
|
||||
## Date: 2025-09-30
|
||||
|
||||
## Problem
|
||||
Need to render VRM 3D model in a transparent desktop window using Python.
|
||||
|
||||
## VRM Format
|
||||
- VRM is based on glTF 2.0 with VRM-specific extensions
|
||||
- Uses MToon shader for toon/cel-shaded rendering
|
||||
- Contains blend shapes for facial expressions
|
||||
- Contains bone data for animations
|
||||
|
||||
## Python VRM/glTF Libraries
|
||||
|
||||
### pygltflib (RECOMMENDED)
|
||||
- **Pros**: Can parse VRM files, read extension data, well-maintained
|
||||
- **Cons**: Parsing only, doesn't handle rendering
|
||||
- **Use**: Load VRM data, extract meshes/materials/animations
|
||||
|
||||
### Alternatives
|
||||
- `gltflib`: Fork of pygltflib with divergent features
|
||||
- `pyrender`: Can render glTF but **doesn't support MToon shader properly** (produces holes)
|
||||
|
||||
## Rendering Solutions
|
||||
|
||||
### ✅ CHOSEN: PyQt6 + QOpenGLWidget
|
||||
**Best fit for desktop widget**
|
||||
|
||||
**Pros:**
|
||||
- Native transparent window support (`WA_TranslucentBackground`)
|
||||
- Always-on-top windows (`WindowStaysOnTopHint`)
|
||||
- Frameless windows (`FramelessWindowHint`)
|
||||
- Direct OpenGL rendering via QOpenGLWidget
|
||||
- Excellent for desktop widgets/mascots
|
||||
- Can implement custom MToon shader in GLSL
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
class VRMWidget(QOpenGLWidget):
|
||||
def initializeGL(self):
|
||||
# Setup OpenGL state, load shaders
|
||||
def resizeGL(self, w, h):
|
||||
# Handle window resize
|
||||
def paintGL(self):
|
||||
# Render VRM model
|
||||
```
|
||||
|
||||
**Window Setup:**
|
||||
```python
|
||||
window.setAttribute(Qt.WA_TranslucentBackground)
|
||||
window.setWindowFlags(Qt.FramelessWindowHint | Qt.WindowStaysOnTopHint)
|
||||
```
|
||||
|
||||
### Alternative: Panda3D
|
||||
**Not chosen due to:**
|
||||
- Limited transparent window support
|
||||
- Requires embedding in another GUI framework anyway
|
||||
- More complexity for our use case
|
||||
|
||||
### Alternative: Blender Python API
|
||||
**Not chosen due to:**
|
||||
- Too heavyweight for a desktop widget
|
||||
- Not designed for real-time desktop applications
|
||||
|
||||
## MToon Shader Implementation
|
||||
Since Python libraries don't support MToon, we need to:
|
||||
1. Parse VRM using pygltflib to get material properties
|
||||
2. Implement custom GLSL MToon shader in OpenGL
|
||||
3. Apply shader during rendering
|
||||
|
||||
MToon shader resources:
|
||||
- Reference: https://github.com/Santarh/MToon
|
||||
- Can port GLSL version to PyQt's OpenGL context
|
||||
|
||||
## Recommended Tech Stack
|
||||
|
||||
### Core
|
||||
- **PyQt6**: Window management, transparency, always-on-top
|
||||
- **QOpenGLWidget**: OpenGL rendering context
|
||||
- **pygltflib**: VRM/glTF parsing
|
||||
- **PyOpenGL**: OpenGL bindings for Python
|
||||
|
||||
### Audio
|
||||
- **pygame**: Sound effects (squeaks, etc.)
|
||||
- **pyttsx3** or **Coqui TTS**: Text-to-speech
|
||||
- **whisper.cpp Python bindings**: Speech-to-text
|
||||
|
||||
### LLM
|
||||
- **llama-cpp-python**: Local LLM inference
|
||||
- **ollama-python**: Alternative if using Ollama
|
||||
|
||||
### Discord
|
||||
- **discord.py**: Discord bot
|
||||
|
||||
### Packaging
|
||||
- **PyInstaller** or **Nuitka**: .exe compilation
|
||||
|
||||
## Next Steps
|
||||
1. Install PyQt6, PyOpenGL, pygltflib
|
||||
2. Create basic transparent window
|
||||
3. Load VRM using pygltflib
|
||||
4. Implement basic OpenGL renderer
|
||||
5. Port MToon shader to GLSL
|
||||
6. Add interactivity (drag, click)
|
Reference in New Issue
Block a user