docs: map existing codebase

- STACK.md - Technologies and dependencies - ARCHITECTURE.md - System design and patterns - STRUCTURE.md - Directory layout - CONVENTIONS.md - Code style and patterns - TESTING.md - Test structure - INTEGRATIONS.md - External services - CONCERNS.md - Technical debt and issues
2026-01-26 23:14:44 -05:00
parent b1d71bc22b
commit f238a958a0
7 changed files with 1667 additions and 0 deletions
--- a/.planning/codebase/STRUCTURE.md
+++ b/.planning/codebase/STRUCTURE.md
@@ -0,0 +1,258 @@
+# Codebase Structure
+
+**Analysis Date:** 2026-01-26
+
+## Directory Layout
+
+```
+mai/
+├── src/
+│   ├── __main__.py              # CLI entry point
+│   ├── mai.py                   # Core Mai class, orchestration
+│   ├── models/
+│   │   ├── __init__.py
+│   │   ├── adapter.py           # Base model adapter interface
+│   │   ├── ollama_adapter.py    # Ollama/LMStudio implementation
+│   │   ├── model_manager.py     # Model selection and switching logic
+│   │   └── resource_monitor.py  # CPU, RAM, GPU tracking
+│   ├── memory/
+│   │   ├── __init__.py
+│   │   ├── store.py             # SQLite conversation store
+│   │   ├── vector_search.py     # Semantic similarity search
+│   │   ├── compression.py       # History compression and summarization
+│   │   └── pattern_extractor.py # Learning and pattern recognition
+│   ├── conversation/
+│   │   ├── __init__.py
+│   │   ├── engine.py            # Main conversation orchestration
+│   │   ├── context_manager.py   # Token budget and window management
+│   │   ├── turn_handler.py      # Single turn processing
+│   │   └── reasoning.py         # Reasoning transparency and clarification
+│   ├── personality/
+│   │   ├── __init__.py
+│   │   ├── core_rules.py        # Unshakeable core values enforcement
+│   │   ├── learned_behaviors.py # Personality adaptation from interactions
+│   │   ├── guardrails.py        # Safety constraints and refusal logic
+│   │   └── config_loader.py     # YAML personality configuration
+│   ├── safety/
+│   │   ├── __init__.py
+│   │   ├── executor.py          # Docker sandbox execution wrapper
+│   │   ├── risk_analyzer.py     # Risk classification (LOW/MEDIUM/HIGH/BLOCKED)
+│   │   ├── ast_validator.py     # Syntax and import validation
+│   │   └── audit_log.py         # Immutable execution history
+│   ├── selfmod/
+│   │   ├── __init__.py
+│   │   ├── analyzer.py          # Code analysis and improvement detection
+│   │   ├── generator.py         # Improvement code generation
+│   │   ├── scheduler.py         # Periodic and on-demand analysis trigger
+│   │   ├── reviewer.py          # Second-agent review coordination
+│   │   └── git_manager.py       # Git commit integration
+│   ├── interfaces/
+│   │   ├── __init__.py
+│   │   ├── cli.py               # CLI chat interface
+│   │   ├── discord_bot.py       # Discord bot implementation
+│   │   ├── message_handler.py   # Shared message processing
+│   │   ├── approval_handler.py  # Change approval workflow
+│   │   └── offline_queue.py     # Message queueing during disconnection
+│   └── utils/
+│       ├── __init__.py
+│       ├── config.py            # Configuration loading
+│       ├── logging.py           # Structured logging setup
+│       ├── validators.py        # Input validation helpers
+│       └── helpers.py           # Shared utility functions
+├── config/
+│   ├── personality.yaml         # Core personality configuration
+│   ├── models.yaml              # Model definitions and resource limits
+│   ├── safety_rules.yaml        # Risk assessment rules
+│   └── logging.yaml             # Logging configuration
+├── tests/
+│   ├── unit/
+│   │   ├── test_models.py
+│   │   ├── test_memory.py
+│   │   ├── test_conversation.py
+│   │   ├── test_personality.py
+│   │   ├── test_safety.py
+│   │   └── test_selfmod.py
+│   ├── integration/
+│   │   ├── test_conversation_flow.py
+│   │   ├── test_selfmod_workflow.py
+│   │   └── test_interfaces.py
+│   └── fixtures/
+│       ├── mock_models.py
+│       ├── test_data.py
+│       └── sample_conversations.json
+├── scripts/
+│   ├── setup_ollama.py          # Initial model downloading
+│   ├── init_db.py               # Database schema initialization
+│   └── verify_environment.py    # Pre-flight checks
+├── docker/
+│   └── Dockerfile               # Sandbox execution environment
+├── .env.example                 # Environment variables template
+├── pyproject.toml               # Project metadata and dependencies
+├── requirements.txt             # Python dependencies
+├── pytest.ini                   # Test configuration
+├── Makefile                     # Development commands
+└── README.md                    # Project overview
+```
+
+## Directory Purposes
+
+**src/:**
+- Purpose: All application code
+- Contains: Python modules organized by architectural layer
+- Key files: `mai.py` (core), `__main__.py` (CLI entry)
+
+**src/models/:**
+- Purpose: Model inference abstraction
+- Contains: Adapter interfaces, Ollama client, resource monitoring
+- Key files: `model_manager.py` (selection logic), `resource_monitor.py` (constraints)
+
+**src/memory/:**
+- Purpose: Persistent storage and retrieval
+- Contains: SQLite operations, vector search, compression
+- Key files: `store.py` (main interface), `vector_search.py` (semantic search)
+
+**src/conversation/:**
+- Purpose: Multi-turn conversation orchestration
+- Contains: Turn handling, context windowing, reasoning transparency
+- Key files: `engine.py` (main coordinator), `context_manager.py` (token budget)
+
+**src/personality/:**
+- Purpose: Values enforcement and personality adaptation
+- Contains: Core rules, learned behaviors, guardrails
+- Key files: `core_rules.py` (unshakeable values), `learned_behaviors.py` (adaptation)
+
+**src/safety/:**
+- Purpose: Code execution sandboxing and risk assessment
+- Contains: Docker wrapper, AST validation, risk classification, audit logging
+- Key files: `executor.py` (sandbox wrapper), `risk_analyzer.py` (classification)
+
+**src/selfmod/:**
+- Purpose: Autonomous code improvement and review
+- Contains: Code analysis, improvement generation, approval workflow
+- Key files: `analyzer.py` (detection), `reviewer.py` (second-agent coordination)
+
+**src/interfaces/:**
+- Purpose: External communication adapters
+- Contains: CLI handler, Discord bot, approval system
+- Key files: `cli.py` (terminal UI), `discord_bot.py` (Discord integration)
+
+**src/utils/:**
+- Purpose: Shared utilities and helpers
+- Contains: Configuration loading, logging, validation
+- Key files: `config.py` (env/file loading), `logging.py` (structured logs)
+
+**config/:**
+- Purpose: Non-code configuration files
+- Contains: YAML personality, models, safety rules definitions
+- Key files: `personality.yaml` (core values), `models.yaml` (resource profiles)
+
+**tests/:**
+- Purpose: Test suites organized by type
+- Contains: Unit tests (layer isolation), integration tests (flows), fixtures (test data)
+- Key files: Each test file mirrors `src/` structure
+
+**scripts/:**
+- Purpose: One-off setup and maintenance scripts
+- Contains: Database initialization, environment verification
+- Key files: `setup_ollama.py` (first-time model setup)
+
+**docker/:**
+- Purpose: Container configuration for sandboxed execution
+- Contains: Dockerfile for isolation environment
+- Key files: `Dockerfile` (build recipe)
+
+## Key File Locations
+
+**Entry Points:**
+- `src/__main__.py`: CLI entry, `python -m mai` launches here
+- `src/interfaces/discord_bot.py`: Discord bot main loop
+- `src/mai.py`: Core Mai class, system initialization
+
+**Configuration:**
+- `config/personality.yaml`: Core values, interaction patterns, refusal rules
+- `config/models.yaml`: Available models, resource requirements, context windows
+- `.env.example`: Required environment variables template
+
+**Core Logic:**
+- `src/mai.py`: Main orchestration
+- `src/conversation/engine.py`: Conversation turn processing
+- `src/selfmod/analyzer.py`: Improvement opportunity detection
+- `src/safety/executor.py`: Safe code execution
+
+**Testing:**
+- `tests/unit/`: Layer-isolated tests (no dependencies between layers)
+- `tests/integration/`: End-to-end flow tests
+- `tests/fixtures/`: Mock objects and test data
+
+## Naming Conventions
+
+**Files:**
+- Module files: `snake_case.py` (e.g., `model_manager.py`)
+- Entry points: `__main__.py` for packages, standalone scripts at package root
+- Config files: `snake_case.yaml` (e.g., `personality.yaml`)
+- Test files: `test_*.py` (e.g., `test_conversation.py`)
+
+**Directories:**
+- Feature areas: `snake_case` (e.g., `src/selfmod/`)
+- No abbreviations except `selfmod` (self-modification) which is project standard
+- Each layer is a top-level directory under `src/`
+
+**Functions/Classes:**
+- Classes: `PascalCase` (e.g., `ModelManager`, `ConversationEngine`)
+- Functions: `snake_case` (e.g., `generate_response()`, `validate_code()`)
+- Constants: `UPPER_SNAKE_CASE` (e.g., `MAX_CONTEXT_TOKENS`)
+- Private methods/functions: prefix with `_` (e.g., `_internal_method()`)
+
+**Types:**
+- Use type hints throughout: `def process(msg: str) -> str:`
+- Complex types in `src/utils/types.py` or local to module
+
+## Where to Add New Code
+
+**New Feature (e.g., new communication interface like Slack):**
+- Primary code: `src/interfaces/slack_adapter.py` (new adapter following discord_bot.py pattern)
+- Tests: `tests/unit/test_slack_adapter.py` and `tests/integration/test_slack_interface.py`
+- Configuration: Add to `src/interfaces/__init__.py` imports and `config/interfaces.yaml` if needed
+- Entry hook: Modify `src/mai.py` to initialize new adapter
+
+**New Component/Module (e.g., advanced memory with graph databases):**
+- Implementation: `src/memory/graph_store.py` (new module in appropriate layer)
+- Interface: Follow existing patterns (e.g., inherit from `src/memory/store.py` base)
+- Tests: Corresponding test in `tests/unit/test_memory.py` or new file if complex
+- Integration: Modify `src/mai.py` initialization to use new component with feature flag
+
+**Utilities (e.g., new helper function):**
+- Shared helpers: `src/utils/helpers.py` (functions) or new file like `src/utils/math_utils.py` if substantial
+- Internal helpers: Keep in the module where used (don't over-extract)
+- Tests: Add to `tests/unit/test_utils.py`
+
+**Configuration:**
+- Static rules: Add to appropriate YAML in `config/`
+- Dynamic config: Load in `src/utils/config.py`
+- Env-driven: Add to `.env.example` with documentation
+
+## Special Directories
+
+**tests/fixtures/:**
+- Purpose: Reusable test data and mock objects
+- Generated: No, hand-created
+- Committed: Yes, part of repository
+
+**config/:**
+- Purpose: Non-code configuration
+- Generated: No, hand-maintained
+- Committed: Yes, except secrets (use `.env`)
+
+**.env (not committed):**
+- Purpose: Local environment overrides and secrets
+- Generated: No, copied from `.env.example` and filled locally
+- Committed: No (in .gitignore)
+
+**docker/:**
+- Purpose: Sandbox environment for safe execution
+- Generated: No, hand-maintained
+- Committed: Yes
+
+---
+
+*Structure analysis: 2026-01-26*