docs(04): create phase plan

Phase 04: Memory & Context Management
- 4 plan(s) in 3 wave(s)
- 2 parallel, 2 sequential
- Ready for execution

.planning/phases/04-memory-context-management/04-03-PLAN.md

@@ -0,0 +1,172 @@
+---
+phase: 04-memory-context-management
+plan: 03
+type: execute
+wave: 2
+depends_on: ["04-01"]
+files_modified: ["src/memory/backup/__init__.py", "src/memory/backup/archival.py", "src/memory/backup/retention.py", "src/memory/storage/compression.py", "src/memory/__init__.py"]
+autonomous: true
+
+must_haves:
+  truths:
+    - "Old conversations are automatically compressed to save space"
+    - "Compression preserves important information while reducing size"
+    - "JSON archival system stores compressed conversations"
+    - "Smart retention keeps important conversations longer"
+    - "7/30/90 day compression tiers are implemented"
+  artifacts:
+    - path: "src/memory/storage/compression.py"
+      provides: "Progressive conversation compression"
+      min_lines: 80
+    - path: "src/memory/backup/archival.py"
+      provides: "JSON export/import for long-term storage"
+      min_lines: 60
+    - path: "src/memory/backup/retention.py"
+      provides: "Smart retention policies based on conversation importance"
+      min_lines: 50
+    - path: "src/memory/__init__.py"
+      provides: "MemoryManager with archival capabilities"
+      exports: ["MemoryManager", "CompressionEngine"]
+  key_links:
+    - from: "src/memory/storage/compression.py"
+      to: "src/memory/storage/sqlite_manager.py"
+      via: "conversation data retrieval for compression"
+      pattern: "sqlite_manager\\.get_conversation"
+    - from: "src/memory/backup/archival.py"
+      to: "src/memory/storage/compression.py"
+      via: "compressed conversation data"
+      pattern: "compression_engine\\.compress"
+    - from: "src/memory/backup/retention.py"
+      to: "src/memory/storage/sqlite_manager.py"
+      via: "conversation importance analysis"
+      pattern: "sqlite_manager\\.update_importance_score"
+---
+
+<objective>
+Implement progressive compression and archival system to manage memory growth efficiently. This ensures the memory system can scale without indefinite growth while preserving important information.
+
+Purpose: Automatically compress and archive old conversations to maintain performance and storage efficiency
+Output: Working compression engine with JSON archival and smart retention policies
+</objective>
+
+<execution_context>
+@~/.opencode/get-shit-done/workflows/execute-plan.md
+@~/.opencode/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/phases/04-memory-context-management/04-CONTEXT.md
+@.planning/phases/04-memory-context-management/04-RESEARCH.md
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+
+# Reference storage foundation
+@.planning/phases/04-memory-context-management/04-01-SUMMARY.md
+
+# Reference compression research patterns
+@.planning/phases/04-memory-context-management/04-RESEARCH.md
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Implement progressive compression engine</name>
+  <files>src/memory/storage/compression.py</files>
+  <action>
+Create src/memory/storage/compression.py with CompressionEngine class:
+
+1. Implement progressive compression following research pattern:
+   - 7 days: Full content (no compression)
+   - 30 days: Key points extraction (70% retention)
+   - 90 days: Brief summary (40% retention)
+   - 365+ days: Metadata only
+
+2. Add transformers to requirements.txt for summarization
+3. Implement compression methods:
+   - extract_key_points(conversation: Conversation) -> str
+   - generate_summary(conversation: Conversation, target_ratio: float = 0.4) -> str
+   - extract_metadata_only(conversation: Conversation) -> dict
+
+4. Use hybrid extractive-abstractive approach:
+   - Extract key sentences using NLTK or simple heuristics
+   - Generate abstractive summary using transformers pipeline
+   - Preserve important quotes, facts, and decision points
+
+5. Include compression quality metrics:
+   - Information retention scoring
+   - Compression ratio calculation
+   - Quality validation checks
+
+6. Add methods:
+   - compress_by_age(conversation: Conversation) -> CompressedConversation
+   - get_compression_level(age_days: int) -> CompressionLevel
+   - decompress(compressed: CompressedConversation) -> ConversationSummary
+
+Follow existing error handling patterns from src/models/ modules.
+  </action>
+  <verify>python -c "from src.memory.storage.compression import CompressionEngine; ce = CompressionEngine(); print('Compression engine created successfully')"</verify>
+  <done>Compression engine can compress conversations at different levels</done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Create JSON archival and smart retention systems</name>
+  <files>src/memory/backup/__init__.py, src/memory/backup/archival.py, src/memory/backup/retention.py, src/memory/__init__.py</files>
+  <action>
+Create archival and retention components:
+
+1. Create src/memory/backup/archival.py with ArchivalManager:
+   - JSON export/import for compressed conversations
+   - Archival directory structure by year/month
+   - Batch archival operations
+   - Import capabilities for restoring conversations
+   - Methods: archive_conversations(), restore_conversation(), list_archived()
+
+2. Create src/memory/backup/retention.py with RetentionPolicy:
+   - Value-based retention scoring
+   - User-marked important conversations
+   - High engagement detection (length, back-and-forth)
+   - Smart retention overrides compression rules
+   - Methods: calculate_importance_score(), should_retain_full(), update_retention_policy()
+
+3. Update src/memory/__init__.py to integrate archival:
+   - Add archival methods to MemoryManager
+   - Implement automatic compression triggering
+   - Add archival scheduling capabilities
+   - Provide manual archival controls
+
+4. Include backup integration:
+   - Integrate with existing system backup processes
+   - Ensure archival data is included in regular backups
+   - Provide restore verification and validation
+
+Follow existing patterns for data management and error handling. Ensure archival JSON structure is human-readable and versioned for future compatibility.
+  </action>
+  <verify>python -c "from src.memory import MemoryManager; mm = MemoryManager(':memory:'); print('Memory manager with archival created successfully')"</verify>
+  <done>Memory manager can compress and archive conversations automatically</done>
+</task>
+
+</tasks>
+
+<verification>
+After completion, verify:
+1. Compression engine works at all 4 levels (7/30/90/365+ days)
+2. JSON archival stores compressed conversations correctly
+3. Smart retention keeps important conversations from over-compression
+4. Archival directory structure is organized and navigable
+5. Integration with storage layer works for compression triggers
+6. Restore functionality brings back conversations correctly
+</verification>
+
+<success_criteria>
+- Progressive compression reduces storage usage while preserving information
+- JSON archival provides human-readable long-term storage
+- Smart retention policies preserve important conversations
+- Compression ratios meet research recommendations (70%/40%/metadata)
+- Archival system integrates with existing backup processes
+- Memory manager provides unified interface for compression and archival
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/04-memory-context-management/04-03-SUMMARY.md`
+</output>