Mai/.planning/phases/04-memory-context-management/04-03-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	must_haves
04-memory-context-management	03	execute	2	04-01	src/memory/backup/__init__.py src/memory/backup/archival.py src/memory/backup/retention.py src/memory/storage/compression.py src/memory/__init__.py	true	truths artifacts key_links Old conversations are automatically compressed to save space Compression preserves important information while reducing size JSON archival system stores compressed conversations Smart retention keeps important conversations longer 7/30/90 day compression tiers are implemented path provides min_lines src/memory/storage/compression.py Progressive conversation compression 80 path provides min_lines src/memory/backup/archival.py JSON export/import for long-term storage 60 path provides min_lines src/memory/backup/retention.py Smart retention policies based on conversation importance 50 path provides exports src/memory/__init__.py MemoryManager with archival capabilities MemoryManager CompressionEngine from to via pattern src/memory/storage/compression.py src/memory/storage/sqlite_manager.py conversation data retrieval for compression sqlite_manager.get_conversation from to via pattern src/memory/backup/archival.py src/memory/storage/compression.py compressed conversation data compression_engine.compress from to via pattern src/memory/backup/retention.py src/memory/storage/sqlite_manager.py conversation importance analysis sqlite_manager.update_importance_score

Implement progressive compression and archival system to manage memory growth efficiently. This ensures the memory system can scale without indefinite growth while preserving important information.

Purpose: Automatically compress and archive old conversations to maintain performance and storage efficiency Output: Working compression engine with JSON archival and smart retention policies

<execution_context> @/.opencode/get-shit-done/workflows/execute-plan.md @/.opencode/get-shit-done/templates/summary.md </execution_context>

@.planning/phases/04-memory-context-management/04-CONTEXT.md @.planning/phases/04-memory-context-management/04-RESEARCH.md @.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md

Reference storage foundation

@.planning/phases/04-memory-context-management/04-01-SUMMARY.md

Reference compression research patterns

@.planning/phases/04-memory-context-management/04-RESEARCH.md

Task 1: Implement progressive compression engine src/memory/storage/compression.py Create src/memory/storage/compression.py with CompressionEngine class:

1. Implement progressive compression following research pattern:

   • 7 days: Full content (no compression)
   • 30 days: Key points extraction (70% retention)
   • 90 days: Brief summary (40% retention)
   • 365+ days: Metadata only

2. Add transformers to requirements.txt for summarization

3. Implement compression methods:

   • extract_key_points(conversation: Conversation) -> str
   • generate_summary(conversation: Conversation, target_ratio: float = 0.4) -> str
   • extract_metadata_only(conversation: Conversation) -> dict

4. Use hybrid extractive-abstractive approach:

   • Extract key sentences using NLTK or simple heuristics
   • Generate abstractive summary using transformers pipeline
   • Preserve important quotes, facts, and decision points

5. Include compression quality metrics:

   • Information retention scoring
   • Compression ratio calculation
   • Quality validation checks

6. Add methods:

   • compress_by_age(conversation: Conversation) -> CompressedConversation
   • get_compression_level(age_days: int) -> CompressionLevel
   • decompress(compressed: CompressedConversation) -> ConversationSummary

Follow existing error handling patterns from src/models/ modules. python -c "from src.memory.storage.compression import CompressionEngine; ce = CompressionEngine(); print('Compression engine created successfully')" Compression engine can compress conversations at different levels

Task 2: Create JSON archival and smart retention systems src/memory/backup/__init__.py, src/memory/backup/archival.py, src/memory/backup/retention.py, src/memory/__init__.py Create archival and retention components:

1. Create src/memory/backup/archival.py with ArchivalManager:

   • JSON export/import for compressed conversations
   • Archival directory structure by year/month
   • Batch archival operations
   • Import capabilities for restoring conversations
   • Methods: archive_conversations(), restore_conversation(), list_archived()

2. Create src/memory/backup/retention.py with RetentionPolicy:

   • Value-based retention scoring
   • User-marked important conversations
   • High engagement detection (length, back-and-forth)
   • Smart retention overrides compression rules
   • Methods: calculate_importance_score(), should_retain_full(), update_retention_policy()

3. Update src/memory/init.py to integrate archival:

   • Add archival methods to MemoryManager
   • Implement automatic compression triggering
   • Add archival scheduling capabilities
   • Provide manual archival controls

4. Include backup integration:

   • Integrate with existing system backup processes
   • Ensure archival data is included in regular backups
   • Provide restore verification and validation

Follow existing patterns for data management and error handling. Ensure archival JSON structure is human-readable and versioned for future compatibility. python -c "from src.memory import MemoryManager; mm = MemoryManager(':memory:'); print('Memory manager with archival created successfully')" Memory manager can compress and archive conversations automatically

After completion, verify: 1. Compression engine works at all 4 levels (7/30/90/365+ days) 2. JSON archival stores compressed conversations correctly 3. Smart retention keeps important conversations from over-compression 4. Archival directory structure is organized and navigable 5. Integration with storage layer works for compression triggers 6. Restore functionality brings back conversations correctly

<success_criteria>

• Progressive compression reduces storage usage while preserving information
• JSON archival provides human-readable long-term storage
• Smart retention policies preserve important conversations
• Compression ratios meet research recommendations (70%/40%/metadata)
• Archival system integrates with existing backup processes
• Memory manager provides unified interface for compression and archival </success_criteria>

After completion, create `.planning/phases/04-memory-context-management/04-03-SUMMARY.md`