mystiatech/Mai
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

docs(04): create gap closure plans for memory and context management

Browse Source 
Phase 04: Memory & Context Management
- 3 gap closure plans to address verification issues
- 04-05: Personality learning integration (PersonalityAdaptation, MemoryManager integration, src/personality.py)
- 04-06: Vector Store missing methods (search_by_keyword, store_embeddings)
- 04-07: Context-aware search metadata integration (get_conversation_metadata)
- All gaps from verification report addressed
- Updated roadmap to reflect 7 total plans
This commit is contained in:
Mai Development
2026-01-28 12:08:47 -05:00
parent 7cd12abe0c
commit 47e4864049
4 changed files with 536 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.planning/ROADMAP.md
View File

@@ -51,11 +51,15 @@ Mai's development is organized into three major milestones, each delivering dist
- Distill long-term patterns into personality layers
- Proactively surface relevant context from memory
**Plans:** 4 plans in 3 waves
**Status:** 3 gap closure plans needed to complete integration
**Plans:** 7 plans in 4 waves
- [x] 04-01-PLAN.md — Storage foundation with SQLite and sqlite-vec
- [x] 04-02-PLAN.md — Semantic search and context-aware retrieval
- [x] 04-03-PLAN.md — Progressive compression and JSON archival
- [x] 04-04-PLAN.md — Personality learning and adaptive layers
- [ ] 04-05-PLAN.md — Personality learning integration gap closure
- [ ] 04-06-PLAN.md — Vector Store missing methods gap closure
- [ ] 04-07-PLAN.md — Context-aware search metadata gap closure
### Phase 5: Conversation Engine
- Multi-turn context preservation

211
.planning/phases/04-memory-context-management/04-05-PLAN.md Normal file
View File

@@ -0,0 +1,211 @@
---
phase: 04-memory-context-management
plan: 05
type: execute
wave: 1
depends_on: ["04-04"]
files_modified: ["src/memory/personality/adaptation.py", "src/memory/__init__.py", "src/personality.py"]
autonomous: true
gap_closure: true
must_haves:
truths:
- "Personality layers learn from conversation patterns"
- "Personality system integrates with existing personality.py"
artifacts:
- path: "src/memory/personality/adaptation.py"
provides: "Dynamic personality updates"
min_lines: 50
- path: "src/memory/__init__.py"
provides: "Complete MemoryManager with personality learning"
exports: ["PersonalityLearner"]
- path: "src/personality.py"
provides: "Updated personality system with memory integration"
min_lines: 20
key_links:
- from: "src/memory/personality/adaptation.py"
to: "src/memory/personality/layer_manager.py"
via: "layer updates for adaptation"
pattern: "layer_manager\\.update_layer"
- from: "src/memory/__init__.py"
to: "src/memory/personality/adaptation.py"
via: "PersonalityLearner integration"
pattern: "PersonalityLearner.*update_personality"
- from: "src/personality.py"
to: "src/memory/personality/layer_manager.py"
via: "personality overlay application"
pattern: "layer_manager\\.get_active_layers"
---
<objective>
Complete personality learning integration by implementing missing PersonalityAdaptation class and connecting all personality learning components to the MemoryManager and existing personality system.
Purpose: Close the personality learning integration gap identified in verification
Output: Working personality learning system fully integrated with memory and personality systems
</objective>
<execution_context>
@~/.opencode/get-shit-done/workflows/execute-plan.md
@~/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/04-memory-context-management/04-CONTEXT.md
@.planning/phases/04-memory-context-management/04-RESEARCH.md
@.planning/phases/04-memory-context-management/04-memory-context-management-VERIFICATION.md
# Reference existing personality components
@src/memory/personality/pattern_extractor.py
@src/memory/personality/layer_manager.py
@src/resource/personality.py
# Reference memory manager
@src/memory/__init__.py
</context>
<tasks>
<task type="auto">
<name>Task 1: Implement PersonalityAdaptation class</name>
<files>src/memory/personality/adaptation.py</files>
<action>
Create src/memory/personality/adaptation.py with PersonalityAdaptation class to close the missing file gap:
1. PersonalityAdaptation class with time-weighted learning:
- update_personality_layer(patterns, layer_id, adaptation_rate)
- calculate_adaptation_rate(conversation_history, user_feedback)
- apply_stability_controls(proposed_changes, current_state)
- integrate_user_feedback(feed_data, layer_weights)
2. Time-weighted learning implementation:
- Recent conversations have less influence (exponential decay)
- Historical patterns provide stable baseline
- Prevent rapid personality swings with rate limiting
- Confidence scoring for pattern reliability
3. Stability controls:
- Maximum change per update (e.g., 10% weight shift)
- Cooling period between major adaptations
- Core value protection (certain aspects never change)
- Reversion triggers for unwanted changes
4. Integration methods:
- import_pattern_data(pattern_extractor, conversation_range)
- export_layer_config(layer_manager, output_format)
- validate_layer_consistency(layers, core_personality)
5. Configuration and persistence:
- Learning rate configuration (slow/medium/fast)
- Adaptation history tracking
- Rollback capability for problematic changes
- Integration with existing memory storage
Follow existing error handling patterns from layer_manager.py. Use similar data structures and method signatures for consistency.
</action>
<verify>python -c "from src.memory.personality.adaptation import PersonalityAdaptation; pa = PersonalityAdaptation(); print('PersonalityAdaptation created successfully')"</verify>
<done>PersonalityAdaptation class provides time-weighted learning with stability controls</done>
</task>
<task type="auto">
<name>Task 2: Integrate personality learning with MemoryManager</name>
<files>src/memory/__init__.py</files>
<action>
Update src/memory/__init__.py to integrate personality learning and export PersonalityLearner:
1. Import PersonalityAdaptation in memory/personality/__init__.py:
- Add from .adaptation import PersonalityAdaptation
- Update __all__ to include PersonalityAdaptation
2. Create PersonalityLearner class in MemoryManager:
- Combines PatternExtractor, LayerManager, and PersonalityAdaptation
- Methods: learn_from_conversations(conversation_range), apply_learning(), get_current_personality()
- Learning triggers: after conversations, periodic updates, manual requests
3. Integration with existing MemoryManager:
- Add personality_learner attribute to MemoryManager.__init__
- Implement learning_workflow() method for coordinated learning
- Add personality data persistence to existing storage
- Provide learning controls (enable/disable, rate, triggers)
4. Export PersonalityLearner from memory/__init__.py:
- Add PersonalityLearner to __all__
- Ensure it's importable as from src.memory import PersonalityLearner
5. Learning workflow integration:
- Hook into conversation storage for automatic learning triggers
- Periodic learning schedule (e.g., daily pattern analysis)
- Integration with existing configuration system
- Memory usage monitoring for learning processes
Update existing MemoryManager methods to support personality learning without breaking current functionality. Follow the existing pattern of having feature-specific managers within the main MemoryManager.
</action>
<verify>python -c "from src.memory import PersonalityLearner; pl = PersonalityLearner(); print('PersonalityLearner imported successfully')"</verify>
<done>PersonalityLearner is integrated with MemoryManager and available for import</done>
</task>
<task type="auto">
<name>Task 3: Create src/personality.py with memory integration</name>
<files>src/personality.py</files>
<action>
Create src/personality.py to integrate with memory personality learning system:
1. Core personality system:
- Import PersonalityLearner from memory system
- Maintain core personality values (immutable)
- Apply learned personality layers as overlays
- Protect core values from learned modifications
2. Integration with existing personality:
- Import and extend src/resource/personality.py functionality
- Add memory integration to existing personality methods
- Hybrid system prompt + behavior configuration
- Context-aware personality layer activation
3. Personality application methods:
- get_personality_response(context, user_input) -> enhanced_response
- apply_personality_layers(base_response, context) -> final_response
- get_active_layers(conversation_context) -> List[PersonalityLayer]
- validate_personality_consistency(applied_layers) -> bool
4. Configuration and control:
- Learning enable/disable flag
- Layer activation rules
- Core value protection settings
- User feedback integration for personality tuning
5. Integration points:
- Connect to MemoryManager.PersonalityLearner
- Use existing personality.py from src/resource as base
- Ensure compatibility with existing conversation systems
- Provide clear separation between core and learned personality
Follow the pattern established in src/resource/personality.py but extend it with memory learning integration. Ensure core personality values remain protected while allowing learned layers to enhance responses.
</action>
<verify>python -c "from src.personality import get_personality_response; print('Personality system integration working')"</verify>
<done>src/personality.py integrates with memory learning while protecting core values</done>
</task>
</tasks>
<verification>
After completion, verify:
1. PersonalityAdaptation class exists and implements time-weighted learning
2. PersonalityLearner is integrated into MemoryManager and exportable
3. src/personality.py exists and integrates with memory personality system
4. Personality learning workflow connects all components (PatternExtractor -> LayerManager -> PersonalityAdaptation)
5. Core personality values are protected from learned modifications
6. Learning system can be enabled/disabled through configuration
</verification>
<success_criteria>
- Personality learning integration gap is completely closed
- All personality components work together as a cohesive system
- Personality layers learn from conversation patterns over time
- Core personality values remain protected while allowing adaptive learning
- Integration follows existing patterns and maintains code consistency
- System is ready for testing and eventual user verification
</success_criteria>
<output>
After completion, create `.planning/phases/04-memory-context-management/04-05-SUMMARY.md`
</output>

161
.planning/phases/04-memory-context-management/04-06-PLAN.md Normal file
View File

@@ -0,0 +1,161 @@
---
phase: 04-memory-context-management
plan: 06
type: execute
wave: 1
depends_on: ["04-01"]
files_modified: ["src/memory/storage/vector_store.py"]
autonomous: true
gap_closure: true
must_haves:
truths:
- "User can search conversations by semantic meaning"
artifacts:
- path: "src/memory/storage/vector_store.py"
provides: "Vector storage and retrieval with sqlite-vec"
contains: "search_by_keyword method"
contains: "store_embeddings method"
key_links:
- from: "src/memory/retrieval/semantic_search.py"
to: "src/memory/storage/vector_store.py"
via: "vector similarity search operations"
pattern: "vector_store\\.search_by_keyword"
- from: "src/memory/retrieval/semantic_search.py"
to: "src/memory/storage/vector_store.py"
via: "embedding storage operations"
pattern: "vector_store\\.store_embeddings"
---
<objective>
Complete VectorStore implementation by adding missing search_by_keyword and store_embeddings methods that are called by SemanticSearch but not implemented.
Purpose: Close the vector store methods gap to enable full semantic search functionality
Output: Complete VectorStore with all required methods for semantic search operations
</objective>
<execution_context>
@~/.opencode/get-shit-done/workflows/execute-plan.md
@~/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/04-memory-context-management/04-CONTEXT.md
@.planning/phases/04-memory-context-management/04-memory-context-management-VERIFICATION.md
# Reference existing vector store implementation
@src/memory/storage/vector_store.py
# Reference semantic search that calls these methods
@src/memory/retrieval/semantic_search.py
</context>
<tasks>
<task type="auto">
<name>Task 1: Implement search_by_keyword method in VectorStore</name>
<files>src/memory/storage/vector_store.py</files>
<action>
Add missing search_by_keyword method to VectorStore class to close the verification gap:
1. search_by_keyword method implementation:
- search_by_keyword(self, query: str, limit: int = 10) -> List[Dict]
- Perform keyword-based search on message content using FTS if available
- Fall back to LIKE queries if FTS not enabled
- Return results in same format as vector search for consistency
2. Keyword search implementation:
- Use SQLite FTS (Full-Text Search) if virtual tables exist
- Query message_content and conversation_summary fields
- Support multiple keywords with AND/OR logic
- Rank results by keyword frequency and position
3. Integration with existing vector operations:
- Use same database connection as existing methods
- Follow existing error handling patterns
- Return results compatible with hybrid_search in SemanticSearch
- Include message_id, conversation_id, content, and relevance score
4. Performance optimizations:
- Add appropriate indexes for keyword search if missing
- Use query parameters to prevent SQL injection
- Limit result sets for performance
- Cache frequent keyword queries if beneficial
5. Method signature matching:
- Match the expected signature from semantic_search.py line 248
- Return format: List[Dict] with message_id, conversation_id, content, score
- Handle edge cases: empty queries, no results, database errors
The method should be called by SemanticSearch.hybrid_search at line 248. Verify the exact signature and return format by checking semantic_search.py before implementation.
</action>
<verify>python -c "from src.memory.storage.vector_store import VectorStore; vs = VectorStore(); result = vs.search_by_keyword('test', limit=5); print(f'search_by_keyword returned {len(result)} results')"</verify>
<done>VectorStore.search_by_keyword method provides keyword-based search functionality</done>
</task>
<task type="auto">
<name>Task 2: Implement store_embeddings method in VectorStore</name>
<files>src/memory/storage/vector_store.py</files>
<action>
Add missing store_embeddings method to VectorStore class to close the verification gap:
1. store_embeddings method implementation:
- store_embeddings(self, embeddings: List[Tuple[str, List[float]]]) -> bool
- Batch store multiple embeddings efficiently
- Handle conversation_id and message_id associations
- Return success/failure status
2. Embedding storage implementation:
- Use existing vec_entries virtual table from current implementation
- Insert embeddings with proper rowid mapping to messages
- Support batch inserts for performance
- Handle embedding dimension validation
3. Integration with existing storage patterns:
- Follow same database connection patterns as other methods
- Use existing error handling and transaction management
- Coordinate with sqlite_manager for message metadata
- Maintain consistency with existing vector storage
4. Method signature compatibility:
- Match expected signature from semantic_search.py line 363
- Accept list of (id, embedding) tuples
- Return boolean success indicator
- Handle partial failures gracefully
5. Performance and reliability:
- Use transactions for batch operations
- Validate embedding dimensions before insertion
- Handle database constraint violations
- Provide detailed error logging for debugging
The method should be called by SemanticSearch at line 363. Verify the exact signature and expected behavior by checking semantic_search.py before implementation. Ensure compatibility with the existing vec_entries table structure and sqlite-vec extension usage.
</action>
<verify>python -c "from src.memory.storage.vector_store import VectorStore; import numpy as np; vs = VectorStore(); test_emb = [('test_id', np.random.rand(1536).tolist())]; result = vs.store_embeddings(test_emb); print(f'store_embeddings returned: {result}')"</verify>
<done>VectorStore.store_embeddings method provides batch embedding storage functionality</done>
</task>
</tasks>
<verification>
After completion, verify:
1. search_by_keyword method exists and is callable from SemanticSearch
2. store_embeddings method exists and is callable from SemanticSearch
3. Both methods follow the exact signatures expected by semantic_search.py
4. Methods integrate properly with existing VectorStore database operations
5. SemanticSearch.hybrid_search can now call these methods without errors
6. Keyword search returns properly formatted results compatible with vector search
</verification>
<success_criteria>
- VectorStore missing methods gap is completely closed
- SemanticSearch can perform hybrid search combining keyword and vector search
- Methods follow existing VectorStore patterns and error handling
- Database operations are efficient and properly transactional
- Integration with semantic search is seamless and functional
- All anti-patterns related to missing method calls are resolved
</success_criteria>
<output>
After completion, create `.planning/phases/04-memory-context-management/04-06-SUMMARY.md`
</output>

159
.planning/phases/04-memory-context-management/04-07-PLAN.md Normal file
View File

@@ -0,0 +1,159 @@
---
phase: 04-memory-context-management
plan: 07
type: execute
wave: 1
depends_on: ["04-01"]
files_modified: ["src/memory/storage/sqlite_manager.py"]
autonomous: true
gap_closure: true
must_haves:
truths:
- "Context-aware search prioritizes current topic discussions"
artifacts:
- path: "src/memory/storage/sqlite_manager.py"
provides: "SQLite database operations and schema management"
contains: "get_conversation_metadata method"
key_links:
- from: "src/memory/retrieval/context_aware.py"
to: "src/memory/storage/sqlite_manager.py"
via: "conversation metadata for topic analysis"
pattern: "sqlite_manager\\.get_conversation_metadata"
---
<objective>
Complete SQLiteManager by adding missing get_conversation_metadata method to enable ContextAwareSearch topic analysis functionality.
Purpose: Close the metadata integration gap to enable context-aware search prioritization
Output: Complete SQLiteManager with metadata access for topic-based search enhancement
</objective>
<execution_context>
@~/.opencode/get-shit-done/workflows/execute-plan.md
@~/.opencode/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/phases/04-memory-context-management/04-CONTEXT.md
@.planning/phases/04-memory-context-management/04-memory-context-management-VERIFICATION.md
# Reference existing sqlite manager implementation
@src/memory/storage/sqlite_manager.py
# Reference context aware search that needs this method
@src/memory/retrieval/context_aware.py
</context>
<tasks>
<task type="auto">
<name>Task 1: Implement get_conversation_metadata method in SQLiteManager</name>
<files>src/memory/storage/sqlite_manager.py</files>
<action>
Add missing get_conversation_metadata method to SQLiteManager class to close the verification gap:
1. get_conversation_metadata method implementation:
- get_conversation_metadata(self, conversation_ids: List[str]) -> Dict[str, Dict]
- Retrieve comprehensive metadata for specified conversations
- Include topics, timestamps, message counts, user engagement metrics
- Return structured data suitable for topic analysis
2. Metadata fields to include:
- Conversation metadata: title, summary, created_at, updated_at
- Topic information: main_topics, topic_frequency, topic_sentiment
- Engagement metrics: message_count, user_message_ratio, response_times
- Temporal data: time_of_day patterns, day_of_week patterns
- Context clues: related_conversations, conversation_chain_position
3. Database queries for metadata:
- Query conversations table for basic metadata
- Aggregate message data for engagement metrics
- Join with message metadata if available
- Calculate topic statistics from existing topic fields
- Use existing indexes for efficient querying
4. Integration with existing SQLiteManager patterns:
- Follow same connection and cursor management
- Use existing error handling and transaction patterns
- Return data in formats compatible with existing methods
- Handle missing or incomplete data gracefully
5. Performance optimizations:
- Batch queries when multiple conversation_ids provided
- Use appropriate indexes for metadata fields
- Cache frequently accessed metadata
- Limit result size for large conversation sets
The method should support the needs identified in ContextAwareSearch for topic analysis. Check context_aware.py to understand the specific metadata requirements and expected return format.
</action>
<verify>python -c "from src.memory.storage.sqlite_manager import SQLiteManager; sm = SQLiteManager(); result = sm.get_conversation_metadata(['test_id']); print(f'get_conversation_metadata returned: {type(result)} with keys: {list(result.keys()) if result else \"None\"}')"</verify>
<done>SQLiteManager.get_conversation_metadata method provides comprehensive conversation metadata</done>
</task>
<task type="auto">
<name>Task 2: Integrate metadata access in ContextAwareSearch</name>
<files>src/memory/retrieval/context_aware.py</files>
<action>
Update ContextAwareSearch to use the new get_conversation_metadata method for proper topic analysis:
1. Import and use sqlite_manager.get_conversation_metadata:
- Update imports if needed to access sqlite_manager
- Replace any mock or placeholder metadata calls with real method
- Integrate metadata results into topic analysis algorithms
- Handle missing metadata gracefully
2. Topic analysis enhancement:
- Use real conversation metadata for topic relevance scoring
- Incorporate temporal patterns and engagement metrics
- Weight recent conversations appropriately in topic matching
- Use conversation chains and relationships for context
3. Context-aware search improvements:
- Enhance topic analysis with real metadata
- Improve current topic discussion prioritization
- Better handle multi-topic conversations
- More accurate context relevance scoring
4. Error handling and fallbacks:
- Handle cases where metadata is incomplete or missing
- Provide fallback to basic topic analysis
- Log metadata access issues for debugging
- Maintain search functionality even with metadata failures
5. Integration verification:
- Ensure ContextAwareSearch calls sqlite_manager.get_conversation_metadata
- Verify metadata is properly used in topic analysis
- Test with various conversation metadata scenarios
- Confirm search results improve with real metadata
Update the existing ContextAwareSearch implementation to leverage the new metadata capability while maintaining backward compatibility and handling edge cases appropriately.
</action>
<verify>python -c "from src.memory.retrieval.context_aware import ContextAwareSearch; cas = ContextAwareSearch(); print('ContextAwareSearch ready for metadata integration')"</verify>
<done>ContextAwareSearch integrates with SQLiteManager metadata for enhanced topic analysis</done>
</task>
</tasks>
<verification>
After completion, verify:
1. get_conversation_metadata method exists in SQLiteManager and is callable
2. Method returns comprehensive metadata suitable for topic analysis
3. ContextAwareSearch successfully calls and uses the metadata method
4. Topic analysis is enhanced with real conversation metadata
5. Context-aware search results are more accurate with metadata integration
6. No broken method calls or missing imports remain
</verification>
<success_criteria>
- Metadata integration gap is completely closed
- ContextAwareSearch can access conversation metadata for topic analysis
- Topic analysis is enhanced with real engagement and temporal data
- Current topic discussion prioritization works with real metadata
- Integration follows existing patterns and maintains performance
- All verification issues related to metadata access are resolved
</success_criteria>
<output>
After completion, create `.planning/phases/04-memory-context-management/04-07-SUMMARY.md`
</output>