docs(04): create gap closure plans for memory and context management
Phase 04: Memory & Context Management - 3 gap closure plans to address verification issues - 04-05: Personality learning integration (PersonalityAdaptation, MemoryManager integration, src/personality.py) - 04-06: Vector Store missing methods (search_by_keyword, store_embeddings) - 04-07: Context-aware search metadata integration (get_conversation_metadata) - All gaps from verification report addressed - Updated roadmap to reflect 7 total plans
This commit is contained in:
161
.planning/phases/04-memory-context-management/04-06-PLAN.md
Normal file
161
.planning/phases/04-memory-context-management/04-06-PLAN.md
Normal file
@@ -0,0 +1,161 @@
|
||||
---
|
||||
phase: 04-memory-context-management
|
||||
plan: 06
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: ["04-01"]
|
||||
files_modified: ["src/memory/storage/vector_store.py"]
|
||||
autonomous: true
|
||||
gap_closure: true
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "User can search conversations by semantic meaning"
|
||||
artifacts:
|
||||
- path: "src/memory/storage/vector_store.py"
|
||||
provides: "Vector storage and retrieval with sqlite-vec"
|
||||
contains: "search_by_keyword method"
|
||||
contains: "store_embeddings method"
|
||||
key_links:
|
||||
- from: "src/memory/retrieval/semantic_search.py"
|
||||
to: "src/memory/storage/vector_store.py"
|
||||
via: "vector similarity search operations"
|
||||
pattern: "vector_store\\.search_by_keyword"
|
||||
- from: "src/memory/retrieval/semantic_search.py"
|
||||
to: "src/memory/storage/vector_store.py"
|
||||
via: "embedding storage operations"
|
||||
pattern: "vector_store\\.store_embeddings"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Complete VectorStore implementation by adding missing search_by_keyword and store_embeddings methods that are called by SemanticSearch but not implemented.
|
||||
|
||||
Purpose: Close the vector store methods gap to enable full semantic search functionality
|
||||
Output: Complete VectorStore with all required methods for semantic search operations
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.opencode/get-shit-done/workflows/execute-plan.md
|
||||
@~/.opencode/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/phases/04-memory-context-management/04-CONTEXT.md
|
||||
@.planning/phases/04-memory-context-management/04-memory-context-management-VERIFICATION.md
|
||||
|
||||
# Reference existing vector store implementation
|
||||
@src/memory/storage/vector_store.py
|
||||
|
||||
# Reference semantic search that calls these methods
|
||||
@src/memory/retrieval/semantic_search.py
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Implement search_by_keyword method in VectorStore</name>
|
||||
<files>src/memory/storage/vector_store.py</files>
|
||||
<action>
|
||||
Add missing search_by_keyword method to VectorStore class to close the verification gap:
|
||||
|
||||
1. search_by_keyword method implementation:
|
||||
- search_by_keyword(self, query: str, limit: int = 10) -> List[Dict]
|
||||
- Perform keyword-based search on message content using FTS if available
|
||||
- Fall back to LIKE queries if FTS not enabled
|
||||
- Return results in same format as vector search for consistency
|
||||
|
||||
2. Keyword search implementation:
|
||||
- Use SQLite FTS (Full-Text Search) if virtual tables exist
|
||||
- Query message_content and conversation_summary fields
|
||||
- Support multiple keywords with AND/OR logic
|
||||
- Rank results by keyword frequency and position
|
||||
|
||||
3. Integration with existing vector operations:
|
||||
- Use same database connection as existing methods
|
||||
- Follow existing error handling patterns
|
||||
- Return results compatible with hybrid_search in SemanticSearch
|
||||
- Include message_id, conversation_id, content, and relevance score
|
||||
|
||||
4. Performance optimizations:
|
||||
- Add appropriate indexes for keyword search if missing
|
||||
- Use query parameters to prevent SQL injection
|
||||
- Limit result sets for performance
|
||||
- Cache frequent keyword queries if beneficial
|
||||
|
||||
5. Method signature matching:
|
||||
- Match the expected signature from semantic_search.py line 248
|
||||
- Return format: List[Dict] with message_id, conversation_id, content, score
|
||||
- Handle edge cases: empty queries, no results, database errors
|
||||
|
||||
The method should be called by SemanticSearch.hybrid_search at line 248. Verify the exact signature and return format by checking semantic_search.py before implementation.
|
||||
</action>
|
||||
<verify>python -c "from src.memory.storage.vector_store import VectorStore; vs = VectorStore(); result = vs.search_by_keyword('test', limit=5); print(f'search_by_keyword returned {len(result)} results')"</verify>
|
||||
<done>VectorStore.search_by_keyword method provides keyword-based search functionality</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Implement store_embeddings method in VectorStore</name>
|
||||
<files>src/memory/storage/vector_store.py</files>
|
||||
<action>
|
||||
Add missing store_embeddings method to VectorStore class to close the verification gap:
|
||||
|
||||
1. store_embeddings method implementation:
|
||||
- store_embeddings(self, embeddings: List[Tuple[str, List[float]]]) -> bool
|
||||
- Batch store multiple embeddings efficiently
|
||||
- Handle conversation_id and message_id associations
|
||||
- Return success/failure status
|
||||
|
||||
2. Embedding storage implementation:
|
||||
- Use existing vec_entries virtual table from current implementation
|
||||
- Insert embeddings with proper rowid mapping to messages
|
||||
- Support batch inserts for performance
|
||||
- Handle embedding dimension validation
|
||||
|
||||
3. Integration with existing storage patterns:
|
||||
- Follow same database connection patterns as other methods
|
||||
- Use existing error handling and transaction management
|
||||
- Coordinate with sqlite_manager for message metadata
|
||||
- Maintain consistency with existing vector storage
|
||||
|
||||
4. Method signature compatibility:
|
||||
- Match expected signature from semantic_search.py line 363
|
||||
- Accept list of (id, embedding) tuples
|
||||
- Return boolean success indicator
|
||||
- Handle partial failures gracefully
|
||||
|
||||
5. Performance and reliability:
|
||||
- Use transactions for batch operations
|
||||
- Validate embedding dimensions before insertion
|
||||
- Handle database constraint violations
|
||||
- Provide detailed error logging for debugging
|
||||
|
||||
The method should be called by SemanticSearch at line 363. Verify the exact signature and expected behavior by checking semantic_search.py before implementation. Ensure compatibility with the existing vec_entries table structure and sqlite-vec extension usage.
|
||||
</action>
|
||||
<verify>python -c "from src.memory.storage.vector_store import VectorStore; import numpy as np; vs = VectorStore(); test_emb = [('test_id', np.random.rand(1536).tolist())]; result = vs.store_embeddings(test_emb); print(f'store_embeddings returned: {result}')"</verify>
|
||||
<done>VectorStore.store_embeddings method provides batch embedding storage functionality</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
After completion, verify:
|
||||
1. search_by_keyword method exists and is callable from SemanticSearch
|
||||
2. store_embeddings method exists and is callable from SemanticSearch
|
||||
3. Both methods follow the exact signatures expected by semantic_search.py
|
||||
4. Methods integrate properly with existing VectorStore database operations
|
||||
5. SemanticSearch.hybrid_search can now call these methods without errors
|
||||
6. Keyword search returns properly formatted results compatible with vector search
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- VectorStore missing methods gap is completely closed
|
||||
- SemanticSearch can perform hybrid search combining keyword and vector search
|
||||
- Methods follow existing VectorStore patterns and error handling
|
||||
- Database operations are efficient and properly transactional
|
||||
- Integration with semantic search is seamless and functional
|
||||
- All anti-patterns related to missing method calls are resolved
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/04-memory-context-management/04-06-SUMMARY.md`
|
||||
</output>
|
||||
Reference in New Issue
Block a user