Commit Graph

85 Commits

Author SHA1 Message Date
Mai Development
26543d0402 docs(04-06): complete VectorStore gap closure plan
Tasks completed: 2/2
- Implemented search_by_keyword method with FTS/LIKE hybrid search
- Implemented store_embeddings method with transactional batch operations
- Fixed VectorStore schema for sqlite-vec extension compatibility
- Resolved all missing method calls from SemanticSearch.hybrid_search

SUMMARY: .planning/phases/04-memory-context-management/04-06-SUMMARY.md
Updated STATE.md to reflect Phase 4 completion
2026-01-28 13:33:13 -05:00
Mai Development
cc24b54b7c feat(04-06): implement store_embeddings method in VectorStore
- Added store_embeddings method for batch embedding storage
- Supports transactional batch operations with error handling
- Validates embedding dimensions before storage
- Fixed schema compatibility with sqlite-vec extension using separate metadata tables
- Handles partial failures gracefully and reports success/failure status
- Integrates with existing VectorStore patterns and error handling
- Fixed row handling issues in keyword search methods
2026-01-28 13:28:45 -05:00
Mai Development
0bf62661b5 feat(04-06): implement search_by_keyword method in VectorStore
- Added search_by_keyword method for keyword-based search functionality
- Supports FTS (Full-Text Search) when available, falls back to LIKE queries
- Includes helper methods _check_fts_available, _search_with_fts, _search_with_like
- Fixed schema to separate vector and metadata tables for sqlite-vec compatibility
- Returns properly formatted results compatible with SemanticSearch.hybrid_search
- Handles multiple keywords with AND/OR logic and relevance scoring
2026-01-28 13:20:54 -05:00
Mai Development
8969d382a9 docs(04-07): complete metadata integration plan
Tasks completed: 2/2
- Implemented get_conversation_metadata method in SQLiteManager
- Integrated metadata access in ContextAwareSearch

SUMMARY: .planning/phases/04-memory-context-management/04-07-SUMMARY.md
2026-01-28 13:17:29 -05:00
Mai Development
346a013a6f feat(04-07): integrate SQLiteManager metadata in ContextAwareSearch
- Enhanced _calculate_topic_relevance with conversation metadata support
- Added metadata-based topic boosts for primary topics and engagement
- Incorporated temporal patterns for recent activity preference
- Updated prioritize_by_topic to use get_conversation_metadata
- Enhanced get_topic_summary with comprehensive metadata insights
- Added related conversation context and engagement metrics
- Maintained backward compatibility with existing functionality
2026-01-28 13:15:17 -05:00
Mai Development
1e4ceec820 feat(04-07): implement get_conversation_metadata and get_recent_messages methods
- Added get_conversation_metadata method for comprehensive conversation metadata
- Added get_recent_messages method for retrieving recent messages by conversation
- Methods support topic analysis and engagement metrics
- Includes temporal patterns, context clues, and relationship analysis
- Follows existing SQLiteManager patterns and error handling
2026-01-28 13:12:59 -05:00
Mai Development
47e4864049 docs(04): create gap closure plans for memory and context management
Phase 04: Memory & Context Management
- 3 gap closure plans to address verification issues
- 04-05: Personality learning integration (PersonalityAdaptation, MemoryManager integration, src/personality.py)
- 04-06: Vector Store missing methods (search_by_keyword, store_embeddings)
- 04-07: Context-aware search metadata integration (get_conversation_metadata)
- All gaps from verification report addressed
- Updated roadmap to reflect 7 total plans
2026-01-28 12:08:47 -05:00
Mai Development
7cd12abe0c feat(04-04): create pattern extraction system
- Created src/memory/personality/__init__.py module structure
- Implemented PatternExtractor class with multi-dimensional analysis:
  - Topics: Track frequently discussed subjects and user interests
  - Sentiment: Analyze emotional tone and sentiment patterns
  - Interaction: Response times, question asking, information sharing
  - Temporal: Communication style by time of day/week
  - Response styles: Formality level, verbosity, emoji/humor use
- Pattern extraction methods for all dimensions with confidence scoring
- Lightweight analysis techniques to avoid computational overhead
- Pattern validation with stability tracking and outlier detection
2026-01-28 00:33:38 -05:00
Mai Development
a8b7a35baa docs(04-03): complete progressive compression and JSON archival plan
Tasks completed: 2/2
- Progressive compression engine with 4-tier age-based levels
- JSON archival system with gzip compression and organized structure
- Smart retention policies with importance-based scoring
- MemoryManager integration with unified archival interface

SUMMARY: .planning/phases/04-memory-context-management/04-03-SUMMARY.md
2026-01-28 00:00:12 -05:00
Mai Development
8c58b1d070 feat(04-03): create JSON archival and smart retention systems
- Added ArchivalManager for JSON export/import with gzip compression
- Implemented organized directory structure by year/month
- Added batch archival operations and restore functionality
- Created RetentionPolicy with importance-based scoring
- Smart retention considers engagement, topics, user-marked importance
- MemoryManager integrates compression and archival automatically
- Added automatic compression triggering and archival scheduling
- Comprehensive archival statistics and retention recommendations
- Support for backup integration and restore verification
2026-01-27 23:56:49 -05:00
Mai Development
017df5466d feat(04-03): implement progressive compression engine
- Added CompressionEngine class with 4-tier age-based compression
- 7 days: Full content (no compression)
- 30 days: Key points extraction (70% retention)
- 90 days: Brief summary (40% retention)
- 365+ days: Metadata only
- Hybrid extractive-abstractive summarization with fallbacks
- Compression quality metrics and validation
- Support for missing dependencies (NLTK/transformers)
- Added transformers and nltk to requirements.txt
2026-01-27 23:42:20 -05:00
Mai Development
bb7205223d docs(04-02): complete memory retrieval system plan
Tasks completed: 2/2
- Semantic search with sentence-transformers embeddings
- Context-aware search with topic-based prioritization
- Timeline search with date filtering and temporal proximity
- Enhanced MemoryManager with unified search interface

SUMMARY: .planning/phases/04-memory-context-management/04-02-SUMMARY.md
Updated STATE.md progress to 2/4 in Phase 4
2026-01-27 23:28:42 -05:00
Mai Development
dd4715643c feat(04-02): implement context-aware and timeline search capabilities
- Completed Task 2: Context-aware and timeline search
- ContextAwareSearch class with topic classification and result prioritization
- TimelineSearch class with date-range filtering and temporal proximity
- Enhanced MemoryManager with unified search interface
- Supports semantic, keyword, context-aware, timeline, and hybrid search
- Added search result dataclasses with relevance scoring
- Integrated all search strategies into MemoryManager.search() method

All search modes operational:
- Semantic search with sentence-transformers embeddings
- Context-aware search with topic-based prioritization
- Timeline search with date filtering and recency weighting
- Hybrid search combining multiple strategies

Search results include conversation context and relevance scoring as required.
2026-01-27 23:25:04 -05:00
Mai Development
b9aba97086 feat(04-02): create semantic search with embedding-based retrieval
- Added sentence-transformers to requirements.txt for semantic embeddings
- Created src/memory/retrieval/ module with search capabilities
- Implemented SemanticSearch class with embedding generation and vector similarity
- Added SearchResult and SearchQuery dataclasses for structured search results
- Included hybrid search combining semantic and keyword matching
- Added conversation indexing for semantic search
- Followed lazy loading pattern for embedding model performance

Files created:
- src/memory/retrieval/__init__.py
- src/memory/retrieval/search_types.py
- src/memory/retrieval/semantic_search.py
- Updated src/memory/__init__.py with enhanced MemoryManager

Note: sentence-transformers installation requires proper venv setup in production
2026-01-27 23:22:50 -05:00
Mai Development
bdba17773c feat(04-01): create memory module structure and SQLite manager
- Created src/memory module with MemoryManager stub
- Created src/memory/storage subpackage
- Implemented SQLiteManager with connection management and thread safety
- Database schema supports conversations, messages, and metadata
- Includes proper indexing and error handling

Schema:
- conversations table: id, title, timestamps, metadata, session stats
- messages table: id, conversation_id, role, content, importance, embedding_ref
- Foreign key constraints and performance indexes
- Thread-local connections with WAL mode for concurrency
2026-01-27 22:50:02 -05:00
Mai Development
61db47e8d6 docs(04): create phase plan
Phase 04: Memory & Context Management
- 4 plan(s) in 3 wave(s)
- 2 parallel, 2 sequential
- Ready for execution
2026-01-27 22:04:42 -05:00
Mai Development
9cdb1e7f6c docs(04): create phase plan
Phase 04: Memory & Context Management
- 4 plan(s) in 3 wave(s)
- 2 parallel, 2 sequential
- Ready for execution
2026-01-27 21:53:07 -05:00
Mai Development
c09ea8c8f2 docs(04): research phase 4 memory & context management domain
Phase 04: Memory & Context Management
- Standard stack identified: SQLite + sqlite-vec + sentence-transformers
- Architecture patterns documented: hybrid storage, progressive compression, vector search
- Pitfalls cataloged: embedding drift, memory bloat, personality overfitting
- Code examples provided from official sources
2026-01-27 20:12:40 -05:00
Mai Development
3e88d33bd3 docs(04): capture phase context
Phase 04: memory-context-management
- Implementation decisions documented
- Hybrid storage strategy with SQLite + JSON archives
- Progressive compression and smart retention policies
- Multi-dimensional pattern learning approach
- Phase boundary established
2026-01-27 19:59:52 -05:00
Mai Development
27fa6b654f docs(03): complete resource management phase
Phase 03: resource-management
- Enhanced GPU detection with pynvml support
- Hardware tier detection and management system
- Proactive scaling with hybrid monitoring
- Personality-driven resource communication
- All phase goals verified
2026-01-27 19:17:14 -05:00
Mai Development
9b4ce96ff5 removed discord sync due to errors 2026-01-27 19:11:38 -05:00
Mai Development
5dda3d2f55 fix(02): orchestrator corrections
Some checks are pending
Discord Webhook / git (push) Waiting to run
Add missing Phase 2 Plan 2 SUMMARY and Discord integration artifacts
2026-01-27 19:10:31 -05:00
Mai Development
087974fa88 docs(03-04): complete personality-driven resource communication plan
Some checks failed
Discord Webhook / git (push) Has been cancelled
Tasks completed: 2/2
- Implemented ResourcePersonality with dere-tsun gremlin persona
- Integrated personality-aware model switching with degradation notifications

SUMMARY: .planning/phases/03-resource-management/03-04-SUMMARY.md
2026-01-27 19:07:41 -05:00
Mai Development
1c9764526f feat(03-04): integrate personality with model management
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Added ResourcePersonality import and initialization to ModelManager
- Created personality_aware_model_switch() method for graceful degradation notifications
- Only notifies users about capability downgrades, not upgrades (per requirements)
- Includes optional technical tips for resource optimization
- Updated proactive scaling callbacks to use personality-aware switching
- Enhanced failure handling with personality-driven resource requests
- Added _is_capability_downgrade() helper for capability comparison
2026-01-27 19:04:19 -05:00
Mai Development
dd3a75f0f0 feat(03-04): implement ResourcePersonality with dere-tsun gremlin persona
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Created ResourcePersonality class with Drowsy Dere-Tsun Onee-san Hex-Mentor Gremlin personality
- Includes mood system with sleepy, grumpy, helpful, gremlin, and mentor states
- Personality-specific vocabularies for different emotional responses
- Optional technical tips with hexadecimal/coding references
- generate_resource_message() for contextual resource communications
- Support for resource requests, degradation notices, system status, and scaling recommendations
2026-01-27 18:57:13 -05:00
Mai Development
54f0decb40 docs(03-03): complete proactive scaling plan
Some checks failed
Discord Webhook / git (push) Has been cancelled
Tasks completed: 2/2
- Implemented ProactiveScaler class with hybrid monitoring
- Integrated proactive scaling into ModelManager

Proactive scaling system with hybrid monitoring, graceful degradation cascades, and intelligent stabilization periods.

SUMMARY: .planning/phases/03-resource-management/03-03-SUMMARY.md
2026-01-27 18:50:18 -05:00
Mai Development
53b8ef7c1b feat(03-03): integrate proactive scaling into ModelManager
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Added ProactiveScaler integration with HardwareTierDetector
- Implemented pre-flight resource checks before model inference
- Enhanced model selection with scaling recommendations
- Added graceful degradation handling for resource constraints
- Integrated performance metrics tracking for scaling decisions
- Added proactive upgrade execution with stabilization periods
- Enhanced status reporting with scaling information
- Maintained silent switching behavior per Phase 1 decisions
2026-01-27 18:47:10 -05:00
Mai Development
4d7749da7b feat(03-03): implement ProactiveScaler class with hybrid monitoring
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Created ProactiveScaler class for proactive resource management
- Implemented continuous background monitoring with configurable intervals
- Added pre-flight resource checks before operations
- Implemented graceful degradation cascades with stabilization periods
- Added trend analysis for predictive scaling decisions
- Included hysteresis to prevent model switching thrashing
- Provided callbacks for integration with ModelManager
- Thread-safe implementation with proper shutdown handling
2026-01-27 18:40:58 -05:00
Mai Development
4c3cab9dd9 docs(03-02): complete hardware tier detection plan
Some checks failed
Discord Webhook / git (push) Has been cancelled
Tasks completed: 3/3
- Resource module structure with proper exports
- Configurable hardware tier definitions in YAML
- HardwareTierDetector class with classification logic

SUMMARY: .planning/phases/03-resource-management/03-02-SUMMARY.md
2026-01-27 18:35:41 -05:00
Mai Development
8857ced92a feat(03-02): implement HardwareTierDetector class
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Created comprehensive hardware tier detection system
- Loads configurable tier definitions from YAML
- Classifies systems based on RAM, CPU cores, and GPU capabilities
- Provides model recommendations and performance characteristics
- Includes caching for performance and error handling
- Integrates with ResourceMonitor for real-time data
2026-01-27 18:32:07 -05:00
Mai Development
0b4c270632 feat(03-02): create configurable hardware tier definitions
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Added comprehensive tier definitions for low_end, mid_range, high_end
- Configurable thresholds for RAM, CPU cores, GPU requirements
- Model size recommendations per tier (1B-70B parameter range)
- Performance characteristics and scaling thresholds
- Global settings for model selection and scaling behavior
2026-01-27 18:30:42 -05:00
Mai Development
5d93e9715f feat(03-02): create resource module structure
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Created src/resource/__init__.py with module docstring
- Exported HardwareTierDetector (to be implemented)
- Established resource management module foundation
2026-01-27 18:29:38 -05:00
Mai Development
a1db08c72c docs(03-01): complete enhanced GPU detection plan
Some checks failed
Discord Webhook / git (push) Has been cancelled
Tasks completed: 2/2
- Added pynvml>=11.0.0 dependency for NVIDIA GPU monitoring
- Enhanced ResourceMonitor with pynvml GPU detection and graceful fallbacks
- Optimized performance with caching and failure tracking (~50ms per call)

SUMMARY: .planning/phases/03-resource-management/03-01-SUMMARY.md
2026-01-27 18:25:01 -05:00
Mai Development
0ad2b393a5 perf(03-01): optimize ResourceMonitor performance
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Added caching for GPU info to avoid repeated pynvml initialization
- Added pynvml failure tracking to skip repeated failed attempts
- Optimized CPU measurement interval from 1.0s to 0.05s
- Reduced monitoring overhead from ~1000ms to ~50ms per call
- Maintained accuracy while significantly improving performance
2026-01-27 18:21:01 -05:00
Mai Development
8cf9e9ab04 feat(03-01): enhance ResourceMonitor with pynvml GPU detection
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Added pynvml import with graceful fallback handling
- Enhanced _get_gpu_info() method using pynvml for NVIDIA GPUs
- Added detailed GPU metrics: total/used/free VRAM, utilization, temperature
- Updated get_current_resources() to include comprehensive GPU info
- Maintained backward compatibility with existing gpu_vram_gb field
- Added gpu-tracker fallback for AMD/Intel GPUs
- Proper error handling for pynvml initialization failures
- Ensured pynvmlShutdown() always called in finally-style logic
2026-01-27 18:17:12 -05:00
Mai Development
e2023754eb feat(03-01): add pynvml dependency for GPU monitoring
Some checks failed
Discord Webhook / git (push) Has been cancelled
- Added pynvml>=11.0.0 to main dependencies
- Enables NVIDIA GPU VRAM monitoring capabilities
- Required for enhanced resource detection in Phase 3
2026-01-27 18:14:23 -05:00
Mai Development
1e071398ff docs(03): create phase plan
Some checks failed
Discord Webhook / git (push) Has been cancelled
Phase 3: Resource Management
- 4 plan(s) in 2 wave(s)
- 2 parallel, 2 sequential
- Ready for execution
2026-01-27 17:58:09 -05:00
Mai Development
a37b61acce docs(03): research phase domain
Some checks failed
Discord Webhook / git (push) Has been cancelled
Phase 03: Resource Management
- Standard stack identified: psutil, pynvml, gpu-tracker
- Architecture patterns documented: hybrid monitoring, tier-based management, graceful degradation
- Pitfalls catalogued: GPU detection, aggressive switching, memory leaks, over-technical communication
- Don't-hand-roll items listed for custom implementations
- Code examples provided with official source references
2026-01-27 17:52:47 -05:00
Mai Development
2d24f8f93f docs(03): capture phase context
Some checks failed
Discord Webhook / git (push) Has been cancelled
Phase 03: resource-management
- Implementation decisions documented
- Resource threshold strategy with dynamic adjustment
- Efficiency-first model selection behavior
- Bottleneck detection with hybrid approach
- Personality-driven user communication
- Drowsy Dere-Tsun Onee-san Hex-Mentor Gremlin persona
2026-01-27 17:46:33 -05:00
Mai Development
f815f4fecf docs(02): complete phase execution
Some checks failed
Discord Webhook / git (push) Has been cancelled
Phase 02: Safety & Sandboxing
- 4 plans executed across 3 waves
- Security assessment, sandbox execution, audit logging, integration
- Verification passed - all must-haves verified
- Ready for Phase 3: Resource Management
2026-01-27 16:12:18 -05:00
Mai Development
1413433d89 docs(02-04): Add execution summary
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 16:06:39 -05:00
Mai Development
543fe75150 feat(02-04): Create integration tests for safety system
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 16:05:46 -05:00
Mai Development
26a77e612d feat(02-04): Implement safety API interface
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:56:50 -05:00
Mai Development
73155af6be feat(02-04): Create safety coordinator
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:55:07 -05:00
Mai Development
df5ca04c5a docs(02-03): Create comprehensive execution summary for tamper-proof audit logging implementation
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:48:49 -05:00
Mai Development
387c39d90f feat(02-03): Configure comprehensive audit policies with retention and hash chain settings
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:47:47 -05:00
Mai Development
241b9d2dbb feat(02-03): Implement audit logging interface with comprehensive security event methods
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:44:28 -05:00
Mai Development
7ab8e7a983 feat(02-03): Create tamper-proof audit logger with SHA-256 hash chains
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:42:03 -05:00
Mai Development
8b4e31bd47 feat(02-02): Configure sandbox policies
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:37:22 -05:00
Mai Development
9b79107fb3 feat(02-02): Implement sandbox execution interface
Some checks failed
Discord Webhook / git (push) Has been cancelled
2026-01-27 15:36:58 -05:00