This directory contains comprehensive documentation for the Let's Talk system.
- Architecture Documentation - Complete system architecture overview
- First-Time Startup Process - Automatic first-time detection and setup
- First-Time Startup Quick Reference - Quick commands and troubleshooting
- Migration Guide - Migrate from legacy to new structure
- Pipeline Usage Guide - Comprehensive guide for using the pipeline
- Pipeline Quick Reference - Quick commands and common scenarios
- Pipeline Scheduling API - FastAPI scheduling endpoints
- Scheduler Documentation - Complete scheduler documentation
- Pipeline Configuration Template - Environment variables template
- Incremental Indexing Final Report - Detailed technical implementation report
- Scheduler Implementation Report - Scheduler implementation details
# Copy and customize configuration
cp docs/PIPELINE_CONFIG_TEMPLATE.env .env
# Edit .env with your settings
nano .env
# Create initial vector store
cd py-src && uv run python -m lets_talk.core.pipeline.engine --force-recreate# Start the FastAPI server
cd py-src && uv run python lets_talk/main.py
# Or use the web frontend
cd frontend && pnpm dev# Update with incremental indexing
cd py-src && uv run python -m lets_talk.core.pipeline.engine --auto-incremental
# Use scheduler for automation
cd py-src && uv run python -m lets_talk.core.scheduler.cli start --daemon# Use programmatic API
cd py-src && uv run python -c "
from lets_talk.core.pipeline.engine import run_pipeline
from lets_talk.agents import create_rag_agent
from lets_talk.shared.config import load_configuration_with_prompts
# Run pipeline
result = run_pipeline(data_dir='data/', force_recreate=False)
# Create agents
config = load_configuration_with_prompts()
agent = create_rag_agent(config)
"# Health check
cd py-src && uv run python -m lets_talk.core.pipeline.engine --health-check-only
# See what would be processed
cd py-src && uv run python -m lets_talk.core.pipeline.engine --dry-run --incremental
# Check imports
cd py-src && uv run python -c "import lets_talk; print('All imports working')"docs/
├── README.md # This file
├── PIPELINE_USAGE_GUIDE.md # Complete usage guide
├── PIPELINE_QUICK_REFERENCE.md # Quick commands reference
├── PIPELINE_CONFIG_TEMPLATE.env # Configuration template
└── INCREMENTAL_INDEXING_FINAL_REPORT.md # Technical implementation details
- Automatic detection of new, modified, and deleted documents
- Configurable fallback thresholds
- Metadata tracking for efficient updates
- Environment variable-based configuration
- Customizable output files and directories
- Performance tuning options
- Comprehensive system health checks
- Vector store validation
- Metadata integrity verification
- Batch processing for large collections
- Concurrent operations
- Memory-efficient chunking
- Dry-run mode for testing changes
- Detailed logging configuration
- Common problem solutions
- For usage questions: See Pipeline Usage Guide
- For quick commands: See Pipeline Quick Reference
- For configuration: Use Pipeline Configuration Template
- For technical details: See Incremental Indexing Report
| Use Case | Documentation | Command |
|---|---|---|
| First-time setup | Usage Guide | --force-recreate |
| Regular updates | Quick Reference | --auto-incremental |
| Debugging issues | Usage Guide | --health-check-only |
| Configuration | Config Template | Edit .env file |
| Performance tuning | Usage Guide | Adjust environment variables |
- ✅ Parameterized Configuration: All hardcoded values are now configurable via environment variables
- ✅ Comprehensive Documentation: Complete usage guides and quick reference
- ✅ Configuration Templates: Ready-to-use environment variable templates
- ✅ Troubleshooting Guides: Common issues and solutions documented
For the latest updates and source code, see the pipeline implementation and configuration.