Create utils/IngestManual.js to recursively walk a directory of markdown files and ingest them into the Semem system via HTTP POST requests to the MCP API server's /tell endpoint.
Following the codebase pattern from BookmarkIngest.js, use Node's built-in parseArgs instead of yargs:
--host(default:http://localhost:4101) - Semem MCP server host URL--dir(default:docs/manual) - Root directory to crawl for .md files--dry-run/-d- Preview files without ingesting--verbose/-v- Enable detailed logging--help/-h- Show help message
Follow BookmarkIngest.js structure but simplified for HTTP client:
- Minimal dependencies - no Config, no SimpleVerbsService (those are server-side)
- Pure HTTP client using native
fetchor minimal http library - Recursive directory walking using
fs.promisesandpath - Progress reporting with file count and success/error tracking
POST to /tell endpoint with payload:
{
"content": "<markdown file content>",
"type": "document",
"metadata": {
"source": "manual-ingestion",
"filename": "config.md",
"filepath": "docs/manual/config.md",
"format": "markdown",
"title": "<extracted from first # header>",
"ingestionDate": "<ISO timestamp>"
}
}- Recursively find all
.mdfiles in target directory - Read each file content
- Extract title from first
#header (if present) - Create metadata object with file information
- POST to
/tellendpoint - Track success/failure for reporting
- Continue on individual failures (don't abort entire batch)
- Network errors: Retry with exponential backoff (max 3 attempts)
- File read errors: Log and continue to next file
- API errors: Log response and continue
- Final summary report showing successes/failures
- Dry run mode: List files that would be ingested without making API calls
- Progress tracking: Show current file and completion percentage
- Batch processing: Process files sequentially to avoid overwhelming the API
- Summary report: Total files found, ingested, failed, with error details
-
Create file structure:
utils/IngestManual.jswith proper shebang and executable permissions -
Implement CLI class:
ManualIngestCLIwith methods:parseArguments()- Parse CLI argsshowHelp()- Display usage instructionsfindMarkdownFiles(dir)- Recursively find .md filesextractTitle(content)- Get title from markdowningestFile(filepath, host)- POST single file to APIexecuteIngestion(options)- Main ingestion logicrun()- Entry point
-
Implement file discovery: Recursive walk using
fs.promises.readdirwithrecursive: trueoption (Node 18+) or manual recursion -
Implement HTTP client: Use native
fetch(Node 18+) with proper error handling and retry logic -
Add progress reporting: Console output showing:
- Files discovered count
- Current file being processed
- Success/failure indicators
- Final summary statistics
-
Testing considerations: Can be tested against running MCP server on localhost:4101
- The user originally requested yargs, but the codebase pattern uses Node's built-in
parseArgsfromutil - Following existing patterns in
BookmarkIngest.jsfor consistency - Unlike
BookmarkIngest.js, this is a pure HTTP client and doesn't need server-side dependencies like Config or SimpleVerbsService - The MCP API server at
/tellendpoint will handle chunking, embedding generation, and storage - The script's responsibility is limited to file discovery and HTTP POSTing
- Test with dry-run mode first to verify file discovery
- Test with small subset using --limit flag (if implemented)
- Test error handling with invalid host
- Test with full docs/manual directory
- Verify ingestion by querying the system with ask verb
- Batch processing with configurable concurrency
- Resume capability (skip already ingested files)
- Watch mode for continuous ingestion
- Integration with git hooks for automatic ingestion on commits
- Filtering by file modification date