Skip to content

Add Async $text Operator for LLM-Integrated Data Generation with Ollama#42

Open
omkarkhair wants to merge 20 commits into
rueckstiess:masterfrom
omkarkhair:master
Open

Add Async $text Operator for LLM-Integrated Data Generation with Ollama#42
omkarkhair wants to merge 20 commits into
rueckstiess:masterfrom
omkarkhair:master

Conversation

@omkarkhair

@omkarkhair omkarkhair commented Jul 28, 2024

Copy link
Copy Markdown

This PR introduces a new $text operator to Mongo's mgenerate tool, allowing integration with Large Language Models (LLMs) using Ollama API to generate contextually relevant text data based on user-defined prompts. This enhancement significantly improves the tool's capability to create more specific and meaningful dummy data, addressing various application use cases, such as:

  • Application-Specific Data: Generate tailored data for specific domains (e.g., healthcare job titles).
  • Long Text Generation: Produce coherent, context-appropriate long text (e.g., product reviews).
  • Regional Contextualization: Generate data with regional relevance (e.g., Indian names).

Key Changes:

  • Added a new $text operator in mgenerate with Ollama integration.
  • Integrated LLM model via Ollama
  • Converted mgenerate into an asynchronous library to support LLM integration.
  • Updated documentation to include usage examples and details for the new $text operator.

Example Usage:

{
    "name": "$name",
    "Role": {
        "$text": {
            "prompt": "Rare Designation or job title found in Healthcare",
            "maxWordCount": "4"
        }
    },
    "lastLogin": "$now"
}

Example Output (model: mistral-nemo):

{
    "name": "Virginia Blair",
    "Role": "Medical Assistant",
    "lastLogin": {
        "$date": "2024-07-28T12:53:00.267Z"
    }
}

@omkarkhair omkarkhair marked this pull request as ready for review August 1, 2024 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant