Send in parameters with query

[mstensmo]

Hi guys, I'm trying to find a way to send in parameters such as temperature and max_new_tokens when you send a query to a langcorn server. Something like this:

{"question": "tell me a story about dogs.",
"parameters": {"temperature": 1}
}

There is no complaint when you send in things so you can add it.

Is this possible and if so, how do you do it?

Thanks

[msoedov]

Hi @mstensmo! Thx for your suggestion! It's not supported in the current version but I can add it similarly to x-llm-api-key HTTP header in the new release.

x-llm-temperature: 0.7
x-max-tokens: 256
x-model-name: '...'

[msoedov]

I just released this change to pypi

[mstensmo]

Hi @msoedov, wow that's quick. Was going to say thank for getting back to me. I'll check it out.

By the way, how do you send in those arguments to a chain in langchain. Can you do that to chain.run?

Thanks a lot!

[msoedov]

Those arguments can be set to LLM during initialization.

class BaseOpenAI(BaseLLM):
    """Base OpenAI large language model class."""

    @property
    def lc_secrets(self) -> Dict[str, str]:
        return {"openai_api_key": "OPENAI_API_KEY"}

    @property
    def lc_serializable(self) -> bool:
        return True

    client: Any  #: :meta private:
    model_name: str = Field("text-davinci-003", alias="model")
    """Model name to use."""
    temperature: float = 0.7
    """What sampling temperature to use."""
    max_tokens: int = 256
    """The maximum number of tokens to generate in the completion.
    -1 returns as many tokens as possible given the prompt and
    the models maximal context size."""
    top_p: float = 1
    """Total probability mass of tokens to consider at each step."""
    

[mstensmo]

Yes, you set them when you initialize the LLM, if I understand you right. I'm trying to avoid that and want to send in new parameters when I run the llm. The reason is that there's overhead when you create the llm, especially if you have to load it locally.

For example:
model = AutomodelForCausalLM("bigscience/bloomz-560m")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, temperature=0, max_new_tokens=256)
llm = HuggingfacePipeline(pipeline=pipe)
then you create a chain out of this
prompt = PromptTemplate(...)
chain = LLMChain(prompt=prompt, llm=llm)
then use that chain with LangCorn to set up a server

So, I don't want to reload the llm each time. Did I understand you right?