🚀 Describe the new functionality needed
This week I came across headroom, a proxy that uses several techniques to reduce the number of tokens sent to LLMs (see How compression works) and also reduce the number of output tokens generated by them (see Output Token Reduction).
Headroom implements a proxy, similar to ogxserver, that supports multiple providers.
I was wondering whether ogx could implement some of these capabilities within the inference providers, or alternatively define a new Compressor API with inline and remote compressors that can be plugged into the inference providers.
💡 Why is this needed? What if we don't build it?
Agents are increasing token usage compared to traditional chatbots, and providers will likely increase token prices in the future. The popularity of headroom seems to be evidence that these kinds of features are needed, or at least that there is significant interest in them.
Other thoughts
No response
🚀 Describe the new functionality needed
This week I came across headroom, a proxy that uses several techniques to reduce the number of tokens sent to LLMs (see How compression works) and also reduce the number of output tokens generated by them (see Output Token Reduction).
Headroom implements a proxy, similar to ogxserver, that supports multiple providers.
I was wondering whether ogx could implement some of these capabilities within the inference providers, or alternatively define a new Compressor API with inline and remote compressors that can be plugged into the inference providers.
💡 Why is this needed? What if we don't build it?
Agents are increasing token usage compared to traditional chatbots, and providers will likely increase token prices in the future. The popularity of headroom seems to be evidence that these kinds of features are needed, or at least that there is significant interest in them.
Other thoughts
No response