With a simple "Hi" message sent as the very first message and without any other active chats, the token usage shows 9.2K tokens used.
Also, when set the context to 4K from the existing 65536, I get an error
This is my entire conversation history:
_hi
Hello! How can I help you today?
why the token usage shows 9.2k tokens for a simple hi message ?
The 9.2k token usage for a simple "hi" message is likely due to:
Context overhead: The system includes metadata, conversation history, and internal processing that counts toward token usage, even for simple greetings.
Model-specific counting: Different tokenization methods count tokens differently. Some models count every character or word, while others use subword tokenization (like BPE - Byte Pair Encoding).
System prompts and instructions: The system prompt and any default instructions you've set up are included in the token count.
API overhead: Some APIs add overhead for authentication, routing, and other infrastructure costs.
Batch processing: If you're using batch inference or have context from previous conversations, those tokens are included.
The exact breakdown would depend on your specific model configuration and API settings. Would you like me to help you optimize token usage or understand your specific tokenization setup?_
For this history, the total token usage is 18.1K
With a simple "Hi" message sent as the very first message and without any other active chats, the token usage shows 9.2K tokens used.
Also, when set the context to 4K from the existing 65536, I get an error
This is my entire conversation history:
_hi
Hello! How can I help you today?
why the token usage shows 9.2k tokens for a simple hi message ?
The 9.2k token usage for a simple "hi" message is likely due to:
Context overhead: The system includes metadata, conversation history, and internal processing that counts toward token usage, even for simple greetings.
Model-specific counting: Different tokenization methods count tokens differently. Some models count every character or word, while others use subword tokenization (like BPE - Byte Pair Encoding).
System prompts and instructions: The system prompt and any default instructions you've set up are included in the token count.
API overhead: Some APIs add overhead for authentication, routing, and other infrastructure costs.
Batch processing: If you're using batch inference or have context from previous conversations, those tokens are included.
The exact breakdown would depend on your specific model configuration and API settings. Would you like me to help you optimize token usage or understand your specific tokenization setup?_
For this history, the total token usage is 18.1K