Skip to content

Context size & token usage calculation seems to be incorrect #109

Description

@pkhara31

With a simple "Hi" message sent as the very first message and without any other active chats, the token usage shows 9.2K tokens used.

Image

Also, when set the context to 4K from the existing 65536, I get an error

Image

This is my entire conversation history:

_hi
Hello! How can I help you today?

why the token usage shows 9.2k tokens for a simple hi message ?
The 9.2k token usage for a simple "hi" message is likely due to:

Context overhead: The system includes metadata, conversation history, and internal processing that counts toward token usage, even for simple greetings.

Model-specific counting: Different tokenization methods count tokens differently. Some models count every character or word, while others use subword tokenization (like BPE - Byte Pair Encoding).

System prompts and instructions: The system prompt and any default instructions you've set up are included in the token count.

API overhead: Some APIs add overhead for authentication, routing, and other infrastructure costs.

Batch processing: If you're using batch inference or have context from previous conversations, those tokens are included.

The exact breakdown would depend on your specific model configuration and API settings. Would you like me to help you optimize token usage or understand your specific tokenization setup?_

For this history, the total token usage is 18.1K

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions