There is a naming convention in GGUF files where the [A-Za-z_]+ slug after the last - is the quantization type. Today I tried loading a model from https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF where the files are named like this:
Step-3.7-Flash-APEX-Balanced.gguf
Step-3.7-Flash-APEX-Compact.gguf
Step-3.7-Flash-APEX-I-Balanced.gguf
Step-3.7-Flash-APEX-I-Compact.gguf <- I have this one downloaded
Step-3.7-Flash-APEX-I-Mini.gguf
Step-3.7-Flash-APEX-I-Quality.gguf
Step-3.7-Flash-APEX-Quality.gguf
When using llama serve it detected the following model id from my huggingface cache:
mudler/Step-3.7-Flash-APEX-GGUF:COMPACT
If you were to name the files like Step-3.7-Flash-APEX-I_Compact.gguf it would be unambiguous and matching llama.cpp's conventions. The specific issue here is that both -I-Compact and -Compact resolve to :COMPACT and there isn't really a clean way to disambiguate this.
There is a naming convention in GGUF files where the
[A-Za-z_]+slug after the last-is the quantization type. Today I tried loading a model from https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF where the files are named like this:When using
llama serveit detected the following model id from my huggingface cache:If you were to name the files like
Step-3.7-Flash-APEX-I_Compact.ggufit would be unambiguous and matching llama.cpp's conventions. The specific issue here is that both-I-Compactand-Compactresolve to:COMPACTand there isn't really a clean way to disambiguate this.