Goal: don't just freehand the context, engineer it.
How do we know if CONTEXT.yaml is better than a CLAUDE.md? We don't, without proper testing.
But how can we test? What criteria are we looking for. Broadly, good context results in better solutions with fewer tokens used to get there (in that order).
But this is maybe hard to test for, especially when it needs to be well suited to a complex code base with nuance.
In testing against Backstage and Zudoku, quality of answers was hard to quantify. Is the simple solution more desirable than the complex one that fixes underlying architecture issues? One phrase of context changing can have a huge impact on which solution results.
But also with LLMs, this stuff is not 100% deterministic.
We either need to identify a deterministic "proxy" measurement measurement that eliminates the LLM variability, or we need a larger sample size to "prove" context quality through result quality. Even then, result quality is hard to quantify for nuanced code bases which have very different contextual needs.
It's a hard problem to solve, but if we figure it out, it could be valuable.
Goal: don't just freehand the context, engineer it.
How do we know if CONTEXT.yaml is better than a CLAUDE.md? We don't, without proper testing.
But how can we test? What criteria are we looking for. Broadly, good context results in better solutions with fewer tokens used to get there (in that order).
But this is maybe hard to test for, especially when it needs to be well suited to a complex code base with nuance.
In testing against Backstage and Zudoku, quality of answers was hard to quantify. Is the simple solution more desirable than the complex one that fixes underlying architecture issues? One phrase of context changing can have a huge impact on which solution results.
But also with LLMs, this stuff is not 100% deterministic.
We either need to identify a deterministic "proxy" measurement measurement that eliminates the LLM variability, or we need a larger sample size to "prove" context quality through result quality. Even then, result quality is hard to quantify for nuanced code bases which have very different contextual needs.
It's a hard problem to solve, but if we figure it out, it could be valuable.