Make --run aware that produced counts come from the LAVA manifest - #2011
Merged
Conversation
validate_sample_data.py --run takes each artifact's produced row count from the run's _lava_data.lava manifest. An artifact whose output_types exclude LAVA never reaches that manifest, so it read as zero however many rows it produced, and every count it declares was reported as an error. knowledgeC_AppUsage_EndTime is the only artifact this reaches today. It declares "timeline", so its rows go to tl.db alone, and it carries 14 recorded counts. Runs on magnet_ios16 and hc_ios18_7 produce 202 and 184 rows, which is what it declares and what the timeline database holds, while the manifest holds neither. --run now asks the core's own check_output_types whether an artifact writes to LAVA and reports the ones that do not as uncheckable. The import is lazy and only reached from --run, which already runs ileapp.py, so the structure step CI runs stays free of third party imports. If the import fails, every count is compared as before, so a real regression stays visible. The run log is also written beside the output rather than inside it, so it outlives the cleanup. A produced count of zero can mean the artifact found nothing, that its input is absent from the corpus, or that a tool it needs is not installed, and the artifact usually logged which. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
--runaware that it reads produced row counts from the LAVA manifest.output_typesexclude LAVA never reaches that manifest, so it read as zero whatever it produced and every count it declares was reported as an error. Today that isknowledgeC_AppUsage_EndTimealone. Those are now reported as uncheckable instead of compared.check_output_types, imported lazily so the structure step CI runs stays import free. If that import fails, every count is compared as before.