Skip to content

CHECKS.md

Latest commit

 

History

History
108 lines (78 loc) · 7.57 KB

CHECKS.md

File metadata and controls

108 lines (78 loc) · 7.57 KB
Raw

VEILSCAN Checks Reference

This file summarizes the major scan families in VEILSCAN and what each one is looking for.

VEILSCAN findings are emitted as check_ids such as PI-001, SERIAL-003, or RAG-005. Those raw findings are later grouped into checklist views such as OWASP LLM, OWASP ML, MITRE ATLAS, and the compliance checklist.

How To Read This File

  • Family is the broad check module
  • Prefixes are the finding ID ranges emitted by that module
  • What it looks for describes the main risk area
  • Typical examples gives representative patterns, not an exhaustive list

Check Families

Family Prefixes What it looks for Typical examples
Prompt Injection & Insecure Output Handling PI-* Unsafe prompt construction and unsafe use of model output User input concatenated into prompts, user data inserted into system prompts, eval(response), exec(output), prompt-to-SQL
Unsafe Serialization SERIAL-* Risky model/file formats and unsafe deserialization .pkl, .joblib, .pt, .bin, pickle.load, torch.load without weights_only=True, unsafe yaml.load
Model Integrity INTEG-* Tampering, backdoors, and integrity gaps Missing checksums, loader code with sockets or subprocess execution, compressed pickle artifacts, known bypass patterns
Data Leakage & Sensitive Information DL-* Secrets and sensitive data exposure API keys, Hugging Face tokens, AWS keys, PEM private keys, database URIs with creds, PII patterns
Excessive Agency & Model Theft AGENCY-* Overly autonomous model behavior or data/model exposure Disabled confirmation gates, unsafe tool permissions, prompt grants like "do anything", raw logits/embeddings/state export
Plugin / Tool Security PLUGIN-* Dangerous tool integrations User-controlled URLs to tool calls, weak parameter validation, dangerous action surfaces
Training Data Poisoning TRAIN-* Unsafe training input pipelines Remote datasets, user-controlled training files, poisoned prompts in .jsonl, direct reward-model abuse
Model Configuration CONFIG-* Risky HF/tokenizer/generation settings trust_remote_code=true, auto_map, risky tokenizer classes/tokens, huge generation limits
RAG Security RAG-* Retrieval and vector-store attack surface Untrusted loaders, user-controlled vector stores, unsanitized retrieved context, direct query injection
Model Denial of Service DOS-* Resource exhaustion risks Missing max_tokens or max_new_tokens, no API rate limiting, infinite loops, oversized generation config
Privacy Risk PRIV-* Information leakage from outputs or APIs Confidence score exposure, hidden states, prompt leakage, personal-data training without privacy controls
Adversarial Robustness ADV-* Missing defenses against adversarial inputs No robustness tooling, no input validation, poor preprocessing, no robustness documentation
Transfer Learning Security XFER-* Unsafe base-model fine-tuning practices Dynamic from_pretrained() paths, missing provenance, trust_remote_code=True on base models
Output Integrity OI-* Integrity problems in serving pipelines Missing audit logs, HTTP serving without TLS, forwarding model output to external systems without safeguards
License Compliance LIC-* Legal and licensing issues Unknown license, GPL/AGPL, non-commercial restrictions, missing declared license
Bias & Fairness BIAS-* Fairness and governance gaps No fairness library, missing bias section in docs, protected attributes in training features, high-risk-use documentation gaps
Model Drift DRIFT-* Missing monitoring after deployment No drift tooling, no monitoring on serving path, no retraining/evaluation pipeline
Dependency Vulnerabilities DEP-* Version-based package issues Known vulnerable torch, transformers, fastapi, mlflow, onnx, and similar dependency rules
Supply Chain & Provenance SC-* Trust and provenance gaps around shipped models/projects Missing model cards, missing provenance, missing source documentation, weak supply-chain hygiene

Examples By Area

Prompt and Output Risks

These checks focus on how applications construct prompts and how they consume LLM output.

Representative findings:

  • user input directly embedded in prompts
  • user input injected into system messages
  • model output passed to eval, exec, or shell/process calls
  • model output turned into SQL and executed directly
  • jailbreak phrases embedded in docs, prompts, or templates
  • remote or file content inserted into prompts without sanitization

Serialization and Artifact Safety

These checks are especially important for downloaded models and checkpoints.

Representative findings:

  • pickle-derived model files present in the target
  • PyTorch checkpoints that imply pickle-backed load behavior
  • code that calls pickle.load, joblib.load, or unsafe torch.load
  • compressed pickle artifacts used to evade scanners
  • code or configs that enable remote code execution during model load

RAG and Agent Risks

These checks cover the AI application layer rather than only the model artifact.

Representative findings:

  • user-controlled document or URL ingestion
  • unsafe vector-store loading
  • unsanitized retrieved context injected into prompts
  • direct similarity search with raw attacker-controlled queries
  • agents configured to act without approval
  • tools with dangerous capabilities or broad access

Privacy, Governance, and Compliance

These checks look at operational and regulatory risk, not just exploit paths.

Representative findings:

  • confidence scores or logits exposed by APIs
  • hidden states or embeddings returned to callers
  • personal data training without differential privacy controls
  • missing fairness tooling or bias reporting
  • high-risk use cases without AI Act-style documentation
  • unclear or restrictive licensing

Checklist Frameworks

VEILSCAN maps findings into these checklist views:

  • owasp-llm
  • owasp-ml
  • mitre-atlas
  • real-world
  • compliance

These checklists do not replace the raw findings. They provide a framework-aligned view of the same underlying scan results.

Notes

  • VEILSCAN is primarily a heuristic static scanner for files, code, configs, and metadata.
  • probe is separate from the static checks and performs live HTTP-based runtime testing against running endpoints.
  • A finding means "this pattern deserves review", not "this is guaranteed exploitable in every environment".