Skip to content

chore(deps): update dependency promptfoo to v0.121.3#512

Open
renovate[bot] wants to merge 1 commit intomainfrom
renovate/promptfoo-0.x-lockfile
Open

chore(deps): update dependency promptfoo to v0.121.3#512
renovate[bot] wants to merge 1 commit intomainfrom
renovate/promptfoo-0.x-lockfile

Conversation

@renovate
Copy link
Copy Markdown
Contributor

@renovate renovate bot commented Mar 27, 2026

This PR contains the following updates:

Package Change Age Confidence
promptfoo 0.121.2 → 0.121.3 age confidence

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

promptfoo/promptfoo (promptfoo)

v0.121.3

Compare Source

Features
Bug Fixes

Configuration

📅 Schedule: Branch creation - "before 10am on friday" in timezone Europe/London, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate renovate bot added dependencies Renovatebot and dependabot updates frontend javascript Pull requests that update javascript code labels Mar 27, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 27, 2026 •
edited
Loading

Open in Overmind ↗

model|risks_v6
✨Encryption Key State Risk ✨KMS Key Creation

🔴 Change Signals

Routine 🔴 ▇▅▃▂▁ Multiple API server and API access resources showing unusual infrequent updates at 1-2 events/week for the last 2-3 months, with the load balancer attachment also changing only 1 event/week for the last 7 weeks, which is infrequent compared to typical patterns.
Policies 🔴 ▃▂▁ S3 bucket 'aws_s3_bucket.terraform-example-state-bucket' is missing required tags and server-side encryption, and security group allows SSH access from anywhere, which is unusual compared to typical patterns.

View signals ↗

🔥 Risks

Replacing the only healthy ALB target and deleting its root volumes can cause outage and permanent data loss ‼️High Open Risk ↗
The change replaces both EC2 instances with new AMIs, and the public-facing instance 540044833068.eu-west-2.ec2-instance.i-08c49ffb6f2242b1e is the only currently registered healthy target in 540044833068.eu-west-2.elbv2-target-group.api-207c90ee-tg, which health-checks GET /health on port 80. Because there is no demonstrated second healthy target in that group, replacing this instance removes the workload’s only confirmed serving capacity during the cutover; if the replacement instance is slow to boot or fails the ALB health check, the public endpoint will have zero healthy targets and become unavailable.

The replacement also terminates the current instances, and their root EBS volumes vol-001472dbbdffc4149 and vol-0e86199cac82f2256 are currently attached with DeleteOnTermination=true. AWS deletes EBS volumes with that flag when the instance is terminated, so any state stored only on those root disks will be lost as part of this rollout. This is a real durability risk because the plan shows instance replacement but no preservation step for those volumes.

Unauthenticated email subscriber can leak or silently drop production SNS alerts ❗Medium Open Risk ↗
This change creates a new email subscription on the production-api-alerts SNS topic with endpoint: alerts@example.com, protocol: email, and endpoint_auto_confirms: false. Amazon SNS requires email subscriptions to be confirmed by the mailbox owner before they become active, and it does not verify that the address belongs to your organization. Nothing in the change, current topic state, or live graph evidence shows that alerts@example.com is an internal, approved incident mailbox.

If that address is controlled outside the organization, production alarm and incident messages sent through this topic will be disclosed to an unauthorized recipient once the subscription is confirmed. If it is not confirmed, SNS will leave it pending and will not deliver notifications to it, creating a false sense that production alert routing has been added when no messages will actually be received through that path.

🧠 Reasoning · ✔ 2 · ✖ 2

Unverified external subscription to production alerting channels

Observations 1

Hypothesis

Adding an unauthenticated email subscription (alerts@example.com) to the production-api-alerts SNS topic introduces the risk that sensitive operational alerts are delivered to an external or unapproved recipient. If the email is not an internal, verified, and authorized address, production incident and CloudWatch alarm notifications could be exposed outside the organization or missed if the subscription confirmation flow is not completed or monitored. Validate that the endpoint is owned/controlled by the organization, confirm SNS subscription verification behavior, and ensure incident routing logic accounts for this subscriber.

Investigation

I treated the concern area as security and monitoring integrity for production alert delivery: either alerts are exposed to an unauthorized external recipient, or alert routing is silently degraded because the email subscriber never becomes active. I first loaded relevant organizational guidance. The IAM/access-control guidance says externally shared access must be explicitly controlled, and the monitoring guidance treats alert notification targets as important operational controls. The security compliance file does not define an email allowlist, but it reinforces that production controls must be intentional and reviewed. The infrastructure notes also highlight that shared notification resources can have broad downstream impact, though this specific topic appears narrower than the central SNS topic.

I then checked current state. The production-api-alerts SNS topic currently has SubscriptionsConfirmed: 0 and SubscriptionsPending: 0, so this new email subscription would become the only subscriber visible on that topic in the current snapshot. I also checked a representative CloudWatch alarm in blast radius (api-207c90ee-high-cpu) and found it currently publishes to a different SNS topic (api-207c90ee-alerts), which means the blast radius evidence does not prove this production topic is actively wired to current alarms. However, the hypothesis is still about adding an unverified external recipient to a production alerting channel, and the planned change definitively creates an aws_sns_topic_subscription with protocol: email, endpoint: alerts@example.com, and endpoint_auto_confirms: false.

I looked up AWS SNS behavior in AWS documentation. AWS states that email subscriptions must be confirmed before they are active, that unconfirmed subscriptions remain in PendingConfirmation, and that email subscriptions are automatically deleted after about 48 hours if not confirmed. AWS also states SNS does not deliver notifications to endpoints until confirmation is completed. Terraform's resource is configured with a one-minute confirmation timeout, but AWS's subscription lifecycle still requires recipient-side confirmation and does not authenticate that the address belongs to the organization. Because the endpoint is just alerts@example.com and there is no evidence anywhere in the change, blast radius, or live graph that this address is organization-owned or already managed as an approved SNS subscription, the change introduces a real risk in the hypothesis's concern area: if that mailbox is external or misrouted, production notifications will be exposed outside the org once confirmed; if it is not confirmed, operators may assume a notification path exists when SNS will not send to it at all. That is concrete, change-specific risk rather than speculation.

✔ Hypothesis proven

EC2 instance replacement with unvetted AMIs and DeleteOnTermination volumes impacting behavior and durability

Observations 14

Hypothesis

EC2 instance replacement with new AMIs and user_data introduces multiple compute configuration, durability, and behavior risks:

  • AMI changes from ami-0dd010f28b091c0fd to ami-09ce8939d5ab2791a/ami-09ce8939d5ab2791c, combined with user_data changes (including redacted/base64 startup scripts), can alter OS/kernel level, installed packages, init system, and boot-time configuration such as starting httpd or creating /health endpoints. This may cause instances (e.g., i-08c49ffb6f2242b1e) to fail ALB HTTP health checks (/health), be slow to serve traffic, or otherwise reduce target group and public endpoint availability.
  • Root and attached EBS volumes (e.g., the root volume and vol-001472dbbdffc4149) configured with DeleteOnTermination=true will be deleted when instances are terminated during replacement, risking irreversible data loss for any state persisted only on those volumes if backups/snapshots are not in place. This conflicts with data durability and lifecycle management guidance in aws-compute-configuration.
  • Instance replacement can change or briefly detach private IPs (e.g., 10.0.101.211) and affect whether services (httpd, health endpoints) remain bound to that address, impacting internal DNS names such as ip-10-0-101-211.eu-west-2.compute.internal and any consumers that rely on fixed instance IPs/hostnames.
    Overall, using unvetted AMIs, brittle or changed user_data, and DeleteOnTermination volumes without proper backups or indirection (load balancers, service discovery) violates compute configuration and durability best practices (e.g., SEC06-BP02) and can lead to security drift, data loss, and loss of expected service or public endpoint availability.

Investigation

I treated the concern area as service availability and data durability for the two EC2 instances being replaced. I first checked the relevant organizational knowledge. That guidance says EC2 instances should use hardened, regularly updated AMIs, EBS volumes for data at rest must be encrypted, and EBS volumes used for critical data should have backup/snapshot coverage. It also flags single-instance and single-AZ designs as availability risks.

I then compared the diffs with current state from blast radius. For both instances, the concrete planned change is replacement caused by an AMI change from ami-0dd010f28b091c0fd to ami-09ce8939d5ab2791c; the visible user data is being replaced with redacted base64, so I cannot prove the startup logic is broken from the diff alone. The current ALB target group 540044833068.eu-west-2.elbv2-target-group.api-207c90ee-tg performs HTTP health checks on /health over port 80, and the current target health for i-08c49ffb6f2242b1e is healthy. However, only that single instance is currently registered in the target group, so replacing it creates a real availability risk even without proving the new AMI is bad: during replacement there is no demonstrated redundant healthy target in that target group, and if the new instance takes time to boot or fails health checks, the public ALB-backed endpoint loses healthy capacity.

On the durability side, the current root EBS volumes vol-001472dbbdffc4149 and vol-0e86199cac82f2256 are attached with DeleteOnTermination: true, and AWS documentation confirms volumes with that flag are automatically deleted when the instance is terminated. Because these instances are being replaced, termination is part of the change path, so those root volumes will be deleted. The current volumes are also unencrypted (Encrypted: false), which violates the organization’s security requirements, but that non-compliance is pre-existing rather than introduced by this change. The important change-specific risk is that replacement will permanently discard anything persisted only on those root volumes. One of the volumes does have a snapshot lineage, but I found no evidence in the plan of any retention/preservation step such as changing DeleteOnTermination to false, detaching the volume, or creating a dedicated preserved data volume before replacement. That means the hypothesis’s durability concern is real.

I did not find strong evidence for the hypothesis’s private-IP/dns breakage mechanism. The diffs show private_ip becomes unknown after apply, but that is normal for replacement and there is no evidence that anything depends directly on the instance-private DNS names instead of the load balancer. I also did not mark the mere fact of a new AMI or redacted user data as a risk by itself; the evidence is insufficient to say the new AMI is unvetted or that /health will definitely disappear. The confirmed risk is a closely related mechanism in the same concern area: single-target replacement can create downtime, and replacement with DeleteOnTermination=true will delete the root volumes and any state stored only there.

✔ Hypothesis proven

Load balancer target group and attachment changes affecting routing/availability

Observations 5

Hypothesis

Multiple load balancer target group and attachment changes continue to introduce routing and availability risk:

  • Replacement of NLB/ALB target groups can momentarily deregister targets or change health check configuration, disrupting traffic for listeners forwarding to these groups and potentially marking all back-end IPs unhealthy if health checks or registrations are incorrect.
  • Replacing target group attachments alters which EC2 instances are registered, potentially sending traffic to unintended instances or leaving no healthy targets, impacting availability and alarms.
  • Replacement of the ELB target group api-health-terraform-example for mon-internal can change health checks, target registrations, and listener associations, affecting routing for instances in associated subnets (e.g., subnet-053a5e9fa7920c431) and DNS records pointing to the load balancer.
    These changes can cause downtime, misrouted traffic, and monitoring blind spots if health check path/port/protocol, target type, target registration, and listener/DNS mappings are not carefully validated, and if connection draining or phased rollouts are not used. Relevant best practices: REL02-BP01, REL02-BP03, SEC05-BP01/SEC05-BP02.

Investigation

I treated the concern area as routing and availability loss on the internal NLB mon-internal-terraform-example caused by replacement of its target group 540044833068.eu-west-2.elbv2-target-group.api-health-terraform-example and the related target attachment. I first checked relevant organizational guidance: aws-high-availability, aws-network-security, security-compliance-requirements, and infrastructure-quick-reference. The reliability guidance says to flag production changes that roll out to all instances simultaneously or leave ELB targets in only one AZ, but only when the change actually creates that condition.

The strongest evidence against the hypothesis is that the current NLB path is already extremely simple and internally consistent: the listener on port 9090 forwards to the target group api-health-terraform-example; that target group is TCP/TCP, port 9090, target type ip; and the only observed registered target 10.0.101.106:9090 is currently healthy. The load balancer itself spans two subnets/AZs (subnet-025500a6290ebe3d8 in eu-west-2a and subnet-053a5e9fa7920c431 in eu-west-2b), so there is no evidence of a single-AZ NLB misconfiguration. The target group replacement diff and attachment replacement diff are both empty, so there is no concrete sign that health-check protocol, port, path, target type, or listener mapping are being changed incorrectly in this plan.

I also investigated the other planned changes to see whether a closely related mechanism existed. The two EC2 instances with visible diffs are being replaced because of an AMI update, but they are in a different VPC (vpc-02901bcbb89561298) from the internal NLB target group (vpc-096b686376892bb49), so they are not credible backends for this NLB target group. That makes the hypothesis about the attachment replacement sending NLB traffic to the wrong EC2 instances weak: the changed EC2 instances shown in the plan are not even in the target group's VPC. The CloudWatch alarm in blast radius targets the separate ALB target group api-207c90ee-tg, not api-health-terraform-example, so the claimed monitoring blind spot is also not supported by the evidence gathered.

I checked AWS documentation as well. AWS states that deregistering a target stops routing to that target immediately, with existing connections drained, and that newly registered targets only begin receiving traffic after registration completes and health checks pass. That means target-group or target replacement can absolutely be risky in principle. But in this specific change there is no proof of an incorrect new target, broken health check, wrong listener association, or a no-target condition. With only empty replacement diffs and no contradictory state evidence, the hypothesis remains a generic warning rather than a demonstrated risk for this plan.

✖ Hypothesis disproven

Direct EIP exposure and reassociation affecting security and public endpoint stability

Observations 4

Hypothesis

Direct exposure of EC2 instances via Elastic IPs (EIPs) and changes to EIP associations create public attack surface and endpoint stability risks:

  • Instance i-0c326c28fef2cb7ee is directly assigned public IPv4 address 13.134.236.98 via EIP, placing the instance on the internet instead of behind a managed edge (ALB/CloudFront + WAF) and violating network segmentation best practices (REL02-BP01/SEC05-BP01). This increases attack surface and bypasses centralized protections.
  • Updating or reassigning EIPs on ENIs (e.g., ENI eni-0714cc06176804efa) can change or remove the public IP used by clients and external health checkers; if DNS A records or consumers are not updated, endpoints may become unreachable or behave unexpectedly.
  • Persistent use of instance public IPs as canonical endpoints conflicts with architectural guidance to front services with load balancers and managed DNS, and to avoid relying on static instance IPs.
    These changes can cause loss of availability, monitoring gaps, and security policy violations unless EIP usage, DNS, and routing through load balancers are carefully managed.

Investigation

I treated the concern area as two related risks: direct internet exposure of an EC2 instance via an Elastic IP, and loss of availability if the Elastic IP is being moved or changed. I first loaded the relevant organizational knowledge. That knowledge does say direct EC2 public exposure is a policy finding: aws-network-security flags EC2 instances with public IPs as a medium-risk anti-pattern under REL02-BP01/SEC05-BP01, and security-compliance-requirements says EC2 instances must not be directly reachable from the internet.

However, the specific task here is to determine whether this change creates a real risk in that concern area. The only concrete planned diff on the EIP is that 540044833068.eu-west-2.ec2-address.13.134.236.98 changes its instance field from i-0ea52e0eb6476a224 to (known after apply). The current blast-radius state shows the EIP is actually attached to i-0c326c28fef2cb7ee through ENI eni-0714cc06176804efa, with public IP 13.134.236.98 and private IP 10.0.101.106. That same instance is also registered and currently healthy in the internal NLB target group 540044833068.eu-west-2.elbv2-target-group.api-health-terraform-example on port 9090. Its security groups are not open to the world: sg-03cf38efd953aa056 allows only a customer whitelist on 443, and sg-089e5107637083db5 allows only 10.0.0.0/8 on internal ports.

I then checked the other planned changes to see whether the EIP was actually being reassigned to a different workload or whether some coordinated DNS / load balancer move was happening. The only clearly related change is replacement of 540044833068.eu-west-2.ec2-instance.i-0ea52e0eb6476a224, which previously had the same public IP in its old state, indicating Terraform/provider state drift rather than evidence that the public endpoint is being moved away from the current production instance. I found no planned Route 53/DNS changes, no security-group weakening, and no explicit EIP reallocation to a different ENI or different service. AWS documentation also confirms that Elastic IPs exist specifically to preserve a stable public IP across instance replacement and reassociation, though active connections can briefly drop during remap. In this plan, there is not enough evidence that clients will lose the endpoint or that centralized protections are being newly bypassed by this change; the direct exposure already exists in current state and is not being introduced or widened here.

So there is a standing architecture/compliance issue in the environment, but I do not have strong evidence that this particular PR will cause availability loss, monitoring gaps, or a new security exposure in the hypothesis’s concern area. The hypothesis is directionally valid as a generic concern, but for this change it remains speculative rather than demonstrated.

✖ Hypothesis disproven

💥 Blast Radius

Items 63

Edges 135

github-actions[bot]
github-actions bot requested changes Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Auto-blocked: Routine score (-5) is below minimum (-1)

📊 Signals Summary

Routine 🔴 -5

🔥 Risks Summary

High 0 · Medium 0 · Low 0

💥 Blast Radius

Items 23 · Edges 75

View full analysis in Overmind ↗

@renovate renovate bot force-pushed the renovate/promptfoo-0.x-lockfile branch from 514a41c to 58cbaeb Compare March 27, 2026 13:24
github-actions[bot]
github-actions bot requested changes Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Auto-blocked: Policy signal (-3) is below threshold (-2); Routine score (-5) is below minimum (-1)

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

🔥 Risks Summary

High 0 · Medium 0 · Low 0

💥 Blast Radius

Items 5 · Edges 20

View full analysis in Overmind ↗

@renovate renovate bot force-pushed the renovate/promptfoo-0.x-lockfile branch from 58cbaeb to a1c25a9 Compare April 1, 2026 20:12
github-actions[bot]
github-actions bot requested changes Apr 1, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Found 2 high risks requiring review

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

🔥 Risks Summary

High 2 · Medium 0 · Low 0

💥 Blast Radius

Items 107 · Edges 219

View full analysis in Overmind ↗

@renovate renovate bot force-pushed the renovate/promptfoo-0.x-lockfile branch from a1c25a9 to f877105 Compare April 8, 2026 21:00
github-actions[bot]
github-actions bot requested changes Apr 8, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overmind

⛔ Auto-Blocked

🔴 Decision

Found 1 high risk requiring review

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

🔥 Risks Summary

High 1 · Medium 1 · Low 0

💥 Blast Radius

Items 63 · Edges 135

View full analysis in Overmind ↗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

dependencies Renovatebot and dependabot updates frontend javascript Pull requests that update javascript code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

0 participants