Skip to content

Failing PCF integration tests #1129

New issue
New issue
Closed
#1133
Closed
Failing PCF integration tests#1129
#1133
Labels
investigatingassignee is investigating to determine root cause and fixtestsaffects unit tests or integration tests

Description

@jranson
jranson
opened on Sep 3, 2026

2 tests are failing intermittently. This may be due to changes in #1123. Appears to be a test harness defect rather than a PCF or Proxy issue. No actual data corruption observed; the expected bodies are correct, but the harnesses' request counters are higher than expected.

Likely causes:

  • Health check probes are causing the counters used in these tests to increase unexpectedly. Should disable health checking for PCF integration tests that rely on app-wide counters for verification. 1123 changed the startup timing of the health check probes which might be provoking this.
  • The integration harness cancels each Trickster daemon during cleanup, but does not wait for all its background workers to terminate before moving to the next test case, allowing shutdown/startup overlap and taint subsequent tests.
=== RUN   TestPCFRefusesIneligibleResponses/vary
time=2026-09-03T13:57:08.283388814Z app=trickster level=error event="http listener stopping" caller=pkg/proxy/listener/listener.go:343 detail="http: Server closed" listenerName=listener.mgmt.http
time=2026-09-03T13:57:08.283445529Z app=trickster level=error event="http listener stopping" caller=pkg/proxy/listener/listener.go:343 detail="http: Server closed" listenerName=listener.default.http
time=2026-09-03T13:57:08.283565242Z app=trickster level=error event="http listener stopping" caller=pkg/proxy/listener/listener.go:343 detail="http: Server closed" listenerName=listener.metrics.http
time=2026-09-03T13:57:08.354416877Z app=trickster level=info event="healthcheck status changed" caller=pkg/backends/healthcheck/target.go:351 status=available targetName=prom2 threshold=1
time=2026-09-03T13:57:08.379691898Z app=trickster level=warn event="listener \"default\" TLS port is disabled because no mapped backend provides a TLS certificate" caller=pkg/daemon/setup/setup.go:138
time=2026-09-03T13:57:08.38004636Z app=trickster level=info event="application loaded from configuration" caller=pkg/daemon/setup/setup.go:412 buildTime= commitID= config=/tmp/TestPCFRefusesIneligibleResponsesvary1639929487/001/trickster.yaml goArch=amd64 goOS=linux goVersion=go1.26.8 logLevel=info name= pid=50161 version=
time=2026-09-03T13:57:08.383151647Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rpc3 backendProvider=rpc upstreamHost=127.0.0.1:9090
time=2026-09-03T13:57:08.383912706Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb3 backendProvider=alb upstreamHost=
time=2026-09-03T13:57:08.383927522Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=click-native backendProvider=clickhouse upstreamHost=127.0.0.1:9000
time=2026-09-03T13:57:08.384140369Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=flux2 backendProvider=influxdb upstreamHost=127.0.0.1:8086
time=2026-09-03T13:57:08.384219958Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=sim1 backendProvider=prometheus upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.38443971Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=graphite1 backendProvider=graphite upstreamHost=127.0.0.1:8081
time=2026-09-03T13:57:08.384802049Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=influx3 backendProvider=influxdb upstreamHost=127.0.0.1:8181
time=2026-09-03T13:57:08.384886004Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=pcfvary backendProvider=reverseproxy upstreamHost=127.0.0.1:33519
time=2026-09-03T13:57:08.384964381Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=prom3 backendProvider=prometheus upstreamHost=127.0.0.1:9090
time=2026-09-03T13:57:08.385181514Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=sim2 backendProvider=prometheus upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.38547452Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=click1 backendProvider=clickhouse upstreamHost=127.0.0.1:8123
time=2026-09-03T13:57:08.385547994Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=flux-sim-1 backendProvider=influxdb upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.385625785Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=flux1 backendProvider=influxdb upstreamHost=127.0.0.1:8086
time=2026-09-03T13:57:08.385720997Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=mysql1 backendProvider=mysql upstreamHost=127.0.0.1:3306
time=2026-09-03T13:57:08.385738844Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb1 backendProvider=alb upstreamHost=
time=2026-09-03T13:57:08.385785418Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=lm1 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.385858922Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=lm2 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.385922672Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=prom2 backendProvider=prometheus upstreamHost=127.0.0.1:9090
time=2026-09-03T13:57:08.386180441Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rp1 backendProvider=proxy upstreamHost=127.0.0.1:9090
time=2026-09-03T13:57:08.386234541Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rpc1 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.386345507Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rpc2 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T13:57:08.386474098Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb2 backendProvider=alb upstreamHost=
time=2026-09-03T13:57:08.386545239Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb4 backendProvider=alb upstreamHost=
time=2026-09-03T13:57:08.386560266Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=prom1 backendProvider=prometheus upstreamHost=127.0.0.1:9090
time=2026-09-03T13:57:08.387390387Z app=trickster level=info event="registering default backend handler paths" caller=pkg/routing/routing.go:469 backendName=prom1
time=2026-09-03T13:57:08.388949473Z app=trickster level=info event="registering pprof /debug routes" caller=pkg/observability/pprof/pprof.go:31 routerName=metrics
time=2026-09-03T13:57:08.388968606Z app=trickster level=info event="registering pprof /debug routes" caller=pkg/observability/pprof/pprof.go:31 routerName=mgmt
time=2026-09-03T13:57:08.391089888Z app=trickster level=info event="clickhouse listener starting" caller=pkg/proxy/listener/protocol.go:72 address=127.0.0.1 listenerName=listener.clickhouse-native-origin-native.clickhouse port=45063
time=2026-09-03T13:57:08.391723482Z app=trickster level=info event="http listener starting" caller=pkg/proxy/listener/listener.go:313 address=127.0.0.1 listenerName=listener.default.http port=41167
time=2026-09-03T13:57:08.391772274Z app=trickster level=info event="http listener starting" caller=pkg/proxy/listener/listener.go:313 address=127.0.0.1 listenerName=listener.mgmt.http port=44491
time=2026-09-03T13:57:08.391807952Z app=trickster level=info event="clickhouse listener starting" caller=pkg/proxy/listener/protocol.go:72 address=127.0.0.1 listenerName=listener.clickhouse-native.clickhouse port=37929
time=2026-09-03T13:57:08.39187367Z app=trickster level=info event="http listener starting" caller=pkg/proxy/listener/listener.go:313 address=127.0.0.1 listenerName=listener.metrics.http port=34813
time=2026-09-03T13:57:08.391951009Z app=trickster level=info event="all listeners ready" caller=pkg/daemon/daemon.go:121
time=2026-09-03T13:57:08.391961446Z app=trickster level=info event="mysql listener starting" caller=pkg/proxy/listener/protocol.go:72 address=0.0.0.0 listenerName=listener.mysql1.mysql port=34177
time=2026-09-03T13:57:08.44158634Z app=trickster level=info event="healthcheck status changed" caller=pkg/backends/healthcheck/target.go:351 status=available targetName=sim1 threshold=3
    pcf_test.go:169: 
        	Error Trace:	/home/runner/work/trickster/trickster/integration/pcf_test.go:169
        	Error:      	Not equal: 
        	            	expected: int(2)
        	            	actual  : int32(7)
        	Test:       	TestPCFRefusesIneligibleResponses/vary
        	Messages:   	vary response must not collapse
time=2026-09-03T13:57:08.829601514Z app=trickster level=error event="http listener stopping" caller=pkg/proxy/listener/listener.go:343 detail="http: Server closed" listenerName=listener.mgmt.http
time=2026-09-03T13:57:08.829742574Z app=trickster level=error event="http listener stopping" caller=pkg/proxy/listener/listener.go:343 detail="http: Server closed" listenerName=listener.metrics.http
--- FAIL: TestPCFRefusesIneligibleResponses (1.63s)
    --- PASS: TestPCFRefusesIneligibleResponses/private (0.54s)
    --- PASS: TestPCFRefusesIneligibleResponses/set-cookie (0.54s)
    --- FAIL: TestPCFRefusesIneligibleResponses/vary (0.55s)
FAIL





=== RUN   TestPCFNeverCollapsesPost
time=2026-09-03T14:37:02.690834711Z app=trickster level=info event="healthcheck status changed" caller=pkg/backends/healthcheck/target.go:351 status=available targetName=prom7 threshold=1
time=2026-09-03T14:37:02.777118313Z app=trickster level=warn event="listener \"default\" TLS port is disabled because no mapped backend provides a TLS certificate" caller=pkg/daemon/setup/setup.go:138
time=2026-09-03T14:37:02.777579454Z app=trickster level=info event="application loaded from configuration" caller=pkg/daemon/setup/setup.go:412 buildTime= commitID= config=/tmp/TestPCFNeverCollapsesPost1168396863/001/trickster.yaml goArch=amd64 goOS=linux goVersion=go1.26.8 logLevel=info name= pid=54167 version=
time=2026-09-03T14:37:02.782837783Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb3 backendProvider=alb upstreamHost=
time=2026-09-03T14:37:02.782873927Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb1 backendProvider=alb upstreamHost=
time=2026-09-03T14:37:02.782887728Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=flux1 backendProvider=influxdb upstreamHost=127.0.0.1:8086
time=2026-09-03T14:37:02.784270149Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=flux2 backendProvider=influxdb upstreamHost=127.0.0.1:8086
time=2026-09-03T14:37:02.784265062Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb4 backendProvider=alb upstreamHost=
time=2026-09-03T14:37:02.784286904Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=prom2 backendProvider=prometheus upstreamHost=127.0.0.1:9090
time=2026-09-03T14:37:02.784282062Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=pcfpost backendProvider=reverseproxy upstreamHost=127.0.0.1:39545
time=2026-09-03T14:37:02.784300645Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rpc3 backendProvider=rpc upstreamHost=127.0.0.1:9090
time=2026-09-03T14:37:02.784785632Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=click-native backendProvider=clickhouse upstreamHost=127.0.0.1:9000
time=2026-09-03T14:37:02.784798556Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=sim2 backendProvider=prometheus upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.784828266Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=alb2 backendProvider=alb upstreamHost=
time=2026-09-03T14:37:02.784920834Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=flux-sim-1 backendProvider=influxdb upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.785226212Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=graphite1 backendProvider=graphite upstreamHost=127.0.0.1:8081
time=2026-09-03T14:37:02.786324608Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=influx3 backendProvider=influxdb upstreamHost=127.0.0.1:8181
time=2026-09-03T14:37:02.78635298Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=lm1 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.786367672Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=sim1 backendProvider=prometheus upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.786970275Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rpc2 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.78700691Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=click1 backendProvider=clickhouse upstreamHost=127.0.0.1:8123
time=2026-09-03T14:37:02.787799768Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=prom3 backendProvider=prometheus upstreamHost=127.0.0.1:9090
time=2026-09-03T14:37:02.787858676Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rp1 backendProvider=proxy upstreamHost=127.0.0.1:9090
time=2026-09-03T14:37:02.787819768Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=lm2 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.787841941Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=mysql1 backendProvider=mysql upstreamHost=127.0.0.1:3306
time=2026-09-03T14:37:02.787877495Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=rpc1 backendProvider=reverseproxycache upstreamHost=127.0.0.1:8482
time=2026-09-03T14:37:02.787967309Z app=trickster level=info event="registering route paths" caller=pkg/routing/routing.go:307 backendName=prom1 backendProvider=prometheus upstreamHost=127.0.0.1:9090
time=2026-09-03T14:37:02.789318664Z app=trickster level=info event="registering default backend handler paths" caller=pkg/routing/routing.go:469 backendName=prom1
time=2026-09-03T14:37:02.792466591Z app=trickster level=info event="registering pprof /debug routes" caller=pkg/observability/pprof/pprof.go:31 routerName=metrics
time=2026-09-03T14:37:02.792791759Z app=trickster level=info event="registering pprof /debug routes" caller=pkg/observability/pprof/pprof.go:31 routerName=mgmt
time=2026-09-03T14:37:02.796354107Z app=trickster level=info event="clickhouse listener starting" caller=pkg/proxy/listener/protocol.go:72 address=127.0.0.1 listenerName=listener.clickhouse-native-origin-native.clickhouse port=45247
time=2026-09-03T14:37:02.798020168Z app=trickster level=info event="clickhouse listener starting" caller=pkg/proxy/listener/protocol.go:72 address=127.0.0.1 listenerName=listener.clickhouse-native.clickhouse port=42571
time=2026-09-03T14:37:02.798164494Z app=trickster level=info event="http listener starting" caller=pkg/proxy/listener/listener.go:313 address=127.0.0.1 listenerName=listener.default.http port=43095
time=2026-09-03T14:37:02.798339086Z app=trickster level=info event="http listener starting" caller=pkg/proxy/listener/listener.go:313 address=127.0.0.1 listenerName=listener.metrics.http port=37333
time=2026-09-03T14:37:02.798410838Z app=trickster level=info event="all listeners ready" caller=pkg/daemon/daemon.go:121
time=2026-09-03T14:37:02.7984952Z app=trickster level=info event="mysql listener starting" caller=pkg/proxy/listener/protocol.go:72 address=0.0.0.0 listenerName=listener.mysql1.mysql port=43147
time=2026-09-03T14:37:02.798508635Z app=trickster level=info event="http listener starting" caller=pkg/proxy/listener/listener.go:313 address=127.0.0.1 listenerName=listener.mgmt.http port=37951
time=2026-09-03T14:37:03.083332574Z app=trickster level=info event="healthcheck status changed" caller=pkg/backends/healthcheck/target.go:351 status=available targetName=sim1 threshold=3
time=2026-09-03T14:37:03.447133156Z app=trickster level=info event="healthcheck status changed" caller=pkg/backends/healthcheck/target.go:351 status=available targetName=sim1 threshold=3
    pcf_test.go:187: 
        	Error Trace:	/home/runner/work/trickster/trickster/integration/pcf_test.go:187
        	Error:      	Not equal: 
        	            	expected: int(2)
        	            	actual  : int32(10)
        	Test:       	TestPCFNeverCollapsesPost
        	Messages:   	POST must never collapse
time=2026-09-03T14:37:03.563353882Z app=trickster level=error event="http listener stopping" caller=pkg/proxy/listener/listener.go:343 detail="http: Server closed" listenerName=listener.metrics.http
--- FAIL: TestPCFNeverCollapsesPost (0.97s)

Activity

  1. jranson
    added
    testsaffects unit tests or integration tests
    investigatingassignee is investigating to determine root cause and fix
    on Sep 3, 2026

  2. jranson commented on Sep 7, 2026

    @jranson
    jranson
    on Sep 7, 2026
    MemberAuthor

    Also random unit test failure:

    === RUN   TestCorrectnessInvariant
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min json (pass 0) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[52831.0, 1788187200], [53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 1788188700], [54631.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min json (pass 1) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[52831.0, 1788187200], [53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 1788188700], [54631.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min raw (pass 0) differs from the origin
         got: dev.medium.orders.us-east.count,1788187200,1788792300,300|52831.0,53131.0,53431.0,53731.0,54031.0,54331.0,54631.0,54931.0,55231.0,55531.0,55831.0,56131.0,56431.0,56731.0,57031.0,57331.0,57631.0,57931.0,58231.0,58531.0,58831.0,59131.0,59431.
        want: dev.medium.orders.us-east.count,1788187500,1788792300,300|53131.0,53431.0,53731.0,54031.0,54331.0,54631.0,54931.0,55231.0,55531.0,55831.0,56131.0,56431.0,56731.0,57031.0,57331.0,57631.0,57931.0,58231.0,58531.0,58831.0,59131.0,59431.0,59731.
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min raw (pass 1) differs from the origin
         got: dev.medium.orders.us-east.count,1788187200,1788792300,300|52831.0,53131.0,53431.0,53731.0,54031.0,54331.0,54631.0,54931.0,55231.0,55531.0,55831.0,56131.0,56431.0,56731.0,57031.0,57331.0,57631.0,57931.0,58231.0,58531.0,58831.0,59131.0,59431.
        want: dev.medium.orders.us-east.count,1788187500,1788792300,300|53131.0,53431.0,53731.0,54031.0,54331.0,54631.0,54931.0,55231.0,55531.0,55831.0,56131.0,56431.0,56731.0,57031.0,57331.0,57631.0,57931.0,58231.0,58531.0,58831.0,59131.0,59431.0,59731.
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min csv (pass 0) differs from the origin
         got: dev.medium.orders.us-east.count,2026-08-31 14:40:00,52831.0
        dev.medium.orders.us-east.count,2026-08-31 14:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 14:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 14:55:00,53731
        want: dev.medium.orders.us-east.count,2026-08-31 14:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 14:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 14:55:00,53731.0
        dev.medium.orders.us-east.count,2026-08-31 15:00:00,54031
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min csv (pass 1) differs from the origin
         got: dev.medium.orders.us-east.count,2026-08-31 14:40:00,52831.0
        dev.medium.orders.us-east.count,2026-08-31 14:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 14:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 14:55:00,53731
        want: dev.medium.orders.us-east.count,2026-08-31 14:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 14:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 14:55:00,53731.0
        dev.medium.orders.us-east.count,2026-08-31 15:00:00,54031
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min msgpack (pass 0) differs from the origin
         got: ���name�dev.medium.orders.us-east.count�start�j��@�end�j���step��,�values����@����@��`�@����@�<`�@�a��@�`�@���@��`�@����@��`�@�B��@�h`�@���@�`�@����@��`�@�#��@�I`�
        want: ���name�dev.medium.orders.us-east.count�start�j��l�end�j���step��,�values����@��`�@����@�<`�@�a��@�`�@���@��`�@����@��`�@�B��@�h`�@���@�`�@����@��`�@�#��@�I`�@�n��
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min msgpack (pass 1) differs from the origin
         got: ���name�dev.medium.orders.us-east.count�start�j��@�end�j���step��,�values����@����@��`�@����@�<`�@�a��@�`�@���@��`�@����@��`�@�B��@�h`�@���@�`�@����@��`�@�#��@�I`�
        want: ���name�dev.medium.orders.us-east.count�start�j��l�end�j���step��,�values����@��`�@����@�<`�@�a��@�`�@���@��`�@����@��`�@�B��@�h`�@���@�`�@����@��`�@�#��@�I`�@�n��
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min maxDataPoints=7 (pass 0) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43114.73702422145, 1788187500], [43115.77508650519, 1788274200], [43116.813148788926, 1788360900], [43117.851211072666, 1788
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43081.0, 1788220800], [43081.0, 1788307200], [43081.0, 1788393600], [43081.0, 1788480000], [43081.0, 1788566400], [43081.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min maxDataPoints=7 (pass 1) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43114.73702422145, 1788187500], [43115.77508650519, 1788274200], [43116.813148788926, 1788360900], [43117.851211072666, 1788
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43081.0, 1788220800], [43081.0, 1788307200], [43081.0, 1788393600], [43081.0, 1788480000], [43081.0, 1788566400], [43081.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min maxDataPoints=1 (pass 0) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43085.833911750124, 1788187200]]}]
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43081.0, 1788187500]]}]
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min maxDataPoints=1 (pass 1) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43085.833911750124, 1788187200]]}]
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[43081.0, 1788187500]]}]
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min maxDataPoints=1000 (pass 0) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [54031.0, 1788188400], [54931.0, 1788189300], [55831.0, 1788190200], [56731.0, 1788191100], [57631.0, 
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[54031.0, 1788188400], [54931.0, 1788189300], [55831.0, 1788190200], [56731.0, 1788191100], [57631.0, 1788192000], [58531.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min maxDataPoints=1000 (pass 1) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [54031.0, 1788188400], [54931.0, 1788189300], [55831.0, 1788190200], [56731.0, 1788191100], [57631.0, 
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[54031.0, 1788188400], [54931.0, 1788189300], [55831.0, 1788190200], [56731.0, 1788191100], [57631.0, 1788192000], [58531.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min noNullPoints (pass 0) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[52831.0, 1788187200], [53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 1788188700], [54631.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min noNullPoints (pass 1) differs from the origin
         got: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[52831.0, 1788187200], [53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 
        want: [{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 1788188700], [54631.0, 
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min pretty (pass 0) differs from the origin
         got: [
          {
            "target": "dev.medium.orders.us-east.count",
            "tags": {
              "name": "dev.medium.orders.us-east.count"
            },
            "datapoints": [
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min jsonp (pass 0) differs from the origin
         got: cb([{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[52831.0, 1788187200], [53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.
        want: cb([{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 1788188700], [54631.
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min jsonp (pass 1) differs from the origin
         got: cb([{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[52831.0, 1788187200], [53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.
        want: cb([{"target": "dev.medium.orders.us-east.count", "tags": {"name": "dev.medium.orders.us-east.count"}, "datapoints": [[53131.0, 1788187500], [53431.0, 1788187800], [53731.0, 1788188100], [54031.0, 1788188400], [54331.0, 1788188700], [54631.
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min tz (pass 0) differs from the origin
         got: dev.medium.orders.us-east.count,2026-08-31 10:40:00,52831.0
        dev.medium.orders.us-east.count,2026-08-31 10:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 10:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 10:55:00,53731
        want: dev.medium.orders.us-east.count,2026-08-31 10:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 10:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 10:55:00,53731.0
        dev.medium.orders.us-east.count,2026-08-31 11:00:00,54031
    invariant_test.go:109: dev.medium.orders.us-east.count -604801s..-5min tz (pass 1) differs from the origin
         got: dev.medium.orders.us-east.count,2026-08-31 10:40:00,52831.0
        dev.medium.orders.us-east.count,2026-08-31 10:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 10:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 10:55:00,53731
        want: dev.medium.orders.us-east.count,2026-08-31 10:45:00,53131.0
        dev.medium.orders.us-east.count,2026-08-31 10:50:00,53431.0
        dev.medium.orders.us-east.count,2026-08-31 10:55:00,53731.0
        dev.medium.orders.us-east.count,2026-08-31 11:00:00,54031
    invariant_test.go:143: 3740 responses verified byte-identical to the origin; confidence mix: exact=561 derived=374 configured=187 unknown=748
--- FAIL: TestCorrectnessInvariant (25.18s)
  3. jranson
    closed this as completedin #1133on Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    investigatingassignee is investigating to determine root cause and fixtestsaffects unit tests or integration tests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions