Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions giga/deps/tasks/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package tasks

import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/metric"
)

// taskMetrics mirrors sei-cosmos/tasks/metrics.go's instruments of the same
// name/scope so the sei-cosmos and giga fork schedulers merge into single
// scheduler_retries/scheduler_incarnations series. Keep description/unit
// byte-identical across both declarations or the OTel SDK stops deduping.
var (
meter = otel.Meter("seicosmos_tasks")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This file is a byte-identical copy of sei-cosmos/tasks/metrics.go (same blob), but it's the only one of the three mirrored metrics files without the "same name/scope so the series merge; keep description/unit byte-identical or the OTel SDK stops deduping" comment that giga/deps/xbank/keeper/metrics.go and sei-cosmos/x/bank/keeper/metrics.go both carry. It's also the copy holding the corrected scheduler_incarnations description that must stay in lockstep with sei-cosmos — the place drift is most likely. Please add the same comment here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This file is byte-identical to sei-cosmos/tasks/metrics.go and intentionally reuses the seicosmos_tasks meter scope and instrument names so the two forks' series merge — but unlike the three bank_new_account declarations, it carries no comment saying so. The seicosmos_ scope name inside the Giga fork reads like a copy-paste slip without one. Worth mirroring the comment style used in giga/deps/xbank/keeper/metrics.go so a future description tweak on either side doesn't silently break the dedupe.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This file is a byte-identical copy of sei-cosmos/tasks/metrics.go, including the seicosmos_tasks meter scope, so both packages register the same instruments and depend on the SDK deduping them. Unlike the bank counterparts (which this PR annotates in both sei-cosmos/x/bank/keeper/metrics.go and giga/deps/xbank/keeper/metrics.go), neither scheduler copy carries the "mirrored by … keep description/unit byte-identical" comment — and the scheduler_incarnations description fix in this same PR is exactly the drift that comment prevents. Worth adding the note here and in sei-cosmos/tasks/metrics.go.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This file is byte-identical to sei-cosmos/tasks/metrics.go (same git blob 9c8d709342) and declares the same two instruments in the same seicosmos_tasks scope — so it relies on the same "keep description/unit byte-identical or the SDK stops deduping" invariant that the bank instruments got an explicit comment for. Neither tasks file says so.

This PR is itself the demonstration: it had to edit the scheduler_incarnations description in two places to keep them matching. Please add the same cross-reference comment here and in sei-cosmos/tasks/metrics.go, so the next description edit doesn't silently desync the pair into two conflicting Prometheus streams.


taskMetrics = struct {
retries metric.Int64Counter
incarnations metric.Int64Counter
}{
retries: must(meter.Int64Counter(
"scheduler_retries",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This PR establishes a good convention for bank_new_account — exported BankNewAccount* consts in all three declaring packages plus utils/metrics/bank_new_account_mirror_test.go pinning them — but the scheduler instruments don't get it. scheduler_retries / scheduler_incarnations are duplicated here and in sei-cosmos/tasks/metrics.go as inline string literals, with only a prose comment ("Keep description/unit byte-identical ... or the OTel SDK stops deduping") enforcing the invariant.

The two files are byte-identical today, so this is purely preventative — but it's exactly the drift the bank mirror test exists to catch, and the failure mode is silent (a split series, discovered on a dashboard rather than in CI). Suggest exporting SchedulerRetries* / SchedulerIncarnations* consts from both packages and extending the mirror test to cover them.

Relevant here: the scheduler_incarnations description is being changed in this very PR, which is the concrete case where the two copies could have drifted.

metric.WithDescription("Number of OCC scheduler transaction retries"),
metric.WithUnit("{count}"),
)),
incarnations: must(meter.Int64Counter(
"scheduler_incarnations",
metric.WithDescription("Sum of per-round maximum incarnations in the OCC scheduler"),
metric.WithUnit("{count}"),
)),
}
)

func must[V any](v V, err error) V {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This makes four identical copies of must[V any] in the tree (sei-cosmos/tasks, sei-cosmos/x/bank/keeper, giga/deps/xbank/keeper, and here). Harmless, but a single shared helper would be one less thing to copy the next time an OTel instrument is added.

if err != nil {
panic(err)
}
return v
}
33 changes: 29 additions & 4 deletions giga/deps/tasks/scheduler.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"runtime/debug"
"sort"
"strings"
"sync"
Expand Down Expand Up @@ -279,9 +281,32 @@ type schedulerMetrics struct {
retries int
}

func (s *scheduler) emitMetrics() {
telemetry.IncrCounter(float32(s.metrics.retries), "scheduler", "retries")
telemetry.IncrCounter(float32(s.metrics.maxIncarnation), "scheduler", "incarnations")
func (s *scheduler) emitMetrics(ctx context.Context) {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
// TODO(PLT-353): remove once scheduler_retries verified
func() {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
telemetry.IncrCounter(float32(s.metrics.retries), "scheduler", "retries")
}()
taskMetrics.retries.Add(ctx, int64(s.metrics.retries))
// TODO(PLT-353): remove once scheduler_incarnations verified
Comment thread
cursor[bot] marked this conversation as resolved.
func() {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
telemetry.IncrCounter(float32(s.metrics.maxIncarnation), "scheduler", "incarnations")
}()
taskMetrics.incarnations.Add(ctx, int64(s.metrics.maxIncarnation))
}

func (s *scheduler) ProcessAll(ctx sdk.Context, reqs []*sdk.DeliverTxEntry) ([]types.ResponseDeliverTx, error) {
Expand All @@ -295,7 +320,7 @@ func (s *scheduler) ProcessAll(ctx sdk.Context, reqs []*sdk.DeliverTxEntry) ([]t
s.conflictKeyCounts = make(map[string]int)
s.executeCh = make(chan func(), len(tasks))
s.validateCh = make(chan func(), len(tasks))
defer s.emitMetrics()
defer s.emitMetrics(ctx.Context())

// default to number of tasks if workers is negative or 0 by this point
workers := s.workers
Expand Down
71 changes: 71 additions & 0 deletions giga/deps/xbank/keeper/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
package keeper

import (
"context"
"fmt"
"os"
"runtime/debug"

"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/metric"

"github.com/sei-protocol/sei-chain/sei-cosmos/telemetry"
)

// BankNewAccount* must stay byte-identical to the same consts in
// sei-cosmos/x/bank/keeper and utils/metrics so the three Int64Counter
// declarations merge into one series.
const (
BankNewAccountMeter = "seicosmos_x_bank_keeper"
BankNewAccountName = "bank_new_account"
BankNewAccountDescription = "Number of new accounts created during bank transfers"
BankNewAccountUnit = "{count}"
)

// bankMetrics.newAccount mirrors sei-cosmos/x/bank/keeper/metrics.go's
// instrument of the same name/scope so the two dual-emit paths merge into a

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] "the two dual-emit paths" — there are three declarations of this instrument (this file, sei-cosmos/x/bank/keeper/metrics.go, and utils/metrics.bankNewAccountCounter); the other two comments correctly say "all three." More importantly, this comment doesn't name utils/metrics, so someone editing the description here would only know to check one of the two other sites. Worth listing all three, given this comment is the only mechanism enforcing the invariant.

// single bank_new_account series.
var (
meter = otel.Meter(BankNewAccountMeter)

bankMetrics = struct {
newAccount metric.Int64Counter
}{
newAccount: must(meter.Int64Counter(
BankNewAccountName,
metric.WithDescription(BankNewAccountDescription),
metric.WithUnit(BankNewAccountUnit),
)),
}
)

func must[V any](v V, err error) V {
if err != nil {
panic(err)
}
return v
}

// recordNewAccounts dual-emits the legacy new-account counter and its OTel
// counterpart (bank_new_account). Runs from consensus-critical send paths, so
// a telemetry fault here must not panic into the caller.
func recordNewAccounts(ctx context.Context, count int64) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This is a byte-for-byte copy of sei-cosmos/x/bank/keeper.recordNewAccounts, but the two live in differently-named files — metrics.go here, send.go there. For a pair of functions whose entire contract is "these must stay identical," putting them at matching paths makes the mirror relationship visible to anyone diffing the two trees. metrics.go (this file) is the better home for both, since it's where the instrument it feeds is declared.

Also note the mirror test pins the four consts but not this function body, so the dual-emit logic itself can drift between the fork and upstream without CI noticing.

if count <= 0 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] count <= 0 is unreachable — both call sites in send.go pass a literal 1, and neither sibling helper (utils/metrics.RecordBankNewAccount, the sei-cosmos inline emission) has an equivalent guard. Either drop it, or drop the count parameter and match the other two helpers' signature.

return
}
defer func() {
if e := recover(); e != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] e is bound but never used beyond the nil check, so a telemetry panic in production leaves an unattributed stack trace with no message — and since the whole point is that the fault stays invisible to the caller, there's nothing else to alert on. Logging the recovered value (seilog is already available in these packages) would make the swallowed fault diagnosable. Same in giga/deps/tasks/scheduler.go:286, sei-cosmos/tasks/scheduler.go:271, and utils/metrics/metrics_util.go:81. (The existing SafeTelemetryIncrCounter has the same gap, so this is pre-existing style — but four new copies is the moment to fix it.)

fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
// TODO(PLT-353): remove once bank_new_account verified
func() {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
telemetry.IncrCounter(float32(count), "new", "account")
}()
bankMetrics.newAccount.Add(ctx, count)
}
9 changes: 6 additions & 3 deletions giga/deps/xbank/keeper/send.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import (
"github.com/sei-protocol/sei-chain/giga/deps/xbank/types"
"github.com/sei-protocol/sei-chain/sei-cosmos/codec"
"github.com/sei-protocol/sei-chain/sei-cosmos/store/prefix"
"github.com/sei-protocol/sei-chain/sei-cosmos/telemetry"
sdk "github.com/sei-protocol/sei-chain/sei-cosmos/types"
sdkerrors "github.com/sei-protocol/sei-chain/sei-cosmos/types/errors"
cosmosbanktypes "github.com/sei-protocol/sei-chain/sei-cosmos/x/bank/types"
Expand Down Expand Up @@ -145,7 +144,9 @@ func (k BaseSendKeeper) InputOutputCoins(ctx sdk.Context, inputs []types.Input,
// such as delegated fee messages.
accExists := k.ak.HasAccount(ctx, outAddress)
if !accExists {
defer telemetry.IncrCounter(1, "new", "account")
defer func() {
recordNewAccounts(ctx.Context(), 1)
}()
k.ak.SetAccount(ctx, k.ak.NewAccountWithAddress(ctx, outAddress))
}
}
Expand All @@ -166,7 +167,9 @@ func (k BaseSendKeeper) SendCoins(ctx sdk.Context, fromAddr sdk.AccAddress, toAd
// such as delegated fee messages.
accExists := k.ak.HasAccount(ctx, toAddr)
if !accExists {
defer telemetry.IncrCounter(1, "new", "account")
defer func() {
recordNewAccounts(ctx.Context(), 1)
}()
k.ak.SetAccount(ctx, k.ak.NewAccountWithAddress(ctx, toAddr))
}

Expand Down
2 changes: 1 addition & 1 deletion precompiles/bank/bank.go
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ func (p PrecompileExecutor) sendNative(ctx sdk.Context, method *abi.Method, args
}
accExists := p.accountKeeper.HasAccount(ctx, receiverSeiAddr)
if !accExists {
defer metrics.SafeTelemetryIncrCounter(1, "new", "account")
defer metrics.RecordBankNewAccount(ctx.Context())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Worth noting alongside the Giga change: the defer here means the counter still fires when sendNative returns an error later and the EVM reverts the account creation. That's unchanged from defer metrics.SafeTelemetryIncrCounter(...), so nothing regresses — but it's the exact opposite of the new Giga InputOutputCoins behavior, which now drops the count on error. Same metric, opposite revert semantics across the two producers.

p.accountKeeper.SetAccount(ctx, p.accountKeeper.NewAccountWithAddress(ctx, receiverSeiAddr))
}

Expand Down
6 changes: 5 additions & 1 deletion sei-cosmos/tasks/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ import (
"go.opentelemetry.io/otel/metric"
)

// taskMetrics mirrors giga/deps/tasks/metrics.go's instruments of the same
// name/scope so the sei-cosmos and giga fork schedulers merge into single
// scheduler_retries/scheduler_incarnations series. Keep description/unit
// byte-identical across both declarations or the OTel SDK stops deduping.
var (
meter = otel.Meter("seicosmos_tasks")

Expand All @@ -19,7 +23,7 @@ var (
)),
incarnations: must(meter.Int64Counter(
"scheduler_incarnations",
metric.WithDescription("Maximum incarnation seen in OCC scheduler round"),
metric.WithDescription("Sum of per-round maximum incarnations in the OCC scheduler"),
metric.WithUnit("{count}"),
)),
}
Expand Down
33 changes: 27 additions & 6 deletions sei-cosmos/tasks/scheduler.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"runtime/debug"
"sort"
"strings"
"sync"
Expand Down Expand Up @@ -265,13 +267,32 @@ type schedulerMetrics struct {
retries int
}

func (s *scheduler) emitMetrics() {
taskMetrics.retries.Add(context.Background(), int64(s.metrics.retries))
func (s *scheduler) emitMetrics(ctx context.Context) {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
// TODO(PLT-353): remove once scheduler_retries verified
telemetry.IncrCounter(float32(s.metrics.retries), "scheduler", "retries")
taskMetrics.incarnations.Add(context.Background(), int64(s.metrics.maxIncarnation))
func() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This anonymous-func-wrapping-a-recover shape is added ten times across four files in this PR (twice each in both scheduler.gos, once each in both recordNewAccounts). AGENTS.md § Structural corrections is directly on point: guard at the choke point rather than at each call site, and let the step name carry the what while a doc comment carries the why.

Two concrete things:

  1. The rationale is undocumented. The inner recover isn't redundant with the outer one — it exists so a panic in the legacy telemetry.IncrCounter doesn't skip the subsequent taskMetrics.*.Add(ctx, ...). That's a load-bearing invariant and nothing in the code says it, so the next person to "simplify" this by deleting the inner recovers will silently couple the two emit paths back together.

  2. utils/metrics already has the right shape. RecordBankNewAccount reads as a clean sequence of named steps because it calls SafeTelemetryIncrCounter, which owns the recover. A telemetry.SafeIncrCounter (or a safeEmit(func())) in sei-cosmos/telemetry would let all four of these functions read the same way and would cover any dual-emit site added later, rather than relying on the next author remembering to paste the closure.

defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
telemetry.IncrCounter(float32(s.metrics.retries), "scheduler", "retries")
}()
taskMetrics.retries.Add(ctx, int64(s.metrics.retries))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The reordering plus a single recover around all four emissions makes legacy telemetry a single point of failure for the new metrics. Before this PR, taskMetrics.retries.Add ran first; now a panic from telemetry.IncrCounter on the line above (the exact failure mode SafeTelemetryIncrCounter exists to absorb) is caught, but silently drops both OTel Adds and the second legacy counter. Emitting the OTel adds first, or protecting each emission independently, keeps a legacy-telemetry fault from taking the replacement metric down with it. Same ordering in giga/deps/tasks/scheduler.go:291.

// TODO(PLT-353): remove once scheduler_incarnations verified
telemetry.IncrCounter(float32(s.metrics.maxIncarnation), "scheduler", "incarnations")
func() {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
telemetry.IncrCounter(float32(s.metrics.maxIncarnation), "scheduler", "incarnations")
}()
taskMetrics.incarnations.Add(ctx, int64(s.metrics.maxIncarnation))
Comment thread
cursor[bot] marked this conversation as resolved.
}

func (s *scheduler) ProcessAll(ctx sdk.Context, reqs []*sdk.DeliverTxEntry) ([]types.ResponseDeliverTx, error) {
Expand All @@ -285,7 +306,7 @@ func (s *scheduler) ProcessAll(ctx sdk.Context, reqs []*sdk.DeliverTxEntry) ([]t
s.conflictKeyCounts = make(map[string]int)
s.executeCh = make(chan func(), len(tasks))
s.validateCh = make(chan func(), len(tasks))
defer s.emitMetrics()
defer s.emitMetrics(ctx.Context())

// default to number of tasks if workers is negative or 0 by this point
workers := s.workers
Expand Down
22 changes: 18 additions & 4 deletions sei-cosmos/x/bank/keeper/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,18 @@ import (
"go.opentelemetry.io/otel/metric"
)

// BankNewAccount* must stay byte-identical to the same consts in
// utils/metrics and giga/deps/xbank/keeper so the three Int64Counter
// declarations merge into one series.
const (
BankNewAccountMeter = "seicosmos_x_bank_keeper"
BankNewAccountName = "bank_new_account"
BankNewAccountDescription = "Number of new accounts created during bank transfers"
BankNewAccountUnit = "{count}"
)

var (
meter = otel.Meter("seicosmos_x_bank_keeper")
meter = otel.Meter(BankNewAccountMeter)

bankMetrics = struct {
sendAmount metric.Int64Gauge
Expand All @@ -17,10 +27,14 @@ var (
metric.WithDescription("Amount sent in the last MsgSend transaction by denomination"),
metric.WithUnit("{utoken}"),
)),
// newAccount is mirrored by utils/metrics.bankNewAccountCounter (bank
// precompiles) and giga/deps/xbank/keeper.bankMetrics.newAccount (the
// Giga fork) on the same name/scope, so all three merge into one
// bank_new_account series.
newAccount: must(meter.Int64Counter(
"bank_new_account",
metric.WithDescription("Number of new accounts created during bank transfers"),
metric.WithUnit("{count}"),
BankNewAccountName,
metric.WithDescription(BankNewAccountDescription),
metric.WithUnit(BankNewAccountUnit),
)),
}
)
Expand Down
36 changes: 30 additions & 6 deletions sei-cosmos/x/bank/keeper/send.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ package keeper

import (
"bytes"
"context"
"fmt"
"os"
"runtime/debug"
"strings"

"github.com/sei-protocol/sei-chain/sei-cosmos/codec"
Expand Down Expand Up @@ -146,9 +150,7 @@ func (k BaseSendKeeper) InputOutputCoins(ctx sdk.Context, inputs []types.Input,
accExists := k.ak.HasAccount(ctx, outAddress)
if !accExists {
defer func() {
bankMetrics.newAccount.Add(ctx.Context(), 1)
// TODO(PLT-353): remove once bank_new_account verified
telemetry.IncrCounter(1, "new", "account")
recordNewAccounts(ctx.Context(), 1)
}()
k.ak.SetAccount(ctx, k.ak.NewAccountWithAddress(ctx, outAddress))
}
Expand All @@ -171,16 +173,38 @@ func (k BaseSendKeeper) SendCoins(ctx sdk.Context, fromAddr sdk.AccAddress, toAd
accExists := k.ak.HasAccount(ctx, toAddr)
if !accExists {
defer func() {
bankMetrics.newAccount.Add(ctx.Context(), 1)
// TODO(PLT-353): remove once bank_new_account verified
telemetry.IncrCounter(1, "new", "account")
recordNewAccounts(ctx.Context(), 1)
}()
k.ak.SetAccount(ctx, k.ak.NewAccountWithAddress(ctx, toAddr))
}

return nil
}

// recordNewAccounts dual-emits the legacy new-account counter and its OTel
// counterpart (bank_new_account). Runs from consensus-critical send paths, so
// a telemetry fault here must not panic into the caller.
func recordNewAccounts(ctx context.Context, count int64) {
if count <= 0 {
return
}
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
// TODO(PLT-353): remove once bank_new_account verified
func() {
defer func() {
if e := recover(); e != nil {
fmt.Fprintf(os.Stderr, "telemetry panic: %v\n%s", e, debug.Stack())
}
}()
telemetry.IncrCounter(float32(count), "new", "account")
}()
bankMetrics.newAccount.Add(ctx, count)
}

func (k BaseSendKeeper) SendCoinsWithoutAccCreation(ctx sdk.Context, fromAddr sdk.AccAddress, toAddr sdk.AccAddress, amt sdk.Coins) error {
return k.sendCoinsWithoutAccCreation(ctx, fromAddr, toAddr, amt, true)
}
Expand Down
Loading
Loading