Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,20 @@ Each minor release also has a narrative overview in `docs/new-changed-<version>.

## [Unreleased]

### Added

- Any value in a configuration file can reference an environment variable, whatever its type, with `${NAME}`, `${NAME:-default}` or `${NAME:?message}`. A value that is only a reference to an unset variable leaves the setting at its default. See [Environment Variable References](./docs/configuring.md#environment-variable-references).

### Changed

- A literal `${` in a configuration value now needs to be written `$${`. Inside `request_rewriters` instructions, only `TRK_`-prefixed variables are expanded, so capture tokens like `${1}` and `${site}` are unchanged.
- The config view shows `<secret>` instead of `*****` for a Redis `password` and the Graphite `origin_password` and `origin_authorization`.

### Fixed

- `${VAR}` references were never expanded, even in the fields documented to support them, such as a Redis `password` or authenticator `users`.
- The config view showed a user router's `to_credential`, discovery `password`, `bearer_token` and headers, tracing headers and credential-named `request_params` in plain text.

## [2.2.0] - 2026-10-26

See [What's New in Trickster 2.2](./docs/new-changed-2.2.md) for an overview.
Expand Down
41 changes: 35 additions & 6 deletions docs/configuring.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,20 +65,49 @@ current key is not set; configuration dumps show only the current key. Setting b
| `backfill_tolerance_points` | `volatile_window_points` |
| `fastforward_ttl` | `partial_bucket_ttl` |

### Configuring Secrets or Sensitive Information
### Environment Variable References

Trickster supports Environment variable substitution in its configuration file where sensitive information is expected.
- Supported via the following fields:
- `caches[*].redis.password`, `backends[*].healthcheck.headers`, `backends[*].cors.headers`, `backends[*].paths[*].cors.headers`, `backends[*].paths[*].request_headers`, `backends[*].paths[*].request_params`, `backends[*].paths[*].response_headers`, `backends[*].alb.user_router.users[*].to_credential`, `authenticators[*].users`, `discovery[*].headers`
Any value in a configuration file can reference an environment variable, whatever the field's type:

Usage `${ENV_VAR_NAME}`, example:
```yaml
frontend:
listen_port: ${TRK_PORT:-8480}
caches:
default:
provider: redis
redis:
password: "${MY_REDIS_PW}"
endpoint: ${REDIS_HOST}:6379
password: "${MY_REDIS_PW:?the Redis password is required}"
backends:
default:
max_ttl: ${MAX_TTL}
```

| Form | Value |
| --- | --- |
| `${NAME}` | The value of `NAME`. If `NAME` is not set, see below. |
| `${NAME:-default}` | The value of `NAME`, or `default` when `NAME` is unset or empty. |
| `${NAME:?message}` | The value of `NAME`. When it is unset or empty, loading fails with an error that names `NAME` and includes `message`. |
| `$${` | A literal `${`: `"$${NOT_A_VAR}"` loads as `${NOT_A_VAR}`. |

- **Unset variables.** When a value is just `${NAME}` and `NAME` is not set, the setting is left unset, so it keeps its default. Inside a longer value, or as an item in a list, an unset variable expands to an empty string. Trickster logs a warning naming each unset variable at startup.
- **Types.** An unquoted reference takes the type of what it expands to, so `listen_port: ${PORT}` loads as a number and `enabled: ${ON}` as a boolean. Quoting the reference (`"${PORT}"`) always makes it a string. A value that expands to `null` or `~` stays a string. Inside a `[...]` or `{...}` flow collection, YAML requires the reference to be quoted.
- **Names.** Only the braced form is expanded, and only for valid variable names (letters, digits and `_`, not starting with a digit). A bare `$`, as in `$NAME` or a bcrypt hash like `$2y$10$...`, is kept as written. Defaults and messages are literal text and cannot contain `${` or `}`.
- **Single pass.** A variable's value is used as is; a `${...}` inside it is not expanded again.
- **Keys.** Mapping keys, such as backend and cache names, are never expanded.
- **Request rewriters.** In `request_rewriters[*].instructions`, `${0}`, `${1}` and `${name}` are the rewriter's own capture tokens. There, only variables whose names start with `TRK_` are expanded, such as `${TRK_ORIGIN_DOMAIN}`, and every other `${...}` is left for the rewriter.
- **Kubernetes.** Configuration that the Gateway/Ingress controller generates from cluster objects is never expanded, so a route cannot read Trickster's environment.

### Configuring Secrets or Sensitive Information

Supply credentials through environment variable references rather than writing them into the file. Credential fields are redacted wherever Trickster shows its configuration, such as the management config endpoint:
- `<secret>` replaces these: `caches[*].redis.password`, `backends[*].alb.user_router.users[*].to_credential`, `backends[*].graphite.origin_password` and `origin_authorization`, `discovery[*].http.password` and `bearer_token`, and the other fields that hold keys and tokens.
- `*****` replaces these:
- every value of `discovery[*].http.headers` and `tracing[*].headers`
- path, CORS and health check header and `request_params` values whose names look like credentials (`Authorization`, `Cookie`, or a name containing `token`, `key`, `secret`, `password` or `auth`, for example)
- authenticator `users`
- passwords embedded in URLs

## Environment Variables

Trickster will then check for and evaluate the following Environment Variables:
Expand Down
9 changes: 4 additions & 5 deletions docs/kubernetes-deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,11 +155,10 @@ the environment.

### Secrets

Credentials never go in a ConfigMap. The configuration expands `${VARIABLE}`
in its credential-bearing fields, listed under
[Configuring Secrets](./configuring.md#configuring-secrets-or-sensitive-information):
authenticator users, a Redis `password`, a user router's `to_credential`, and
header values in path, CORS, health check and discovery blocks. Mount a
Credentials never go in a ConfigMap. Any configuration value can be a
[`${VARIABLE}` reference](./configuring.md#environment-variable-references),
such as authenticator users, a Redis `password`, a user router's
`to_credential`, or a header value. Mount a
Secret into the environment
(`envFrom.secretRef` in `gateway-deployment.yaml`) and reference its keys.
A container's environment is fixed at start, so a rotated Secret reaches
Expand Down
8 changes: 3 additions & 5 deletions docs/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -570,11 +570,9 @@ The global `TRK_ORIGIN_URL` and `TRK_ORIGIN_TYPE` variables map only to the
default backend and are suitable only for a basic single-backend deployment.
There are no one-to-one `TRK_*` variables for MySQL listener limits, backend
limits, TLS, health, or User Router options; configure those in YAML.
Authenticator `users` values support `${VARIABLE}` expansion, and supported
users files and secret volume mounts are preferred for production. The
`origin_url` field does not expand `${VARIABLE}` in structured YAML; store a
configuration containing an embedded DSN as a protected secret, not a public
ConfigMap.
Every YAML value, including authenticator `users` and an `origin_url` with an
embedded DSN password, supports [`${VARIABLE}` references](configuring.md#environment-variable-references).
Supported users files and secret volume mounts are preferred for production.

Validate before rollout:

Expand Down
2 changes: 1 addition & 1 deletion docs/paths.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ As an example, if the client request provides a `Cache-Control: no-store` header

##### Environment Variable Substitution

The `request_headers`, `request_params` and `response_headers` sections support environment variable substitution. This means you can use environment variables in your header or query parameter values. Example:
Like every configuration value, `request_headers`, `request_params` and `response_headers` values can [reference environment variables](./configuring.md#environment-variable-references). Example:

```yaml
backends:
Expand Down
3 changes: 1 addition & 2 deletions integration/moq_webtransport_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ import (
"github.com/trickstercache/trickster/v2/pkg/backends/providers"
tkconfig "github.com/trickstercache/trickster/v2/pkg/config"
listenerconfig "github.com/trickstercache/trickster/v2/pkg/config/listener"
configtypes "github.com/trickstercache/trickster/v2/pkg/config/types"
"github.com/trickstercache/trickster/v2/pkg/moq/codec"
"github.com/trickstercache/trickster/v2/pkg/moq/message"
"github.com/trickstercache/trickster/v2/pkg/moq/session"
Expand Down Expand Up @@ -75,7 +74,7 @@ func moqWebTransportHarness(t *testing.T, origin *moqOrigin, site string, authen
if authenticated {
b.AuthenticatorName = moqWebBackend
c.Authenticators[moqWebBackend] = &autho.Options{
Provider: "basic", Users: configtypes.EnvStringMap{moqWebUser: moqWebSecret},
Provider: "basic", Users: map[string]string{moqWebUser: moqWebSecret},
}
}
c.Backends[moqWebBackend] = b
Expand Down
3 changes: 1 addition & 2 deletions integration/pgwire_conformance_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ import (
"github.com/trickstercache/trickster/v2/pkg/backends/providers"
tkconfig "github.com/trickstercache/trickster/v2/pkg/config"
"github.com/trickstercache/trickster/v2/pkg/config/listener"
configtypes "github.com/trickstercache/trickster/v2/pkg/config/types"
autho "github.com/trickstercache/trickster/v2/pkg/proxy/authenticator/options"

"github.com/jackc/pgx/v5/pgconn"
Expand Down Expand Up @@ -269,7 +268,7 @@ func pgwireHarness(t *testing.T, target pgwireTarget) (tricksterHarness, string)
backend.CacheName = "mem1"
backend.AuthenticatorName = pgwireConformanceName
c.Authenticators[pgwireConformanceName] = &autho.Options{
Provider: "basic", Users: configtypes.EnvStringMap{
Provider: "basic", Users: map[string]string{
target.ClientUser: target.ClientPassword, pgwireSecondUser: target.ClientPassword,
},
}
Expand Down
4 changes: 2 additions & 2 deletions pkg/backends/alb/mech/ur/options/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import (
"maps"
"net/http"

"github.com/trickstercache/trickster/v2/pkg/config/types"
"github.com/trickstercache/trickster/v2/pkg/secret"
"github.com/trickstercache/trickster/v2/pkg/util/pointers"
"github.com/trickstercache/trickster/v2/pkg/util/sets"

Expand Down Expand Up @@ -60,7 +60,7 @@ type UserMappingOptions struct {
// ToUser is the User Name that will be substituted in the upstream request
ToUser string `yaml:"to_user"`
// ToCredential is the Credential that will be substituted in the upstream request
ToCredential types.EnvString `yaml:"to_credential"`
ToCredential secret.Secret `yaml:"to_credential,omitempty"`
}

type UserMappingOptionsByUser map[string]*UserMappingOptions
Expand Down
3 changes: 2 additions & 1 deletion pkg/backends/graphite/health_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (
ho "github.com/trickstercache/trickster/v2/pkg/backends/healthcheck/options"
bo "github.com/trickstercache/trickster/v2/pkg/backends/options"
"github.com/trickstercache/trickster/v2/pkg/proxy/headers"
"github.com/trickstercache/trickster/v2/pkg/secret"

"github.com/stretchr/testify/require"
)
Expand Down Expand Up @@ -111,7 +112,7 @@ func TestFinalizeHealthCheckOptionsCasingCollisions(t *testing.T) {
newClient := func(t *testing.T, originAuth string) *Client {
o := bo.New()
o.Graphite = gro.New()
o.Graphite.OriginAuthorization = originAuth
o.Graphite.OriginAuthorization = secret.Secret(originAuth)
b, err := NewClient("test", o, nil, nil, nil, nil)
require.NoError(t, err)
return b.(*Client)
Expand Down
7 changes: 4 additions & 3 deletions pkg/backends/graphite/options/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"slices"

"github.com/trickstercache/trickster/v2/pkg/parsing/timeconv"
"github.com/trickstercache/trickster/v2/pkg/secret"
"github.com/trickstercache/trickster/v2/pkg/util/pointers"

"go.yaml.in/yaml/v3"
Expand All @@ -33,11 +34,11 @@ type Options struct {
TimeZone string `yaml:"time_zone,omitempty"`
// OriginUsername and OriginPassword form a static HTTP Basic credential
// every request to the origin carries, proxied and synthetic alike
OriginUsername string `yaml:"origin_username,omitempty"`
OriginPassword string `yaml:"origin_password,omitempty"`
OriginUsername string `yaml:"origin_username,omitempty"`
OriginPassword secret.Secret `yaml:"origin_password,omitempty"`
// OriginAuthorization is a verbatim Authorization header value (e.g.,
// 'Bearer <token>'); mutually exclusive with OriginUsername/OriginPassword
OriginAuthorization string `yaml:"origin_authorization,omitempty"`
OriginAuthorization secret.Secret `yaml:"origin_authorization,omitempty"`
// PassthroughMaxDataPoints forwards the client's maxDataPoints to the origin
// instead of consolidating in Trickster; such requests are not accelerated
PassthroughMaxDataPoints bool `yaml:"passthrough_max_data_points,omitempty"`
Expand Down
4 changes: 2 additions & 2 deletions pkg/backends/graphite/options/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func (o *Options) AuthHeader() (string, error) {
if o.OriginUsername != "" || o.OriginPassword != "" {
return "", ErrOriginAuthConflict
}
return o.OriginAuthorization, nil
return string(o.OriginAuthorization), nil
}
if o.OriginUsername == "" {
if o.OriginPassword != "" {
Expand All @@ -60,7 +60,7 @@ func (o *Options) AuthHeader() (string, error) {
return "", nil
}
return "Basic " + base64.StdEncoding.EncodeToString(
[]byte(o.OriginUsername+":"+o.OriginPassword)), nil
[]byte(o.OriginUsername+":"+string(o.OriginPassword))), nil
}

// ValidateWithPaths checks the declarative origin-auth option combinations,
Expand Down
3 changes: 2 additions & 1 deletion pkg/backends/graphite/originauth_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
"github.com/trickstercache/trickster/v2/pkg/proxy/methods"
po "github.com/trickstercache/trickster/v2/pkg/proxy/paths/options"
"github.com/trickstercache/trickster/v2/pkg/proxy/request"
"github.com/trickstercache/trickster/v2/pkg/secret"
)

func TestOriginAuthHeader(t *testing.T) {
Expand Down Expand Up @@ -70,7 +71,7 @@ func TestOriginAuthInjection(t *testing.T) {
o := bo.New()
o.Graphite = gro.New()
o.Graphite.OriginUsername = user
o.Graphite.OriginPassword = pass
o.Graphite.OriginPassword = secret.Secret(pass)
return o
}

Expand Down
4 changes: 2 additions & 2 deletions pkg/backends/healthcheck/options/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ type Options struct {
// Query provides the HTTP query parameters to use when making an upstream health check
Query string `yaml:"query,omitempty"`
// Headers provides the HTTP Headers to apply when making an upstream health check
Headers types.EnvStringMap `yaml:"headers,omitempty"`
Headers map[string]string `yaml:"headers,omitempty"`
// Body provides a body to apply when making an upstream health check request
Body string `yaml:"body,omitempty"`
// Timeout is the amount of time a health check probe should wait for a response
Expand Down Expand Up @@ -135,7 +135,7 @@ func (o *Options) Validate() (bool, error) {
func (o *Options) Clone() *Options {
c := pointers.Clone(o)
if o.Headers != nil {
c.Headers = types.EnvStringMap(headers.Lookup(o.Headers).Clone())
c.Headers = map[string]string(headers.Lookup(o.Headers).Clone())
}
if o.ExpectedHeaders != nil {
c.ExpectedHeaders = headers.Lookup(o.ExpectedHeaders).Clone()
Expand Down
3 changes: 1 addition & 2 deletions pkg/backends/healthcheck/options/options_validate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ import (
"testing"
"time"

ct "github.com/trickstercache/trickster/v2/pkg/config/types"
"github.com/trickstercache/trickster/v2/pkg/parsing/timeconv"
"github.com/trickstercache/trickster/v2/pkg/proxy/headers"
)
Expand Down Expand Up @@ -102,7 +101,7 @@ func TestCloneCopiesSlicesAndMaps(t *testing.T) {
o := &Options{
Verb: "GET",
ExpectedCodes: []int{200},
Headers: ct.EnvStringMap{"X-Test": "1"},
Headers: map[string]string{"X-Test": "1"},
ExpectedHeaders: map[string]string{headers.NameContentType: "text/plain"},
}
cl := o.Clone()
Expand Down
9 changes: 4 additions & 5 deletions pkg/backends/mysql/mysql_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ import (
trickstercache "github.com/trickstercache/trickster/v2/pkg/cache"
cacheoptions "github.com/trickstercache/trickster/v2/pkg/cache/options"
cachestatus "github.com/trickstercache/trickster/v2/pkg/cache/status"
configtypes "github.com/trickstercache/trickster/v2/pkg/config/types"
"github.com/trickstercache/trickster/v2/pkg/observability/metrics"
"github.com/trickstercache/trickster/v2/pkg/parsing/sqlanalyzer"
autho "github.com/trickstercache/trickster/v2/pkg/proxy/authenticator/options"
Expand All @@ -60,7 +59,7 @@ func validBackendOptions() *bo.Options {
o.OriginURL = "mysql://origin:origin-password@db.example/trickster"
o.AuthenticatorName = "mysql-clients"
o.AuthOptions = &autho.Options{
Users: configtypes.EnvStringMap{"client": "client-password"},
Users: map[string]string{"client": "client-password"},
}
return o
}
Expand Down Expand Up @@ -124,13 +123,13 @@ func TestProtocolConfigurationErrors(t *testing.T) {
{"observe only", func(o *bo.Options) { o.AuthOptions.ObserveOnly = true }, "cannot be observe_only"},
{"no users", func(o *bo.Options) { o.AuthOptions.Users = nil }, "has no users"},
{"empty username", func(o *bo.Options) {
o.AuthOptions.Users = configtypes.EnvStringMap{"": "password"}
o.AuthOptions.Users = map[string]string{"": "password"}
}, "cannot be empty"},
{"empty password", func(o *bo.Options) {
o.AuthOptions.Users = configtypes.EnvStringMap{"client": ""}
o.AuthOptions.Users = map[string]string{"client": ""}
}, "cannot be empty"},
{"password hash", func(o *bo.Options) {
o.AuthOptions.Users = configtypes.EnvStringMap{"client": "$2a$10$abcdefghijklmnopqrstuv"}
o.AuthOptions.Users = map[string]string{"client": "$2a$10$abcdefghijklmnopqrstuv"}
}, "requires a plaintext credential"},
{"missing users file", func(o *bo.Options) {
o.AuthOptions.UsersFile = "/definitely/not/a/mysql-users-file"
Expand Down
5 changes: 2 additions & 3 deletions pkg/backends/mysql/native_listener_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ import (
"github.com/trickstercache/trickster/v2/pkg/backends/providers"
"github.com/trickstercache/trickster/v2/pkg/config"
listenerconfig "github.com/trickstercache/trickster/v2/pkg/config/listener"
configtypes "github.com/trickstercache/trickster/v2/pkg/config/types"
autho "github.com/trickstercache/trickster/v2/pkg/proxy/authenticator/options"
"github.com/trickstercache/trickster/v2/pkg/proxy/listener/native"
)
Expand Down Expand Up @@ -431,7 +430,7 @@ func routedRestartTestConfig() *config.Config {
}
router.AuthenticatorName = "mysql-listener-clients"
router.AuthOptions = &autho.Options{
Users: configtypes.EnvStringMap{"alice": "alice-password"},
Users: map[string]string{"alice": "alice-password"},
}

newTarget := func(name string) *bo.Options {
Expand All @@ -441,7 +440,7 @@ func routedRestartTestConfig() *config.Config {
target.OriginURL = "mysql://origin:password@127.0.0.1/database"
target.AuthenticatorName = "mysql-clients"
target.AuthOptions = &autho.Options{
Users: configtypes.EnvStringMap{"alice": "alice-password"},
Users: map[string]string{"alice": "alice-password"},
}
return target
}
Expand Down
Loading
Loading