observed.composite.resource.spec missing fields on first reconcile causes silent resource drops

[johncarpio-sumo]

Bug Description

When using observed.composite.resource.spec in a Go template, fields that are present in the XR may not be available on the first reconcile cycle. This causes {{- with .fieldName }} blocks to silently skip, resulting in composed resources never being created.

Root Cause

On the first reconcile after XR creation, observed.composite.resource.spec may not contain all user-specified fields. The desired.composite.resource.spec always has the full spec. When combined with resources that fail quickly (triggering the circuit breaker), the template never gets a second chance to execute with the now-populated observed state.

Environment

• function-go-templating: v0.11.0
• Crossplane: v1.19.x (latest stable)
• Cluster: Kind (local development)

Steps to Reproduce

1. Create a Composition with an inline template that uses {{- with .observed.composite.resource.spec -}} and accesses multiple sub-objects via {{- with .service }}, {{- with .autoscaling }}, etc.
2. Include a resource that will fail immediately (e.g., AppAutoscaling Policy that depends on a Target that doesn't exist yet)
3. Create an XR with all fields populated

Expected Behavior

All with blocks should execute on every reconcile since the XR spec is fully populated.

Actual Behavior

• First reconcile: some with blocks don't execute because observed.composite.resource.spec is missing those fields
• The failing resource triggers the circuit breaker
• Subsequent reconciles are throttled — the template never re-executes with the complete observed state
• Result: some composed resources are never created

Evidence

• Decoder test: yaml.NewYAMLOrJSONDecoder correctly decodes 6+ documents from well-formed YAML
• convertToMap uses protojson.Marshal(req) with default options — proto3 omits zero/empty fields
• When the same template uses only 3 resources (no failing ones), all fields are available on the second reconcile (circuit breaker doesn't activate)
• Switching from observed to desired composite spec resolves the issue completely

Workaround

Use desired with fallback to observed:

{{- $spec := (.desired.composite.resource.spec | default .observed.composite.resource.spec) -}}
{{- with $spec -}}
  ...
{{- end -}}

This ensures the full user-specified spec is always available regardless of reconcile timing.

Recommendation

Consider documenting that desired.composite.resource.spec should be preferred over observed.composite.resource.spec for resource creation templates, since desired always reflects the user's intent while observed may lag behind. Alternatively, the function could pre-merge desired and observed composite state before template execution.

[johncarpio-sumo]

Additional Investigation

After deeper analysis, the YAML decoder itself is NOT the issue — tested with 6 well-formed documents and all decode correctly.

The actual root cause appears to be a race condition in the observed composite state:

1. The YAML decoder (yaml.NewYAMLOrJSONDecoder with 1024 buffer) handles 6+ documents correctly
2. convertToMap uses protojson.Marshal with default options (which omits zero-value fields)
3. On the first reconcile cycle, observed.composite.resource.spec may not contain all fields that were set in the XR's desired state — specifically, sub-objects like .service may not be present in the observed state on early reconcile passes

The failure pattern:

• First reconcile: observed.composite.resource.spec.service is absent → {{- with .service }} evaluates to nil → Service document not produced
• Subsequent reconciles: circuit breaker opens (due to other failing resources) → template never re-executes with the now-populated observed state
• Result: Service never created

Evidence:

• Without autoscaling (only 3 resources), everything converges fast enough that the second reconcile happens before circuit breaker, allowing the Service to be created
• With autoscaling (6 resources), the failing policies trigger the circuit breaker before the second reconcile can produce the Service

Workaround: Use {{- with (.desired.composite.resource.spec).service }} instead of {{- with .service }} within the observed spec scope, OR split into multiple pipeline steps so each has fewer resources and converges faster.

This may actually be a Crossplane core issue (observed state population timing) rather than a function-go-templating bug. Consider closing if the Crossplane project confirms this is expected behavior for the first reconcile cycle.

[bobh66]

Is this still a problem with the latest Crossplane and function-go-templating releases? 1.19 is pretty old and would not get a fix for the problem if it ends up being a core Crossplane issue.