Skip to content

Hessian via forward-over-reverse - #85

Open
dpsanders wants to merge 1 commit into
forward-mode-adfrom
hessian
Open

dpsanders wants to merge 1 commit into
forward-mode-adfrom
hessian

Conversation

@dpsanders

Copy link
Copy Markdown
Member

Summary

Stacked on #84 (forward-mode AD), which is stacked on #83 (derivative hardening). Adds hessian(ex, vars) as the third AD entry point, reusing the same scalar rule table (unary_rule / binary_rule) as gradient and tangent.

How it works

Forward-over-reverse:

  1. gradient_code(ex, vars) — one reverse-mode pass. Yields an SSA whose variables.gradient = [g₁, …, gₙ] are symbolic handles into the code.
  2. binarize_ssa — decompose any n-ary muls the adjoint left behind (e.g. 2 * _b̄ * x) so the tangent pass can consume them.
  3. tangent_code — attach a forward-mode tangent pass on top. Produces tangent assignments for every SSA variable, including the gᵢs.
  4. Compiled (inputs, direction) → (value, gradient, gradient_tangent). Invoking it with direction eᵢ gives row i of the Hessian. The user-facing hessian closure loops over i = 1..n and assembles the n × n matrix.

API

@variables x y z
H = hessian(x^2 * y + y^3 + z^3, [x, y, z])
val, grad, M = H((2.0, 3.0, 1.0))
# val :: Float64
# grad :: Tuple{Float64, Float64, Float64}
# M :: Matrix{Float64}  (3×3, Schwarz-symmetric for C² inputs)

Also in this PR

tangent_eq is hardened to accept non-call rhs's. Reverse-mode SSA contains _c̄ := 1 (constant seed) and identity copies like _b̄ := _c̄; the previous tangent_eq assumed every rhs was a function call and errored the moment gradient SSA was used as forward-mode input. Tangent of a bare constant is 0, tangent of a bare variable is the variable's own tangent — both fall out of dotted dispatching on Real vs BasicSymbolic.

Test plan

  • 95/95 tests pass locally (68 inherited, +27 new for Hessian).
  • Closed-form check against x² + y² (diagonal [2, 2]).
  • Cross-term x·y (off-diagonal 1s).
  • x²y + y³, sin(xy) — matches hand-derived Hessian.
  • 3-variable case x² + y² + z² + xyz — dense 3×3.
  • Schwarz symmetry check on three non-trivial expressions.

Bump to v0.8.0.

Performance note

The runtime cost is n tangent-pass invocations per hessian(…)(inputs) call. Fine for the small-n symbolic regime the package targets; denser cases can compile all n tangent passes into a single SSA later.

🤖 Generated with Claude Code

Composes the two AD passes introduced in the previous PRs:

- One reverse-mode call (`gradient_code`) builds the SSA for the
  gradient.
- `binarize_ssa` decomposes any n-ary adjoints the reverse pass left
  behind (e.g. `2 * _b̄ * x`) so `tangent_eq` can consume them.
- `tangent_code` attaches a forward-mode tangent pass on top of the
  gradient SSA.
- The compiled function invokes the tangent pass `n` times at runtime
  with unit-vector directions and assembles the `n × n` Hessian matrix.

Uses exactly the same `unary_rule` / `binary_rule` table as the first
two passes — a third source of truth would be a bug farm.

Also fixes `tangent_eq` to handle non-call assignments (bare-variable
identity copies, bare-constant seeds). The reverse-mode initialization
(`_c̄ := 1`) and identity copies (`_b̄ := _c̄`) produced by
`gradient_code` are non-call rhs's; the previous `tangent_eq` required
every rhs to be a function call, which was fine for `cse_equations(ex)`
output but blew up the moment a gradient SSA fed into forward-mode.

Bumps to v0.8.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant