Decompositions transform complex PyTorch operations into simpler, primitive operations. They serve as a bridge between high-level PyTorch APIs and lower-level representations that backends can handle.
Key benefits:
- Simplification: Complex ops become sequences of basic ops
- Backend compatibility: Backends only need to implement core operations
- Optimization: Simpler ops enable better analysis and optimization
PyTorch Op → Decomposition → Simpler Ops → Backend IR → Compiled Code
The @register_decomposition decorator registers a function as the decomposition rule for specific operations. It extends PyTorch's decomposition system.
These are defined in decompositions.py:
aten.as_strided.defaultaten.topk.defaultaten._scaled_dot_product_fused_attention_overrideable.defaultaten.scalar_tensor.defaultaten.index_copy.defaultaten.nonzero_static.defaultaten.linalg_vector_norm.defaultaten.sigmoid_backward.defaultaten.linear_backward
The base decomposition table is built using torch._decomp.get_decompositions():
from torch._decomp import get_decompositions
neuron_decompositions = get_decompositions([
aten._unsafe_index,
aten.split,
aten.unbind,
aten.embedding,
# ... more ops
])This includes decompositions for:
aten.split,aten.unbind,aten.embeddingaten.native_layer_norm_backward,aten.nll_loss_backwardaten.gelu_backward,aten.softplus_backward- And more
To add additional PyTorch decompositions, add the op to the list in get_decompositions() within decompositions.py:
neuron_decompositions = get_decompositions([
# Existing ops...
aten.my_new_op, # Add new op here
])See PyTorch decompositions for available decompositions.
Some decompositions may be candidates for upstreaming to PyTorch if they are generally useful across backends. Backend-specific decompositions (e.g., those optimized for Neuron hardware) will remain in this repository.
Decompositions live in:
torch_neuronx/neuron_dynamo_backend/decompositions.py
Simply annotate your decomposition function with @register_decomposition:
from torch_neuronx.neuron_dynamo_backend.decompositions import register_decomposition
@register_decomposition([aten.my_op.default])
def my_op_decomposition(x, y):
"""Decompose my_op into simpler operations."""
return x + yThe decorator:
- Registers the function as the decomposition for the specified op(s)
- Accepts a single op or list of ops
- Raises
ValueErrorif the op is already registered
@register_decomposition([aten.scalar_tensor.default])
def scalar_tensor(
value,
dtype=None,
layout=None,
device=None,
pin_memory=None,
):
"""Decompose scalar_tensor to full() for better StableHLO support."""
return torch.full((), value, dtype=dtype, device=device)@register_decomposition([aten.index_copy.default])
def index_copy(
input: torch.Tensor, dim: int, index: torch.Tensor, source: torch.Tensor
) -> torch.Tensor:
"""Decompose index_copy into functional scatter."""
if dim < 0:
dim = input.dim() + dim
target_shape = [1] * source.dim()
target_shape[dim] = index.numel()
index_expanded = index.view(target_shape).expand(source.shape)
return torch.scatter(input, dim, index_expanded, source)@register_decomposition([aten.sigmoid_backward.default])
def sigmoid_backward_decomposition(grad_output, output):
"""Decompose sigmoid_backward into primitive ops."""
return grad_output * (output * (1 - output))When writing decompositions, follow these guidelines to ensure correctness and compatibility:
Note: When implementing op decompositions, you must strictly follow the pattern used in PyTorch meta registration. This ensures consistency with PyTorch's internal decomposition system and prevents unexpected behavior.
If the op you want to implement doesn't have a meta registration, you must also create and register one. Use
register_fake(also known asregister_meta) to define how the op behaves on meta tensors (computing output shapes/dtypes without actual data):from torch.library import register_fake @register_fake("mylib::my_op") def my_op_fake(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: """Meta implementation: compute output shape/dtype without real computation.""" # Return a tensor with the correct shape and dtype return x.new_empty(x.shape)For more details, see PyTorch custom_ops.py.
if dim < 0:
dim = input.dim() + dimMixing CPU and neuron tensors can result in errors under Dynamo.
device = input_tensor.device
idx = torch.zeros(size, dtype=torch.long, device=device)# Let PyTorch handle cases we don't need to decompose
if dim == x.ndim - 1 and largest and sorted:
return NotImplementedSee tests/neuron_dynamo_backend/unit/test_decompositions.py for existing test patterns.