Propose a gradient step
Starting from feasible weights, take a task-loss gradient step and scale it to the local trust region.
IFAC World Congress 2026 / Oral presentation
SCPOSampling-Based
Weight-Space Projection
Improve the policy.
Preserve feasibility.
SCPO projects each policy update into a locally feasible region of weight space built from sampled safety evaluations. No analytic gradients of the constraints are needed.
Constrain the learning stepWeight space
01 / StartA known feasible policy
02 / SampleEvaluate nearby updates
03 / ProjectPreserve the safety constraints
The question
What if the learning signal
points somewhere unsafe?
A task objective can reward behavior that violates constraints. When demonstrations are harmful, better imitation can mean a worse controller.
SCPO constrains the update itself. It evaluates the safety of candidate weight changes, builds a conservative local feasible region from those evaluations, and projects the raw gradient step into it.
Only the task loss needs gradients. Safety constraints are evaluated pointwise, so no constraint Jacobian is required.
Explore the geometry
Step through
a toy run of SCPO.
At each iteration the raw gradient step is projected into the region certified from the candidate bank, and the accepted update becomes a candidate for the next iteration. Step through the run to watch the bank form and the iterate stop at the constraint boundary.
Computing the sampled bound…
A two-parameter toy of Equation (29) with a known curvature bound. The task loss is a quadratic whose minimum lies outside the feasible disk, so every raw step points toward infeasible weights. The green region is derived from evaluations, not a convex hull of the candidate points.
Loss at current weights—L(θₖ), decreasing over the run
Constraint at current weights—Feasible when g ≤ 0
Distance from raw step—‖Δθproj − Δθraw‖
Accepted step length—After Armijo backtracking
| Candidate | g(θₖ + candidate) | Projected coefficient |
|---|
What is computed in this example?
The toy constraint is g(θ) = ½[(θ₁ + 0.25)² + (θ₂ − 0.15)²] − 0.405 ≤ 0. Its Hessian is the identity, so ℓ = 1 is an exact curvature bound; the conservativeness slider scales ℓ up to 3. The toy loss is L(θ) = ½(θ − θ*)ᵀA(θ − θ*), with A having eigenvalues 1 and 8 and the minimum θ* outside the feasible disk. The raw step is Δθraw = −η∇L(θₖ), clipped to a trust radius of 0.5, within which the smoothness bound is used.
The bank at iteration k is D = [Δθₖ₋₁, Δθraw]: the previous accepted update and the current raw step. When the two are nearly collinear the bank drops to the raw step alone. With sampled values Gᵢ = g(θₖ + dᵢ), S = DᵀD, and g₀ = g(θₖ), the green region is the set of updates Dξ inside the trust ball that satisfy Equation (29):
(1 − 1ᵀξ)g₀ + Gξ + ½ℓ [ξᵀSξ + |ξ|ᵀdiag(S)] ≤ 0.
The projected step minimizes ‖Dξ − Δθraw‖² over this region; coefficients may be negative, so the region is not a convex hull of the candidates. Armijo backtracking then shortens the step if needed. The region is convex and contains the origin, so a shortened step stays certified. The accepted update becomes the next iteration’s first candidate.
Only point evaluations of g and the bound ℓ enter the construction; the true feasible disk is drawn for comparison only. In two dimensions a single disk constraint leaves no certified direction once the iterate reaches the boundary, so the steps shrink to zero there. In a high-dimensional weight space, far more directions can decrease the loss without pushing on an active constraint.
The method
Evaluate. Approximate.
Project. Check.
Each update is chosen from the span of a small bank of candidate updates, so the projection is solved over a few coefficients rather than over every network weight.
Evaluate safety
Evaluate the constraints at a bank of candidate weight changes, and combine the samples with local smoothness bounds to form a feasible region.
Solve the projection
Solve a small convex QCQP for the feasible combination of candidates closest to the raw step.
Validate and accept
Backtrack until the loss decreases and re-check the constraints. If the check fails, tighten the bounds and re-solve.
Work in an update subspace
Δθ = Dξ
The columns of D are sampled candidate updates; SCPO solves for their coefficients ξ. The size of the projection problem is set by the number of candidates and constraints, not by the number of network weights.
See Section 5 and Algorithm 1 ↗A safe starting point for control
A frozen backup.
A trainable residual.
For control tasks, a zero-initialized residual policy is added to a stabilizing backup controller, so training starts from a controller that is exactly the backup. Near the target, the backup acts alone.
Guarantees
Feasibility by induction.
Descent by construction.
Feasibility: starting from a feasible policy, every accepted iterate satisfies the constraints, provided the local smoothness bounds are valid.
Descent: a nonzero projected step is a descent direction for the task loss, and Armijo backtracking guarantees decrease under standard smoothness assumptions.
Control stability: when the input, invariance, and Lyapunov-margin conditions hold on the target region, the closed loop remains invariant and locally asymptotically stable.
The evidence
Learning under
harmful supervision.
Two experiments test whether projection rejects harmful updates while still letting the policy change.
01
Constrained regression
Fit a reference function that sometimes exceeds the allowed output range.
±1.4 output bounds64 validation grid points[−3, 3] constrained input interval
Safety is checked on a uniform grid of 64 points over [−3, 3]. The soft-penalty baseline uses constraint gradients; SCPO does not.
02
Double-integrator imitation
Adapt a stabilizing backup controller toward a demonstrator, including one that drives the state away from the origin.
2 demonstrators3 initial states25 training epochs
Unsafe teacher: a diverging proportional controller
The demonstrator pushes the state away from the origin. Over training, the learned policy moves toward the demonstrator’s behavior, yet every displayed rollout still converges to the origin.
Gray: safe controller, the backup (a stabilizing proportional controller). Orange dashed: demonstrator (a diverging proportional controller for the unsafe teacher; an LQR controller for the safe teacher). Blue: trained policy. Each frame shows closed-loop rollouts from three initial states at one training epoch. This experiment is from the IFAC presentation and does not appear in the arXiv manuscript.
Perspective
A constrained update.
An adaptable policy.
SCPO connects projected gradient learning with rollout-based constraint evaluation. Its local geometry comes from candidate updates and safety metrics in weight space.
The projection can be conservative. It recombines directions already in the sample bank and may return a zero update when that bank offers no feasible descent direction.
The paper identifies better safe exploration and safety-validation design as directions for future work.
Read & build on the work
Explore SCPO.
The manuscript contains the full derivation, the guarantees with proofs, and the experiments.
01 / arXiv
Abstract page ↗
Abstract, version history, and metadata. Accepted to IFAC World Congress 2026.
02 / PDFFull manuscript ↗
Method, guarantees, experiments, and proofs.
Cite this work
Constrained Policy Optimization via Sampling-Based Weight-Space Projection
Shengfan Cao, Francesco Borrelli, and Eunhyek Joa
arXiv preprint, v3 (May 2026). Accepted for publication at IFAC World Congress 2026.
@misc{cao2025scpo,
title = {Constrained Policy Optimization via
Sampling-Based Weight-Space Projection},
author = {Shengfan Cao and Francesco Borrelli and Eunhyek Joa},
year = {2025},
eprint = {2512.13788},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2512.13788},
note = {Accepted for publication at IFAC World Congress 2026}
}