# Profiling and Performance

`benchmarks/profile_pipeline.py` profiles a full `run_simulation` call
rather than a micro-benchmark of one kernel, because the relevant costs
are distributional: which stage dominates depends on N, on how often
sticking fails, and on how many retries a regime forces. It rolls
`cProfile` output up into the pipeline's own stages, so the report
identifies what to optimize rather than only what is hot.

```
devenv shell -- uv run python benchmarks/profile_pipeline.py --n 1024 --regime medium
```

## Findings

The first profiling pass found four separate problems, none of them in
the overlap kernels (already JIT-compiled, and never the bottleneck).

### `np.cross` on 3-vectors costs 21 µs

`np.cross` supports 2- and 3-component inputs over arbitrary axes with
broadcasting, and pays for that generality on every call. Against three
lines of explicit component arithmetic:

| Operation | Time | Ratio |
|---|---:|---:|
| `np.cross(vec3, vec3)` | 21.0 µs | — |
| explicit `cross3` | 1.7 µs | 12.4x |
| `np.linalg.norm(vec3)` | 1.28 µs | — |
| `math.sqrt` inline | 0.63 µs | 2.0x |

The CCA sticking path made ~19,500 `norm` and ~1,550 `cross` calls per
N=512 aggregate, all on plain 3-vectors. `geometry.norm3`/`cross3` now
serve those sites. These are deliberately not JIT-compiled: they are
called with plain Python floats and small arrays from interpreted code,
where numba's dispatch overhead would exceed the gain.

### The per-aggregate quality record was O(N²) in memory

`quality.max_self_overlap` built an `(N, N, 3)` difference array, and
runs on every aggregate — a regression introduced together with the
quality record itself:

| N | Before | After (pdist) |
|---:|---:|---:|
| 512 | 12.2 ms | ~0.4 ms |
| 1024 | 49.3 ms | ~1.1 ms |
| 2048 | 211 ms, 101 MB temporary | ~5.5 ms |

`scipy.spatial.distance.pdist` walks the upper triangle directly in C.

### Disabled telemetry was still being computed

`_leaf_mask_for_cluster` is O(n²) per cluster and ran on every sticking
call, as did `_candidate_score` per candidate attempt. Under the
production defaults neither result is read: the baseline candidate
policy ignores leaf classification, and the leaf/score counters are
only printed when their `profile_*` flags are set. Both are now
computed only when something consumes them — an experimental candidate
policy, a profiling flag, or an attached merge log.

### The sticking placement was scalar glue around fast kernels

`_cca_sticking_v1` ran 2,176 times per N=1024 aggregate and accounted
for ~26% of the run, almost none of it arithmetic: ~30k `norm` calls,
~24k `np.array` allocations and ~20k `np.zeros`, all on 3-vectors and
small temporaries between operations that were already fast.

`cca_kernels.cca_sticking_v1_kernel` fuses the whole default
(`ext_case=0`) placement — translate cluster 2, sample the first
contact point, rotate cluster 1, sample the second contact point,
rotate cluster 2 — into one compiled function, including both cluster
transforms as tight loops. Its self time dropped 5.8×, from 133 ms to
23 ms.

Randomness is hoisted out of the kernel: the two angles the
sphere-sphere intersections would have sampled are passed in as
arguments, so the caller retains sole ownership of the RNG stream and
seeded runs stay reproducible. This also makes the kernel a pure
function, which is what allows `tests/test_sticking_kernel.py` to pin
it against the interpreted reference exactly — seeding both identically
makes the comparison bit-for-bit rather than statistical. `ext_case=1`
(spherical-cap sampling) still runs the interpreted path, and
`_cca_sticking_v1_interpreted` is retained as the readable definition
of correct behaviour.

## Measured effect

End-to-end medians, 5 seeds per cell, single-threaded:

| Regime | N | Before | After | Speedup |
|---|---:|---:|---:|---:|
| easy | 256 | 130.9 ms | 111.4 ms | 1.18x |
| easy | 512 | 267.4 ms | 203.5 ms | 1.31x |
| easy | 1024 | 498.8 ms | 329.9 ms | 1.51x |
| medium | 256 | 174.4 ms | 125.2 ms | 1.39x |
| medium | 512 | 450.7 ms | 251.1 ms | 1.80x |
| medium | 1024 | 998.7 ms | 571.9 ms | 1.75x |

The gain grows with N and with difficulty, as expected: the O(N²) fixes
scale with size, and the sticking kernel scales with how many candidate
placements a regime forces.

Results are unchanged: a seeded run remains bit-identical to itself and
produces the same overlap-free geometry. Aggregates generated by older
versions at the same seed may differ in the last bits, since
`math.sqrt` of an explicit sum and `np.linalg.norm` need not associate
identically; this shifts a trajectory without changing its statistics.

## A negative result on log-call guards

Recorded to prevent repetition. Three `logger.trace(f"...")` calls in
the rotation loop evaluate their f-string before `trace()` can check
the level — a genuine and common defect — but guarding them produced no
measurable change, because only ~900 rotations run per aggregate and a
float format costs ~1 µs. The guards were kept as a correctness fix,
not as an optimization.

The expectation that they would matter came from trusting cProfile's
*self time*, which charges some per-call instrumentation overhead to
the calling function and made `_perform_cca_sticking`'s loop body
appear to hold ~47% of the run. Wall-clock A/B measurement is the
arbiter; the profiler is useful for pointing at candidates only.

## Remaining time

After the changes above, the profile at N=1024/medium is dominated by
`_perform_cca_sticking`'s own loop body, with the geometry and sticking
stages largely gone (`_cca_sticking_v1` fell from 14% of the run to
under 4%). The self-time caveat above applies when reading that
figure: a loop making thousands of calls is exactly the shape cProfile
over-attributes.

Further gains would require compiling the whole candidate/rotation
loop, not just the placement inside it. That is a substantially harder
change — the loop interleaves RNG draws, overlap checks and the retry
policy, and would have to keep the RNG draw order identical to remain
reproducible — and the payoff has shrunk: PCA no longer registers on
the profile, and a representative N=1024 aggregate builds in about half
a second.