Failure Statistics: the Structured Event Log#

Ordinary logging records whether an individual run worked. Aggregate analysis — over thousands of runs, where generation fails, by what mechanism, and how severely, sliced by the physical parameters (Df, kf, σ_p,geo, N) — requires structured records: free-text log lines cannot be aggregated, and the in-memory diagnostics dict accepted by run_simulation is per-call and never persisted.

pyfracval/event_log.py writes one JSONL file serving this purpose, enabled by a single config key:

event_log_path = "benchmark_results/events.jsonl"

Nothing is written and no file is opened when the key is unset. Setting it also enables the overlap census automatically, since the failure records exist chiefly to answer “how many particles overlap, and how badly”, and the census is the instrument that measures it.

Record kinds#

Three record kinds share one file, each stamped with the same run context so they can be sliced or joined:

Kind

One per

Records

merge

CCA merge attempt

round, cluster pair, Γ, search consumed, outcome, particles overlapping at give-up

pca_failure

PCA subcluster that could not be built

failing particle index, candidate count, search/swap attempts used

run

aggregate generation attempt

outcome, failure stage and reason, attempts used, wall time, final geometry quality

Every record carries run_id, pid, and the simulation parameters, so a sweep can write all its workers to one path and remain separable.

pca_failure is a distinct kind because PCA and CCA fail by different mechanisms, and a taxonomy that cannot distinguish them is of limited use. PCA failures occur while growing a single subcluster particle by particle — typically because no already-placed particle sits at a workable distance for the next monomer’s Γ — and were previously visible only as free text.

Two overlap denominators#

The codebase measures overlap in two conventions, which differ by a large factor for wide size distributions:

Field

Denominator

Meaning

max_overlap_of_rsum

r_i + r_j

the convention used by tol_ov, overlap.py and quality.py — comparable to the configured tolerance

max_overlap_of_rmin

min(r_i, r_j)

penetration depth of the smaller particle (densify.py’s convention)

Before these were split into separate fields, the census reported only the min(r) form under the bare name “overlap fraction” while the same record’s min_overlap used the r_sum form — two silently different scales in one row. Measured on real failures, the two medians are 0.618 and 1.785, nearly a factor of three apart on the same pairs; the field names now state the denominator explicitly.

Analysis#

devenv shell -- uv run python benchmarks/analyze_event_log.py events.jsonl
devenv shell -- uv run python benchmarks/analyze_event_log.py events.jsonl --by Df rp_gstd

Applied to the full boundary sweep — the same 4200-trial grid as boundary_sweep_v2.md, re-run with logging on (configs/boundary_sweep_v3_eventlog.toml, 523,812 records, 323 MB):

RUNS: where does generation fail?
  total 4164   success 3437 ( 82.5%)
    failed at TIMEOUT           420   10.1%
    failed at CCA               307    7.4%
  successful runs with invalid geometry: 0  (  0.0%)
  Rg error vs scaling law (%): min=-2.60 median=+0.58 max=+10.68
  wall time per run (s): min=0.04 median=1.03 max=157.84

PCA FAILURES: none recorded

CCA MERGES: attempts 519648   stuck 84150 ( 16.2%)
    failed_overlap             435372   83.8%
    failed_no_candidates          126    0.0%
   round    stuck   failed  fail rate
       1    49224   188662      79.3%
       2    21197   148276      87.5%
       3     7976    70437      89.8%
       4     4409    22180      83.4%
       7       52      124      70.5%
      10        6        0       0.0%
  merges that needed a later partner (backtracking): 24547  ( 29.2% of stuck)

OVERLAP AT FAILURE (226,578 censused failures)
  offending particles       : min=2 median=13 max=132
  cluster-pair size         : min=10 median=36 max=1024
  offending fraction        : min=0.02 median=0.36 max=1.00
  worst overlap / (ri+rj)   : median=0.639   <- comparable to tol_ov
  worst overlap / min(ri,rj): median=1.804   <- penetration of smaller particle
  best overlap reached      : median=2.327e-01
  failures with <=10% of the pair offending: 2445  (  1.1%)

Sliced, the two axes that matter:

σ

runs

success

merge fail

med offending

med fraction

1.0

1400

93.5%

34.2%

18

0.21

1.5

1400

83.2%

75.8%

11

0.33

1.9

1364

70.6%

92.3%

14

0.38

Df

runs

success

merge fail

med offending

med fraction

1.8

525

100.0%

0.0%

7

0.23

2.0

525

100.0%

1.3%

7

0.42

2.1

525

100.0%

11.0%

12

0.29

2.2

525

94.7%

69.9%

17

0.30

2.3

514

80.0%

86.5%

15

0.34

2.4

514

54.7%

89.2%

13

0.36

2.5

511

29.0%

89.7%

12

0.38

Findings#

Generation fails in CCA, essentially never in PCA. Across 4200 trials spanning the full grid — including σ=1.9 and Df=2.5 — there were zero PCA failure records. Every failure is a CCA merge failure or a timeout. This is worth stating precisely because PCA can fail (the mechanism exists and is now instrumented; a σ=3.0 probe triggers it), it simply does not within the parameter range anyone sweeps. Effort spent on the PCA stage is effort spent on a non-problem.

Timeouts are the largest single failure category, at 10.1% against 7.4% for genuine CCA failure. These are runs that hit the 120 s per-trial budget rather than exhausting their retries, so a portion of them are “too slow” rather than “impossible”. Any success rate quoted from a timeout-bounded sweep is a statement about the budget as much as the algorithm — see the caveat below.

Failures are not near-misses. The best overlap a failing merge reaches has median 0.233 against a tol_ov of 1e-6 — five orders of magnitude. No refinement of the rotation search closes that, which is consistent with every search-strategy experiment in experiments.md coming out flat.

Failures are not localized. Only 1.1% of 226,578 censused failures have 10% or less of the cluster pair offending; the median is 36%. This is the premise the drop-rescue mechanism rests on, measured now at full scale, and it is quantitatively false — agreeing with drop_rescue.md, which reaches the same conclusion by an independent route.

Difficulty peaks at round 3, it does not decline monotonically. An earlier 18-run sample on this page suggested a monotonic fall with round; the full sweep shows failure rate rising from 79.3% (round 1) to 89.8% (round 3) before falling away. Round 1 merges fresh PCA subclusters, which are small and easy to place; the squeeze comes a couple of rounds later, when clusters are large enough to be awkward but the pool is still big enough that most pairings are attempted. Backtracking still applies — it operates within whichever round is difficult.

No successful run produced invalid geometry, across 3437 aggregates. That is the overlap-acceptance and densify fixes holding at scale, and it is the check that would have caught the catalog overlap leak had it existed then.

Caveat: timeouts make a bounded sweep non-deterministic#

The same grid and seeds gave 80.3% (v2, no logging) and 81.8% (v3, with logging). Neither the log nor the census consumes randomness, so the difference is not the instrumentation changing the algorithm: it is that trial_timeout is wall-clock, so a trial near the budget can land either side of it depending on machine load. Quoting a success rate from a timeout-bounded sweep therefore carries a machine-dependent component, and comparisons between sweeps should either use the same hardware and load or drop the timeout.

Note also that 4164 run records were written for 4200 trials. The 36 missing are failures whose worker was killed by the sweep’s own timeout before the record could be written; all 3437 successes are present, and the run-level totals agree with the sweep’s independent summary.

Cost#

The census runs once per failed merge, off the hot path, and costs one full non-early-exit pairwise scan of the two clusters. Attaching a log also makes the sticking loop compute a true (non-early-exit) overlap per exhausted candidate, so that the recorded min_overlap is exact rather than the incremental scan’s lower bound. Both costs are pure overhead when no statistics are being collected, which is why they are gated on the log being attached.