reikhelm/.dev/2026-06-15-diffusion-pipeline-reliability.md

Diffusion pipeline reliability — making the depth ControlNet actually followed

Started: 2026-06-15 · ongoing (/loop) Problem (user): "controlnet maps generated are barely followed; images of the dungeons are very hit and miss."

Root cause found (iteration 1)

The FPV depth maps fed to the Z-Image Fun depth ControlNet used only a sliver of the tonal range. Histogram over all 68 baked seed-7 frames:

• median dynamic range 193 / 255, worst frame only 54
• 0/68 frames reached true black (<32); the deepest recesses topped out ~mid-gray
• only 35/68 reached near-white at the near surfaces

Cause in reikhelm-web/fpv.js: a fixed t = dpix / maxView with maxView = 12. The nearest floor sits ~1 cell away (never hit white) and a wall 3 cells ahead landed at mid-gray (never black), so small rooms collapsed into the 160–214 band. Depth ControlNets are trained on full-range MiDaS / depth-anything maps, so a washed-out map gives almost no structural signal → the model freelances.

Fix shipped — commit 018909b

Two-pass raycast in computeFPVDepth:

• Pass 1 records each column's wall distance + the frame's true near/far.
• Pass 2 maps depth through a per-frame window: near → white, deepest visible recess → black, every frame.
• Far plane tracks the deepest real surface but is clamped to minSpan (5 cells) so a shallow dead-end nook is NOT stretched into a fake tunnel — its faced wall stays an honest mid-gray slab (no black recess → no phantom-door bait).
• gamma 2.0 → 1.5 (normalization now owns the range). nearPlane/minSpan are opts.

Verified offline (headless bakeAtlasDepths(7), 68 frames): median range 193 → 255; near-white frames 35/68 → 68/68; worst range 54 → 106. Visual before/after (/tmp/ba.png during the session): open frames now frame a black passage; closed frames a clean mid-gray wall.

Next — needs the render workstation (192.168.1.26:8188, was OFFLINE this iteration)

The depth signal is now strong, so the old strength is almost certainly wrong. With a weak signal you crank strength to force adherence; a strong signal wants less. bake_atlas.py defaults --strength 0.85.

1. Re-probe the workstation: python3 tools/comfy-spike/comfy.py probe http://192.168.1.26:8188 (2026-06-15 iter 2: host pings but port 8188 closed — ComfyUI not running; needs a manual start on the workstation before any of the below.)
2. A/B + strength sweep in one run (tool written & validated; old rev-2 maps persisted to tools/out/compare/rev2_depths/ so the fix is provable, not just "looks better"):

   python3 tools/comfy-spike/sweep_strength.py http://192.168.1.26:8188 \
       --seed-dir tools/out/atlas/7 --alt-work tools/out/compare/rev2_depths \
       --open r8_N --closed r0_N --strengths 0.55,0.7,0.85,1.0
   

   → tools/out/compare/sweep_<run>.png: each frame gets a NEW row (full-range maps) and an OLD row (washed-out rev-2) across every strength. Read adherence: does the rendered passage land on the black recess? does the closed wall stay a wall (no phantom door)? Expect NEW ≫ OLD; pick the lowest strength that still honors geometry, update bake_atlas.py --strength 0.85 default. (Drop --alt-work once the win is confirmed.)
3. Then re-bake a full atlas at the chosen strength and spot-check a montage vs the old acc8137/a12bcbb bakes.

Open levers if still hit-and-miss after strength tuning

• nearPlane/minSpan/gamma are now tunable per-export — A/B if near surfaces read too blown-out (the AFTER maps are ~47% near-white; may want a slightly deeper nearPlane to retain near-field texture).
• Subtle masonry-block relief on closed-wall slices so flat walls render as detailed stone (offline-buildable, but verify on GPU it doesn't read as structure).
• Strength start_percent/end_percent scheduling (currently 0→1 full) — easing control off in late steps frees texture while locking composition.