phatch/reikhelm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(render): full-range per-frame depth normalization for FPV ControlNet

Browse source 
Diagnosis: every baked FPV depth map used only the upper-mid grays
(typ. 160–214, 0/68 frames reaching true black, only 35/68 reaching
near-white). A fixed 0..maxView=12 scale meant the nearest floor (~1
cell) never hit white and a wall 3 cells ahead never went past mid-gray,
so a small room collapsed into a low-contrast band. A depth ControlNet
trained on full-range MiDaS/depth-anything maps gets almost no structural
signal from that — the root cause of "controlnet barely followed / hit
and miss" renders.

Fix: two-pass raycast. Pass 1 records each column's wall distance and the
frame's true near/far. Pass 2 maps depth through a per-frame window so
near surfaces read white and the deepest visible recess reads black every
frame. The far plane tracks the deepest real surface but is clamped to a
minSpan floor so a shallow dead-end nook is NOT stretched into a fake
tunnel — its faced wall stays an honest mid-gray slab (no black recess
for the model to paint a phantom door into). gamma default 2.0→1.5 since
normalization now owns the range. nearPlane/minSpan exposed as opts.

Verified offline (headless bake, seed 7, 68 frames): median dynamic range
193→255; frames reaching true-white near surfaces 35/68→68/68; worst-frame
range 54→106. Visual before/after confirms open frames now frame a black
passage and closed frames a clean mid-gray wall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Parley Hatch 2026-06-15 20:24:29 -06:00
parent df02911a1b
commit 018909b2cd
1 changed files with 34 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
reikhelm-web/fpv.js
View file

@ -124,7 +124,8 @@ export function vantage(env, opts = {}) {
}
// Build a first-person depth field. Returns { width, height, gray, at, dir }.
// opts: { stage?, at?[x,y], dir?'N'|'E'|'S'|'W', width?, height?, fov?deg, maxView?, gamma? }
// opts: { stage?, at?[x,y], dir?'N'|'E'|'S'|'W', width?, height?, fov?deg,
// maxView?, gamma?, nearPlane?, minSpan? }
export function computeFPVDepth(env, opts = {}) {
const { tiles } = getStage(env, opts.stage);
const w = env.width, h = env.height;
@ -132,7 +133,7 @@ export function computeFPVDepth(env, opts = {}) {
const fov = (opts.fov ?? 80) * Math.PI / 180; // wider horizontal FOV to fill the widescreen frame
const planeLen = Math.tan(fov / 2);
const maxView = opts.maxView ?? 12; // how far the eye sees (cells) before it's pure dark
const gamma = opts.gamma ?? 2.0; // >1 deepens the falloff to black (matches the proven corridor look)
const gamma = opts.gamma ?? 1.5; // >1 deepens the falloff to black; normalization owns the range now
const at = opts.at ?? vantage(env, opts);
const dirName = opts.dir ?? 'N';
@ -146,6 +147,17 @@ export function computeFPVDepth(env, opts = {}) {
const centerX = W >> 1;
let ahead = maxView; // straight-ahead wall distance (cells) — big = an opening leads onward
// Pass 1 — cast every column, recording the wall-hit distance (or maxView when
// the ray escapes off-map / through a gap). We need the frame's true near/far
// span BEFORE mapping to gray. A fixed 0..maxView scale wastes most of the
// tonal range on small rooms: the nearest floor sits ~1 cell off (never quite
// white) and a wall 3 cells ahead lands at mid-gray (never black), so a whole
// depth map collapses into the 160–214 band — almost no contrast for the depth
// ControlNet to honor. Normalizing each frame to its own near/far instead
// makes near surfaces white and the deepest recess black every time.
const colDist = new Float32Array(W);
let nearRaw = 1.0; // the floor at the screen's bottom edge always sits ~1 cell away
let farRaw = 0;
for (let x = 0; x < W; x++) {
const camX = 2 * x / W - 1; // −1 … 1 across the screen
const rayX = dirX + planeX * camX;
@ -168,8 +180,27 @@ export function computeFPVDepth(env, opts = {}) {
}
const perp = side === 0 ? sideX - deltaX : sideY - deltaY;
const dist = hit ? clamp(perp, 0.02, maxView) : maxView;
colDist[x] = dist;
if (dist < nearRaw) nearRaw = dist;
if (dist > farRaw) farRaw = dist;
if (x === centerX) ahead = dist;
}
// Per-frame normalization window. The far plane tracks the deepest surface
// actually visible (so a real passage reads black), but is clamped to a floor
// (minSpan) so a shallow dead-end nook isn't stretched into a dramatic fake
// tunnel — its faced wall stays an honest mid-gray slab, not a black recess
// the model would paint a phantom door into. `gamma` then shapes the falloff
// within that window.
const nearPlane = opts.nearPlane ?? nearRaw;
const minSpan = opts.minSpan ?? 5;
const farPlane = clamp(farRaw, nearPlane + minSpan, maxView);
const span = Math.max(farPlane - nearPlane, 1e-3);
// Pass 2 — paint each column: a wall slice at its distance, floor/ceiling
// ramping toward the camera, every depth mapped through the per-frame window.
for (let x = 0; x < W; x++) {
const dist = colDist[x];
// Project the 1-cell-tall wall: lineH = H/dist (fills the screen at dist 1).
const lineH = H / dist;
const ds = clamp(Math.floor(half - lineH / 2), 0, H);
@ -185,7 +216,7 @@ export function computeFPVDepth(env, opts = {}) {
const p = y < half ? (half - y) : (y - half + 1);
dpix = Math.min((0.5 * H) / p, maxView);
}
const t = clamp(dpix / maxView, 0, 1);
const t = clamp((dpix - nearPlane) / span, 0, 1);
gray[y * W + x] = Math.round(255 * Math.pow(1 - t, gamma));
}
}