The runtime gives you geometry, not meaning

The gap, stated plainly

Hit testing returns a point and a normal. Plane detection returns polygons that grow and merge. Depth sensing returns distances. All three are geometric facts about the room, and none of them answers the question your application actually has, which is almost always about function: can I put something on this, is it safe to walk here, is this a surface the user thinks of as theirs.

The distance between those two kinds of statement is the whole difficulty. "There is a horizontal plane at 0.74 metres" and "there is a desk" are not the same claim, and applications are written against the second one. Every AR experience that places a virtual object somewhere sensible is doing this translation, and most of them are doing it with rules nobody wrote down.

What the specification does offer

The plane detection module does include a semantic label on detected planes, and runtimes commonly populate it with values along the lines of floor, wall, ceiling and table. Where it is present it is worth using, and checking it costs nothing.

The caution is that it is advisory rather than guaranteed. The set of values is not closed, coverage varies between runtimes and between sessions on the same runtime, and a plane can be reported with no label at all before later acquiring one. Code that branches on the label without a path for "unlabelled" will work on the developer's device and fall through on someone else's. Checked against the specification and shipping behaviour in September 2026; this is an area that moves, and it is worth re-reading before relying on it.

Treat the label as a strong hint that arrives sometimes. Build the fallback first and let the label improve the result when it shows up, rather than building on the label and discovering the fallback is missing.

How far plain geometry gets you

Further than people expect, and it costs almost nothing. Three numbers you already have — height above the floor, orientation, and area — separate most of the cases that matter.

Anything horizontal at zero is floor. Anything vertical is wall, and a vertical plane whose lower edge is well above the floor is more likely a window or a picture than a wall. Horizontal surfaces cluster by height in ways that map onto furniture: roughly 0.4 to 0.5 metres is seat height, 0.7 to 0.8 is table and desk height, around 0.9 is kitchen counter, and above 1.2 is shelving. Area then disambiguates: a 0.75-metre horizontal plane of two square metres is a dining table, and one of a tenth of that is a side table.

This is not machine learning and it does not need to be. It is twenty lines of arithmetic, it runs in microseconds, it works identically on every device, and it gets the common cases right. Reaching for a model before exhausting this is a way of spending a great deal of complexity on a problem that a height threshold already solved.

Classifying a plane by geometry, label optional

The height bands are the part that needs local verification. Counter and table heights differ enough between regions that a threshold tuned in one market misclassifies in another, and this is exactly the sort of assumption that is invisible until someone reports that your application thinks their desk is a kitchen.

// Height bands are approximate and regional. Verify against furniture
// in the markets you ship to rather than treating these as universal.
const BANDS = [
  { max: 0.15, kind: 'floor' },
  { max: 0.55, kind: 'seat' },
  { max: 0.85, kind: 'table' },
  { max: 1.05, kind: 'counter' },
  { max: Infinity, kind: 'shelf' },
];

function classify(plane, floorY) {
  // A label, when present, beats anything geometry can infer.
  if (plane.semanticLabel) return plane.semanticLabel;

  const normal = planeNormal(plane);
  const vertical = Math.abs(normal.y) < 0.3;
  if (vertical) return 'wall';

  const height = planeCentre(plane).y - floorY;
  return BANDS.find((band) => height <= band.max).kind;
}

Checking the label first and falling through to geometry means the same code improves on runtimes that report labels without breaking on runtimes that do not. The reverse order — geometry first, label as an override — produces different results on the same hardware depending on when the label arrives.

Where geometry alone stops working

These are the cases that send people looking for a model, and they are worth being specific about, because some of them a model would not fix either.

A bed and a table are the same height

Both present a large horizontal surface around 0.5 to 0.7 metres. Geometrically they are near-identical and functionally they are nothing alike. Area helps a little; nothing else in the geometry does.

Occupied surfaces

A desk with a monitor, a keyboard and a coffee cup on it is reported as a plane, because plane detection describes the surface rather than what is resting on it. Placing an object at the centre of that plane puts it inside a laptop.

Glass and gloss

Glass tables, mirrors and high-gloss surfaces are unreliable for the underlying tracking, so they arrive as planes that flicker, plane fragments, or nothing at all. This is a sensing limitation and no amount of semantic reasoning recovers it.

Rugs, thresholds and steps

Small height discontinuities near the floor are exactly the range where the floor estimate itself is uncertain. Whether a rug is floor is not a hard question for a person and is genuinely ambiguous to a threshold.

Ownership and social meaning

Whether a surface is one the user would mind you covering is not a geometric property, a semantic label, or something a vision model has access to. Someone else's desk is shaped exactly like your own.

The line the web platform has drawn

The obvious response to the list above is to run a scene-understanding model. On a native headset application that is a reasonable path: request camera access, run a segmentation model on the frames, get labels the geometry cannot provide.

On the web that path is closed, and closed on purpose. WebXR does not hand your page camera frames. You receive abstractions the runtime has already computed — planes, meshes, depth buffers, a light estimate — precisely so that a page cannot reconstruct the room the user is standing in. A camera feed from a headset contains the user's home, the people in it, the documents on their desk and the medication on their counter, and the platform's position is that a web page should not receive that in exchange for a permission prompt nobody reads carefully.

This is worth understanding as a design constraint rather than as a missing feature. It means the ceiling on spatial semantics for a web application is whatever the runtime chooses to expose, and that a native application will always be able to know more about the room than a page can. If your product depends on fine-grained scene understanding, that dependency is an argument about which platform you are shipping on, and it is better to have that argument early.

The upside is not small. The abstraction boundary means a WebXR page carries a genuinely different privacy posture from a native app, which is a real thing to be able to tell users, and it means the expensive perception work happens once in the runtime rather than repeatedly in every page.

Designing so that you do not need to know

The most robust applications in this space are the ones that arranged not to need the answer.

Letting the user place things is the obvious version, and it is underrated because it looks like giving up. It is not: the user knows which surface is a desk, knows what is already on it, and knows whether they mind. A placement interaction that takes two seconds removes an entire category of failure, and users do not experience being asked where to put something as a defect.

Where automatic placement is genuinely required, designing for graceful wrongness beats designing for accuracy. An object that lands somewhere plausible and can be dragged is fine. An object that lands inside a wall because a plane was misclassified, with no way to move it, is the same classification error with a much worse outcome. The error rate is a property of a hard problem; the consequence of an error is a design decision.

None of this is an argument against models in XR. It is an argument that the geometry-to-meaning gap is narrower than it looks once you use the three numbers you already have, and that the remaining width is not, on the web, something you can close by adding intelligence — because the information that would close it was deliberately not given to you.

Related pages

• Plane detection — Where the polygons come from, how they merge, and why a plane's identity is less stable than it looks.
• Depth sensing — The per-pixel distance buffer, its resolution, and what it is and is not good for.
• AR hit testing — The one ray and one surface that most placement decisions are actually made from.
• Anchors — Keeping a placed object where the user put it while the runtime revises its map.
• WebXR Plane Detection Module — The draft specification, including the semantic label and its deliberately open value set.