Understanding the real world

This is the largest section on the site, and the one where the mental model matters most. In VR your scene is the whole truth. In AR it is a guest in a room the runtime is still figuring out, and every API here is a different answer to the same question: what does the device actually know about that room, and how much should you trust it?

The five pages form a progression from a single point to a whole scene. Hit testing gives you one ray and one surface under it. Plane detection gives you polygons that grow, merge and change identity as the runtime sees more. Anchors are the admission that the runtime's map is being rewritten underneath you, and that content placed in world coordinates will drift unless you say otherwise. Depth sensing hands you a per-pixel distance buffer, low-resolution and noisy, which is enough to make virtual objects hide behind real ones. Lighting estimation closes the loop by telling you which way the light falls, so the object you placed does not look pasted on.

What unifies them is that none of these APIs return facts. They return the runtime's current best guess, at a resolution and confidence that varies by device and improves as the user moves. Code that treats a detected plane as a fixed rectangle, or a depth value as a measurement, will work in the demo and fail in a real room. Every page in this section spends time on the specific way its API is allowed to be wrong, because that is the part the specification states plainly and most tutorials skip.

In this section

1. 01AR hit testingA hit test asks the runtime where a ray from the device meets a surface it has actually recognised — and the answer arrives with an orientation, which is the part most first implementations throw away.
2. 02Plane detectionPlane detection does not give you a list of the surfaces in a room. It gives you the runtime's current opinion, and that opinion grows, splits, merges and forgets — every frame, without telling you.
3. 03AnchorsAn anchor is a promise from the runtime that it will keep correcting a pose as it learns more about the room — which means the pose you read this frame is not the pose you read last frame, and that is the entire point.
4. 04Depth sensingOcclusion is not an effect you draw — it is a per-fragment comparison between how far away the real surface is and how far away your pixel is, and the depth buffer that answers it is an order of magnitude coarser than the camera image.
5. 05Lighting estimationLighting estimation answers one question — what does the light in this room look like from here — in three different currencies, and using only the cheapest of them is why so much AR content still looks pasted on.

Read these in order

Hit testing first: it is the smallest possible version of the problem, one ray against one surface, and it introduces the idea that the answer arrives asynchronously and can be absent.

Planes, then anchors. Planes generalise the single surface into a set that changes over time; anchors are the direct consequence of that instability and only make sense after you have seen it.

Depth and lighting last. They are the two that do not place anything — they change how what you already placed is drawn, which is a later problem than getting it into the right spot.