How AI Motion Tracking Works: A Plain-English Guide to Pose Estimation

Backlighting, baggy clothing, occlusion, cropping and non-standard gait break it. A camera watches you jump, and a character on the TV jumps with you. No controller, no wristband, no strap on your leg.

How AI motion tracking works comes down to one problem: turning a flat grid of colored dots into a moving skeleton, thirty to sixty times a second, on a chip that costs less than dinner. The last section covers failure, which is the part readers actually need.

Short answer

●       A camera captures brightness values. Every notion of "arm" or "knee" is added afterward by software.

●       A pose model outputs labeled points on the body, called keypoints or nodes. Nex Playground uses 18.

●       One camera cannot measure distance, so 3D pose is inferred rather than measured.

●       The Kinect measured depth with infrared hardware. Neural networks replaced it because silicon got cheap.

●       On-device inference keeps video inside the box, protecting privacy and cutting latency.

A camera sensor does not see a person. It reports light.

A 1280 by 720 color frame is about 2.7 million numbers, each the intensity of red, green or blue at one point on a grid. At 30 frames per second that is roughly 83 million numbers a second, none of them labeled "elbow."

Hand-written rules once closed that gap: subtract the empty room, find skin-colored blobs, treat strong edges as limbs. All of it collapsed in real living rooms, where a moving curtain or a sofa-colored shirt breaks the assumptions.

The modern approach writes no rules. A neural network trains on a very large set of images in which people have already labeled the joints, and learns the mapping from pixels to joints.

Pose estimation takes an image and returns the coordinates of specific anatomical landmarks. Each landmark is a keypoint, also called a node or a joint, and it is two or three numbers plus a confidence score. Left wrist: x=412, y=308, confidence 0.94. That confidence value is how the system knows to distrust itself when a limb is hidden. How many keypoints there are is a design decision, not a law of nature.

Eighteen nodes is a sensible middle for full-body play: four limbs, the hips and shoulders that define torso orientation, and enough head detail to know which way you face. No fingers, which is why camera consoles have no games needing a pinch or grip.

Connect those points and you get the stick figure everyone recognizes. The game reads that skeleton, not the picture: a jump is "both ankles rose past a threshold within four frames while the hips rose too."

2D pose estimation returns x and y in image space. It is cheap, robust and enough for many games. 3D returns x, y and z, so the skeleton has depth and can be rotated, which you need for judging form or side-on movement.

Here is the honest part. A single ordinary camera cannot measure distance, because depth is destroyed the moment a three-dimensional scene is projected onto a flat sensor. Any 3D pose from one lens is inferred, not measured.

Inference works because bodies are strongly constrained. Limbs have known proportions, a limb pointing at the camera looks shortened by an amount that implies its angle, your size in frame narrows the plausible distances, and consecutive frames have to agree.

Plausible is the right word for the result. Two ambiguities are baked into the mathematics: a small person standing close and a tall person further back project to nearly identical images, and a forearm angled toward the camera projects the same as one angled away. A limb that snaps the wrong way on screen is usually one of those.

The Xbox Kinect solved depth in hardware, and understanding why explains why today's boxes are so small.

The 2010 Kinect projected a fixed pattern of infrared dots across the room with a laser emitter and read the pattern's distortion with an infrared camera. The 2013 Kinect for Xbox One used time-of-flight: modulated infrared light, with the sensor measuring how long it took to return, per pixel, for a direct distance reading. Either way the sensor delivered a depth image before tracking began, and software then sorted each depth pixel into a body part. It never had to guess how far away your knee was.

It was also expensive. The module needed a laser emitter, a second imager and its own optics and power budget, and sunlight swamps infrared while glass reflects it. Microsoft discontinued the sensor in 2017; we cover what actually replaced the Xbox Kinect separately.

Then two curves crossed. Pose models got far better and far smaller, helped by mobile-first architectures and by quantization that runs a model in 8-bit integer math instead of floating point, while phones pushed neural accelerators into cheap commodity chips. Once a small accelerator could run a good pose model at 30 frames per second on a couple of watts, the case for a depth module evaporated.

Top-down runs a person detector first, boxes each human, then runs a single-person pose model inside each box. Accuracy is high because the model sees a clean crop, but inference time grows with the number of people, and anyone the detector misses is invisible.

Bottom-up finds every keypoint in one pass, then solves the grouping problem: which of these six elbows belong to which of these three people. OpenPose introduced part affinity fields for this, learned vector fields along limbs indicating which elbow connects to which shoulder. Cost stays constant whatever the crowd size, and grouping fails when people overlap.

Above pose sits a tracking layer that matches skeletons between frames so player two stays player two after a crossover, using position and motion continuity rather than facial recognition. If two players swap sides and the game swaps their scores, that is what failed. Systems differ here, worth checking when you compare alternatives to the Nex Playground.

On-device means the model runs on the console's own processor: frames go into memory, through the network, out as coordinates, then discarded. No video is uploaded, so there is none to intercept or leak.

The engineering argument is as strong as the privacy one. At 60 frames per second you have 16.7 ms per frame. Exposure and readout eat part of that, inference takes several milliseconds, and rendering costs another frame or two. Then the television adds its own delay, which on many sets runs 60 ms or more unless game mode is on.

As a rule of thumb, whole-body action feels attached to you when total motion-to-photon delay stays under roughly 100 ms, and rhythm games want tighter, because your ear judges timing more harshly than your eye. Shipping frames to a server would add tens to hundreds of milliseconds and make the game unplayable on a bad connection.

In rough order of how often they ruin a session:

Backlighting. A window or lamp behind the player is the most common failure. The sensor exposes for the bright background, the player becomes a silhouette, and the model finds an outline but cannot separate arm from torso. Android Police's reviewer found window light "tampered with" the Nex Playground's tracking.

Low light. In a dim room the sensor raises gain, adding noise, and lengthens exposure, smearing fast limbs into blur. Blurred wrists produce low-confidence keypoints and a jittery skeleton, so ordinary room lighting matters.

Clothing. Contrast is what the model works with, so a charcoal hoodie against a charcoal wall leaves very little edge information. Loose clothes are a different problem: the model estimates the joint under the fabric, and baggy sweatpants or a long skirt hide the line of the limb.

Occlusion. When one player passes in front of another, the rear player's joints are not visible, so the system extrapolates or drops confidence. An arm behind your back is a guess.

Cropping. Keypoints outside the frame do not exist. Raise your arms near the camera and your wrists leave the field of view; stand too close and your feet go. Android Police found "not enough room to fully capture players' feet or hands when reaching upwards, especially if adults play." That is room geometry, and our guide to how much space a motion console needs has the numbers.

Pets, toddlers and pictures of people. A person detector is a probability machine. A large dog on its hind legs, a crawling toddler, a poster, or people on a screen behind you can all score high enough to read as a player.

Unusual gait, and wheelchairs. This is an accessibility limitation, not a quirk, and it deserves naming plainly. Models learn from their training data, and public pose datasets are dominated by standing, able-bodied adults. A wheelchair occludes the legs and hips, crutches and walkers add limb-shaped objects that are not limbs, and an atypical gait sits outside most of what the model has seen. Tracking is worst for exactly the people who stand to gain most from accessible movement games. Seated modes have to be trained for deliberately, so confirm seated support before buying.