What a Camera Actually Gives You
Point a camera at a room and it captures an image. That image is, at its most fundamental level, a grid of numbers. Each number represents the brightness of a tiny area of the scene — a pixel. A typical camera might capture a grid of 1920 by 1080 pixels, thirty times per second. That's over 62 million numbers per second, each one representing a small piece of the light that entered the lens.
This is what a camera actually gives a robot: numbers. Not objects. Not distances. Not meanings. Just numbers representing light intensity at each point in the frame.
The gap between those numbers and useful information — "there is a cardboard box in the lower left of the frame, oriented at 45 degrees, approximately 60 centimeters away" — is what computer vision exists to bridge. Everything in this guide is about how that gap gets crossed.
Image Sensors: How Light Becomes Data
Before software can process an image, hardware must capture it. The image sensor is the component that converts light into electrical signals, which are then converted into the numerical pixel values the computer works with.
CCD and CMOS Sensors
Two sensor technologies dominate: CCD (Charge-Coupled Device) and CMOS (Complementary Metal-Oxide-Semiconductor). Both work by using photosensitive elements that generate an electrical charge proportional to the amount of light that hits them. The difference is in how that charge is read out and processed.
CCD sensors historically produced higher image quality with less noise, making them the choice for scientific and high-end imaging applications. CMOS sensors are cheaper to manufacture, consume less power, and can integrate processing circuitry directly on the sensor chip. Modern CMOS sensors have closed most of the quality gap, and CMOS is now the dominant technology in robotics cameras.
Color: RGB Channels
Most image sensors are inherently monochrome — they measure light intensity but not color. Color is captured by placing a grid of color filters over the sensor, typically in a pattern called a Bayer filter, with alternating red, green, and blue filters. Each pixel captures only one color channel; the other two are interpolated from neighboring pixels. The result is an image with three channels — red, green, and blue — that together represent the full color of the scene.
For many robotics applications, color is useful but not essential. A robot navigating a warehouse cares more about the shape and position of obstacles than their color. But color becomes important for tasks like identifying specific products by their packaging, reading labels, or distinguishing objects that are similar in shape but different in appearance.
Shutter Type: Rolling vs Global
When a camera captures a frame, it doesn't capture all pixels simultaneously — at least not with a rolling shutter. A rolling shutter reads out the sensor row by row, from top to bottom. If the camera or the scene is moving during this readout, different rows capture the scene at slightly different moments, causing a characteristic distortion: vertical lines appear skewed, and fast-moving objects appear warped.
A global shutter captures all pixels simultaneously, eliminating this distortion. Global shutter sensors are more expensive but important for robotics applications where the camera or the objects in the scene are moving quickly — a robot arm moving at speed, a fast-moving conveyor belt, or an AMR navigating at walking pace.
Exposure and Dynamic Range
Exposure controls how much light the sensor collects during each frame. Too little light and the image is dark and noisy; too much and bright areas are overexposed and lose detail. Dynamic range is the ratio between the brightest and darkest details the sensor can capture simultaneously.
Real environments often have challenging lighting: a warehouse with bright skylights and dark corners, a factory floor with reflective metal surfaces next to matte packaging. Sensors with high dynamic range handle these conditions better, capturing detail in both bright and dark areas without sacrificing one for the other.
Depth Perception: Seeing in Three Dimensions
A standard camera produces a two-dimensional image. The world is three-dimensional. Bridging that gap — recovering depth information from camera data — is one of the central challenges of robot vision.
Why a Single Camera Can't Directly Measure Distance
A single camera image tells you where objects appear in the frame, but not how far away they are. A large object far away and a small object close up can produce identical images. Without additional information, there's no way to distinguish them.
Humans solve this problem using two eyes — binocular vision — plus a lifetime of learned cues about how objects look at different distances. Robots use several different technical approaches, each with different trade-offs.
Stereo Vision
Stereo vision uses two cameras separated by a known distance — the baseline — to capture the same scene from slightly different viewpoints. The same object appears at slightly different positions in the two images; this difference is called disparity. Objects that are close to the cameras have large disparity; objects that are far away have small disparity.
By measuring the disparity of each point in the image and knowing the baseline distance and the cameras' focal lengths, the system can calculate the distance to each point using triangulation — the same geometry a surveyor uses to measure distances by taking sightings from two known positions.
Stereo vision works well in textured environments where the matching algorithm can find corresponding points between the two images. It struggles in areas with uniform texture (a plain white wall) or repetitive patterns, where it's hard to determine which point in one image corresponds to which point in the other.
Structured Light
Structured light systems project a known pattern — typically a grid of dots or a series of stripes — onto the scene using an infrared projector. A camera then captures how that pattern is distorted by the surfaces it falls on. Because the pattern is known, the distortion can be used to calculate the depth of each surface point.
Structured light works well in controlled environments and produces dense, accurate depth maps. Its limitations: it requires its own light source (the projector), it can be confused by other structured light sources or strong ambient infrared light (including sunlight), and it typically has a limited range of a few meters.
Time-of-Flight Sensors
Time-of-flight (ToF) sensors measure depth by emitting pulses of infrared light and measuring how long they take to return after reflecting off surfaces. The speed of light is constant, so the round-trip time directly gives the distance. ToF sensors produce depth maps at video frame rates and work in a wide range of lighting conditions.
ToF sensors are compact, fast, and increasingly inexpensive. Their limitations include lower spatial resolution than stereo or structured light systems, susceptibility to interference from other ToF sensors operating nearby, and reduced accuracy at longer ranges.
RGB-D Cameras
RGB-D cameras combine a standard color camera with a depth sensor — typically structured light or ToF — in a single unit. They produce synchronized color and depth images, where each pixel has both a color value and a distance measurement. This combination is particularly useful for manipulation tasks: the color image helps identify what an object is, and the depth image tells the robot exactly where it is in three-dimensional space.
Machine Vision vs Computer Vision
These terms are often used interchangeably, but they describe different approaches with different strengths.
Machine vision is the older discipline. It typically refers to vision systems designed for specific, controlled industrial tasks: inspecting parts on a production line, reading barcodes, measuring dimensions, detecting defects. Machine vision systems work in carefully controlled environments with consistent lighting, fixed camera positions, and known object types. They use rule-based algorithms — explicit instructions for what to look for and how to measure it — rather than learned models.
Computer vision is broader and more flexible. It encompasses machine learning-based approaches that can generalize across varied environments, object types, and conditions. A computer vision system can be trained to recognize thousands of different objects in varied lighting, from different angles, in cluttered scenes — without being explicitly programmed for each case.
In modern robotics, the two approaches coexist. A robot arm on a production line might use traditional machine vision for precise dimensional measurement of known parts, while using computer vision for flexible bin picking of varied items. The choice depends on how controlled the environment is and how much variability the system needs to handle.
Object Detection
Object detection answers the question: where are the objects in this image, and what are they? The output is typically a set of bounding boxes — rectangles drawn around each detected object — along with a class label ("person", "box", "forklift") and a confidence score indicating how certain the system is about each detection.
How Detection Works
Modern object detection systems are built on deep neural networks, specifically convolutional neural networks (CNNs). The network processes the image through many layers, each one extracting increasingly abstract features — edges in early layers, shapes in middle layers, object parts and whole objects in later layers.
Detection networks are trained on large datasets of labeled images — images where humans have drawn bounding boxes around objects and labeled them. The network learns to associate visual patterns with object classes by adjusting its internal parameters to minimize the difference between its predictions and the human-labeled ground truth.
At inference time (when the trained network is processing new images), it scans the image at multiple scales and positions, generating candidate detections and then filtering them to produce the final set of bounding boxes.
Confidence and Thresholds
Every detection comes with a confidence score between 0 and 1. A score of 0.95 means the network is highly confident that object is present; a score of 0.3 means it's uncertain. The system applies a threshold — typically set by the application developer — below which detections are discarded.
Setting this threshold involves a trade-off. A high threshold reduces false positives (detecting objects that aren't there) but increases false negatives (missing objects that are there). For a safety system that needs to detect people near a robot, a low threshold is appropriate — it's better to occasionally slow down unnecessarily than to miss a person. For a picking system that needs to identify specific items, a higher threshold may be appropriate to avoid picking the wrong object.
Real-Time Requirements
Robotics applications often require detection to run in real time — fast enough that the robot can act on the results before the situation changes. A robot arm moving toward a bin needs to know what's in the bin before it arrives, not several seconds later. An AMR navigating a busy corridor needs to detect people fast enough to slow down before reaching them.
Real-time detection requires either fast hardware, efficient