Shadows occurring in outdoor scenes can have a significant impact on the content within captured images. In object tracking applications they can grossly extend the apparent shape of an object due to the reduction in illumination intensity within cast-shadows. They have typically been removed from images using observed properties such as shape, intensity, texture retention, and shadow direction. However, these methods often fail with grey or shadow-like objects on complex backgrounds that vary in colour or texture. We present a new nonlinear method of shadow removal for object tracking based on a surface's response to sunlight and skylight. The method returns full-colour images, operates on complex scenes in outdoor environments, and requires only 2 images for calibration.