Visual processing is strongly influenced by the recent stimulus history – a phenomenon termed adaptation. Prominent theories cast adaptation as a consequence of optimized encoding of visual information, by exploiting the temporal statistics of the world. However, this would require the visual system to track the history of individual briefly experienced events, within a stream of visual input, to build up statistical representations over longer timescales. Here, using an openly available dataset from the Allen Brain Observatory, we show that neurons in the early visual cortex of the mouse indeed maintain long-term traces of individual past stimuli that persist despite the presentation of several intervening stimuli, leading to long-term and stimulus-specific adaptation over dozens of seconds. Long-term adaptation was selectively expressed in cortical, but not in thalamic neurons, which only showed short-term adaptation. Early visual cortex thus maintains concurrent stimulus-specific memory traces of past input, enabling the visual system to build up a statistical representation of the world to optimize the encoding of new information in a changing environment.