Prior knowledge about the visual world can change how a visual stimulus is processed. Two forms of prior knowledge are often distinguished: stimulus familiarity (i.e., whether a stimulus has been seen before) and stimulus expectation (i.e., whether a stimulus is expected to occur, based on the context). Neurophysiological studies in monkeys have shown suppression of spiking activity both for expected and for familiar items in object-selective inferotemporal cortex (IT). It is an open question, however, if and how these types of knowledge interact in their modulatory effects on the sensory response. In order to address this issue and to examine whether previous findings generalize to non-invasively measured neural activity in humans of both sexes, we separately manipulated stimulus familiarity and expectation, while non-invasively recording human brain activity using magnetoencephalography (MEG). We observed independent suppression of neural activity by familiarity and expectation, specifically in the lateral occipital complex (LOC), the putative human homologue of monkey IT. Familiarity also led to sharpened response dynamics, which was predominantly observed in early visual cortex. Together, these results show that distinct types of sensory knowledge jointly determine the amount of neural resources dedicated to object processing in the visual ventral stream.