In this review, we explore the association among physical activity, cardiorespiratory fitness, and exercise on gray matter volume in older adults. We conclude that higher cardiorespiratory fitness levels are routinely associated with greater gray matter volume in the prefrontal cortex and hippocampus and less consistently in other regions. We also conclude that physical activity is associated with greater gray matter volume in the same regions that are associated with cardiorespiratory fitness including the prefrontal cortex and hippocampus. Some heterogeneity in the literature may be explained by effect moderation by age, stress, or other factors. Finally, we report promising results from randomized exercise interventions that suggest that the volume of the hippocampus and prefrontal cortex remain pliable and responsive to moderate intensity exercise for 6 months–1 year. Physical activity appears to be a propitious method for influencing gray matter volume in late adulthood, but additional well-controlled studies are necessary to inform public policies about the potential protective or therapeutic effects of exercise on brain volume.