In the present cross-sectional study, we examined physical activity (PA) and its possible association with cognitive skills and brain structure in 331 cognitively healthy elderly. Based on the number of self-reported light and hard activities for at least 30 minutes per week, participants were assigned to 4 groups representing different levels of PA. The cognitive skills were assessed by the Mini Mental State Examination score, a verbal fluency task, and the Trail-making test as a measure of visuospatial orientation ability. Participants also underwent a magnetic resonance imaging of the brain. Multiple regression analysis revealed that greater PA was associated with a shorter time to complete the Trail-making test, and higher levels of verbal fluency. Further, the level of self-reported PA was positively correlated with brain volume, white matter, as well as a parietal lobe gray matter volume, situated bilaterally at the precuneus. These present cross-sectional results indicate that PA is a lifestyle factor that is linked to brain structure and function in late life.