Social development has been the focus of a great deal of neuroscience based research over the past decade. In this review, we focus on providing a framework for understanding how changes in facets of social development may correspond with changes in brain function. We argue that (1) distinct phases of social behavior emerge based on whether the organizing social force is the mother, peer play, peer integration, or romantic intimacy; (2) each phase is marked by a high degree of affect-driven motivation that elicits a distinct response in subcortical structures; (3) activity generated by these structures interacts with circuits in prefrontal cortex that guide executive functions, and occipital and temporal lobe circuits, which generate specific sensory and perceptual social representations. We propose that the direction, magnitude and duration of interaction among these affective, executive, and perceptual systems may relate to distinct sensitive periods across development that contribute to establishing long-term patterns of brain function and behavior.