Humans show reproducible sex-differences in cognition and psychopathology that may be contributed to by influences of gonadal sex-steroids and/or sex-chromosomes on regional brain development. Gonadal sex-steroids are well known to play a major role in sexual differentiation of the vertebrate brain, but far less is known regarding the role of sex-chromosomes. Our review focuses on this latter issue by bridging together two literatures that have to date been largely disconnected. We first consider “bottom-up” genetic and molecular studies focused on sex-chromosome gene content and regulation. This literature nominates specific sex-chromosome genes that could drive developmental sex-differences by virtue of their sex-biased expression and their functions within the brain. We then consider the complementary “top down” view, from magnetic resonance imaging studies that map sex- and sex chromosome effects on regional brain anatomy, and link these maps to regional gene-expression within the brain. By connecting these top-down and bottom-up approaches, we emphasize the potential role of X-linked genes in driving sex-biased brain development and outline key goals for future work in this field.