Previous work indicates that first impressions of autistic adults are more favorable when neurotypical raters know their clinical diagnosis and have high understanding about autism, suggesting that social experiences of autistic adults are affected by the knowledge and beliefs of the neurotypical individuals they encounter. Here, we examine these patterns in more detail by assessing variability in first impression ratings of autistic adults (N = 20) by neurotypical raters (N = 505). Variability in ratings was driven more by characteristics of raters than those of autistic adults, particularly for items related to "intentions to interact." Specifically, variability in rater stigma toward autism and autism knowledge contributed to first impression ratings. Only ratings of "awkwardness" were driven more by characteristics of the autistic adults than characteristics of the raters. Furthermore, although first impressions of autistic adults generally improved when raters were informed of their autism status, providing a diagnosis worsened impressions made by neurotypical raters with high stigma toward autism. Variations in how the diagnosis was labeled (e.g. "autistic" vs "has autism") did not affect results. These findings indicate a large role of neurotypical perceptions and biases in shaping the social experiences for autistic adults that may be improved by reducing stigma and increasing acceptance.