Evaluation and Selection Process for Symposia and Single Presenter Submissions

Below, we describe the evaluation process in an effort to improve transparency surrounding the convention.

Who are the reviewers? How are they selected?

1. New this year: A Review Board has been created to review submissions for the SPSP Annual Convention. All single-presenter podium presentations and symposia submissions will be evaluated by three or more Review Board members. Posters and round table unconference submissions will be reviewed by two Review Board members.
2. SPSP Review Board members with a PhD are able to review all the submission types. Graduate students are eligible to review posters.
3. The Science Program Review Co-Chairs decide on the final pool of reviewers from the self-nominated group, seeking to ensure a range of expertise areas.
4. Once the submission portal has closed, all submissions are matched to reviewers using keywords, primarily matching the first keyword listed. We aim to give reviewers an equal number of submissions to review.

Podium Presentations: Reviewers evaluate all submissions based on criteria specified by the Science Program Review Co-Chairs. What are those criteria and how are they used?

1. Submissions are masked, so the reviewers cannot identify the authors; submissions are not evaluated based on the names of the people involved.
2. Reviewers consider several dimensions when evaluating submissions:
   • Strength and rigor: Does the research reflect best practices, including issues of statistical power? Are studies well-designed to answer the research question(s)? If the session includes applied or non-empirical talks, do these present strong arguments or clear evidence toward the goals of the session?
   • Contribution: Does the submission address a question or set of questions that substantially advances our knowledge of a theoretical and/or practical/applied contribution in social and/or personality psychology?
   • Interest-value: Will the submission cut across subfields or bridge basic and applied work in an integrative way? Will it spark or make a meaningful contribution to conversations in social and/or personality psychology? Will the audience think they have learned something new? Are there other reasons to expect it to strongly appeal to social-personality psychologists?
   • Rating Scale:
     4: Excellent (top 25% of the submissions you are reviewing, top quartile)
     3: Very Good (falls in the second quartile)
     2: Good (falls in third quartile)
     1: Weak (falls in bottom quartile)
     
     Reviewers are asked for a rectangular distribution in which they assign only 3-4 excellent, only 3-4 very good, only 3-4 good, and only 3-4 weak. The quality of symposia submitted is usually very high, and reviewers must make relative judgments within the set of reviews assigned, even if they might be inclined to rate most of the assigned submissions as "excellent".
   • Reviewers will be instructed to check a box to indicate that the submission contains content that may be of concern (e.g. hate speech, harmful or racist content, potential AI, questionable scientific integrity, unclear or invalid methodology). These submissions will receive additional review by the designated member of the Convention Committee to ensure that the submission meets the SPSP's scientific standards and to evaluate the potential risk of harm of the submission. If additional expertise is required, it will be shared with the Diversity & Climate Committee for review.
   • The review process is independent each year; the content of prior conventions does not factor into the evaluation process, with one exception: If there is a significant amount of feedback during the post-convention survey indicating attendees would like to see more representation of a particular topic, the committee may increase representation of the topic at the following year's convention.
   • When making a final decision about which submissions to accept, the Science Program Review Co-Chairs rely primarily on the reviewers' rubric scores. The Science Program Review Co-Chairs will first consider the average scientific rigor scores to identify high quality research and then will look at overall ratings. Submissions are unmasked at this stage, allowing the Science Program Review Co-Chairs to ensure that the program features a diverse range of speakers. When making the final selection, reviewers aim to accept high-quality submissions while creating a balanced and diverse program. There are no acceptance quotas for research subjects, but there may be general targets to ensure certain topics are adequately represented. Beyond that, any themes that may emerge do so organically based on having a high number of high-quality submissions that year.
   • Diversity, Equity, Inclusion, or Anti-Racism (DEIA) - To aid in the process of creating a balanced and diverse program, we ask submitters to indicate whether their work is of diversity, equity, inclusion, or anti-racism (DEIA) interest. This information will be used as one of the many factors considered when the Convention Committee is making the final selections among the pool of submissions that reviewers have identified as being of high scientific quality and interest.
   • Data blitzes and single-presenter symposia that receive the highest scores from reviewers are examined carefully to create these sessions. In many cases, several high-quality submissions will cluster together in terms of content; these will be collected into a symposium and the presenters will be invited to work together to select a chair, title, and description of the program. The other highest-rated submissions that don't cluster into a symposium set will be included in one of several data blitz sessions scheduled for the convention (provided the speaker is eligible for a data blitz).
   • Submissions that do not score highly enough to be accepted as a single presenter talk are then considered for poster submissions (if the submitter indicated they wished to be considered for a poster). Single presenter submissions that score high enough to clear the threshold for acceptance as a poster are then added to the program.

Note: Responses from the convention exit survey indicated attendees would like more research on personality, relationships, and international research. We encourage submissions in this area, but cannot guarantee acceptance.

Poster and Roundtable Unconference Submission Reviewing Process

When reviewing posters, many of them are theoretical and/or won't have data completely collected which is permissible to submit. Reviewers consider several dimensions when evaluating poster submissions, though they do not provide scores for each dimension:

• ✓ Strength and rigor: Does the research reflect best practices, including issues of statistical power? Are studies well-designed to answer the research question(s)? If the session includes applied or non-empirical talks, do these present strong arguments or clear evidence toward the goals of the session?
• ✓ Contribution: Does the submission address a question or set of questions that substantially advances our knowledge of a theoretical and/or practical/applied contribution in social and/or personality psychology?
• ✓  Interest-value: Will the submission cut across subfields or bridge basic and applied work in an integrative way? Will it spark or make a meaningful contribution to conversations in social and/or personality psychology? Will the audience think they have learned something new? Are there other reasons to expect it to strongly appeal to social-personality psychologists?

Rating Scale for Posters

• Acceptable for poster
• Not acceptable for poster (Note if a reviewer deems a submission as unacceptable for a poster, we ask them to provide comments to be able to share with the committee and submitter for explaining this rating.)
• Reviewers will be instructed to check a box to indicate that the submission requires further review due to containing content that may be of concern (e.g., inappropriate topic. hate speech, harmful or racist content, potential AI, questionable scientific integrity), they can flag, unclear or invalid methodology). These submissions will receive additional review by the designated member of the Convention Committee to ensure that the submission meets the SPSP's scientific standards and to evaluate the potential risk of harm of the submission. If additional expertise is required, it will be shared with the Diversity & Climate Committee for the decision-making team.
• We do not anticipate many "Not acceptables", so this rating should only be given if the submission is totally irrelevant to personality and social psychology, incomprehensible, or lacks scientific strength and rigor.

If the reviewer believes the submission requires further review due to concerning content (e.g., inappropriate topic, potential AI, questionable scientific integrity), they can flag the submission for the decision-making team. 

When making a final decision about which submissions to accept, the Science Program Review Co-Chairs rely primarily on the reviewers' rubric scores.

How is the final schedule determined?

1. SPSP staff members create a grid that ensures primary keywords do not overlap on blocks of the schedule.
2. The Convention Committee and Science Program Review Co-Chairs adjust the grid if they see sessions with overlapping content areas that are scheduled at the same time.
3. SPSP staff members send a list of all accepted symposia to the symposium chairs, who indicate up to three sessions that should not be scheduled at the same time due to content overlap. Organizers consider this information when finalizing the grid.

What are the historical rates of acceptance for prior conferences? 

Since we have significantly revised our submission evaluation process in recent years, the data below represents up to 2024:

• Submission and acceptance rates
• Submission and acceptance rates by primary keyword