98

I’ve submitted a paper to a reputable journal and received two reviews asking for a revision. One of the reviews is on-topic and helpful. The other, however, I suspect was generated by an AI based on the abstract only. I don’t want to debate the decision (the second reviewer is fair), but I wonder if I should mention my suspicions to the journal editor.

Should I? I doubt I can prove the use of AI. All I have is a strong suspicion. Plus, I don’t want to hurt my article. Yet, using AI for generating (not merely styling) reviews seems like such a bad practice that maybe it’s worth letting the editor know.

My suspicions are based on several things:

  • First, the style is off. The review consists only of very long questions like:

    In terms of the paper's stated goal of contributing to the literature, how well does it provide [contribution mentioned in abstract] for [use-case mentioned in abstract]?

    There are no suggestions, no feedback, and no statements. It’s only broad questions suspiciously rephrasing each line of the abstract (and heavily re-using entire phrases from it).

  • Second, the list of suggested articles is strange and includes existing papers (only names) from irrelevant fields.

  • Third, out of curiosity, I've run the text via several online AI-detection tools, and all of them conclude it’s AI (which I am aware is not a valid proof on its own).

Improve this question
6
  • 7
    To clarify some points, have you gone back and glanced over the entire review? Have you attempted to check into any of the suggested articles that, at surface-level, don't make sense? More or less, have you done due diligence to assert your suspicion? In my opinion, you should be treating this as if you're about to accuse the reviewer of misconduct.
    – David S
    Commented Nov 30, 2023 at 0:34
  • 4
    Anecdotally, I've found that many people are using AI as a super spell checker, running whatever they wrote through AI to make the writing better. Assuming good faith, that might not necessarily mean the reviewer is asking AI to review.
    – Passer By
    Commented Nov 30, 2023 at 4:38
  • 7
    @DavidS Yes. My suspicions are based on the entire review, not some selected sentences. I've checked that all articles exist, then quickly went through 3 out of 6, but I don't see how they can be relevant. If there is a higher wisdom in those links, I don't see it. As for accusations, I don't know what kind of due diligence I'm supposed to do. All I have is a suspicion and no fail-proof method to prove or disprove Al-use.
    – pintor
    Commented Nov 30, 2023 at 9:15
  • 3
    @PasserBy I don't mind Al being used for styling, grammar checking (I'm surely guilty of that), and even expanding ideas into a text (within reason). But this review doesn't read like Al-edited. My feeling is it was written entirely by Al with minimal human involvement. However, again, I have no proof.
    – pintor
    Commented Nov 30, 2023 at 9:25
  • Apart from any other consideration, I think it's worth considering the apparent reviewer's motive. Specifically, was he being paid even a nominal fee for doing the work, or in some way accumulating credit for advice given and (gratefully) received?
    – Mark Morgan Lloyd
    Commented Dec 1, 2023 at 7:16

5 Answers 5

Reset to default
120

I think you should contact the editor with basically the same content as your post here: explain why you think the review could be AI generated, including the limitations of each point of evidence. I think most editors see their job as letting reviewers guide their decision rather than reviewing the reviews, so I would start from the assumption that the editor would not have considered checking the reviews for suspicion of gen AI.

Explain why you think this is concerning, which I think at minimum would include the integrity of the peer review process (you and your colleagues are expecting papers to be reviewed by humans, not AI) and the protection of your intellectual property (having an AI review your paper involves inserting your work into a 3rd party tool which may incorporate it into the training data and is a definite breach of the confidentiality that reviewers are sworn to). I would check the journal's website to see if they have an explicit statement about use of AI in peer review; if so, you can refer to that as well. I think you can do this all in one sentence to not appear to lecture the editor.

Finally, make clear that you're not protesting the decision for a revision and plan to work on revising and resubmitting your paper. I'd include a note thanking the editor and the reviewers for their time.

From there, let the editor decide what should be done next; it's their call, your only duty here is to raise their awareness. It's possible they'll want to bring in another reviewer. It's also possible they'll decide it's okay. At that point you'll have to decide whether you're satisfied with their effort and whether that affects your opinion of the journal as an appropriate place to submit your work.

Improve this answer
17
  • 29
    Yes (YESSS!), editors should be made aware of this so that appropriate policies can be developed. It goes beyond the concerns of a particular author or paper and requires deep thought.
    – Buffy
    Commented Nov 29, 2023 at 16:31
  • 4
    Thank you! I've checked peer-reviewing guidelines, but it says nothing about explicitly prohibiting Al-use for reviewers. At least not at first glance. Though, implicitly, it can be tied to copyrights (as you mentioned). However, I found that the journal has an ethics team. Do you think it's better to ask them instead of the editor?
    – pintor
    Commented Nov 29, 2023 at 16:59
  • 21
    @pintor Not too surprising, this genAI stuff is still pretty new and publishing moves slowly, and I agree with you that it's likely already covered by standard confidentiality policies even if not explicit. I'd start by working with the editor. You might possibly ask the editor if this is something the ethics team could help out with, but I'm guessing they exist more as a check on the editorial team when editors run amok (or, cynically, they just exist to quash ethical complaints). Going to them directly may be seen as an accusation against the editor, which you don't intend to make.
    – Bryan Krause ♦
    Commented Nov 29, 2023 at 17:02
  • 3
    While I agree with the first, third, and fourth paragraph, I feel your recommendations in the second one might be overdoing it. It feels like lecturing the editor on something they should know. I would only go to these length if the editor doesn’t see a problem with an AI-generated review.
    – Wrzlprmft ♦
    Commented Nov 30, 2023 at 7:41
  • 1
    @Wrzlprmft They certainly will have to be mindful about the phrasing, though second paragraph has a very important point. Depending on the AI tool (allegedly) used, users consent to the input becoming a part of the datapool. Therefore, even if the paper has no rules regarding AI use, confidentiality still might have been breached. From my point of view, that is the top possible problem and should be highlighted accordingly.
    – vspmis
    Commented Nov 30, 2023 at 10:58
8

The Committee on Publication Ethics and related organizations are developing reporting and investigational best practices for this (COPE). It may be considered misconduct according to the policies of the journal, and if not disclosed, it most likely is and should be investigated by the editors. If they do not take your report seriously, I would question the integrity of the journal--see related material on predatory publishing.

In addition to reporting this to your editor for investigation, as @Bryan Krause stated above (+1 to everything he said), I would familiarize yourself with the work of the committee on this topic, as they are monitoring these issues closely and developing best practices and even formal guidelines on handling investigations.

COPE also provide significant resources to editors who may actually be less versed in dealing with these issues themselves, so it couldn't hurt to share that information with your editors if they need a bit of guidance.

COPE. 2023. Where next in Peer Review? Part II. https://publicationethics.org/news/where-next-peer-review-part-1.

Improve this answer
5

I have been a journal editor for 20 years, and I would not pass on this review to the author. Definitely alert the journal.

Improve this answer
2

Edit in response to discussions here.

I think my answer is wrong. @BryanKrause and @Buffy and @leonos are right. The OP should contact the editor immediately.

Leaving this post up so the back and forth is visible.

Interesting puzzle.

Since the (possibly AI prepared) review is generally positive, treat it as if it were legitimate. Since the reviewer asks for no changes you need not argue about whether they are appropriate. If any of the suggested references are in fact relevant, include them.

After the paper is accepted you could contact the editor to voice your suspicions, with whatever supporting evidence you have.

Edit after reading @BryanKrause's answer.

Clearly you will revise your manuscript in response to the good suggestions from the unquestionably legitimate review. If you think the work would benefit from a second good review then you should tell the editor now about your suspicions.

Improve this answer
5
  • 14
    This suggestion feels unethical. Only raise the problem if the review was negative? Either using “AI” to write a review is a problem, or it is not, this should be independent of the outcome of the artificial review.
    – user126108
    Commented Nov 29, 2023 at 16:04
  • 3
    @leonos Perhaps. My thought was that the ethics of the situation did not require the OP to risk losing the chance to publish by possible antagonizing the editor. There are plenty of lazy referees out there. I hope the OP will report back here when the dust has settled.
    – Ethan Bolker
    Commented Nov 29, 2023 at 16:09
  • 10
    in my opinion, this is a clear ethical issue and the authors should send a clear and immediate message that it is not acceptable - particularly because the journal might not care that much. Imagine if journals could just dispense from finding reviewers and artificially generate reviews at no additional time cost…a gold mine for predatory journals. It’s quite concerning that the editor did not even notice or care about that report in the first place.
    – user126108
    Commented Nov 29, 2023 at 16:34
  • 2
    @leonos You're right. See my edit.
    – Ethan Bolker
    Commented Nov 29, 2023 at 17:32
  • 5
    Ethan - I saw it! Thanks for the edit and for being open to changing your mind!
    – user126108
    Commented Nov 29, 2023 at 17:38
-8

I suspect one of the journal reviews I got is Al-generated I wonder if I should mention my suspicions to the journal editor.

I don't think that's a relevant criterion. Whether the review is useful is a more important criterion.

You could focus on the lack of constructive feedback or how the comments seemed to closely align with your abstract without offering substantial insights or suggestions for improvement.

Expressing gratitude for the helpful feedback from one reviewer while politely pointing out the absence of actionable feedback from the other might prompt the editor to consider the nature of the review.

Improve this answer
4
  • 12
    Strongly disagree; it's highly relevant, for the reasons I point out in my answer.
    – Bryan Krause ♦
    Commented Nov 29, 2023 at 18:49
  • 21
    In addition, the peer-reviewing system relies on the implicit trust that the editor will choose reviewers that are knowledgeable experts of the relevant field. At this stage, the language models are not knowledgeable, experts, or even concerned about truth at all. Using one to write a peer-review completely misses the point of peer-review which is to promote scientific accuracy.
    – user126108
    Commented Nov 29, 2023 at 20:13
  • 6
    I don't disagree with the notion that an AI reviewer could feasibly give useful feedback on a paper. But no matter how you spin, it is not peer review - a journal claiming their articles are peer reviewed when they are not is a common fraud tactic of predatory journals. Not good company to be among.
    – Nuclear Hoagie
    Commented Nov 30, 2023 at 14:26
  • 1
    @leonos I agree, and the "at this stage" is important.
    – Volker Siegel
    Commented Dec 2, 2023 at 1:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .