Category Archives: Peer Review

To mark Peer Review Week 2024, Amy Lynn caught up with Claire Cameron and Gabriel Eichsteller, editors of the International Journal of Social Pedagogy to discuss the importance, challenges and considerations of peer review.

Amy Lynn (AL): What are the challenges with getting a good peer review?

CC/GE: There are two main challenges: Good reviewers are often very busy academics and have limited time to undertake extra work such as doing a thorough peer review. Also, particularly for niche topics, it can be hard to find peer reviewers who have just the right mix of expertise. Particularly as an international journal, we rely on reviewers who have a detailed understanding of the authors’ national context as well as thorough thematic expertise.

AL: How do you select reviewers for each paper?

CC/GE: We often select editorial board members or previous authors as reviewers, so have built up a pool of reliable reviewers over the last 12 years. We also ask them for recommendations from within their professional networks. And we draw on our own professional networks within the editorial team. Every now and again, we need to do a bit more research to find potential reviewers who have the right expertise.

AL: If you could give three key points for reviewers to consider when they are writing what would you say?

CC/GE: A lot of authors tell us that the peer review process hugely adds to the quality of their publication. And it helps us as editors make sound decisions. That’s easiest when reviewers are constructive (pointing out how authors can strengthen their argument rather than critiquing what’s wrong), are clear and specific (so that authors know exactly where their argument might lack focus or what they might not have considered), and focus on the content and contextual factors that an international with a variety of different backgrounds might need to be aware of.

AL: How would you encourage early career researchers to get involved in the peer review process?

CC/GE: Ask to read a few initial submissions and practice your critical thinking skills, compare the review recommendations you would give with the review reports submitted to the journal, and then check out how these are reflected in the published manuscript. You could also do a peer review together with a more experienced colleague. Definitely play to your strengths, so you can approach journal editors and offer to review any submissions that are clearly within your area of expertise. You can also ask for guidance from the editors and for feedback before you log your peer review. Even some very experienced peer reviewers ask every now and again whether their reviews are sound. And don’t worry about getting it wrong! Authors might disagree with your comments but will reflect on them (which is important), and editors usually appreciate different perspectives and interpretations, which enable them to make better publishing decisions.

AL: How should authors consider peer review comments when revising their paper?

CC/GE: Consider them a gift and try not to become defensive. Instead, read them with a focus on how these comments can help you write an even better paper. Peer reviewers are reflective of your paper’s future readers, so each review tells you something important about how your argument will be received and what you might need to strengthen. Remember that the world’s greatest novelists go through loads of edits before their work is published. It’s an essential part of the writing process.

AL: This year’s theme for peer review week is Innovation and Technology in Peer Review. With the advent of A.I. tools to assist writing up research, do you imagine A.I. could be used to provide a good review?

CC/GE: A.I. can be really helpful in some respects, such as improving an article’s structure and focus, ensuring that the writing is clear and concise, that spelling, grammar, referencing styles, etc. are correct. As A.I. has been trained on an incredible amount of publications, it can also potentially help interrogate an article’s rigour. So it can add to the review process in several important ways. However, I don’t think it can replace the professional judgment which a good peer reviewer can provide, because some of that is about the wider contribution to their field’s discourse and how well this paper might fit into that.

AL: Do you have a message for the journal’s peer reviewers?

CC/GE: You have helped us bring out the very best in the authors who have published with us! I’m hugely appreciative of your reviews – not just because they improve the quality of the papers we accept, but also because they reflect social pedagogical principles of being constructive, supportive and collaborative.

---

About the editors

Amy Lynn is Journals Development Editor at UCL Press. Her portfolio covers a variety of topics includes UCL IOE’s flagship journal London Review of Education and the International Journal of Social Pedagogy.

Claire Cameron is Professor of Social Pedagogy at the Thomas Coram Research Unit, UCL IOE. She is co-editor of the UCL Press open access books Transforming Early Childhood in England (with Peter Moss) and Social Research for our Times (with  Alison Koslowski, Alison Lamont and Peter Moss) in addition to co-editing International Journal of Social Pedagogy..

Gabriel Eichsteller is co-founder of ThemPra Social Pedagogy, a social (pedagogical) enterprise supporting the development of social pedagogy in the UK through short courses, capacity-building programmes and strategic development. He is also co-editor the International Journal of Social Pedagogy.

To mark Peer Review Week 2024, Sunbul Akhtar caught up with the editors of History Education Research Journal (HERJ). In this in-depth Q&A, which originally appeared on the HERJ Blog, they discuss the thought processes involved in peer review, the challenges behind the scenes and AI, providing an insight into a function of academic publishing that has become a cornerstone of research integrity.

Sunbul Akhtar (SA): How do you select reviewers for each paper?

Arthur Chapman (AC): Expertise in the relevant fields and subfields is key. Where this is not possible – for example, because the subfield that the article addresses is quite niche – general expertise in the overall field would be key.

I also look out for conflicts of interest so, if the reviewer and the author are from the same country, I look at if they are at the same university or have an obvious link.

Andreas Körber (AK): In a practical sense, the journal uses an online submission system which allows me to search through the database of past reviewers and authors. I use this to find someone in our database who has worked in the same field, but also not too closely on the same questions.

When a paper combines perspectives/approaches/methods, I try to find reviewers to address each of these.

Lindsay Gibson (LG): I also try to consider inviting reviewers who have not reviewed for HERJ in a while so we don’t call on the same people to review articles. 

Gideon Boadu (GB): I also search through the journal’s database for suitably aligned potential reviewers. In many cases, I do a general Google search for researchers with similar interests. I consider the subject matter expertise of potential reviewers and how it aligns with the paper. Another point I consider is the geographical location or region the paper is coming from. In many cases, selecting reviewers from the same region helps to ensure that contextual factors shaping the paper, like curricula, policy, etc, are considered in the review. Where papers are not tied to specific geographical regions, for example conceptual or theoretical papers, expertise becomes paramount.

SA: What are the challenges with getting a good review?

AC: Availability is the big one. Colleagues in higher education are often very busy and they are often asked to do many reviews in addition to their contracted duties.

LG: Time is a major issue. It takes a long time to complete a review, and academics are not known for having huge amounts of unaccounted for time.

AC: It’s rare to have a poor review – the people I approach usually take the role very seriously and give it the time and attention that the task deserves. On rare occasions, a review is too short or superficial. I would not rule out seeking a new review from a different academic in that situation.

LG: Also, sometimes reviewers do not provide authors with specific and descriptive feedback that can help them improve their article, or their comments are overly critical and do not focus on the positive things about the article.

GB: Since research methodologies and traditions are so diverse, not all reviewers have expertise in the specific methodologies used in a paper. This sometimes affects reviews to a good extent.

AC: On rare occasions reviews can be a little harsh or unkind in their language. In such cases, I would look to mediate that and counter-act it through my editorial comments to the author/s in the message I use to advise of review outcomes.

LG: It’s important to remember that authors are fellow academics and our obligation is to help colleagues improve the quality of their writing and research. 

AC: The huge challenge is getting reviews completed in a timely manner. People are very busy and need a lot of reminding and cajoling.

SA: If you could give three key points for reviewers to consider when they are writing what would you say?

AC: Focus on the criteria that you are asked to review to. Be tough but fair and considerate. The republic of letters should be tough but collegiate. 

Be as precise as possible – a concise set of observations is easy to respond to. Bullet pointing the key issues you wish to see addressed at the end of a review can be very helpful.

Suggest courses of action – if you feel that key literature is not consulted, for example, provide some pointers.

AK: I would say, encourage the reviewers not to be too narrow as to how they would have written the article themselves or to it having been written. Reviews are not about aligning but about a kind of broad gatekeeping.

Specifically, it is not about whether you agree to all points – something must still be left open for community to critically engage with after publication – but as  to whether the article with its specific position, perspective, methods and results is “fit to print” and to enter the discourse.

In some cases, reviewers might even highlight points they do not readily agree with, not in order to have them corrected but to point out that they might be especially valuable for further discussion.

LG: Focus on the review criteria, not your personal preferences. Write the review that you would want to receive as an author. 

GB: Offer constructive feedback that helps the authors to improve on their paperwork.

It is peer-review. Consider how you’d expect your own work to be reviewed by a peer.

A paper cannot be perfect. You are contributing to improving what has been submitted to a journal for consideration.

SA: How would you encourage early career researchers to get involved in the peer review process?

AC: Early career researchers have so much to offer and often make the best reviewers, because they can be the most conscientious reviewers.

Reviewing is great for writing – it helps one understand how one’s own papers will be judged. It’s a little like having a viva on paper. Identify journals that deal with your areas of expertise, and then write to the editor/s offering your services and specifying the areas you feel competent to review. I would predict an enthusiastic response from editors – as I’ve said, getting the reviewers we need is a big challenge in a busy academic world. Volunteers will always be welcome in that context.

Make sure to be firm and to decline an invitation, however, if it’s outside your expertise. Also, I don’t think an academic need do more than 2-4 reviews a year. Don’t allow editors to make too many demands on your time.

LG: Email journal editors to see if they are accepting new reviewers – most are. Only review articles you’re interested and have expertise in. 

GB: Reviewing as an early career researcher helps you not only to engage with others’ work prior to publication but also build collegial relationship with editors, some of whom you are likely to ‘meet’ in the course of your career.

It prepares you for your own future research supervision.

It is a way of contributing to the field, aside from your own papers, and ensuring that the field continues to grow.

SA: This year’s theme for peer review week is Innovation and Technology in Peer Review. With the advent of AI tools to assist writing up research, do you imagine AI could be used to provide a good review?

AC: I don’t have much to say here. Sorry!

AK: I am doubtful. Reviewing is judging, and reviewers should not be identifiable with their clear names, but their professional expertise should be discernible. It may be that some tools may help in formulating, especially if English is not the main or first language of the reviewer, but the risk to become too formulaic may be even higher than that in the original writing of a paper, given that reviewing is a communication between concrete persons – even if anonymous.

LG: I’m also doubtful. Reviewing articles requires background knowledge, application of review criteria, and thoughtful and reasoned judgment. I’m not convinced AI is able to do this. 

SA: Finally, do you have a message to the reviewers of the journal?

GB: Thank you for contributing your time and expertise to HERJ.

AC: I would like to thank all the colleagues who provide, and who have provided, reviews for the History Education Research Journal. A journal – and a field of research – is only as good as its peer reviewers. Peer reviewers are vital to what we do and their contribution to keeping the republic of letters running well is impossible to overstate. Thank you all for helping to ensure and to enhance the quality of history education research!

---

About the authors

Sunbul Akhtar is Journals Development Editor at UCL Press. Her portfolio is wide-ranging includes UCL Open: Environment, History Education Journal and Architecture_MPS.

Arthur Chapman is Professor of History Education and Head of Department, Curriculum, Pedagogy and Assessment at UCL IOE in addition to being Editor-in-Chief of History Education Research Journal. He is co-editor of the UCL Press open access book series Knowledge and The Curriculum, which includes his open access book Knowing History in Schools.

Andreas Körber is Editor of History Education Research Journal and Professor and Head of the History Education Subject Group at Universität Hamburg, Germany

Lindsay Gibson is Editor of History Education Research Journal and Assistant Professor at the Faculty of Education at The University of British Columbia, Canada

Gideon Boadu is Editor of History Education Research Journal and Lecturer of Secondary Curriculum & Pedagogy Education at RMIT University, Australia

When discussing peer review, generally, I find it helpful to remind myself of some of the values as to perhaps why researchers publish in scholarly journals. In essence, it usually comes down to these 4 headings.

1. Knowledge and discovery
2. Evaluation and validation
3. Credit
4. Access to research

Within this environment, peer review is arguably the fundamental gold standard aspect of scholarly and academic publishing and is, at least in its most fundamental use of it, the facilitator for publishers to sell journals and its content.

So then, what is the role for peer review precisely, and what does it serve to accomplish?

An easy question to answer, right? In the book Editorial peer review: It’s strengths and weaknesses, the author writes the role of what peer review serves is, as ‘the goal of the process is to ensure that the valid article is accepted, the messy article cleaned up, and the invalid article rejected,’ thereby ensuring that the article made available to the reader is quality controlled.

In another book titled Peer review: A critical inquiry, the author here writes that the process of peer review also benefits the author, as they are later certified by the process if published proceeding peer review: “Careers are often made or destroyed by the process.”

In scholarly publishing, peer review acts to validate and assess work and is the current system used to assess the quality of a manuscript before it is published. Other experts in the relevant field assesses the research or article for things like fact, validity, and significance, that aid the assessors (i.e. Editors) to determine whether the manuscript should be published in the journal or not. I think it is pertinent to remember here that journals do play a vital role in the scientific and scholarly process, by refining research through peer review and disseminating it to appropriate communities by publication, and it is this role of review by peers that has been a part of scholarly communication since the appearance of the first journal in the 17th Century (see the brilliant book by Professor Aileen Fyfe et al, A History of Scientific Journals: Publishing at the Royal Society, 1665-2015.)

Challenges in peer-review

There has been a lot of discussion around the challenges peer review present, stemming from bias and prejudices towards authors, fraudulent behaviour, non-expertise reviews, and so on. In the article Peer review in a changing world: An international study measuring the attitudes of researchers by Mulligan et al in 2012, notes that:

“Although alternative forms of peer review have evolved to tackle issues of bias, it is less clear what effect, if any, they will have upon fraud. High‐profile cases of fraud and plagiarism have brought the debate about the efficacy of peer review to a wider audience, attracting greater public attention. Such incidences include [certain individuals], tipped to be a Nobel Prize winner, who published a series of fraudulent papers that were withdrawn from Nature, Science, Physical Review, and Applied Physics Letters.”

Journals typically tackle these types of concerns by anonymising authors and reviewers from each other to ‘enable a fairer and just review system’. In this article Mulligan et al surveyed around 40,000 published researchers that were randomly selected from the Web of Science (then known as the Thomson Reuters ISI list) and concluded that the majority of respondents were happy with the current system, but noted the system is imperfect and more can be done to ensure a higher level of efficacy and efficiency.

Now, being led by open science principles, it is largely seen that being more open and transparent with research publication and assessment can we increase scholarly rigour, accountability and trust.

What is open peer-review?

There is a growing evidence base of the challenges and flaws in the current anonymised peer review system (albeit, mainly within the biomedical and clinical sciences), and major publishers and journals are already testing open peer review processes (or have already implemented a practice of it already).

In April 2017, a systematic review of what open peer review is was published online in F1000Research (itself an innovative model of open peer review). It concluded: “Open peer review has neither a standardized definition, nor an agreed schema of its features and implementations. The literature reflects this, with a myriad of overlapping and often contradictory definitions.”

What this review very accurately depicts, is that there are a number of definitions of open peer review that can be collated together into themes and it purports there are 7 open traits to what open peer review concerns itself with, and that open peer review can take either a single aspect, or a multitude or mix of any of these traits, to operate as an open peer review model. Briefly, these are:

1. Open identities, where authors and reviewers are aware of each other’s identity.
2. Open reports, where the review reports are published alongside the relevant article
3. Open participation, where the wider community are able to contribute to the review process
4. Open interaction, where direct reciprocal discussion between author(s) and reviewers, and between the reviewers themselves, is allowed and encouraged
5. Open pre-review manuscripts, essentially, a pre-print server, where manuscripts are made immediately available (e.g., BiorXiv) in advance of any formal peer review procedures
6. Open final-version commenting, where the review or commenting on the final “version of record” is published
7. Open platforms (or “decoupled review”), where review is facilitated by a different organizational entity than the venue of publication

What is UCL Press doing?

At UCL Press, we launched our very own open peer review and open science journal called UCL Open Environment: a fully non-commercial, Open Science journal, publishing high impact, multi-disciplinary research, on real world environmental issues, with the overall aim of benefitting humanity. The journal is for any researcher or professional at knowledge-based universities, institutions, and organisations (including Non-Government Organisations, Think Tanks, Inter-Government Organisations, and the United Nations) and submissions are invited from those at all career stages, including early career researchers, mid-career professionals, and senior scholars. There are also no barriers to the Open Peer Review Process (whereby the identity of the reviewer and the report are made publicly visibly at all times); engagement from all will advance the greatest leaps and discoveries.

Reviewers are firstly asked to sign in to the system using their ORCID account and when they submit their review report, the report is posted up online in the preprint server alongside the article, under the CC-BY licence and assigned a unique DOI. You can find out more information about this at https://journals.uclpress.co.uk/ucloe/site/how-it-works/

Reviewers can therefore attain credit of their report and readers are able to follow the process openly online. We hope this will also aid the development for others (especially earlier career researchers and students) with examples on how a review is written and how an article is revised accordingly, aiming to improve the way we should engage critically and beneficially with research.

Readers of this blog can see for themselves how the journal works (you can see here the list of the latest submissions and open peer reviews, as well as here for publications accepted after peer review). It is my hope that readers will be encouraged to provide more open peer reviews or open comments, adding to the corpus of open debate around research, and consider contributing to UCL Open Environment, as we believe that by removing barriers and innovatively working openly and together will we accelerate finding solutions to the world’s most significant challenges.