are primary research articles peer reviewed

Animal Science

How to identify peer reviewed journals, how to identify primary research articles.

• Reference Sources
• Key Journals
• Writing & Citing
• Self Checkout
• Anatomy Study Resources
• Peer Reviewed Journals Quiz How do I know if a journal is peer reviewed? What is peer review, anyway? Take this short quiz to test your knowledge and perhaps learn something new!
• Primary Research Articles Quiz How do I know if an article is a primary or secondary research article? Are there search techniques that will help me find them? Take this short quiz to test your knowledge and perhaps learn something new!

You must get all answers correct to submit the quiz!

Peer review is defined as “a process of subjecting an author’s scholarly work, research or ideas to the scrutiny of others who are experts in the same field” ( 1 ). Peer review is intended to serve two purposes:

• It acts as a filter to ensure that only high quality research is published, especially in reputable journals, by determining the validity, significance and originality of the study.
• Peer review is intended to improve the quality of manuscripts that are deemed suitable for publication. Peer reviewers provide suggestions to authors on how to improve the quality of their manuscripts, and also identify any errors that need correcting before publication.

How do you determine whether an article qualifies as being a peer-reviewed journal article?

• If you're searching for articles in certain databases, you can limit your search to peer-reviewed sources simply by selecting a tab or checking a box on the search screen.
• If you have an article, an indication that it has been through the peer review process will be the publication history , usually at the beginning or end of the article.
• If you're looking at the journal itself, go to the  editorial statement or instructions to authors  (usually in the first few pages of the journal or at the end) for references to the peer-review process.
• Lookup the journal by title or ISSN in the ProQuest Source Evaluation Aid . 
• Careful! Not all information in a peer-reviewed journal is actually reviewed. Editorials, letters to the editor, book reviews, and other types of information don't count as articles, and may not be accepted by your professor.

What about preprint sites and ResearchGate?

• A preprint is a piece of research that has not yet been peer reviewed and published in a journal. In most cases, they can be considered final drafts or working papers. Preprint sites are great sources of current research - and most preprint sites will provide a link to a later, peer-reviewed version of an article. 
• ResearchGate is a commercial social networking site for scientists and researchers to share papers, ask and answer questions, and find collaborators. Members can upload research output including papers, chapters, negative results, patents, research proposals, methods, presentations, etc. Researchers can access these materials, and also contact members to ask for access to material that has not been shared, usually because of copyright restrictions. There is a filter to limit results to articles, but it can be difficult to determine the publication history of ResearchGate items and whether they have been published in peer reviewed sources.

A primary research article reports on an empirical research study conducted by the authors. The goal of a primary research article is to present the result of original research that makes a new contribution to the body of knowledge. 

Characteristics:

• Almost always published in a peer-reviewed journal
• Asks a research question or states a hypothesis or hypotheses
• Identifies a research population
• Describes a specific research method
• Tests or measures something
• Often (but not always) structured in a standard format called IMRAD: Introduction, Methods, Results, and Discussion
• Words to look for as clues include: analysis, study, investigation, examination, experiment, numbers of people or objects analyzed, content analysis, or surveys.

To contrast, the following are not primary research articles (i.e., they are secondary sources):

• Literature reviews/Review articles
• Meta-Analyses (studies that arrive at conclusions based on research from many other studies)
• Editorials & Letters
• Dissertations

Articles that are NOT primary research articles may discuss the same research, but they are not reporting on original research, they are summarizing and commenting on research conducted and published by someone else. For example, a literature review provides commentary and analysis of research done by other people, but it does not report the results of the author's own study and is not primary research.

• << Previous: Home
• Next: Reference Sources >>
• Last Updated: Aug 24, 2023 2:38 PM
• URL: https://libguides.berry.edu/ans

Finding Scholarly Articles: Home

What's a Scholarly Article?

Your professor has specified that you are to use scholarly (or primary research or peer-reviewed or refereed or academic) articles only in your paper. What does that mean?

Scholarly or primary research articles are peer-reviewed , which means that they have gone through the process of being read by reviewers or referees  before being accepted for publication. When a scholar submits an article to a scholarly journal, the manuscript is sent to experts in that field to read and decide if the research is valid and the article should be published. Typically the reviewers indicate to the journal editors whether they think the article should be accepted, sent back for revisions, or rejected.

To decide whether an article is a primary research article, look for the following:

• The author’s (or authors') credentials and academic affiliation(s) should be given;
• There should be an abstract summarizing the research;
• The methods and materials used should be given, often in a separate section;
• There are citations within the text or footnotes referencing sources used;
• Results of the research are given;
• There should be discussion   and  conclusion ;
• With a bibliography or list of references at the end.

Caution: even though a journal may be peer-reviewed, not all the items in it will be. For instance, there might be editorials, book reviews, news reports, etc. Check for the parts of the article to be sure.   

You can limit your search results to primary research, peer-reviewed or refereed articles in many databases. To search for scholarly articles in  HOLLIS , type your keywords in the box at the top, and select  Catalog&Articles  from the choices that appear next.   On the search results screen, look for the  Show Only section on the right and click on  Peer-reviewed articles . (Make sure to  login in with your HarvardKey to get full-text of the articles that Harvard has purchased.)

Many of the databases that Harvard offers have similar features to limit to peer-reviewed or scholarly articles.  For example in Academic Search Premier , click on the box for Scholarly (Peer Reviewed) Journals  on the search screen.

Review articles are another great way to find scholarly primary research articles.   Review articles are not considered "primary research", but they pull together primary research articles on a topic, summarize and analyze them.  In Google Scholar , click on Review Articles  at the left of the search results screen. Ask your professor whether review articles can be cited for an assignment.

A note about Google searching.  A regular Google search turns up a broad variety of results, which can include scholarly articles but Google results also contain commercial and popular sources which may be misleading, outdated, etc.  Use Google Scholar  through the Harvard Library instead.

About Wikipedia .  W ikipedia is not considered scholarly, and should not be cited, but it frequently includes references to scholarly articles. Before using those references for an assignment, double check by finding them in Hollis or a more specific subject  database .

Still not sure about a source? Consult the course syllabus for guidance, contact your professor or teaching fellow, or use the Ask A Librarian service.

• Last Updated: Oct 3, 2023 3:37 PM
• URL: https://guides.library.harvard.edu/FindingScholarlyArticles

Harvard University Digital Accessibility Policy

Peer-Reviewed Literature: Peer-Reviewed Research: Primary vs. Secondary

• Peer-Reviewed Research: Primary vs. Secondary
• Types of Peer Review
• Identifying Peer-Reviewed Research

Peer Reviewed Research

Published literature can be either peer-reviewed or non-peer-reviewed. Official research reports are almost always peer reviewed while a journal's other content is usually not. In the health sciences, official research can be primary, secondary, or even tertiary. It can be an original experiment or investigation (primary), an analysis or evaluation of primary research (secondary), or findings that compile secondary research (tertiary). If you are doing research yourself, then primary or secondary sources can reveal more in-depth information.

Primary Research

Primary research is information presented in its original form without interpretation by other researchers. While it may acknowledge previous studies or sources, it always presents original thinking, reports on discoveries, or new information about a topic.

Health sciences research that is primary includes both experimental trials and observational studies where subjects may be tested for outcomes or investigated to gain relevant insight.  Randomized Controlled Trials are the most prominent experimental design because randomized subjects offer the most compelling evidence for the effectiveness of an intervention. See the below graphic and below powerpoint for further information on primary research studies.

• Research Design

Secondary Research

Secondary research is an account of original events or facts. It is secondary to and retrospective of the actual findings from an experiment or trial. These studies may be appraised summaries, reviews, or interpretations of primary sources and often exclude the original researcher(s). In the health sciences, meta-analysis and systematic reviews are the most frequent types of secondary research. 

• A meta-analysis is a quantitative method of combining the results of primary research. In analyzing the relevant data and statistical findings from experimental trials or observational studies, it can more accurately calculate effective resolutions regarding certain health topics.
• A systematic review is a summary of research that addresses a focused clinical question in a systematic, reproducible manner. In order to provide the single best estimate of effect in clinical decision making, primary research studies are pooled together and then filtered through an inclusion/exclusion process. The relevant data and findings are then compiled and synthesized to arrive at a more accurate conclusion about a specific health topic. Only peer-reviewed publications are used and analyzed in a methodology which may or may not include a meta-analysis.

• << Previous: Home
• Next: Types of Peer Review >>
• Last Updated: Sep 29, 2023 10:05 AM
• URL: https://ttuhsc.libguides.com/PeerReview

• SUNY Oswego, Penfield Library
• Resource Guides

Biological Sciences Research Guide

Primary research vs review article.

• Research Starters
• Citing Sources
• Open Educational Resources
• Peer Review
• How to Read a Scientific Article
• Conducting a Literature Review
• Interlibrary Loan

Quick Links

• Penfield Library
• Research Guides
• A-Z List of Databases & Indexes

Characteristics of a Primary Research Article

• Goal is to present the result of original research that makes a new contribution to the body of knowledge
• Sometimes referred to as an empirical research article
• Typically organized into sections that include:  Abstract, Introduction, Methods, Results, Discussion/Conclusion, and References.

Example of a Primary Research Article:

Flockhart, D.T.T., Fitz-gerald, B., Brower, L.P., Derbyshire, R., Altizer, S., Hobson, K.A., … Norris, D.R., (2017). Migration distance as a selective episode for wing morphology in a migratory insect. Movement Ecology , 5(1), 1-9. doi: doi.org/10.1186/s40462-017-0098-9

Characteristics of a Review Article

• Goal is to summarize important research on a particular topic and to represent the current body of knowledge about that topic.
• Not intended to provide original research but to help draw connections between research studies that have previously been published.  
• Help the reader understand how current understanding of a topic has developed over time and identify gaps or inconsistencies that need further exploration.

Example of a Review Article:

https://www-sciencedirect-com.ezproxy.oswego.edu/science/article/pii/S0960982218302537

• << Previous: Plagiarism
• Next: Peer Review >>

• Main Library
• Digital Fabrication Lab
• Data Visualization Lab
• Multimedia Studio
• Business Learning Center
• Klai Juba Wald Architectural Studies Library
• NDSU Nursing at Sanford Health Library
• Research Assistance
• Special Collections
• Digital Collections
• Collection Development Policy
• Course Reserves
• Request Library Instruction
• Main Library Services
• Alumni & Community
• Academic Support Services in the Library
• Libraries Resources for Employees
• Book Equipment or Study Rooms
• Librarians by Academic Subject
• Germans from Russia Heritage Collection
• NDSU Archives
• Mission, Vision, and Strategic Plan 2022-2024
• Staff Directory
• Floor Plans
• The Libraries Magazine
• Accommodations for People with Disabilities
• Annual Report
• Donate to the Libraries
• Equity, Diversity and Inclusion
• Faculty Senate Library Committee
• Undergraduate Research Award

What is an original research article?

An original research article is a report of research activity that is written by the researchers who conducted the research or experiment. Original research articles may also be referred to as: “primary research articles” or “primary scientific literature.” In science courses, instructors may also refer to these as “peer-reviewed articles” or “refereed articles.”

Original research articles in the sciences have a specific purpose, follow a scientific article format, are peer reviewed, and published in academic journals.

Identifying Original Research: What to Look For

An "original research article" is an article that is reporting original research about new data or theories that have not been previously published. That might be the results of new experiments, or newly derived models or simulations. The article will include a detailed description of the methods used to produce them, so that other researchers can verify them. This description is often found in a section called "methods" or "materials and methods" or similar. Similarly, the results will generally be described in great detail, often in a section called "results."

Since the original research article is reporting the results of new research, the authors should be the scientists who conducted that research. They will have expertise in the field, and will usually be employed by a university or research lab.

In comparison, a newspaper or magazine article (such as in  The New York Times  or  National Geographic ) will usually be written by a journalist reporting on the actions of someone else.

An original research article will be written by and for scientists who study related topics. As such, the article should use precise, technical language to ensure that other researchers have an exact understanding of what was done, how to do it, and why it matters. There will be plentiful citations to previous work, helping place the research article in a broader context. The article will be published in an academic journal, follow a scientific format, and undergo peer-review.

Original research articles in the sciences follow the scientific format. ( This tutorial from North Carolina State University illustrates some of the key features of this format.)

Look for signs of this format in the subject headings or subsections of the article. You should see the following:

Scientific research that is published in academic journals undergoes a process called "peer review."

The peer review process goes like this:

• A researcher writes a paper and sends it in to an academic journal, where it is read by an editor
• The editor then sends the article to other scientists who study similar topics, who can best evaluate the article
• The scientists/reviewers examine the article's research methodology, reasoning, originality, and sginificance
• The scientists/reviewers then make suggestions and comments to impove the paper
• The original author is then given these suggestions and comments, and makes changes as needed
• This process repeats until everyone is satisfied and the article can be published within the academic journal

For more details about this process see the Peer Reviewed Publications guide.

This journal article  is an example. It was published in the journal  Royal Society Open Science  in 2015. Clicking on the button that says "Review History" will show the comments by the editors, reviewers and the author as it went through the peer review process. The "About Us" menu provides details about this journal; "About the journal" under that tab includes the statement that the journal is peer reviewed.

Review articles

There are a variety of article types published in academic, peer-reviewed journals, but the two most common are original research articles and review articles . They can look very similar, but have different purposes and structures.

Like original research articles, review articles are aimed at scientists and undergo peer-review. Review articles often even have “abstract,” “introduction,” and “reference” sections. However, they will not (generally) have a “methods” or “results” section because they are not reporting new data or theories. Instead, they review the current state of knowledge on a topic.

Press releases, newspaper or magazine articles

These won't be in a formal scientific format or be peer reviewed. The author will usually be a journalist, and the audience will be the general public. Since most readers are not interested in the precise details of the research, the language will usually be nontechnical and broad. Citations will be rare or nonexistent.

Tips for Finding Original research Articles

Search for articles in one of the library databases recommend for your subject area . If you are using Google, try searching in Google Scholar instead and you will get results that are more likely to be original research articles than what will come up in a regular Google search!

For tips on using library databases to find articles, see our Library DIY guides .

Tips for Finding the Source of a News Report about Science

If you've seen or heard a report about a new scientific finding or claim, these tips can help you find the original source:

• Often, the report will mention where the original research was published; look for sentences like "In an article published yesterday in the journal  Nature ..." You can use this to find the issue of the journal where the research was published, and look at the table of contents to find the original article.
• The report will often name the researchers involved. You can search relevant databases for their name and the topic of the report to find the original research that way.
• Sometimes you may have to go through multiple articles to find the original source. For example, a video or blog post may be based on a newspaper article, which in turn is reporting on a scientific discovery published in another journal; be sure to find the original research article.
• Don't be afraid to ask a librarian for help!

Search The Site

Find Your Librarian  

Phone:  Circulation:  (701) 231-8888 Reference:  (701) 231-8888 Administration:  (701) 231-8753

Email:  Administration InterLibrary Loan (ILL)

• Online Services
• Phone/Email Directory
• Registration And Records
• Government Information
• Library DIY
• Subject and Course Guides
• Special Topics
• Collection Highlights
• Digital Horizons
• NDSU Repository (IR)
• Libraries Hours
• News & Events

Understanding research and critical appraisal

• Introduction
• Secondary research

What is primary research?

Quantitative research study designs, qualitative research study designs, mixed methods research study designs.

• Critical appraisal of research papers
• Useful terminology
• Further reading and helpful resources

Primary research articles provide a report of individual, original research studies, which constitute the majority of articles published in peer-reviewed journals. All primary research studies are conducted according to a specified methodology, which will be partly determined by the aims and objectives of the research.

The following sections offer brief summaries of some of the common quantitative, qualitative, and mixed-methods study designs you may encounter. 

Randomised Controlled Trial

A randomised controlled trial (RCT) is a study where participants are randomly allocated to two or more groups. One group receives the treatment that is being tested by the study (treatment or experimental group), and the other group(s) receive an alternative, which is often the current standard treatment or a placebo (control or comparison group). The nature of the control used should always be specified.

An RCT is a good study choice for determining the effectiveness of an intervention or treatment, or for comparing the relative effectiveness of different interventions or treatments. If well implemented, the randomisation of participants in RCTs should ensure that the groups differ only in their exposure to treatment, and that differences in outcomes between the groups are probably attributable to the treatment being studied.

In crossover randomised controlled trials, participants receive all of the treatments and controls being tested in a random order. This means that participants receive one treatment, the effect of which is measured, and then "cross over" into the other treatment group, where the effect of the second treatment (or control) is measured.

RCTs are generally considered to be the most rigorous experimental study design, as the randomisation of participants helps to minimise confounding and other sources of bias.

Cohort study

A cohort study identifies a group of people and follows them over a period of time to see who develops the outcome of interest to the study. This type of study is normally used to look at the effect of suspected risk factors that cannot be controlled experimentally – for example, the effect of smoking on lung cancer.

Also sometimes called longitudinal studies, cohort studies can be either prospective, that is, exposure factors are identified at the beginning of a study and the study population is followed into the future, or retrospective, that is, medical records for the study population are used to identify past exposure factors.

Cohort studies are useful in answering questions about disease causation or progression, or studying the effects of harmful exposures.

Cohort studies are generally considered to be the most reliable observational study design. They are not as reliable as RCTs, as the study groups may differ in ways other than the variable being studied.

Other problems with cohort studies are that they require a large sample size, are inefficient for rare outcomes, and can take long periods of time.

Case-Control Study

A case-control study compares a group of people with a disease or condition, against a control population without the disease or condition, in order to investigate the causes of particular outcomes. The study looks back at the two groups over time to see which risk factors for the disease or condition they have been exposed to.

Case-control studies can be useful in identifying which risk factors may predict a disease, or how a disease progresses over time. They can be especially useful for investigating the causes of rare outcomes.

Case-control studies can be done quickly, and do not require large groups of subjects. However, their reliance on retrospective data which may be incomplete or unreliable (owing to subject ability to accurately recall information such as the appearance of a symptom) can be a difficulty.

Cross-Sectional Study

A cross-sectional study collects data from the study population at one point in time, and considers the relationships between characteristics. Also  sometimes called surveys or prevalence studies.

Cross-sectional studies are generally used to study the prevalence of a risk factor, disease or outcome in a chosen population.

Because cross-sectional studies do not look at trends or changes over time, they cannot establish cause and effect between exposures and outcomes.

Case Series / Case Reports

A case series is a descriptive study of a group of people, who have either received the same treatment or have the same disease, in order to identify characteristics or outcomes in a particular group of people.

Case series are useful for studying rare diseases or adverse outcomes, for illustrating particular aspects of a condition, identifying treatment approaches, and for generating hypotheses for further study.

A case report provides a study of an individual, rather than a group.

Case series and case reports have no comparative control groups, and are prone to bias and chance association.

Expert opinion

Expert opinion draws upon the clinical experience and recommendations of those with established expertise on a topic.

Grounded theory

Grounded theory studies aim to generate theory in order to explain social processes, interactions or issues. This explanatory theory is grounded in, and generated from, the research participant data collected.

Research data typically takes the form of interviews, observations or documents. Data is analysed as it is collected, and is coded and organised into categories which inform the further collection of data, and the construction of theory. This cycle helps to refine the theory, which evolves as more data is gathered.

Phenomenology

A phenomenological study aims to describe the meaning(s) of the lived experience of a phenomenon. Research participants will have some common experience of the phenomenon under examination, but will differ in their precise individual experience, and in other personal or social characteristics.

Research data is typically in the form of observations, interviews or written records, and its analysis sets out to identify common themes in the participants' experience, while also highlighting variations and unique themes.

Ethnography

Ethnography is the study of a specific culture or cultural group, where the researcher seeks an insider perspective by placing themselves as a participant observer within the group under study.

Data is typically formed of observations, interviews and conversation. Ethnography aims to offer direct insight into the lives and the experiences of the group or the culture under study, examining its beliefs, values, practices and behaviours.

A case study offers a detailed description of the experience of an individual, a family, a community or an organisation, often with the aim of highlighting a particular issue. Research data may include documents, interviews and observations.

Content analysis

Content analysis is used to explore the occurrence, meanings and relationships of words, themes or concepts within a set of textual data. Research data might be drawn from any type of written document(s). Data is coded and categorised, with the aim of revealing and examining the patterns and the intentions of language use within the data set.

Narrative inquiry

A narrative inquiry offers in depth detail of a situation or experience from the perspective of an individual or small groups. Research data usually consists of interviews or recordings, which is presented as a structured, chronological narrative. Narrative inquiry studies often seek to give voice to individuals or populations whose perspective is less well established, or not commonly sought.

Action research

Action research is a form of research, commonly used with groups, where the participants take a more active, collaborative role in producing the research. Studies incorporate the lived experiences of the individuals, groups or communities under study, drawing on data which might include observation, interviews, questionnaires or workshops.

Action research is generally aimed at changing or improving a particular context, or a specific practice, alongside the generation of theory.

Explanatory sequential design

In an explanatory sequential study, emphasis is given to the collection and analysis of quantitative data, which occurs during the first phase of the study. The results of this quantitative phase inform the subsequent collection of qualitative data in the next phase.

Analysis of the resultant qualitative data is then used to 'explain' the quantitative results, usually serving to contextualise these, or to otherwise enhance or enrich the initial findings.

Exploratory sequential design

In an exploratory sequential study, the opposite sequence to that outlined above is used. In this case, qualitative data is emphasised, with this being collected and analysed during the first phase of the study. The results of this qualitative phase inform the subsequent collection of quantitative data in the next phase.

The quantitative data can then be used to define or to generalise the qualitative results, or to test these results on the basis of theory emerging from the initial findings.

Convergent design

In a convergent study, qualitative and quantitative data sets are collected and analysed simultaneously and independently of one another.

Results from analysis of both sets of data are brought together to provide one overall interpretation; this combination of data types can be handled in various ways, but the objective is always to provide a fuller understanding of the phenomena under study. Equal emphasis is given to both qualitative and quantitative data in a convergent study.

• << Previous: Secondary research
• Next: Critical appraisal of research papers >>
• Last Updated: Mar 26, 2024 4:38 PM
• URL: https://libguides.sgul.ac.uk/researchdesign

Primary Research Articles

• Library vs. Google
• Background Reading
• Keyword Searching
• Evaluating Sources
• Citing Sources
• Need more help?

How Can I Find Primary Research Articles?

Many of the recommended databases in this subject guide contain primary research articles (also known as empirical articles or research studies). Search in databases like ScienceDirect  and MEDLINE .

Primary Research Articles: How Will I Know One When I See One?

Primary research articles  to conduct and publish an experiment or research study, an author or team of authors designs an experiment, gathers data, then analyzes the data and discusses the results of the experiment. a published experiment or research study will therefore  look  very different from other types of articles (newspaper stories, magazine articles, essays, etc.) found in our library databases. the following guidelines will help you recognize a primary research article, written by the researchers themselves and published in a scholarly journal., structure of a primary research article typically, a primary research article has the following sections:.

• The author summarizes her article
• The author discusses the general background of her research topic; often, she will present a literature review, that is, summarize what other experts have written on this particular research topic
• The author describes the study she designed and conducted
• The author presents the data she gathered during her experiment
• The author offers ideas about the importance and implications of her research findings, and speculates on future directions that similar research might take
• The author gives a References list of sources she used in her paper

The structure of the article will often be clearly shown with headings: Introduction, Method, Results, Discussion.

A primary research article will almost always contains statistics, numerical data presented in tables. Also, primary research articles are written in very formal, very technical language.

• << Previous: Resources
• Next: Research Tips >>
• Last Updated: Aug 1, 2024 5:09 PM
• URL: https://libguides.umgc.edu/science

KPU Library

• Library Home
• Borrowing from the Library
• Borrowing From Other Academic Institutions
• Borrowing and Fine Questions
• Course Reserves
• Equipment & Media Services
• Interlibrary Loans
• Requesting KPU Material
• Student Laptop Loans
• Academic Integrity & Plagiarism
• Articles and Databases
• Citation Style Guides
• Collections
• Journal Titles
• Research Help Guide
• Scholarly Publications
• Subject Guides
• Video Tutorials
• Data Services
• Library Instruction
• Library Services to support Teaching & Learning
• Library Tutorials & Activities
• Link.Scan.Open.
• Open Publishing Suite
• Placing Items on Reserve
• Research, Scholarship & Publishing at KPU
• Teaching Online
• Computers & Wifi
• Library Accessibility Services
• My Library Account & Passwords
• Off Site Access
• Our Facilities
• Printing, Photocopy, Scan
• Study Rooms & Bookings
• About the Library
• Indigenization at KPU Library
• Library Directory
• KPU Library News Blog
• Welcome to KPU Library
• Contact Your Librarian
• Hours & Locations
• Library Events

CRIM 1208: Research Methods

• About this guide
• Developing a research question
• What are scholarly articles?
• What are primary research articles?
• Finding articles in databases
• Where to find government statistics? This link opens in a new window
• Literature reviews
• Citation mapping
• Citing in APA style
• Go to main CRIM library guide

Which of these is an ORIGINAL (PRIMARY) research article?

What are original (primary) research articles.

• Primary & secondary research

Structure of a primary research article

Primary and secondary research articles.

Once researchers complete a project, they will usually (try to) publish their findings in a peer-reviewed journal. These are often called PRIMARY or ORIGINAL research articles because they are the first-publication of new research findings and are written by the researchers themselves. They may also be called EMPIRICAL articles.

Secondary sources of information describe, explain, interpret or summarize primary sources. These include encyclopedias, book reviews, commentaries, literature reviews, and any books or journal articles that simply discuss the original (previously-published) work of others . Although these can be very helpful sources for identifying primary research articles, they are not primary studies themselves.

VIDEO: What is Original (Primary) Research in Criminology? (19:38)

Describes the typical structure of an original research article, with a particular focus on the Methods section. It shows examples of several types of original research articles (qualitative and quantitative, including articles using secondary data, and meta-analyses), as well as several types of secondary articles (book reviews, editorial essays, theoretical analyses and literature reviews). The emphasis is on learning to read the abstract for indications of original research, and checking for a Methods section in the article. Part 1 of the Finding Original Research Articles in Criminology video series.

• Video (Kaltura)
• Video transcript (text file)

• VIDEO: What is Empirical Research? (2:59)
• Finding Peer-Reviewed, Primary Research Articles in Criminology 4-page KPU Library guide.

A primary (original) research article will usually be divided into several labeled sections. The screenshot above is from the video " What is Original (Primary) Research in Criminology? ". You can jump to the 3:10 timestamp to watch the " Sections of an original research article " segment of the video.

• Introduction (which usually includes a literature review)
• Method (often called Methodology or Methods) -- always found in an empirical research article
• Findings or Results
• Conclusions

The names of the parts may vary, but a primary research article will always include a methodology section explaining how the research was conducted (i.e. what type of empirical method was used). Most secondary journal articles do not include a methods section.

• << Previous: What are scholarly articles?
• Next: Finding articles in databases >>
• Last Updated: May 22, 2024 12:51 PM
• URL: https://libguides.kpu.ca/crim/1208

Finding Primary Research Articles in the Sciences: Home

• Advanced Search-Databases
• Primary vs. Secondary
• Analyzing a Primary Research Article
• MLA, APA, and Chicago Style

This guide goes over how to find and analyze primary research articles in the sciences (e.g. nutrition, health sciences and nursing, biology, chemistry, physics, sociology, psychology). In addition, the guide explains how to tell the difference between a primary source and a secondary source in scientific subject areas.

If you are looking for how to find primary sources in the humanities and social sciences, such as direct experience accounts in newspapers, diaries, artwork and so forth, please see   Finding Primary Sources in the Humanities and Social Sciences . 

Recommended Databases

To get started, choose one of the databases below.  Once you log in, enter your search terms to start looking for primary articles. 

• Link to all Polk State College Library databases

Login Required

You must log in to use library databases and eBooks. When prompted to log in, enter your Passport credentials. 

If you have trouble, try  resetting your Passport pin , sending an email to  [email protected] ,  or calling the Help Desk at 863.292.3652 . 

You can also get help from Ask a Librarian . 

Search Tips

Keep your search terms simple.

• No need to type full sentences into the database search box.  Limit your search to 2-3 words.
• There is no need to type "research article" into the search box.

Use the "Advanced Search" feature of the database.

• This will allow you to limit your search to only peer reviewed articles or a certain time frame (for example: 2013 or later).
• Click the red tab above for tips on advanced search strategies .

Re-read the assignment guidelines often

• Does this article satisfy the scope of the assignment (e.g. a study focused on nutrition)?
• Does it meet the criteria for the assignment (e.g. an original research article)?

Not finding what you are looking for?

• Ask a Librarian!

Search and Find a Primary Research Article

Are you looking for a primary research journal article if so, that is an article that reports on the results of an original research study conducted by the authors themselves. .

You can use the library's databases to search for primary research articles.  A research article will almost always be published in a peer-reviewed journal. Therefore, it is a good idea to limit your results to peer-reviewed articles. Click on the  Advanced Search-Databases tab at the top of this guide for instructions. 

The following is _not_ primary research:

Review articles are studies that arrive at conclusions after looking over other studies. Therefore, review articles are not  primary (think "first") research.  There are a variety of review articles, including:

• Literature Reviews
• Systematic Reviews
• Meta-Analyses 
• Scoping Reviews
• Topical Reviews
• A review/assessment of the evidence

Having trouble?  Look for a  method section within the article. If the method section includes the process used to conduct the research, how the data was gathered and analyzed and any limitations or ethical concerns to the study, then it is most likely a primary research article. For example: a research article will describe the number of people (e.g. 175 adults with celiac disease) who participated in the study and who were used to collect data.

If the method section describes how the authors found articles on a topic using search terms or databases , then it is mostly likely a secondary review article and not primary research. If there is no method section, it is not a primary research article.

Other sections in a journal: 

Your search may yield these items, too. You can skip these because they are not full write-ups of research:

• Conference Proceedings 
• Symposium Publications

Example of a primary research article found in the Library's Academic Search Complete database : (these authors conducted an original research study)

• Lumia et al. (2015) Lumia, M., Takkinen, H., Luukkainen, P., Kaila, M., Lehtinen, J. S., Nwaru, B. I., Tuokkola, J., Niemelä, O., Haapala, A., Ilonen, J., Simell, O., Knip, M., Veijola, R., & Virtanen, S. M. (2015). Food consumption and risk of childhood asthma. Pediatric Allergy & Immunology, 26(8), 789–796. https://doi.org/10.1111/pai.12352

Example of a secondary article found in the Library's Academic Search Complete database : (these authors are reviewing the work of other authors)

• Rachmah et al. (2022) Rachmah, Q., Martiana, T., Mulyono, Paskarini, I., Dwiyanti, E., Widajati, N., Ernawati, M., Ardyanto, Y. D., Tualeka, A. R., Haqi, D. N., Arini, S. Y., & Alayyannur, P. A. (2022). The effectiveness of nutrition and health intervention in workplace setting: A systematic review. Journal of Public Health Research, 11(1), 1–8. https://doi.org/10.4081/jphr.2021.2312

How do I know if this article is primary?

You've found an article in the library databases but how do you know if it's primary .

Look for these sections: (terminology may vary)

• abstract  - summarizes paper in one paragraph, states the purpose of the study
• methods  - explaining how the experiment was conducted (note: if the method section discusses how a search was conducted that is _not_ primary research) 
• results  - detailing what happened and providing raw data sets (often as tables or graphs)
• conclusions  - connecting the results with theories and other research
• references  - to previous research or theories that influenced the research

Scan the article you found to see if it includes the sections above. You don't have to read the full article (yet). Look for the clues highlighted in the images below. 

Questions? Use Ask a Librarian

• Next: Advanced Search-Databases >>
• Last Updated: Aug 8, 2024 4:22 PM
• URL: https://libguides.polk.edu/primaryresearch

Polk State College is committed to equal access/equal opportunity in its programs, activities, and employment. For additional information, visit polk.edu/compliance .

University of Denver

University libraries, research guides, a guide to health & medical research.

• Evidence based medicine
• Consumer health info
• Journal articles
• Books & media
• Clinical guidelines, statistics, & more
• Newspapers & magazines
• Get full text of a specific article
• Request sources not at DU Libraries
• Search databases effectively
• Evaluate your sources
• Confirm an article is peer-reviewed

What is a scholarly, peer-reviewed journal?

Confirm that an article is peer-reviewed -- part 1:, confirm that an article is peer-reviewed -- part 2:, primary research articles vs review articles.

• Cite sources properly

First, determine whether the article is published in a peer-reviewed journal. There are two ways to determine whether a journal has a peer-review process in place (which means that it is a scholarly source):

Next, look at the article to see what elements it has in it, and consult the table below to make your final determination:

Why are some of the articles in a peer-reviewed journal NOT peer-reviewed?

The peer-review process take a lot of time and effort, so it's reserved for articles where accuracy is essential -- reports of new and original research, or summaries of research. Other researchers are going to use and build upon the data and information reported in those articles, so it's important that it is accurate.

For articles such as book reviews, accuracy is not as important (after all, book reviews and editorials are highly influenced by personal opinions). Therefore, these articles are checked for grammar by an editor but don't undergo the rigorous peer-review process.

A primary research article presents a first report of original research. It's written by the people who performed the research, and it's usually written for other researchers in the same field. 

Example of a primary research article:  Bramhanwade, K.; Shende, S.; Bonde, S.; Gade, A.; Rai, M. Fungicidal activity of Cu nanoparticles against Fusarium causing crop diseases. Environmental Chemistry Letters 2016 , 14, 229-235.

The structure is designed to be a practical and efficient means of delivering information. It also ensures that key points will be covered in the article, including:

A scholarly review article is a special type of peer-reviewed article that provides an overview of an area of research; it describes major advances, discoveries, and ongoing debates in that field. It can be very useful to look for review articles if you are new to an area of research.

Example of a scholarly review article:  Kasana, R. C.; Panwar, N. R.; Kaul, R. K.; Kumar, P. Biosynthesis and effects of copper nanoparticles on plants.  Environmental Chemistry Letters  2017 ,15, 233-240.

• << Previous: Evaluate your sources
• Next: Cite sources properly >>
• Last Updated: Aug 14, 2024 6:04 PM
• URL: https://libguides.du.edu/health-medical

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Reumatologia
• v.59(1); 2021

Peer review guidance: a primer for researchers

Olena zimba.

1 Department of Internal Medicine No. 2, Danylo Halytsky Lviv National Medical University, Lviv, Ukraine

Armen Yuri Gasparyan

2 Departments of Rheumatology and Research and Development, Dudley Group NHS Foundation Trust (Teaching Trust of the University of Birmingham, UK), Russells Hall Hospital, Dudley, West Midlands, UK

The peer review process is essential for quality checks and validation of journal submissions. Although it has some limitations, including manipulations and biased and unfair evaluations, there is no other alternative to the system. Several peer review models are now practised, with public review being the most appropriate in view of the open science movement. Constructive reviewer comments are increasingly recognised as scholarly contributions which should meet certain ethics and reporting standards. The Publons platform, which is now part of the Web of Science Group (Clarivate Analytics), credits validated reviewer accomplishments and serves as an instrument for selecting and promoting the best reviewers. All authors with relevant profiles may act as reviewers. Adherence to research reporting standards and access to bibliographic databases are recommended to help reviewers draft evidence-based and detailed comments.

Introduction

The peer review process is essential for evaluating the quality of scholarly works, suggesting corrections, and learning from other authors’ mistakes. The principles of peer review are largely based on professionalism, eloquence, and collegiate attitude. As such, reviewing journal submissions is a privilege and responsibility for ‘elite’ research fellows who contribute to their professional societies and add value by voluntarily sharing their knowledge and experience.

Since the launch of the first academic periodicals back in 1665, the peer review has been mandatory for validating scientific facts, selecting influential works, and minimizing chances of publishing erroneous research reports [ 1 ]. Over the past centuries, peer review models have evolved from single-handed editorial evaluations to collegial discussions, with numerous strengths and inevitable limitations of each practised model [ 2 , 3 ]. With multiplication of periodicals and editorial management platforms, the reviewer pool has expanded and internationalized. Various sets of rules have been proposed to select skilled reviewers and employ globally acceptable tools and language styles [ 4 , 5 ].

In the era of digitization, the ethical dimension of the peer review has emerged, necessitating involvement of peers with full understanding of research and publication ethics to exclude unethical articles from the pool of evidence-based research and reviews [ 6 ]. In the time of the COVID-19 pandemic, some, if not most, journals face the unavailability of skilled reviewers, resulting in an unprecedented increase of articles without a history of peer review or those with surprisingly short evaluation timelines [ 7 ].

Editorial recommendations and the best reviewers

Guidance on peer review and selection of reviewers is currently available in the recommendations of global editorial associations which can be consulted by journal editors for updating their ethics statements and by research managers for crediting the evaluators. The International Committee on Medical Journal Editors (ICMJE) qualifies peer review as a continuation of the scientific process that should involve experts who are able to timely respond to reviewer invitations, submitting unbiased and constructive comments, and keeping confidentiality [ 8 ].

The reviewer roles and responsibilities are listed in the updated recommendations of the Council of Science Editors (CSE) [ 9 ] where ethical conduct is viewed as a premise of the quality evaluations. The Committee on Publication Ethics (COPE) further emphasizes editorial strategies that ensure transparent and unbiased reviewer evaluations by trained professionals [ 10 ]. Finally, the World Association of Medical Editors (WAME) prioritizes selecting the best reviewers with validated profiles to avoid substandard or fraudulent reviewer comments [ 11 ]. Accordingly, the Sarajevo Declaration on Integrity and Visibility of Scholarly Publications encourages reviewers to register with the Open Researcher and Contributor ID (ORCID) platform to validate and publicize their scholarly activities [ 12 ].

Although the best reviewer criteria are not listed in the editorial recommendations, it is apparent that the manuscript evaluators should be active researchers with extensive experience in the subject matter and an impressive list of relevant and recent publications [ 13 ]. All authors embarking on an academic career and publishing articles with active contact details can be involved in the evaluation of others’ scholarly works [ 14 ]. Ideally, the reviewers should be peers of the manuscript authors with equal scholarly ranks and credentials.

However, journal editors may employ schemes that engage junior research fellows as co-reviewers along with their mentors and senior fellows [ 15 ]. Such a scheme is successfully practised within the framework of the Emerging EULAR (European League Against Rheumatism) Network (EMEUNET) where seasoned authors (mentors) train ongoing researchers (mentees) how to evaluate submissions to the top rheumatology journals and select the best evaluators for regular contributors to these journals [ 16 ].

The awareness of the EQUATOR Network reporting standards may help the reviewers to evaluate methodology and suggest related revisions. Statistical skills help the reviewers to detect basic mistakes and suggest additional analyses. For example, scanning data presentation and revealing mistakes in the presentation of means and standard deviations often prompt re-analyses of distributions and replacement of parametric tests with non-parametric ones [ 17 , 18 ].

Constructive reviewer comments

The main goal of the peer review is to support authors in their attempt to publish ethically sound and professionally validated works that may attract readers’ attention and positively influence healthcare research and practice. As such, an optimal reviewer comment has to comprehensively examine all parts of the research and review work ( Table I ). The best reviewers are viewed as contributors who guide authors on how to correct mistakes, discuss study limitations, and highlight its strengths [ 19 ].

Structure of a reviewer comment to be forwarded to authors

Some of the currently practised review models are well positioned to help authors reveal and correct their mistakes at pre- or post-publication stages ( Table II ). The global move toward open science is particularly instrumental for increasing the quality and transparency of reviewer contributions.

Advantages and disadvantages of common manuscript evaluation models

Since there are no universally acceptable criteria for selecting reviewers and structuring their comments, instructions of all peer-reviewed journal should specify priorities, models, and expected review outcomes [ 20 ]. Monitoring and reporting average peer review timelines is also required to encourage timely evaluations and avoid delays. Depending on journal policies and article types, the first round of peer review may last from a few days to a few weeks. The fast-track review (up to 3 days) is practised by some top journals which process clinical trial reports and other priority items.

In exceptional cases, reviewer contributions may result in substantive changes, appreciated by authors in the official acknowledgments. In most cases, however, reviewers should avoid engaging in the authors’ research and writing. They should refrain from instructing the authors on additional tests and data collection as these may delay publication of original submissions with conclusive results.

Established publishers often employ advanced editorial management systems that support reviewers by providing instantaneous access to the review instructions, online structured forms, and some bibliographic databases. Such support enables drafting of evidence-based comments that examine the novelty, ethical soundness, and implications of the reviewed manuscripts [ 21 ].

Encouraging reviewers to submit their recommendations on manuscript acceptance/rejection and related editorial tasks is now a common practice. Skilled reviewers may prompt the editors to reject or transfer manuscripts which fall outside the journal scope, perform additional ethics checks, and minimize chances of publishing erroneous and unethical articles. They may also raise concerns over the editorial strategies in their comments to the editors.

Since reviewer and editor roles are distinct, reviewer recommendations are aimed at helping editors, but not at replacing their decision-making functions. The final decisions rest with handling editors. Handling editors weigh not only reviewer comments, but also priorities related to article types and geographic origins, space limitations in certain periods, and envisaged influence in terms of social media attention and citations. This is why rejections of even flawless manuscripts are likely at early rounds of internal and external evaluations across most peer-reviewed journals.

Reviewers are often requested to comment on language correctness and overall readability of the evaluated manuscripts. Given the wide availability of in-house and external editing services, reviewer comments on language mistakes and typos are categorized as minor. At the same time, non-Anglophone experts’ poor language skills often exclude them from contributing to the peer review in most influential journals [ 22 ]. Comments should be properly edited to convey messages in positive or neutral tones, express ideas of varying degrees of certainty, and present logical order of words, sentences, and paragraphs [ 23 , 24 ]. Consulting linguists on communication culture, passing advanced language courses, and honing commenting skills may increase the overall quality and appeal of the reviewer accomplishments [ 5 , 25 ].

Peer reviewer credits

Various crediting mechanisms have been proposed to motivate reviewers and maintain the integrity of science communication [ 26 ]. Annual reviewer acknowledgments are widely practised for naming manuscript evaluators and appreciating their scholarly contributions. Given the need to weigh reviewer contributions, some journal editors distinguish ‘elite’ reviewers with numerous evaluations and award those with timely and outstanding accomplishments [ 27 ]. Such targeted recognition ensures ethical soundness of the peer review and facilitates promotion of the best candidates for grant funding and academic job appointments [ 28 ].

Also, large publishers and learned societies issue certificates of excellence in reviewing which may include Continuing Professional Development (CPD) points [ 29 ]. Finally, an entirely new crediting mechanism is proposed to award bonus points to active reviewers who may collect, transfer, and use these points to discount gold open-access charges within the publisher consortia [ 30 ].

With the launch of Publons ( http://publons.com/ ) and its integration with Web of Science Group (Clarivate Analytics), reviewer recognition has become a matter of scientific prestige. Reviewers can now freely open their Publons accounts and record their contributions to online journals with Digital Object Identifiers (DOI). Journal editors, in turn, may generate official reviewer acknowledgments and encourage reviewers to forward them to Publons for building up individual reviewer and journal profiles. All published articles maintain e-links to their review records and post-publication promotion on social media, allowing the reviewers to continuously track expert evaluations and comments. A paid-up partnership is also available to journals and publishers for automatically transferring peer-review records to Publons upon mutually acceptable arrangements.

Listing reviewer accomplishments on an individual Publons profile showcases scholarly contributions of the account holder. The reviewer accomplishments placed next to the account holders’ own articles and editorial accomplishments point to the diversity of scholarly contributions. Researchers may establish links between their Publons and ORCID accounts to further benefit from complementary services of both platforms. Publons Academy ( https://publons.com/community/academy/ ) additionally offers an online training course to novice researchers who may improve their reviewing skills under the guidance of experienced mentors and journal editors. Finally, journal editors may conduct searches through the Publons platform to select the best reviewers across academic disciplines.

Peer review ethics

Prior to accepting reviewer invitations, scholars need to weigh a number of factors which may compromise their evaluations. First of all, they are required to accept the reviewer invitations if they are capable of timely submitting their comments. Peer review timelines depend on article type and vary widely across journals. The rules of transparent publishing necessitate recording manuscript submission and acceptance dates in article footnotes to inform readers of the evaluation speed and to help investigators in the event of multiple unethical submissions. Timely reviewer accomplishments often enable fast publication of valuable works with positive implications for healthcare. Unjustifiably long peer review, on the contrary, delays dissemination of influential reports and results in ethical misconduct, such as plagiarism of a manuscript under evaluation [ 31 ].

In the times of proliferation of open-access journals relying on article processing charges, unjustifiably short review may point to the absence of quality evaluation and apparently ‘predatory’ publishing practice [ 32 , 33 ]. Authors when choosing their target journals should take into account the peer review strategy and associated timelines to avoid substandard periodicals.

Reviewer primary interests (unbiased evaluation of manuscripts) may come into conflict with secondary interests (promotion of their own scholarly works), necessitating disclosures by filling in related parts in the online reviewer window or uploading the ICMJE conflict of interest forms. Biomedical reviewers, who are directly or indirectly supported by the pharmaceutical industry, may encounter conflicts while evaluating drug research. Such instances require explicit disclosures of conflicts and/or rejections of reviewer invitations.

Journal editors are obliged to employ mechanisms for disclosing reviewer financial and non-financial conflicts of interest to avoid processing of biased comments [ 34 ]. They should also cautiously process negative comments that oppose dissenting, but still valid, scientific ideas [ 35 ]. Reviewer conflicts that stem from academic activities in a competitive environment may introduce biases, resulting in unfair rejections of manuscripts with opposing concepts, results, and interpretations. The same academic conflicts may lead to coercive reviewer self-citations, forcing authors to incorporate suggested reviewer references or face negative feedback and an unjustified rejection [ 36 ]. Notably, several publisher investigations have demonstrated a global scale of such misconduct, involving some highly cited researchers and top scientific journals [ 37 ].

Fake peer review, an extreme example of conflict of interest, is another form of misconduct that has surfaced in the time of mass proliferation of gold open-access journals and publication of articles without quality checks [ 38 ]. Fake reviews are generated by manipulating authors and commercial editing agencies with full access to their own manuscripts and peer review evaluations in the journal editorial management systems. The sole aim of these reviews is to break the manuscript evaluation process and to pave the way for publication of pseudoscientific articles. Authors of these articles are often supported by funds intended for the growth of science in non-Anglophone countries [ 39 ]. Iranian and Chinese authors are often caught submitting fake reviews, resulting in mass retractions by large publishers [ 38 ]. Several suggestions have been made to overcome this issue, with assigning independent reviewers and requesting their ORCID IDs viewed as the most practical options [ 40 ].

Conclusions

The peer review process is regulated by publishers and editors, enforcing updated global editorial recommendations. Selecting the best reviewers and providing authors with constructive comments may improve the quality of published articles. Reviewers are selected in view of their professional backgrounds and skills in research reporting, statistics, ethics, and language. Quality reviewer comments attract superior submissions and add to the journal’s scientific prestige [ 41 ].

In the era of digitization and open science, various online tools and platforms are available to upgrade the peer review and credit experts for their scholarly contributions. With its links to the ORCID platform and social media channels, Publons now offers the optimal model for crediting and keeping track of the best and most active reviewers. Publons Academy additionally offers online training for novice researchers who may benefit from the experience of their mentoring editors. Overall, reviewer training in how to evaluate journal submissions and avoid related misconduct is an important process, which some indexed journals are experimenting with [ 42 ].

The timelines and rigour of the peer review may change during the current pandemic. However, journal editors should mobilize their resources to avoid publication of unchecked and misleading reports. Additional efforts are required to monitor published contents and encourage readers to post their comments on publishers’ online platforms (blogs) and other social media channels [ 43 , 44 ].

The authors declare no conflict of interest.

Tutorial: Evaluating Information: Peer Review

• Evaluating Information
• Scholarly Literature Types
• Primary vs. Secondary Articles
• Peer Review
• Systematic Reviews & Meta-Analysis
• Gray Literature
• Evaluating Like a Boss
• Evaluating AV

Types of scholarly literature

You will encounter many types of articles and it is important to distinguish between these different categories of scholarly literature. Keep in mind the following definitions.

Peer-reviewed (or refereed):  Refers to articles that have undergone a rigorous review process, often including revisions to the original manuscript, by peers in their discipline, before publication in a scholarly journal. This can include empirical studies, review articles, meta-analyses among others.

Empirical study (or primary article): An empirical study is one that aims to gain new knowledge on a topic through direct or indirect observation and research. These include quantitative or qualitative data and analysis. In science, an empirical article will often include the following sections: Introduction, Methods, Results, and Discussion.

Review article:  In the scientific literature, this is a type of article that provides a synthesis of existing research on a particular topic. These are useful when you want to get an idea of a body of research that you are not yet familiar with. It differs from a systematic review in that it does not aim to capture ALL of the research on a particular topic.

Systematic review:  This is a methodical and thorough literature review focused on a particular research question. It's aim is to identify and synthesize all of the scholarly research on a particular topic in an unbiased, reproducible way to provide evidence for practice and policy-making. It may involve a meta-analysis (see below). 

Meta-analysis:  This is a type of research study that combines or contrasts data from different independent studies in a new analysis in order to strengthen the understanding of a particular topic. There are many methods, some complex, applied to performing this type of analysis.

Peer Reviewed Article vs. Review Article

TIP :  Review articles and Peer-reviewed articles are not the same thing!  Review articles synthesize and analyze the results of multiple studies on a topic; peer-reviewed articles are articles of any kind that have been vetted for quality by an expert or number of experts in the field. The bibliographies of review articles can be a great source of original, peer-reviewed empirical articles.

Peer Review in Three Minutes

Watch this 3 minute intro to peer review video by North Carolina State University or read this short introduction by the University of Texas at Austin Library.

Is It Peer-Reviewed & How Can I Tell?

There are a couple of ways you can tell if a journal is peer-reviewed:

• If it's online, go to the journal home page and check under About This Journal. Often the brief description of the journal will note that it is peer-reviewed or refereed or will list the Editors or Editorial Board.
• Go to the database Ulrich's and do a Title Keyword search for the journal. If it is peer-reviewed or refereed, the title will have a little umpire shirt symbol by it.
• BE CAREFUL! A journal can be refereed/peer-reviewed and still have non-peer reviewed articles in it. Generally if the article is an editorial, brief news item or short communication, it's not been through the full peer-review process. Databases like Web of Knowledge will let you restrict your search only to articles (and not editorials, conference proceedings, etc).

Checking Peer Review in Ulrich's

Using Ulrich's Periodicals Directory to Verify Peer Review of Your Journal Title

To use this, click on the Ulrich's link to enter the database (or search for it on the main library website).

Change the Quick Search dropdown menu to Title (Keyword) and type your  full   journal title  (not your article title or keywords) into the Quick Search box, then click Search. 

This will give you a list of journal titles which includes the title you typed in. Check the Legend in the upper right corner to view the Refereed symbol ("refereed" is another term for peer-reviewed.) Then check your journal title to make sure it has the refereed symbol next to it.

NOTE: Though a journal can be peer-reviewed, letters to the editor and news reports in those same scientific journals are not! Make sure your article is a primary research article.

• Ulrich's Periodicals Directory Contains information on currently published as well as discontinued periodicals. Includes magazines, journals, newsletters, newspapers, conference proceedings, and electronic publications, together with search and browse indexes. It also contains complete names and addresses of journal publishers.

Peer reviewed articles

They both come out once every month. They're both in English. Both published in the United States. Both of them are "factual".They both have pictures. They even cover some of the same topics.

The difference is that one--Journal of Geography--is peer-reviewed, whereas  National Geographic  is a popular-press title. 

Peer review  is scientists' and other scholars' best effort to publish accurate information. Each article has been submitted by a researcher, and then  reviewed by other scholars in the same field to ensure that it is sound science.  What they are looking for is that:

• The methodology has been fully described (and the study can thus be replicated by another researcher)
• There are no obvious errors of calculation or statistical analysis
• Crucially: The findings support the conclusions. That is, do the results of the research support what the researcher has said about them? The classic problematic example is a scientist claiming that hair growth causes time to pass: The correlation is clearly not causation.

Things to know about peer review:

• It isn't a perfect system:  Scientists make errors (or commit fraud) as often as any other human being and sometimes bad articles slip through. But in general, peer-review ensures that many trained eyes have seen an article before it appears in print.
• Peer-reviewed journals are generally considered "primary source" material:  When a new scientific discovery is made, a peer-reviewed journal is often--but not always--the first place it appears.
• Popular and trade publications  are not peer-reviewed, they are simply edited. That does not mean they are any less potentially truthful or informative--most popular and trade publications take pride in careful fact-checking.* But when the topic is scientific research, the information is generally  "secondary" : It has already appeared elsewhere (usually in a peer-reviewed journal) and has now been "digested" for a broader audience.
• Peer-reviewed journals will always identify themselves as such. If you want to verify that a journal is peer-reviewed, check Ulrich's Periodical Directory .

Some sources of peer-reviewed articles:

• Cornell University Library homepage In particular, check the Articles & Full-Text search and then choose "Limit to articles from scholarly publications, including peer-review" at the top-left.
• Google Scholar Google Scholar takes Google's PageRank algorithm and runs it on a pre-selected set of tables-of-contents and metadata from a preselected set of scholarly journals and papers. These are largely--though not entirely--peer-reviewed. It is a much better option than a regular Google search for scholarly information.
• Web of Science Choosing "All Databases" allows you to search an index of journal articles, conference proceedings, data sets, and other resources in the sciences, social sciences, arts, and humanities.
• << Previous: Primary vs. Secondary Articles
• Next: Systematic Reviews & Meta-Analysis >>
• Last Updated: Sep 6, 2024 12:35 PM
• URL: https://guides.library.cornell.edu/evaluate

Library Research Guides

Peer Review - what is it, and how do I find peer reviewed articles?

• What Is Peer Review?
• Types of Articles
• Finding Peer-reviewed Articles

Primary Study

Empirical study.

• How Do I Know It's Primary?

Ask Me for Help!

Thank you to the librarians at the Mildred E. Sawyer Library at Suffolk University Boston for agreeing to share the content of their Research Guide on this topic.  Your help was invaluable!

What is a Primary Study?

When an author conducts original research in a clinic or laboratory or out in the field, and then writes a detailed article about this research, this is a "primary study." Other related terms are "original research" or "research study." This means that new data or theories discussed in the article have not been previously published.

An original research article will be written by and for experts in their fields or disciplines. They are meant to be read by others in the field.  As such, the article should use precise, technical language to ensure that other researchers have an exact understanding of what was done, how to do it, and why it matters. 

By contrast, a  secondary article is like most of your term papers.  You base your arguments and conclusions on research materials that were written by other people. You did not personally go to Glacier Bay and collect data on the number of humpback whales present in the Bay each summer season, nor did you conduct research on the number and species of fish caught in local lakes, but you based your paper on the research of others, and your paper is considered secondary. Similarly, journalism is almost always considered secondary.  In order to write an article for a journal or magazine, a writer must gather information and data from others and summarize those findings in the article.  However, since the journalist did not do the research, the article would be a secondary secondary resource.

What is an Empirical Study? 

A professor may also use the term "empirical" to describe a study or research article.  In many cases, the term empirical is used to mean the exact same thing as a "primary study" or "original research" article. However, specifically, empirical research is based on direct observations or experiments.  It seeks to answer a specific question or to test a hypothesis.  

Finding an empirical study can be tricky.  The word empirical might be used in describing the study. In a few cases, the subject term "EMPIRICAL research" might even be assigned to a record in a database. Other terms like quantitative data might also be used in the summary.  You can try searching for these words --for example, family violence and empirical-- but otherwise you will need to read and evaluate the article to decide whether that particular research article could be termed an empirical study.

Another important hint: Some scholarly journals use a specific layout, called the "IMRaD" format, to communicate empirical research findings. Such articles typically have four components:

• Introduction: sometimes called "literature review" -- what is currently known about the topic -- usually includes a theoretical framework and/or discussion of previous studies; a statement of the problem and plan for solving the problem.
• Methodology:  sometimes called "research design" -- how to recreate the study -- usually describes the population, research process, and analytical tools, in other words, how the research was conducted.
• Results: sometimes called "findings" -- what was learned through the study -- usually appears as statistical data or as substantial quotations from research participants.
• Discussion: sometimes called "conclusion" or "implications" -- why the study is important -- usually describes how the research results influence professional practices or future studies.
• << Previous: Finding Peer-reviewed Articles
• Next: How Do I Know It's Primary? >>
• Last Updated: Mar 26, 2024 9:38 AM
• URL: https://uas.alaska.libguides.com/peer-review

Peer Reviewed and Primary Research

• Peer Review--What is it?
• Scholarly Journal vs. Magazine
• Finding Peer-Reviewed Articles
• Databases for finding Articles with Primary Research

How to Identify Peer-Reviewed Journal Content

The terms "Academic," "Scholarly," "Peer-Reviewed" and "Refereed" are often used interchangeably.  However, although all Peer-Reviewed Journals ARE Scholarly/Academic, NOT ALL academic or scholarly journals are peer-reviewed or refereed!   As discussed in the previous page, "Peer Review" is something quite specific.  A quarterly journal in a particular field may be written for academic use but the editorial process for selecting articles may not include peer review.  So how do you determine whether a journal is peer-reviewed?  One way is to let the database help you select such journals.  Some databases like JSTOR ( http://www.jstor.org/ ) or  Project Muse ( http://muse.jhu.edu/index.html )  should only contain peer-reviewed journals.  But most general databases include journals (peer-reviewed and not) as well as other non-peer-reviewed materials like magazines, newspapers, conference papers, book chapters and reports.  Here are some tips for limiting to peer reviewed in Academic Search Complete (This works for the EBSCO ERIC database vendor product.)

EbscoHost Databases

You can simply check off the "Limit" box below the search box(es)--even before you do your search--and eliminate all magazine articles, newspaper stories...ANYTHING that is not defined by Ebsco as a Peer-Reviewed Journal:

Or, you may "Refine" your search after you produce an initial results list, by limiting the search from the left frame:

Additional Ways to Determine if a Journal is Peer-Reviewed

•  The webpage for a journal may state this fact.  For example, the homepage for the journal Evolutionary Psychology  includes a statement in the first paragraph that it is "an open-access peer-reviewed journal that aims to foster communication between experimental and theoretical work on the one hand and historical, conceptual and interdisciplinary writings across the whole range of the biological and human sciences on the other."

IMPORTANT NOTE :  When an instructor says that you must use Peer-Reviewed Literature, they often MEAN that the articles must be "Research Articles" or "Primary Articles."   If this is the case, then extra care must be taken in judging the articles you retrieve.  A peer-reviewed or refereed journal usually contains content besides original research articles.  This additional content may include letters, editorials, commentary, book reviews and more.  Generally, your professor does NOT WANT YOU TO USE THIS CONTENT.   They want you to use primary sources.  Look at the next tabbed page for more on this.

• << Previous: Scholarly Journal vs. Magazine
• Next: Databases for finding Articles with Primary Research >>
• Last Updated: Aug 22, 2024 10:35 AM
• URL: https://rwu.libguides.com/peer-reviewed

• Student Services
• Faculty Services

Peer Review and Primary Literature: An Introduction: Finding Peer-Reviewed Articles

• Scholarly Journal vs. Magazine
• Peer Review: What is it?
• Finding Peer-Reviewed Articles
• Primary Journal Literature
• Is it Primary Research? How Do I Know?

How to Identify Peer-Reviewed Journal Content

The terms "Academic," "Scholarly," "Peer-Reviewed" and "Refereed" are often used interchangeably.  However, while all peer-reviewed Journals ARE scholarly/academic, NOT ALL academic or scholarly journals are peer-reviewed or refereed !   As discussed in the previous tab, "Peer Review" is something quite specific.  A journal in a particular field may be written for academic use but the editorial process for selecting articles may not include peer review. 

So, how do you determine whether a journal is peer-reviewed?  One way is to let the database help you select such journals.  Many of our databases will have a specific check-box for limiting your results to only peer-reviewed material.  Also, a few databases like BioOne should only contain peer-reviewed journals.  Most general databases include journals (peer-reviewed and not) as well as other non-peer-reviewed materials like magazines, newspapers, conference papers, book chapters and reports.  Here are some tips for limiting to peer reviewed in two major database vendor products:

EbscoHost Databases (This tip should work for any EbscoHost file, but this example is taken from Academic Search Complete ):

In the Advanced Search options, simply check off the "Scholarly (Peer Reviewed) Journals" box below the search box(es) before you do your search to eliminate anything that is not defined by Ebsco as a Peer-Reviewed Journal:

After you have searched and gotten results, you can filter to peer-reviewed material by checking off that "Scholarly (Peer Reviewed) Journals" box in the left frame:

 Gale Databases (This tip should apply to most Gale academic databases, but this example is taken from Academic OneFile ):

When you do a search in Academic OneFile, the results may or may not already be selected on the "Academic Journals" filter. Before you can get the option to narrow to "only peer-reviewed" content, you must select the "Academic Journals" content filter. After you have filtered your results to "Academic Journals", you will see the "only peer-reviewed" option, as indicated below. Click the box next to that option to filter out any material that has not gone through the peer-review process.

Additional Ways to Determine if a Journal is Peer-Reviewed

•  The webpage for a journal may state this fact.  For example, the homepage for the journal  includes a statement in the first paragraph that it is "an open-access peer-reviewed journal that aims to foster communication between experimental and theoretical work on the one hand and historical, conceptual and interdisciplinary writings across the whole range of the biological and human sciences on the other."

IMPORTANT NOTE :  When an instructor says that you must use Peer-Reviewed Literature, he or she may also mean that the articles must be "Research Articles" or "Primary Articles."   If this is the case, then extra care must be taken in judging the articles you retrieve.  A peer-reviewed or refereed journal usually contains content besides original research articles.  This additional content may include letters, editorials, commentary, book reviews and more.  Generally, your professor does NOT WANT YOU TO USE THIS CONTENT.   They want you to use "primary" literature.  Look at the next tabbed page for more on this.

Subject Guide

• << Previous: Peer Review: What is it?
• Next: Primary Journal Literature >>
• Last Updated: Aug 21, 2024 10:06 AM
• URL: https://suffolk.libguides.com/PeerandPrimary

• UConn Library
• Research Now
• Explore Information
• Understanding & Recognizing Peer Review

Explore Information — Understanding & Recognizing Peer Review

• Getting the Lay of the Land
• Why use Library Information?
• The Information Lifecycle
• Primary & Secondary Sources - Humanities & Social Sciences
• Primary & Secondary Sources - Sciences
• Help & Other Resources
• Research Now Homepage

What Do You Mean by Peer Reviewed Sources?

(Source: NCSU Libraries)

What's so great about peer review?

Peer reviewed articles are often considered the most reliable and reputable sources in that field of study. Peer reviewed articles have undergone review (hence the "peer-review") by fellow experts in that field, as well as an editorial review process. The purpose of this is to ensure that, as much as possible, the finished product meets the standards of the field. 

Peer reviewed publications are one of the main ways researchers communicate with each other. 

Most library databases have features to help you discover articles from scholarly journals. Most articles from scholarly journals have gone through the peer review process. Many scholarly journals will also publish book reviews or start off with an editorial, which are not peer reviewed - so don't be tricked!

So that means I can turn my brain off, right?

Nope! You still need to engage with what you find. Are there additional scholarly sources with research that supports the source you've found, or have you encountered an outlier in the research? Have others been able to replicate the results of the research? Is the information old and outdated? Was this study on toothpaste (for example) funded by Colgate? 

You're engaging with the research - ultimately, you decide what belongs in your project, and what doesn't. You get to decide if a source is relevant or not. It's a lot of responsibility - but it's a lot of authority, too.

Understanding Types of Sources

• Popular vs. Scholarly
• Reading Scholarly Articles
• Check Yourself!

Popular vs. scholarly articles.

When looking for articles to use in your assignment, you should realize that there is a difference between "popular" and "scholarly" articles.

Popular  sources, such as newspapers and magazines, are written by journalists or others for general readers (for example, Time, Rolling Stone, and National Geographic).

Scholarly  sources are written for the academic community, including experts and students, on topics that are typically footnoted and based on research (for example, American Literature or New England Review). Scholarly journals are sometimes referred to as "peer-reviewed," "refereed" or "academic."

How do you find scholarly or "peer-reviewed" journal articles?

The option to select  scholarly or peer-reviewed articles is typically available on the search page of each database.  Just check the box or select the option . You can also search Ulrich's Periodical Directory  to see if the journal is Refereed / Peer-reviewed.  

Popular Sources (Magazines & Newspapers) Inform and entertain the general public.

• Are often written by journalists or professional writers for a general audience
• Use language easily understood by general readers
• Rarely give full citations for sources
• Written for the general public
• Tend to be shorter than journal articles

Scholarly or Academic Sources (Journals & Scholarly Books) Disseminate research and academic discussion among professionals in a discipline. 

• Are written by and for faculty, researchers or scholars (chemists, historians, doctors, artists, etc.)
• Uses scholarly or technical language
• Tend to be longer articles about research
• Include full citations for sources 
• Are often refereed or peer reviewed (articles are reviewed by an editor and other specialists before being accepted for publication)
• Publications may include book reviews and editorials which are not considered scholarly articles

Trade Publications Neither scholarly or popular sources, but could be a combination of both. Allows practitioners in specific industries to share market and production information that improves their businesses.

• Not peer reviewed. Usually written by people in the field or with subject expertise
• Shorter articles that are practical
• Provides information about current events and trends 

What might you find in a scholarly article?

• Title:  what the article is about
• Authors and affiliations:  the writer of the article and the professional affiliations. The credentials may appear below the name or in a footnote.
• Abstract: brief summary of the article. Gives you a general understanding  before you read the whole thing.
• Introduction: general overview of the research topic or problem
• Literature Review: what others have found on the same topic
• Methods:  information about how the authors conducted their research
• Results: key findings of the author's research
• Discussion/Conclusion: summary of the results or findings
• References: Citations to publications by other authors mentioned in the article
• Anatomy of a Scholarly Article This tutorial from the NCSU Libraries provides an interactive module for learning about the unique structure and elements of many scholarly articles.

• << Previous: Primary & Secondary Sources - Sciences
• Next: Help & Other Resources >>
• Last Updated: Aug 2, 2024 4:30 PM
• URL: https://guides.lib.uconn.edu/exploreinfo

Main Navigation Menu

Peer-review and primary research.

• Getting Started With Peer-Reviewed Literature
• What is a Primary Study

Finding Peer-Reviewed Journal Articles

Finding peer-reviewed literature in primo, finding peer-reviewed literature in databases.

• Finding Randomized Controlled Trials (RCTs)
• Evaluating Scholarly Articles
• Google Scholar
• Tips for Reading Journal Articles

Finding Peer Reviewed Journals

• Use a discipline-specific database (Education Full Text, Applied Science and Technology, APA PsycInfo, etc.). While this strategy will help you find peer-reviewed journal articles, you should be aware that not all citations in such indexes are from peer-reviewed journals. 
• Use special features of online databases. Many allow you to limit your search results to peer-reviewed journals.
• Check the Cabell's Directory , available from the Databases A-Z List, to see if it characterizes the journal in question as “peer-reviewed.”
• Check the “Instructions to Authors” section in the journal, where the editor explains the process used to decide whether an article is appropriate for a particular journal.
• If you are in doubt, consult with a librarian or your professor.

Follow some examples below on how to find peer-reviewed journal articles in PRIMO, Databases and Google Scholar.

PRIMO Search  on the library homepage :

Enter a search on your research topic in the search anything box.

After you have searched and gotten results use the filters on the left-hand side to limit your search to peer-reviewed journals.

From the library homepage select  Browse Databases . Choose a subject specific or a multidisciplinary database that relates to your research topic. If you are accessing the database remotely you will be prompted to login with your 700 # and password. 

The example below is for ProQuest databases, in this examples it is ProQuest Central:

Databases will often have an option to limit your results to peer-reviewed articles before you do your search. 

Once you complete a search and go to the results page, you can filter to peer-reviewed material by selecting the "Scholarly Journals" box on the left-hand side.

• << Previous: What is a Primary Study
• Next: Finding Randomized Controlled Trials (RCTs) >>
• Last Updated: Feb 15, 2024 2:45 PM
• URL: https://guides.library.ucmo.edu/peerreview

Psychology 111 - Henares: What's a Peer Reviewed Article?

• What's a Peer Reviewed Article?
• Empirical and Review Articles
• Finding Scholarly Articles in Subject Databases
• Finding Both Sides
• Using LibrarySearch
• Using Google Scholar Effectively
• Searching Google
• Citing sources and using Zotero
• Video on using Zotero
• Consultations

Peer review in 3 minutes

• Peer review video from North Carolina State University Libraries

What is a "Refereed" (Peer-Reviewed) article?

Refereed, or peer-reviewed articles are ones which are reviewed by a board of editors who are expert in a given field.These readers/editors determine if the material submitted for publication adds something significant to a body of knowledge that already exists. Refereed (or peer-reviewed) materials are important to the research and the literature of most academic fields; they assure readers that the information conveyed is reliable and timely .

Note : The term "scholarly materials" is often used to describe refereed (or peer-reviewed) materials. However,the term is not exclusive to refereed material. While non-refereed materials may not by scrutinized as intensely as refereed materials, they can still be considered scholarly (or academic)!

Peer Reviewed Journals

• Audience is general public
• No previous knowledge is required  
• Articles must help sell the magazine  
• Authors are not necessarily  experts  
• Carries lots of advertising, probably more than editorial content  
• Lots of glossy pictures  
• For sale at newsstands

Popular Journals

• Next: Empirical and Review Articles >>
• Last Updated: Sep 12, 2024 10:36 AM
• URL: https://libguides.stthomas.edu/DFC_psych

Biology 231 - Research Methods

• Verifying If An Article Is Peer-Reviewed
• Should I Trust Internet Sources?
• Suggested Databases for Finding Sources
• Critical Analysis
• Creating an Annotated Bibliography
• Creating a Literature Review
• Resources for Writing Position Papers
• Off-Campus Access

Ask A Librarian

chat Text: 1-308-210-3865 email Librarians by Subject Make an Appointment

What Is A Peer-Reviewed Article?

In academic publishing, the goal of  peer review  is to  assess the quality  of articles submitted for publication in a scholarly journal. Before an article is deemed appropriate to be published in a peer-reviewed journal, it must undergo the following process:

• The author of the article must submit it to the journal editor who forwards the article to experts in the field. Because the reviewers specialize in the same scholarly area as the author, they are considered the author’s peers (hence “peer review”).
• These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.
• The peer reviewers check the manuscript for accuracy and assess the validity of the research methodology and procedures.
• If appropriate, they suggest revisions. If they find the article lacking in scholarly validity and rigor, they reject it.

Because a peer-reviewed journal will not publish articles that fail to meet the standards established for a given discipline, peer-reviewed articles that are accepted for publication exemplify the best research practices in a field.

Search Tip:  Peer-reviewed journals may also contain items that are not peer-reviewed, such as letters to the editor, literature reviews, opinion pieces, and book reviews.  Even if you check the peer-review limiter box, you still need to examine the items carefully to be sure they are articles containing original research.

Features of a Peer-Reviewed Article

When you are determining whether or not the article you found is a peer-reviewed article, you should consider the following.

Does the article have the following features?

Also consider...

• Is the journal in which you found the article published or sponsored by a professional scholarly society, professional association, or university academic department? Does it describe itself as a peer-reviewed publication? (To know that, check the journal's website). 
• Did you find a citation for it in one of the  databases that includes scholarly publications? (Academic Search Complete, PsycINFO, etc.)?  Read the database description to see if it includes scholarly publications.
• In the database, did you limit your search to  scholarly  or  peer-reviewed  publications? 
• Is the topic of the article  narrowly focused and explored in depth ?
• Is the article based on either  original research  or  authorities in the field  (as opposed to personal opinion)?
• Is the article written for readers with some prior knowledge of the subject?
• If your field is social or natural science, is the article divided into sections with headings such as those listed below?

How Do I Find Peer-Reviewed Articles?

The easiest and fastest way to find peer-reviewed articles is to  search the online library databases , many of which include peer-reviewed journals. To make sure your results come from peer-reviewed (also called "scholarly" or "academic") journals, do the following:

• Read the database description  to determine if it features peer-reviewed articles.
• When you search for articles, choose the Advanced Search option. On the search screen, look for  a check-box that allows you to limit your results to peer-reviewed  only.
• If you didn't check off the "peer-reviewed articles only" box,  try to see if your results can organized by source . For example, the database Criminal Justice Abstracts will let you choose the tab "Peer-Reviewed Journals."
• << Previous: Home
• Next: Verifying If An Article Is Peer-Reviewed >>
• Last Updated: Sep 4, 2024 6:55 AM
• URL: https://guides.library.unk.edu/BIO231/ResearchMethods
• University of Nebraska Kearney
• Library Policies
• Accessibility
• Research Guides
• Citation Guides
• A to Z Databases
• Open Nebraska
• Interlibrary Loan

2508 11th Avenue, Kearney, NE 68849-2240

Circulation Desk:  308-865-8599 Main Office:  308-865-8535

  Ask A Librarian

GSD 3681/AFRO 3681 Black Central and Nordic Europe

Article Databases

Finding books.

• Libraries Search Search the University of Minnesota Libraries to find books, scholarly journal articles, news, magazines, media, and other items in the University's collection. View this tutorial to learn how to go from a general idea to a very precise set of results of journal articles and scholarly materials.
• WorldCat Find books, journals, articles, maps, music scores, sound recordings, films, theses/dissertations, machine-readable data files, and any other materials available in libraries worldwide. All subject areas are covered. Request items through Interlibrary Loan using the ILLiad link on each item's record. A mobile version of this database is available. Limited to 65 simultaneous users.

SEARCHING EFFECTIVELY

• To search for people of African descent, try first searching using the term "Black people" and the name of the country or "Europe" 
• You may need to use other related terms too, such as "africans," "afro-german" and related concepts like "refugees," "ethnic minorities," "race relations." 
• Also search using the term for people of African descent in the language of the target country: e.g. schwarz, noir, svart, zwart.
• Use the * as truncation to find variations.  E.g. "schwarz*" will search "schwarze" and "schwarzen"
• Below are some sample searches for Germany in  Libraries Search Advanced . Use the limits on the right to narrow the results further. Follo this pattern for other countries.
• "black people" and "germany" as subjects
• "africans" and "germany"  as subjects
• "germany -- race relations"  as subject
• "germany" and "minorities"  as subject

Selected books

Finding Primary Source Materials

• AM Primary Sources for Teaching and Research AM (formerly Adam Matthew) publishes unique primary source collections from archives around the world. Collections span the social sciences and humanities and cover a multitude of topics ranging from Medieval family life and Victorian medicine to 1960s pop culture and global politics.
• EuroDocs: Online Sources for European History Links to European primary historical documents that are transcribed, reproduced in facsimile, or translated. Video, sound files, maps, photographs and other imagery, databases, and other documentation are also available.
• Europeana Single access point to millions of artworks, artefacts, books, films and music from European museums, galleries, libraries and archives.
• German History in Documents and Images/Deutsche Geschichte in Dokumenten und Bildern Primary source materials documenting Germany's political, social, and cultural history from 1500 to the present. It comprises original German texts with new English translations, and a wide range of visual imagery.
• HathiTrust Digital Library This link opens in a new window HathiTrust provides access to millions of books and other materials. Full text searching of most books is available and books in the public domain (generally older books) can be freely viewed and/or downloaded. Books still in copyright have more limited access. Users with print-reading disabilities can apply for special access to digitized works by emailing [email protected].
• Nineteenth Century Collections Online Search primary source collections of the nineteenth century (1800s) with books, newspapers, pamphlets, manuscripts, ephemera, maps, statistics, and more. Topics include British politics, theater and music; European literature, Asian exploration, photography, and more.

Finding Primary Source materials in Libraries Search

Additional primary source materials can be found by searching Libraries Search . Enter one of the terms below and keywords of the event as subject keywords.

• "personal narratives"
• "sources"
• "documentary film" 
• "interviews" 
• "correspondence" 

More information

• << Previous: Home
• Next: Background information and reference >>

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• News & Views
• Legal clarity allows...

Legal clarity allows the use of GnRH analogues in research

• Related content
• Peer review
• Susan Bewley , emeritus professor of obstetrics and women’s health 1 ,
• Kath Checkland , professor of health policy and primary care 2 ,
• Paul Garner , emeritus professor in evidence synthesis 3 ,
• Riittakerttu Kaltiala , professor of adolescent psychiatry , chief psychiatrist 4 5 ,
• Margaret McCartney , senior lecturer 6 ,
• Carl Heneghan , professor of evidence based medicine 7 ,
• Hannah Ryan , clinical pharmacology specialty registrar 8
• 1 King’s College London, London, UK
• 2 University of Manchester, Manchester, UK
• 3 Liverpool School of Tropical Medicine, Liverpool, UK
• 4 Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland
• 5 Department of Adolescent Psychiatry, Tampere University Hospital, Tampere University, Tampere, Finland
• 6 School of Medicine, University of St Andrews, St Andrews, UK
• 7 University of Oxford, Oxford, UK
• 8 Royal Liverpool University Hospital, Liverpool, UK
• susan.bewley{at}kcl.ac.uk

The Cass review of the care of children and adolescents with gender dysphoria was a wake up call. 1 Having commissioned several peer reviewed, systematic reviews of evidence, paediatrician Hilary Cass’s team drew on four years’ comprehensive engagement with service users, parents, clinicians, researchers, and advocacy groups, finding that most children’s gender dysphoria in historical cohorts resolves through puberty; suicidality is equivalent to children with diagnosed mental health …

Log in using your username and password

BMA Member Log In

If you have a subscription to The BMJ, log in:

• Need to activate
• Log in via institution
• Log in via OpenAthens

Log in through your institution

Subscribe from £184 *.

Subscribe and get access to all BMJ articles, and much more.

* For online subscription

Access this article for 1 day for: £50 / $60/ €56 ( excludes VAT )

You can download a PDF version for your personal record.

Buy this article

• Introduction
• Conclusions
• Article Information

AI indicates artificial intelligence; AIPA, artificial intelligence predictive algorithm; CE, Conformité Européene; FDA, US Food and Drug Administration.

CVD indicates cardiovascular disease; UTI, urinary tract infection.

Light blue indicates the public availability of evidence. Orange indicates that the available evidence partially covered the requirement. Dark blue indicates no evidence was publicly available. AI indicates artificial intelligence.

eAppendix 1. Search Strategies Used Up to July 7, 2023

eAppendix 2. Selection Process

eAppendix 3. Online Questionnaire for Information From Authors and Commercial Product Owners

eTable 1. The Evidence Requirements Established per Life Cycle Phase as Described in the Dutch AIPA Guideline

eFigure. Flowchart of Literature Inclusion for Assessment of the Six Phases

eTable 2. Overview of Publication Characteristics per Predictive ML Algorithm

eTable 3. Overview of the Availability of Evidence per Predictive ML Algorithm

Data Sharing Statement

• The Need for Continuous Evaluation of AI Prediction Algorithms JAMA Network Open Invited Commentary September 12, 2024 Nigam H. Shah, MBBS, PhD; Michael A. Pfeffer, MD; Marzyeh Ghassemi, PhD

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health

Primary Care

• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Rakers MM , van Buchem MM , Kucenko S, et al. Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care : A Systematic Review . JAMA Netw Open. 2024;7(9):e2432990. doi:10.1001/jamanetworkopen.2024.32990

Manage citations:

© 2024

• Permissions

Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care : A Systematic Review

• 1 Department of Public Health and Primary Care, Leiden University Medical Centre, ZA Leiden, the Netherlands
• 2 National eHealth Living Lab, Leiden University Medical Centre, ZA Leiden, the Netherlands
• 3 Department of Information Technology and Digital Innovation, Leiden University Medical Center, ZA Leiden, the Netherlands
• 4 Hamburg University of Applied Sciences, Department of Health Sciences, Ulmenliet 20, Hamburg, Germany
• 5 Department of Digital Health, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, CG Utrecht, the Netherlands
• 6 Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, CG Utrecht, the Netherlands
• Invited Commentary The Need for Continuous Evaluation of AI Prediction Algorithms Nigam H. Shah, MBBS, PhD; Michael A. Pfeffer, MD; Marzyeh Ghassemi, PhD JAMA Network Open

Question   Which machine learning (ML) predictive algorithms have been implemented in primary care, and what evidence is publicly available for supporting their quality?

Findings   In this systematic review of 43 predictive ML algorithms in primary care from scientific literature and the registration databases of the US Food and Drug Administration and Conformité Européene, there was limited publicly available evidence across all artificial intelligence life cycle phases from development to implementation. While the development phase (phase 2) was most frequently reported, most predictive ML algorithms did not meet half of the predefined requirements of the Dutch artificial intelligence predictive algorithm guideline.

Meaning   Findings of this study underscore the urgent need to facilitate transparent and consistent reporting of the quality criteria in literature, which could build trust among end users and facilitate large-scale implementation.

Importance   The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.

Objectives   To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.

Evidence Review   PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.

Findings   The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).

Conclusions and Relevance   The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms’ quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.

In most high-income countries, primary health care is affected by the increasing burden of illness experienced by aging and multimorbid populations along with personnel shortages. 1 Primary care generates large amounts of routinely collected coded and free-text clinical data, which can be used by flexible and powerful machine learning (ML) techniques to facilitate early diagnosis, enhance treatment, and prevent adverse effects and outcomes. 2 - 5 Therefore, primary care is a highly interesting domain for implementing predictive ML algorithms in daily clinical practice. 6 - 8

Nevertheless, scientific literature describes the implementation of artificial intelligence (AI), especially predictive ML algorithms, as limited and far behind other sectors in data-driven technology. Predictive ML algorithms in health care often face criticism regarding the lack of comprehensibility and transparency for health care professionals and patients as well as lack of explainability and interpretability. 8 - 11 Additionally, the reporting of peer-reviewed evidence is limited, and the utility of predictive ML algorithms in clinical workflows is often unclear. 8 , 12 - 14 In response to these challenges, the Dutch Ministry of Health, Welfare, and Sports commissioned the development and validation of a Dutch guideline for high-quality diagnostic and prognostic applications of AI in health care. Published in 2022, the Dutch Artificial Intelligence Predictive Algorithm (AIPA) guideline is applicable to predictive ML algorithms. 15 , 16 The guideline encourages the collection of data and evidence consistent with the 6 phases and criteria outlined in the AI life cycle (requirements), providing a comprehensive overview of existing research guideline aspects across the AI life cycle.

In this systematic review, we aimed to (1) systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and (2) ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports, across the AI life cycles. For this purpose, the Dutch AIPA guideline was adapted into a practical evaluation tool to assess the quality criteria of each predictive ML algorithm.

We conducted the systematic review in 2 steps. First, we systematically identified predictive ML algorithms by searching peer-reviewed literature and FDA and CE registration databases. Second, we ascertained the availability of evidence for the identified algorithms across the AI life cycle by systematically searching literature databases and technical reports, examining references in relevant studies, conducting product searches, visiting manufacturer websites, and contacting authors and product owners. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses ( PRISMA ) reporting guideline. 17

Peer-reviewed studies were included if they met all of the following eligibility criteria: (1) published between January 2000 and July 2023; (2) written in English; (3) published as original results; (4) concerned a predictive ML algorithm intended for primary care; and (5) focused on the implementation of the predictive ML algorithm in a research setting or clinical practice with, for example, pilot, feasibility, implementation, or clinical validation study designs. This review examined ML techniques (eg, [deep] neural networks, support vector machines, and random forests) developed for tasks such as computer vision and natural language processing that generated the prediction of health outcomes in individuals. We classified a study as applying ML if it used a nonregression statistical technique to develop or validate a prediction model, similar to Andaur Navarro et al, 9 excluding traditional statistical approaches, such as expert systems and decision trees based on expert knowledge. Studies that addressed predictive ML algorithm development or external validation without implementation in primary care were excluded. Given that over 60% of CE-marked AI tools are not found in electronic research databases, 12 this review included CE-marked or FDA-approved predictive ML algorithms published in FDA and CE registration databases. 18 - 20

Searches were conducted using the following electronic databases: PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar. All databases were searched in July 2023 for entries from January 2000 to July 2023. The search terms, derived from the National Library of Medicine MeSH (Medical Subject Headings) Tree Structures and the review team’s expertise, formed a combination related to AI, primary care, 21 and implementation 22 (defined in Box 1 ). The full search strategy is provided in eAppendix 1 in Supplement 1 .

Definitions

Predictive algorithm.

“An algorithm leading to a prediction of a health outcome in individuals. This includes, but is not limited to, predicting the probability or classification of having (diagnostic or screening predictive algorithm) or developing over time (prognostic or prevention predictive algorithm) desirable or undesirable health outcomes.” 16

Implementation

“An intentional effort designed to change, adapt, or uptake interventions into routines, including pilot and feasibility studies.” 22 Implemented predictive ML algorithms can be placed in phases 5 and 6 of the AI life cycle, as defined by the Dutch AIPA guideline. 16

“Universal access to essential healthcare in communities, facilitated by practical, scientifically sound, socially acceptable methods and technology, sustainably affordable at all developmental stages, fostering self-reliance and self-determination.” 21

Abbreviations: AI, artificial intelligence; AIPA, Artificial Intelligence Predictive Algorithm; ML, machine learning.

Three of us (M.M.R., M.M.vB., and S.K.) conducted an independent review of the selection process, resolving disagreements among us through discussion with a senior reviewer (H.J.A.vO.). The full selection process is detailed in eAppendix 2 in Supplement 1 .

Five strategies were used to gather publicly available evidence for all identified predictive ML algorithms: (1) searches of PubMed and Google Scholar using product, company, and author names; (2) searches of technical reports from the FDA and CE registration online databases; (3) exploration of references within selected studies; (4) visits to predictive ML algorithm manufacturer websites; and (5) solicitation of information from authors and product owners via email or telephone, with a request to complete an online questionnaire about the reported evidence (eAppendix 3 in Supplement 1 ). Accepted data sources included original, peer-reviewed articles in English as well as posters, conference papers, and data management plans (DMPs).

The availability of evidence was categorized according to the life cycle phases ( Box 2 ) and the requirements per phase set forth by the Dutch AIPA guideline. 15 , 16 These requirements ( Table 1 ) are defined as aspects necessary to address during the AI predictive algorithm life cycle. Therefore, developers, researchers, or owners of predictive ML algorithms should ideally provide data and evidence regarding these aspects.

Summary of the 6 Life Cycle Phases a 

Phase 1: preparation and verification of the data.

A DMP should be used to prepare for the collection and management of the necessary data for phases 2 to 5. In this plan, agreements and established procedures for collecting, processing, and storing data and managing this data are captured. During the implementation of the DMP, any changes should be continuously updated.

Phase 2: Development of the AI Model

The development of the AI model, which results from the analysis of the training data, entails the development of the algorithm and the set of algorithm-specific data structures.

Phase 3: Validation of the AI Model

The AI model undergoes external validation, which involves evaluating its performance using data not used in phase 2. The validation process assesses the statistical or predictive value of the model and examines issues related to fairness and algorithmic bias.

Phase 4: Development of the Necessary Software Tool

The focus shifts to developing the necessary software tool around the AI model. This phase encompasses designing, developing, conducting user testing, and defining the system requirements for the software.

Phase 5: Impact Assessment of the AI Model in Combination With the Software

This phase determines the impact or added value of integrating the AI model and software within the intended medical practice or context. It evaluates how these advancements affect medical actions and the health outcomes of the target group, such as patients, clients, or citizens. Additionally, conducting a health technology assessment is part of this phase.

Phase 6: Implementation and Use of the AI Model With Software in Daily Practice

The AI model and software are implemented, monitored, and incorporated into daily practice. Efforts are made to ensure smooth integration, continuous monitoring, and appropriate education and training related to their use.

Abbreviations: AI, artificial intelligence; DMP, data management plan.

a As established in the Dutch AIPA guideline. 15 , 16

The extent to which the evidence of each predictive ML algorithm aligned with the requirements of the Dutch AIPA guideline was assessed per life cycle phase ( Table 1 ; Box 2 ; eTable 1 in Supplement 1 ), using availability scores (2 for complete, 1 for partial, and 0 for none). Two analyses were conducted. First, availability of evidence per requirement was represented as a percentage, considering the requirements per life cycle phase. The availability of evidence per life cycle phase was reported as a percentage and calculated by dividing the sum of scores of a specific life cycle phase by the maximum possible score. Second, evidence availability per predictive ML algorithm was calculated as the sum of values for all requirements divided by the total applicable requirements, excluding the requirements that were not applicable because of the life cycle phase of the algorithm (eTable 1 in Supplement 1 ; requirements are shaded in orange); the denominator value was 48. These availability scores aimed to provide an overview of implemented predictive ML algorithms and evidence per life cycle phase.

The analysis was conducted independently by 3 of us (M.M.R., M.M.vB., and S.K.), who resolved discrepancies through discussion with another author (H.J.A.vO.). Data were analyzed using Microsoft Excel for Windows 11 (Microsoft Corp).

Of the 5994 studies identified initially, 20 (comprising 19 predictive ML algorithms) met the inclusion criteria and were included in this systematic review. One algorithm was excluded after personal communication confirmed that the tool used was a rule-based expert system. 23 Additionally, 25 commercially available CE-marked or FDA-approved predictive ML algorithms in primary care were included. 18 Only 2 predictive ML algorithms were found in the FDA or CE registration databases and the literature databases searched. 24 - 27 Forty-three AIPAs were included in the analysis ( Figure 1 ). 24 - 65

Table 2 provides an overview of the key characteristics of the 43 predictive ML algorithms included in this review. 24 - 67 Most studies (35 [81%]) were published in the past 5 years (2018-2023). 24 , 26 - 36 , 40 - 46 , 50 - 55 , 57 - 62 , 64 - 66 , 68 Most predictive ML algorithms (36 [84%]) fit under the category of clinical decision support systems for either diagnosis or treatment indication ( Figure 2 ). 24 - 29 , 31 - 35 , 37 - 42 , 44 , 46 , 47 , 49 , 51 - 53 , 55 - 61 , 63 - 67 Twenty-seven predictive ML algorithms (63%) focused on cardiovascular diseases and diabetes ( Figure 2 ). 24 - 28 , 32 - 34 , 37 - 47 , 50 , 57 , 60 , 62 , 64 , 66 , 68 Furthermore, 9 AIPAs (21%) mentioned the use of AI in their product descriptions but did not offer any specific details about the AI technique applied to develop the model. 37 - 39 , 42 , 45 , 47 , 48 , 50 , 54 These 9 predictive ML algorithms were FDA approved or CE marked. Twelve of 43 (28%) predictive ML algorithms were implemented (life cycle phase 6) in a research setting but not in practice. 26 , 31 - 34 , 58 , 61 - 64 , 66 , 67

The search for public availability of evidence on the 43 predictive ML algorithms resulted in 1541 hits, of which 33 were duplicates. Eighty-two publications met the inclusion criteria. Additionally, 80 publications were provided by the product owners or authors or obtained from vendors’ websites. A total of 162 publications were included in the study, which included peer-reviewed articles, technical reports, posters, conference papers, and DMPs (eFigure in Supplement 1 ). Nine authors and product owners responded and completed the online questionnaire about the reported evidence of the predictive ML algorithm. An overview of the publication characteristics per predictive ML algorithm is provided in eTable 2 in Supplement 1 . An overview of the availability of evidence per predictive ML algorithm is provided in eTable 3 in Supplement 1 .

An overview of the availability of evidence per requirement and according to the life cycle phase is shown in Figure 3 . The 3 most commonly available types of evidence per requirement were a clear definition of the target use of the AI model, evaluation of its statistical characteristics, and adherence to software standards and regulations (78%, 47%, and 66% availability, respectively). Conversely, the least available evidence pertained to the implementation plan, monitoring, and health technology assessment (2%, 2%, and 14% availability, respectively), largely due to a lack of information from FDA-approved and CE-marked predictive ML algorithms.

The life cycle phase with the most comprehensive evidence was phase 2 (development), where 46% evidence availability for the relevant requirements was identified. This finding was followed by life cycle phase 3 (validation), with a 39% evidence availability. The life cycle phases with the most limited availability of evidence were phase 1 (preparation) at 19% and phase 5 (impact assessment) at 30%. Commercially available CE-marked and FDA-approved AIPAs offered less evidence across all life cycle phases compared with AIPAs found in the literature database search (29% vs 48% of the overall score to be determined).

For 5 predictive ML algorithms (12%), evidence was available only for 2 requirements: definition of the target use and required standards and regulations (availability per predictive ML algorithm score: 4 of 48 possible points). 37 , 38 , 47 , 48 , 69 Twelve (28%) predictive ML algorithms obtained approximately half of their individual maximum attainable evidence availability score. 24 , 26 - 28 , 30 , 31 , 33 , 34 , 36 , 41 , 43 , 65 Twelve (28%) did not reach life cycle phase 6 (implementation). 26 , 31 - 34 , 36 , 58 , 61 - 64 , 66 The predictive ML algorithms that reported the highest availability of evidence per predictive ML algorithm score were a risk-prediction algorithm for identifying undiagnosed atrial fibrillation 28 (36 of 48 possible points) and an AI-powered clinical decision support tool that enabled early diagnosis of low ejection fraction 34 (37 of 42 possible points). Both predictive ML algorithms were neither CE-marked nor FDA-approved at the time of publication. Overall, predictive ML algorithms identified through the peer-reviewed literature database search yielded more publicly available evidence 24 , 26 , 28 - 36 , 58 , 61 - 66 compared with the predictive ML algorithms identified solely from FDA-approved or CE-marked databases 25 , 27 , 37 - 55 , 57 , 59 , 60 , 68 (45% vs 29%) (eTable 2 in Supplement 1 ).

To our knowledge, this systematic review provides the most comprehensive overview of predictive ML algorithms implemented in primary care to date and reveals insufficient public availability of evidence of a broad set of predictive ML algorithm quality criteria. The availability of evidence was highly inconsistent across the included predictive ML algorithms, life cycles, and individual quality criteria. Predictive ML algorithms identified from peer-reviewed literature generally provided more publicly available evidence compared with predictive ML algorithms identified solely from FDA or CE registration databases.

The results align with those of previously published research. The scarcity of evidence is particularly pronounced among predictive ML algorithms that have received FDA approval or CE marking. 9 , 12 , 13 , 70 - 72 Many AI developers in the health care sector are known not to disclose information in the literature about the development, validation, evaluation, or implementation of AI tools. 12 , 19 , 73 There may be tension between protecting intellectual property and being transparent. 74 Moreover, not all evidence requires peer review, including regulatory processes such as obtaining a CE mark, where notified bodies assess the high-risk medical devices’ evidence for compliance. However, concerns may arise regarding the complexity of methodologies when reporting on effectiveness in a clinical setting. In such cases, there might be a preference for a peer-reviewed process to ensure that evaluation does not solely rely on notified bodies. 75 Although the FDA and the European Union Medical Device Regulation and, more recently, the AI Act, have released new initiatives to enhance transparency, disclosure of evidence was not mandatory at the time of writing this systematic review. 76 - 80 It would be interesting to assess the impact of new regulation in the future.

The availability of evidence fosters transparency and trust among end users, allowing other investigators to scrutinize the data and methods used and thus ensuring ethical and unbiased research and development practices. 81 , 82 Researchers can build on previous work, advancing scientific knowledge by making evidence available. If studies lack the necessary details, subsequent researchers may be more likely to create a new AI model instead of validating or updating an existing one. In addition, transparent reporting of predictive ML algorithms encourages vigilance among users, increasing the level of trust humans have in AI, as shown by human factors research. 82 On the other hand, failing to provide evidence can hamper patient safety due to, for example, algorithmically generated outcomes, interpretations, and recommendations that exhibit unfair advantages or disadvantages for specific individuals or groups. 83

The results show that evidence was the most scarce regarding the availability of, or reference to, a DMP. The DMP, while not necessarily required to be publicly accessible, is critical to preparing for collecting, managing, and processing data. The DMP plays an overarching role in the entire trajectory toward structurally implementing and using the AI model in daily practice. 84 It forms an essential component for every stage of the predictive ML algorithm life cycle and can ensure and safeguard data quality, reproducibility, and transparency while striving for Findable, Accessible, Interoperable, and Reusable (FAIR) data. 85 - 88 The FAIR principles aim to support the reuse of scholarly data, including algorithms, and to focus on making data findable and reusable by humans and machines. 87 Although FAIR principles have been widely adopted in academic contexts, the response of the industry has been less consistent. 89

Evidence was also limited regarding the impact and health technology assessments of predictive ML algorithms. The lack of accessible evaluations of the outcome and implementation in everyday clinical practice may hinder the translation of research findings into practical applications in health care. 90 Lack of such information may also impede adoption, as medical professionals need robust evidence to gain trust in these technologies and consistently integrate them into their everyday workflow. Medical professionals stress the importance of adhering to legal, ethical, and quality standards. They voice the need to be trained to interpret the availability of evidence supporting the safety of AI systems, including predictive ML algorithms and their effectiveness. 8 , 81 Without this information, it is challenging to ascertain whether the success of a predictive ML algorithm model is attributable to the model itself, the elements of its implementation, or both. As a result, it can be challenging to inform stakeholders about which, how, and for whom predictive ML algorithms are most effective.

Applying the Dutch AIPA guideline requirements to structure the availability of evidence, as demonstrated in this systematic review, can serve as a blueprint for showcasing to policymakers, primary care practitioners, and patients the reliability, transparency, and advancement of predictive ML algorithms. The guideline also has the potential to accelerate the process of complying with regulations. 16 Although not legally binding, the guideline can be used by developers and researchers as the basis for self-assessment. Furthermore, in the context of Dutch primary care, wherein general practitioners often operate within smaller organizations, limited resources may impede their ability to evaluate complex AI models effectively. 91 Therefore, comprehensive tools for assessing the availability of evidence on predictive ML algorithms, such as the Dutch AIPA guideline and the practical evaluation tool we developed, are valuable to primary care professionals and may aid large-scale adoption of predictive ML algorithms in practice. Since primary care worldwide is under substantial pressure, from a health systems perspective, it is essential to remove barriers to implementing innovation such as predictive ML algorithms.

This study has methodological limitations that should be taken into account. First, the scope of the systematic review excluded regression-based predictive models and simple rule-based systems. Although these approaches can be of substantial value in primary care, the focus on predictive ML algorithms enabled us to provide an in-depth overview of the aspects of model complexity and interpretability. Although most accepted definitions of ML do not exclude simple regression, the scope was beneficial for maintaining a manageable overview of more complex models that pose unique challenges for standardized reporting of model development, validation, and implementation. Second, we restricted the systematic review to articles published in English. We believe this restriction does not substantially affect the generalizability of the results since previous research has found no evidence of systematic bias due to English-language limitations. 92 Third, we could not formally compare the predictive validity across predictive ML algorithms due to the substantial variations in the types of AI models and heterogeneous methods between studies. Additionally, the availability scores presented in this study should be seen as an approximation of the degree of public availability of evidence, in line with the objectives described . Fourth, the Dutch AIPA guideline is a local norm, which is not legally binding. Several international AI guidelines exist that apply to predictive ML algorithms, such as TRIPOD+AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) for development and validation, DECIDE-AI (Developmental and Exploratory Clinical Investigations of Decision Support Systems Driven by Artificial Intelligence) for feasibility studies, and SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) and CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) for impact assessments. 93 - 97 These international guidelines, however, are aimed primarily at researchers. We chose to build on the Dutch AIPA guideline because it provides complete, structured, and pragmatic quality assessment derived from existing guidelines across the entire AI life cycle, and it is specific for predictive algorithms, which are considered to be of great potential in the medical field. 15 It specifically emphasizes implementation aspects and practical applications in clinical practice and may therefore be useful for primary care professionals.

In this systematic review, we comprehensively identified the availability of evidence of predictive ML algorithms in primary care, using the Dutch AIPA guideline as a reference. We found a scarcity of evidence across the AI life cycle phases for implemented predictive ML algorithms, particularly from algorithms published in FDA-approved or CE-marked databases. Adopting guidelines such as the Dutch AIPA guideline can improve the availability of evidence regarding the predictive ML algorithms’ quality criteria. It could facilitate transparent and consistent reporting of the quality criteria in literature, potentially fostering trust among end users and facilitating large-scale implementation.

Accepted for Publication: July 10, 2024.

Published: September 12, 2024. doi:10.1001/jamanetworkopen.2024.32990

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Rakers MM et al. JAMA Network Open .

Corresponding Author: Margot M. Rakers, MD, Department of Public Health and Primary Care, Leiden University Medical Centre, 2333 ZA Leiden, the Netherlands ( [email protected] ).

Author Contributions: Dr van Os had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Rakers, van Buchem, Kant, van Smeden, Moons, Chavannes, van Os.

Acquisition, analysis, or interpretation of data: Rakers, van Buchem, Kucenko, de Hond, van Smeden, Moons, Leeuwenberg, Villalobos-Quesada, van Os.

Drafting of the manuscript: Rakers, van Buchem, Moons, Leeuwenberg, Villalobos-Quesada, van Os.

Critical review of the manuscript for important intellectual content: All authors.

Statistical analysis: van Buchem, Kucenko, van Smeden, Moons, van Os.

Obtained funding: van Os.

Administrative, technical, or material support: Rakers, Moons, Chavannes.

Supervision: de Hond, Kant, van Smeden, Moons, Chavannes, Villalobos-Quesada, van Os.

Conflict of Interest Disclosures: None reported.

Funding/Support: This work was supported by grant LSHM21009 from Innovative Medical Devices Initiative (Dr van Os).

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 2 .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts