healthcare articles that use quantitative research

• Research article
• Open access
• Published: 03 February 2021

A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines

• Ellesha A. Smith   ORCID: orcid.org/0000-0002-4241-7205 1 ,
• Nicola J. Cooper 1 ,
• Alex J. Sutton 1 ,
• Keith R. Abrams 1 &
• Stephanie J. Hubbard 1  

BMC Public Health volume  21 , Article number:  278 ( 2021 ) Cite this article

4655 Accesses

5 Citations

3 Altmetric

Metrics details

The complexity of public health interventions create challenges in evaluating their effectiveness. There have been huge advancements in quantitative evidence synthesis methods development (including meta-analysis) for dealing with heterogeneity of intervention effects, inappropriate ‘lumping’ of interventions, adjusting for different populations and outcomes and the inclusion of various study types. Growing awareness of the importance of using all available evidence has led to the publication of guidance documents for implementing methods to improve decision making by answering policy relevant questions.

The first part of this paper reviews the methods used to synthesise quantitative effectiveness evidence in public health guidelines by the National Institute for Health and Care Excellence (NICE) that had been published or updated since the previous review in 2012 until the 19th August 2019.The second part of this paper provides an update of the statistical methods and explains how they address issues related to evaluating effectiveness evidence of public health interventions.

The proportion of NICE public health guidelines that used a meta-analysis as part of the synthesis of effectiveness evidence has increased since the previous review in 2012 from 23% (9 out of 39) to 31% (14 out of 45). The proportion of NICE guidelines that synthesised the evidence using only a narrative review decreased from 74% (29 out of 39) to 60% (27 out of 45).An application in the prevention of accidents in children at home illustrated how the choice of synthesis methods can enable more informed decision making by defining and estimating the effectiveness of more distinct interventions, including combinations of intervention components, and identifying subgroups in which interventions are most effective.

Conclusions

Despite methodology development and the publication of guidance documents to address issues in public health intervention evaluation since the original review, NICE public health guidelines are not making full use of meta-analysis and other tools that would provide decision makers with fuller information with which to develop policy. There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making.

Peer Review reports

To make well-informed decisions and provide the best guidance in health care policy, it is essential to have a clear framework for synthesising good quality evidence on the effectiveness and cost-effectiveness of health interventions. There is a broad range of methods available for evidence synthesis. Narrative reviews provide a qualitative summary of the effectiveness of the interventions. Meta-analysis is a statistical method that pools evidence from multiple independent sources [ 1 ]. Meta-analysis and more complex variations of meta-analysis have been extensively applied in the appraisals of clinical interventions and treatments, such as drugs, as the interventions and populations are clearly defined and tested in randomised, controlled conditions. In comparison, public health studies are often more complex in design, making synthesis more challenging [ 2 ].

Many challenges are faced in the synthesis of public health interventions. There is often increased methodological heterogeneity due to the inclusion of different study designs. Interventions are often poorly described in the literature which may result in variation within the intervention groups. There can be a wide range of outcomes, whose definitions are not consistent across studies. Intermediate, or surrogate, outcomes are often used in studies evaluating public health interventions [ 3 ]. In addition to these challenges, public health interventions are often also complex meaning that they are made up of multiple, interacting components [ 4 ]. Recent guidance documents have focused on the synthesis of complex interventions [ 2 , 5 , 6 ]. The National Institute for Health and Care Excellence (NICE) guidance manual provides recommendations across all topics that are covered by NICE and there is currently no guidance that focuses specifically on the public health context.

Research questions

A methodological review of NICE public health intervention guidelines by Achana et al. (2014) found that meta-analysis methods were not being used [ 3 ]. The first part of this paper aims to update and compare, to the original review, the meta-analysis methods being used in evidence synthesis of public health intervention appraisals.

The second part of this paper aims to illustrate what methods are available to address the challenges of public health intervention evidence synthesis. Synthesis methods that go beyond a pairwise meta-analysis are illustrated through the application to a case study in public health and are discussed to understand how evidence synthesis methods can enable more informed decision making.

The third part of this paper presents software, guidance documents and web tools for methods that aim to make appropriate evidence synthesis of public health interventions more accessible. Recommendations for future research and guidance production that can improve the uptake of these methods in a public health context are discussed.

Update of NICE public health intervention guidelines review

Nice guidelines.

The National Institute for Health and Care Excellence (NICE) was established in 1999 as a health authority to provide guidance on new medical technologies to the NHS in England and Wales [ 7 ]. Using an evidence-based approach, it provides recommendations based on effectiveness and cost-effectiveness to ensure an open and transparent process of allocating NHS resources [ 8 ]. The remit for NICE guideline production was extended to public health in April 2005 and the first recommendations were published in March 2006. NICE published ‘Developing NICE guidelines: the manual’ in 2006, which has been updated since, with the most recent in 2018 [ 9 ]. It was intended to be a guidance document to aid in the production of NICE guidelines across all NICE topics. In terms of synthesising quantitative evidence, the NICE recommendations state: ‘meta-analysis may be appropriate if treatment estimates of the same outcome from more than 1 study are available’ and ‘when multiple competing options are being appraised, a network meta-analysis should be considered’. The implementation of network meta-analysis (NMA), which is described later, as a recommendation from NICE was introduced into the guidance document in 2014, with a further update in 2018.

Background to the previous review

The paper by Achana et al. (2014) explored the use of evidence synthesis methodology in NICE public health intervention guidelines published between 2006 and 2012 [ 3 ]. The authors conducted a systematic review of the methods used to synthesise quantitative effectiveness evidence within NICE public health guidelines. They found that only 23% of NICE public health guidelines used pairwise meta-analysis as part of the effectiveness review and the remainder used a narrative summary or no synthesis of evidence at all. The authors argued that despite significant advances in the methodology of evidence synthesis, the uptake of methods in public health intervention evaluation is lower than other fields, including clinical treatment evaluation. The paper concluded that more sophisticated methods in evidence synthesis should be considered to aid in decision making in the public health context [ 3 ].

The search strategy used in this paper was equivalent to that in the previous paper by Achana et al. (2014)[ 3 ]. The search was conducted through the NICE website ( https://www.nice.org.uk/guidance ) by searching the ‘Guidance and Advice List’ and filtering by ‘Public Health Guidelines’ [ 10 ]. The search criteria included all guidance documents that had been published from inception (March 2006) until the 19th August 2019. Since the original review, many of the guidelines had been updated with new documents or merged. Guidelines that remained unchanged since the previous review in 2012 were excluded and used for comparison.

The guidelines contained multiple documents that were assessed for relevance. A systematic review is a separate synthesis within a guideline that systematically collates all evidence on a specific research question of interest in the literature. Systematic reviews of quantitative effectiveness, cost-effectiveness evidence and decision modelling reports were all included as relevant. Qualitative reviews, field reports, expert opinions, surveillance reports, review decisions and other supporting documents were excluded at the search stage.

Within the reports, data was extracted on the types of review (narrative summary, pairwise meta-analysis, network meta-analysis (NMA), cost-effectiveness review or decision model), design of included primary studies (randomised controlled trials or non-randomised studies, intermediate or final outcomes, description of outcomes, outcome measure statistic), details of the synthesis methods used in the effectiveness evaluation (type of synthesis, fixed or random effects model, study quality assessment, publication bias assessment, presentation of results, software). Further details of the interventions were also recorded, including whether multiple interventions were lumped together for a pairwise comparison, whether interventions were complex (made up of multiple components) and details of the components. The reports were also assessed for potential use of complex intervention evidence synthesis methodology, meaning that the interventions that were evaluated in the review were made up of components that could potentially be synthesised using an NMA or a component NMA [ 11 ]. Where meta-analysis was not used to synthesis effectiveness evidence, the reasons for this was also recorded.

Search results and types of reviews

There were 67 NICE public health guidelines available on the NICE website. A summary flow diagram describing the literature identification process and the list of guidelines and their reference codes are provided in Additional files  1 and 2 . Since the previous review, 22 guidelines had not been updated. The results from the previous review were used for comparison to the 45 guidelines that were either newly published or updated.

The guidelines consisted of 508 documents that were assessed for relevance. Table  1 shows which types of relevant documents were available in each of the 45 guidelines. The median number of relevant articles per guideline was 3 (minimum = 0, maximum = 10). Two (4%) of the NICE public health guidelines did not report any type of systematic review, cost-effectiveness review or decision model (NG68, NG64) that met the inclusion criteria. 167 documents from 43 NICE public health guidelines were systematic reviews of quantitative effectiveness, cost-effectiveness or decision model reports and met the inclusion criteria.

Narrative reviews of effectiveness were implemented in 41 (91%) of the NICE PH guidelines. 14 (31%) contained a review that used meta-analysis to synthesise the evidence. Only one (1%) NICE guideline contained a review that implemented NMA to synthesise the effectiveness of multiple interventions; this was the same guideline that used NMA in the original review and had been updated. 33 (73%) guidelines contained cost-effectiveness reviews and 34 (76%) developed a decision model.

Comparison of review types to original review

Table  2 compares the results of the update to the original review and shows that the types of reviews and evidence synthesis methodologies remain largely unchanged since 2012. The proportion of guidelines that only contain narrative reviews to synthesise effectiveness or cost-effectiveness evidence has reduced from 74% to 60% and the proportion that included a meta-analysis has increased from 23% to 31%. The proportion of guidelines with reviews that only included evidence from randomised controlled trials and assessed the quality of individual studies remained similar to the original review.

Characteristics of guidelines using meta-analytic methods

Table  3 details the characteristics of the meta-analytic methods implemented in 24 reviews of the 14 guidelines that included one. All of the reviews reported an assessment of study quality, 12 (50%) reviews included only data from randomised controlled trials, 4 (17%) reviews used intermediate outcomes (e.g. uptake of chlamydia screening rather than prevention of chlamydia (PH3)), compared to the 20 (83%) reviews that used final outcomes (e.g. smoking cessation rather than uptake of a smoking cessation programme (NG92)). 2 (8%) reviews only used a fixed effect meta-analysis, 19 (79%) reviews used a random effects meta-analysis and 3 (13%) did not report which they had used.

An evaluation of the intervention information reported in the reviews concluded that 12 (50%) reviews had lumped multiple (more than two) different interventions into a control versus intervention pairwise meta-analysis. Eleven (46%) of the reviews evaluated interventions that are made up of multiple components (e.g. interventions for preventing obesity in PH47 were made up of diet, physical activity and behavioural change components).

21 (88%) of the reviews presented the results of the meta-analysis in the form of a forest plot and 22 (92%) presented the results in the text of the report. 20 (83%) of the reviews used two or more forms of presentation for the results. Only three (13%) reviews assessed publication bias. The most common software to perform meta-analysis was RevMan in 14 (58%) of the reviews.

Reasons for not using meta-analytic methods

The 143 reviews of effectiveness and cost effectiveness that did not use meta-analysis methods to synthesise the quantitative effectiveness evidence were searched for reasons behind this decision. 70 reports (49%) did not give a reason for not synthesising the data using a meta-analysis and 164 reasons were reported which are displayed in Fig.  1 . Out of the remaining reviews, multiple reasons for not using a meta-analysis were given. 53 (37%) of the reviews reported at least one reason due to heterogeneity. 30 (21%) decision model reports did not give a reason and these are categorised separately. 5 (3%) reviews reported that meta-analysis was not applicable or feasible, 1 (1%) reported that they were following NICE guidelines and 5 (3%) reported that there were a lack of studies.

Frequency and proportions of reasons reported for not using statistical methods in quantitative evidence synthesis in NICE PH intervention reviews

The frequency of reviews and guidelines that used meta-analytic methods were plotted against year of publication, which is reported in Fig.  2 . This showed that the number of reviews that used meta-analysis were approximately constant but there is some suggestion that the number of meta-analyses used per guideline increased, particularly in 2018.

Number of meta-analyses in NICE PH guidelines by year. Guidelines that were published before 2012 had been updated since the previous review by Achana et al. (2014) [ 3 ]

Comparison of meta-analysis characteristics to original review

Table  4 compares the characteristics of the meta-analyses used in the evidence synthesis of NICE public health intervention guidelines to the original review by Achana et al. (2014) [ 3 ]. Overall, the characteristics in the updated review have not much changed from those in the original. These changes demonstrate that the use of meta-analysis in NICE guidelines has increased but remains low. Lumping of interventions still appears to be common in 50% of reviews. The implications of this are discussed in the next section.

Application of evidence synthesis methodology in a public health intervention: motivating example

Since the original review, evidence synthesis methods have been developed and can address some of the challenges of synthesising quantitative effectiveness evidence of public health interventions. Despite this, the previous section shows that the uptake of these methods is still low in NICE public health guidelines - usually limited to a pairwise meta-analysis.

It has been shown in the results above and elsewhere [ 12 ] that heterogeneity is a common reason for not synthesising the quantitative effectiveness evidence available from systematic reviews in public health. Statistical heterogeneity is the variation in the intervention effects between the individual studies. Heterogeneity is problematic in evidence synthesis as it leads to uncertainty in the pooled effect estimates in a meta-analysis which can make it difficult to interpret the pooled results and draw conclusions. Rather than exploring the source of the heterogeneity, often in public health intervention appraisals a random effects model is fitted which assumes that the study intervention effects are not equivalent but come from a common distribution [ 13 , 14 ]. Alternatively, as demonstrated in the review update, heterogeneity is used as a reason to not undertake any quantitative evidence synthesis at all.

Since the size of the intervention effects and the methodological variation in the studies will affect the impact of the heterogeneity on a meta-analysis, it is inappropriate to base the methodological approach of a review on the degree of heterogeneity, especially within public health intervention appraisal where heterogeneity seems inevitable. Ioannidis et al. (2008) argued that there are ‘almost always’ quantitative synthesis options that may offer some useful insights in the presence of heterogeneity, as long as the reviewers interpret the findings with respect to their limitations [ 12 ].

In this section current evidence synthesis methods are applied to a motivating example in public health. This aims to demonstrate that methods beyond pairwise meta-analysis can provide appropriate and pragmatic information to public health decision makers to enable more informed decision making.

Figure  3 summarises the narrative of this part of the paper and illustrates the methods that are discussed. The red boxes represent the challenges in synthesising quantitative effectiveness evidence and refers to the section within the paper for more detail. The blue boxes represent the methods that can be applied to investigate each challenge.

Summary of challenges that are faces in the evidence synthesis of public health interventions and methods that are discussed to overcome these challenges

Evaluating the effect of interventions for promoting the safe storage of cleaning products to prevent childhood poisoning accidents

To illustrate the methodological developments, a motivating example is used from the five year, NIHR funded, Keeping Children Safe Programme [ 15 ]. The project included a Cochrane systematic review that aimed to increase the use of safety equipment to prevent accidents at home in children under five years old. This application is intended to be illustrative of the benefits of new evidence synthesis methods since the previous review. It is not a complete, comprehensive analysis as it only uses a subset of the original dataset and therefore the results are not intended to be used for policy decision making. This example has been chosen as it demonstrates many of the issues in synthesising effectiveness evidence of public health interventions, including different study designs (randomised controlled trials, observational studies and cluster randomised trials), heterogeneity of populations or settings, incomplete individual participant data and complex interventions that contain multiple components.

This analysis will investigate the most effective promotional interventions for the outcome of ‘safe storage of cleaning products’ to prevent childhood poisoning accidents. There are 12 studies included in the dataset, with IPD available from nine of the studies. The covariate, single parent family, is included in the analysis to demonstrate the effect of being a single parent family on the outcome. In this example, all of the interventions are made up of one or more of the following components: education (Ed), free or low cost equipment (Eq), home safety inspection (HSI), and installation of safety equipment (In). A Bayesian approach using WinBUGS was used and therefore credible intervals (CrI) are presented with estimates of the effect sizes [ 16 ].

The original review paper by Achana et al. (2014) demonstrated pairwise meta-analysis and meta-regression using individual and cluster allocated trials, subgroup analyses, meta-regression using individual participant data (IPD) and summary aggregate data and NMA. This paper firstly applies NMA to the motivating example for context, followed by extensions to NMA.

Multiple interventions: lumping or splitting?

Often in public health there are multiple intervention options. However, interventions are often lumped together in a pairwise meta-analysis. Pairwise meta-analysis is a useful tool for two interventions or, alternatively in the presence of lumping interventions, for answering the research question: ‘are interventions in general better than a control or another group of interventions?’. However, when there are multiple interventions, this type of analysis is not appropriate for informing health care providers which intervention should be recommended to the public. ‘Lumping’ is becoming less frequent in other areas of evidence synthesis, such as for clinical interventions, as the use of sophisticated synthesis techniques, such as NMA, increases (Achana et al. 2014) but lumping is still common in public health.

NMA is an extension of the pairwise meta-analysis framework to more than two interventions. Multiple interventions that are lumped into a pairwise meta-analysis are likely to demonstrate high statistical heterogeneity. This does not mean that quantitative synthesis could not be undertaken but that a more appropriate method, NMA, should be implemented. Instead the statistical approach should be based on the research questions of the systematic review. For example, if the research question is ‘are any interventions effective for preventing obesity?’, it would be appropriate to perform a pairwise meta-analysis comparing every intervention in the literature to a control. However, if the research question is ‘which intervention is the most effective for preventing obesity?’, it would be more appropriate and informative to perform a network meta-analysis, which can compare multiple interventions simultaneously and identify the best one.

NMA is a useful statistical method in the context of public health intervention appraisal, where there are often multiple intervention options, as it estimates the relative effectiveness of three or more interventions simultaneously, even if direct study evidence is not available for all intervention comparisons. Using NMA can help to answer the research question ‘what is the effectiveness of each intervention compared to all other interventions in the network?’.

In the motivating example there are six intervention options. The effect of lumping interventions is shown in Fig.  4 , where different interventions in both the intervention and control arms are compared. There is overlap of intervention and control arms across studies and interpretation of the results of a pairwise meta-analysis comparing the effectiveness of the two groups of interventions would not be useful in deciding which intervention to recommend. In comparison, the network plot in Fig.  5 illustrates the evidence base of the prevention of childhood poisonings review comparing six interventions that promote the use of safety equipment in the home. Most of the studies use ‘usual care’ as a baseline and compare this to another intervention. There are also studies in the evidence base that compare pairs of the interventions, such as ‘Education and equipment’ to ‘Equipment’. The plot also demonstrates the absence of direct study evidence between many pairs of interventions, for which the associated treatment effects can be indirectly estimated using NMA.

Network plot to illustrate how pairwise meta-analysis groups the interventions in the motivating dataset. Notation UC: Usual care, Ed: Education, Ed+Eq: Education and equipment, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq: Equipment

Network plot for the safe storage of cleaning products outcome. Notation UC: Usual care, Ed: Education, Ed+Eq: Education and equipment, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq: Equipment

An NMA was fitted to the motivating example to compare the six interventions in the studies from the review. The results are reported in the ‘triangle table’ in Table  5 [ 17 ]. The top right half of the table shows the direct evidence between pairs of the interventions in the corresponding rows and columns by either pooling the studies as a pairwise meta-analysis or presenting the single study results if evidence is only available from a single study. The bottom left half of the table reports the results of the NMA. The gaps in the top right half of the table arise where no direct study evidence exists to compare the two interventions. For example, there is no direct study evidence comparing ‘Education’ (Ed) to ‘Education, equipment and home safety inspection’ (Ed+Eq+HSI). The NMA, however, can estimate this comparison through the direct study evidence as an odds ratio of 3.80 with a 95% credible interval of (1.16, 12.44). The results suggest that the odds of safely storing cleaning products in the Ed+Eq+HSI intervention group is 3.80 times the odds in the Ed group. The results demonstrate a key benefit of NMA that all intervention effects in a network can be estimated using indirect evidence, even if there is no direct study evidence for some pairwise comparisons. This is based on the consistency assumption (that estimates of intervention effects from direct and indirect evidence are consistent) which should be checked when performing an NMA. This is beyond the scope of this paper and details on this can be found elsewhere [ 18 ].

NMA can also be used to rank the interventions in terms of their effectiveness and estimate the probability that each intervention is likely to be the most effective. This can help to answer the research question ‘which intervention is the best?’ out of all of the interventions that have provided evidence in the network. The rankings and associated probabilities for the motivating example are presented in Table  6 . It can be seen that in this case the ‘education, equipment and home safety inspection’ (Ed+Eq+HSI) intervention is ranked first, with a 0.87 probability of being the best intervention. However, there is overlap of the 95% credible intervals of the median rankings. This overlap reflects the uncertainty in the intervention effect estimates and therefore it is important that the interpretation of these statistics clearly communicates this uncertainty to decision makers.

NMA has the potential to be extremely useful but is underutilised in the evidence synthesis of public health interventions. The ability to compare and rank multiple interventions in an area where there are often multiple intervention options is invaluable in decision making for identifying which intervention to recommend. NMA can also include further literature in the analysis, compared to a pairwise meta-analysis, by expanding the network to improve the uncertainty in the effectiveness estimates.

Statistical heterogeneity

When heterogeneity remains in the results of an NMA, it is useful to explore the reasons for this. Strategies for dealing with heterogeneity involve the inclusion of covariates in a meta-analysis or NMA to adjust for the differences in the covariates across studies [ 19 ]. Meta-regression is a statistical method developed from meta-analysis that includes covariates to potentially explain the between-study heterogeneity ‘with the aim of estimating treatment-covariate interactions’ (Saramago et al. 2012). NMA has been extended to network meta-regression which investigates the effect of trial characteristics on multiple intervention effects. Three ways have been suggested to include covariates in an NMA: single covariate effect, exchangeable covariate effects and independent covariate effects which are discussed in more detail in the NICE Technical Support Document 3 [ 14 ]. This method has the potential to assess the effect of study level covariates on the intervention effects, which is particularly relevant in public health due to the variation across studies.

The most widespread method of meta-regression uses study level data for the inclusion of covariates into meta-regression models. Study level covariate data is when the data from the studies are aggregated, e.g. the proportion of participants in a study that are from single parent families compared to dual parent families. The alternative to study level data is individual participant data (IPD), where the data are available and used as a covariate at the individual level e.g. the parental status of every individual in a study can be used as a covariate. Although IPD is considered to be the gold standard for meta-analysis, aggregated level data is much more commonly used as it is usually available and easily accessible from published research whereas IPD can be hard to obtain from study authors.

There are some limitations to network meta-regression. In our motivating example, using the single parent covariate in a meta-regression would estimate the relative difference in the intervention effects of a population that is made up of 100% single parent families compared to a population that is made up of 100% dual parent families. This interpretation is not as useful as the analysis that uses IPD, which would give the relative difference of the intervention effects in a single parent family compared to a dual parent family. The meta-regression using aggregated data would also be susceptible to ecological bias. Ecological bias is where the effect of the covariate is different at the study level compared to the individual level [ 14 ]. For example, if each study demonstrates a relationship between a covariate and the intervention but the covariate is similar across the studies, a meta-regression of the aggregate data would not demonstrate the effect that is observed within the studies [ 20 ].

Although meta-regression is a useful tool for investigating sources of heterogeneity in the data, caution should be taken when using the results of meta-regression to explain how covariates affect the intervention effects. Meta-regression should only be used to investigate study characteristics, such as the duration of intervention, which will not be susceptible to ecological bias and the interpretation of the results (the effect of intervention duration on intervention effectiveness) would be more meaningful for the development of public health interventions.

Since the covariate of interest in this motivating example is not a study characteristic, meta-regression of aggregated covariate data was not performed. Network meta-regression including IPD and aggregate level data was developed by Samarago et al. (2012) [ 21 ] to overcome the issues with aggregated data network meta-regression, which is discussed in the next section.

Tailored decision making to specific sub-groups

In public health it is important to identify which interventions are best for which people. There has been a recent move towards precision medicine. In the field of public health the ‘concept of precision prevention may [...] be valuable for efficiently targeting preventive strategies to the specific subsets of a population that will derive maximal benefit’ (Khoury and Evans, 2015). Tailoring interventions has the potential to reduce the effect of inequalities in social factors that are influencing the health of the population. Identifying which interventions should be targeted to which subgroups can also lead to better public health outcomes and help to allocate scarce NHS resources. Research interest, therefore, lies in identifying participant level covariate-intervention interactions.

IPD meta-analysis uses data at the individual level to overcome ecological bias. The interpretation of IPD meta-analysis is more relevant in the case of using participant characteristics as covariates since the interpretation of the covariate-intervention interaction is at the individual level rather than the study level. This means that it can answer the research question: ‘which interventions work best in subgroups of the population?’. IPD meta-analyses are considered to be the gold standard for evidence synthesis since it increases the power of the analysis to identify covariate-intervention interactions and it has the ability to reduce the effect of ecological bias compared to aggregated data alone. IPD meta-analysis can also help to overcome scarcity of data issues and has been shown to have higher power and reduce the uncertainty in the estimates compared to analysis including only summary aggregate data [ 22 ].

Despite the advantages of including IPD in a meta-analysis, in reality it is often very time consuming and difficult to collect IPD for all of the studies [ 21 ]. Although data sharing is becoming more common, it remains time consuming and difficult to collect IPD for all studies in a review. This results in IPD being underutilised in meta-analyses. As an intermediate solution, statistical methods have been developed, such as the NMA in Samarago et al. (2012), that incorporates both IPD and aggregate data. Methods that simultaneously include IPD and aggregate level data have been shown to reduce uncertainty in the effect estimates and minimise ecological bias [ 20 , 21 ]. A simulation study by Leahy et al. (2018) found that an increased proportion of IPD resulted in more accurate and precise NMA estimates [ 23 ].

An NMA including IPD, where it is available, was performed, based on the model presented in Samarago et al. (2012) [ 21 ]. The results in Table  7 demonstrates the detail that this type of analysis can provide to base decisions on. More relevant covariate-intervention interaction interpretations can be obtained, for example the regression coefficients for covariate-intervention interactions are the individual level covariate intervention interactions or the ‘within study interactions’ that are interpreted as the effect of being in a single parent family on the effectiveness of each of the interventions. For example, the effect of Ed+Eq compared to UC in a single parent family is 1.66 times the effect of Ed+Eq compared to UC in a dual parent family but this is not an important difference as the credible interval crosses 1. The regression coefficients for the study level covariate-intervention interactions or the ‘between study interactions’ can be interpreted as the relative difference in the intervention effects of a population that is made up of 100% single parent families compared to a population that is made up of 100% dual parent families.

• Complex interventions

In many public health research settings the complex interventions are comprised of a number of components. An NMA can compare all of the interventions in a network as they are implemented in the original trials. However, NMA does not tell us which components of the complex intervention are attributable to this effect. It could be that particular components, or the interacting effect of multiple components, are driving the effectiveness and other components are not as effective. Often, trials have not directly compared every combination of components as there are so many component combination options, it would be inefficient and impractical. Component NMA was developed by Welton et al. (2009) to estimate the effect of each component of the complex interventions and combination of components in a network, in the absence of direct trial evidence and answers the question: ‘are interventions with a particular component or combination of components effective?’ [ 11 ]. For example, for the motivating example, in comparison to Fig.  5 , which demonstrates the interventions that an NMA can estimate effectiveness, Fig.  6 demonstrates all of the possible interventions of which the effectiveness can be estimated in a component NMA, given the components present in the network.

Network plot that illustrates how component network meta-analysis can estimate the effectiveness of intervention components and combinations of components, even when they are not included in the direct evidence. Notation UC: Usual care, Ed: Education, Eq: Equipment, Installation, Ed+Eq: Education and equipment, Ed+HSI: Education and home safety inspection, Ed+In: Education and installation, Eq+HSI: Equipment and home safety inspection, Eq+In: equipment and installation, HSI+In: Home safety inspection and installation, Ed+Eq+HSI: Education, equipment, and home safety inspection, Ed+Eq+In: Education, equipment and installation, Eq+HSI+In: Equipment, home safety inspection and installation, Ed+Eq+HSI+In: Education, equipment, home safety inspection and installation

The results of the analyses of the main effects, two way effects and full effects models are shown in Table  8 . The models, proposed in the original paper by Welton et al. (2009), increase in complexity as the assumptions regarding the component effects relax [ 24 ]. The main effects component NMA assumes that the components in the interventions each have separate, independent effects and intervention effects are the sum of the component effects. The two-way effects models assumes that there are interactions between pairs of the components, so the effects of the interventions are more than the sum of the effects. The full effects model assumes that all of the components and combinations of the components interact. Component NMA did not provide further insight into which components are likely to be the most effective since all of the 95% credible intervals were very wide and overlapped 1. There is a lot of uncertainty in the results, particularly in the 2-way and full effects models. A limitation of component NMA is that there are issues with uncertainty when data is scarce. However, the results demonstrate the potential of component NMA as a useful tool to gain better insights from the available dataset.

In practice, this method has rarely been used since its development [ 24 – 26 ]. It may be challenging to define the components in some areas of public health where many interventions have been studied. However, the use of meta-analysis for planning future studies is rarely discussed and component NMA would provide a useful tool for identifying new component combinations that may be more effective [ 27 ]. This type of analysis has the potential to prioritise future public health research, which is especially useful where there are multiple intervention options, and identify more effective interventions to recommend to the public.

Further methods / other outcomes

The analysis and methods described in this paper only cover a small subset of the methods that have been developed in meta-analysis in recent years. Methods that aim to assess the quality of evidence supporting a NMA and how to quantify how much the evidence could change due to potential biases or sampling variation before the recommendation changes have been developed [ 28 , 29 ]. Models adjusting for baseline risk have been developed to allow for different study populations to have different levels of underlying risk, by using the observed event rate in the control arm [ 30 , 31 ]. Multivariate methods can be used to compare the effect of multiple interventions on two or more outcomes simultaneously [ 32 ]. This area of methodological development is especially appealing within public health where studies assess a broad range of health effects and typically have multiple outcome measures. Multivariate methods offer benefits over univariate models by allowing the borrowing of information across outcomes and modelling the relationships between outcomes which can potentially reduce the uncertainty in the effect estimates [ 33 ]. Methods have also been developed to evaluate interventions with classes or different intervention intensities, known as hierarchical interventions [ 34 ]. These methods were not demonstrated in this paper but can also be useful tools for addressing challenges of appraising public health interventions, such as multiple and surrogate outcomes.

This paper only considered an example with a binary outcome. All of the methods described have also been adapted for other outcome measures. For example, the Technical Support Document 2 proposed a Bayesian generalised linear modelling framework to synthesise other outcome measures. More information and models for continuous and time-to-event data is available elsewhere [ 21 , 35 – 38 ].

Software and guidelines

In the previous section, meta-analytic methods that answer more policy relevant questions were demonstrated. However, as shown by the update to the review, methods such as these are still under-utilised. It is suspected from the NICE public health review that one of the reasons for the lack of uptake of methods in public health could be due to common software choices, such as RevMan, being limited in their flexibility for statistical methods.

Table  9 provides a list of software options and guidance documents that are more flexible than RevMan for implementing the statistical methods illustrated in the previous section to make these methods more accessible to researchers.

In this paper, the network plot in Figs.  5 and 6 were produced using the networkplot command from the mvmeta package [ 39 ] in Stata [ 61 ]. WinBUGS was used to fit the NMA in this paper by adapting the code in the book ‘Evidence Synthesis for Decision Making in Healthcare’ which also provides more detail on Bayesian methods and assessing convergence of Bayesian models [ 45 ]. The model for including IPD and summary aggregate data in an NMA was based on the code in the paper by Saramago et al. (2012). The component NMA in this paper was performed in WinBUGS through R2WinBUGS, [ 47 ] using the code in Welton et al. (2009) [ 11 ].

WinBUGS is a flexible tool for fitting complex models in a Bayesian framework. The NICE Decision Support Unit produced a series of Evidence Synthesis Technical Support Documents [ 46 ] that provide a comprehensive technical guide to methods for evidence synthesis and WinBUGS code is also provided for many of the models. Complex models can also be performed in a frequentist framework. Code and commands for many models are available in R and STATA (see Table  9 ).

The software, R2WinBUGS, was used in the analysis of the motivating example. Increasing numbers of researchers are using R and so packages that can be used to link the two softwares by calling BUGS models in R, packages such as R2WinBUGS, can improve the accessibility of Bayesian methods [ 47 ]. The new R package, BUGSnet, may also help to facilitate the accessibility and improve the reporting of Bayesian NMA [ 48 ]. Webtools have also been developed as a means of enabling researchers to undertake increasingly complex analyses [ 52 , 53 ]. Webtools provide a user-friendly interface to perform statistical analyses and often help in the reporting of the analyses by producing plots, including network plots and forest plots. These tools are very useful for researchers that have a good understanding of the statistical methods they want to implement as part of their review but are inexperienced in statistical software.

This paper has reviewed NICE public health intervention guidelines to identify the methods that are currently being used to synthesise effectiveness evidence to inform public health decision making. A previous review from 2012 was updated to see how method utilisation has changed. Methods have been developed since the previous review and these were applied to an example dataset to show how methods can answer more policy relevant questions. Resources and guidelines for implementing these methods were signposted to encourage uptake.

The review found that the proportion of NICE guidelines containing effectiveness evidence summarised using meta-analysis methods has increased since the original review, but remains low. The majority of the reviews presented only narrative summaries of the evidence - a similar result to the original review. In recent years, there has been an increased awareness of the need to improve decision making by using all of the available evidence. As a result, this has led to the development of new methods, easier application in standard statistical software packages, and guidance documents. Based on this, it would have been expected that their implementation would rise in recent years to reflect this, but the results of the review update showed no such increasing pattern.

A high proportion of NICE guideline reports did not provide a reason for not applying quantitative evidence synthesis methods. Possible explanations for this could be time or resource constraints, lack of statistical expertise, being unaware of the available methods or poor reporting. Reporting guidelines, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), should be updated to emphasise the importance of documenting reasons for not applying methods, as this can direct future research to improve uptake.

Where it was specified, the most common reported reason for not conducting a meta-analysis was heterogeneity. Often in public health, the data is heterogeneous due to the differences between studies in population, design, interventions or outcomes. A common misconception is that the presence of heterogeneity implies that it is not possible to pool the data. Meta-analytic methods can be used to investigate the sources of heterogeneity, as demonstrated in the NMA of the motivating example, and the use of IPD is recommended where possible to improve the precision of the results and reduce the effect of ecological bias. Although caution should be exercised in the interpretation of the results, quantitative synthesis methods provide a stronger basis for making decisions than narrative accounts because they explicitly quantify the heterogeneity and seek to explain it where possible.

The review also found that the most common software to perform the synthesis was RevMan. RevMan is very limited in its ability to perform advanced statistical analyses, beyond that of pairwise meta-analysis, which might explain the above findings. Standard software code is being developed to help make statistical methodology and application more accessible and guidance documents are becoming increasingly available.

The evaluation of public health interventions can be problematic due to the number and complexity of the interventions. NMA methods were applied to a real Cochrane public health review dataset. The methods that were demonstrated showed ways to address some of these issues, including the use of NMA for multiple interventions, the inclusion of covariates as both aggregated data and IPD to explain heterogeneity, and the extension to component network meta-analysis for guiding future research. These analyses illustrated how the choice of synthesis methods can enable more informed decision making by allowing more distinct interventions, and combinations of intervention components, to be defined and their effectiveness estimated. It also demonstrated the potential to target interventions to population subgroups where they are likely to be most effective. However, the application of component NMA to the motivating example has also demonstrated the issues around uncertainty if there are a limited number of studies observing the interventions and intervention components.

The application of methods to the motivating example demonstrated a key benefit of using statistical methods in a public health context compared to only presenting a narrative review – the methods provide a quantitative estimate of the effectiveness of the interventions. The uncertainty from the credible intervals can be used to demonstrate the lack of available evidence. In the context of decision making, having pooled estimates makes it much easier for decision makers to assess the effectiveness of the interventions or identify when more research is required. The posterior distribution of the pooled results from the evidence synthesis can also be incorporated into a comprehensive decision analytic model to determine cost-effectiveness [ 62 ]. Although narrative reviews are useful for describing the evidence base, the results are very difficult to summarise in a decision context.

Although heterogeneity seems to be inevitable within public health interventions due to their complex nature, this review has shown that it is still the main reported reason for not using statistical methods in evidence synthesis. This may be due to guidelines that were originally developed for clinical treatments that are tested in randomised conditions still being applied in public health settings. Guidelines for the choice of methods used in public health intervention appraisals could be updated to take into account the complexities and wide ranging areas in public health. Sophisticated methods may be more appropriate in some cases than simpler models for modelling multiple, complex interventions and their uncertainty, given the limitations are also fully reported [ 19 ]. Synthesis may not be appropriate if statistical heterogeneity remains after adjustment for possible explanatory covariates but details of exploratory analysis and reasons for not synthesising the data should be reported. Future research should focus on the application and dissemination of the advantages of using more advanced methods in public health, identifying circumstances where these methods are likely to be the most beneficial, and ways to make the methods more accessible, for example, the development of packages and web tools.

There is an evident need to facilitate the translation of the synthesis methods into a public health context and encourage the use of methods to improve decision making. This review has shown that the uptake of statistical methods for evaluating the effectiveness of public health interventions is slow, despite advances in methods that address specific issues in public health intervention appraisal and the publication of guidance documents to complement their application.

Availability of data and materials

The dataset supporting the conclusions of this article is included within the article.

Abbreviations

National institute for health and care excellence

• Network meta-analysis

Individual participant data

Home safety inspection

Installation

Credible interval

Preferred reporting items for systematic reviews and meta-analyses

Dias S, Welton NJ, Sutton AJ, Ades A. NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials: National Institute for Health and Clinical Excellence; 2011, p. 98. (Technical Support Document in Evidence Synthesis; TSD2).

Higgins JPT, López-López JA, Becker BJ, et al.Synthesising quantitative evidence in systematic reviews of complex health interventions. BMJ Global Health. 2019; 4(Suppl 1):e000858. https://doi.org/10.1136/bmjgh-2018-000858 .

Article   PubMed   PubMed Central   Google Scholar  

Achana F, Hubbard S, Sutton A, Kendrick D, Cooper N. An exploration of synthesis methods in public health evaluations of interventions concludes that the use of modern statistical methods would be beneficial. J Clin Epidemiol. 2014; 67(4):376–90.

Article   PubMed   Google Scholar  

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new medical research council guidance. Int J Nurs Stud. 2013; 50(5):587–92.

Caldwell DM, Welton NJ. Approaches for synthesising complex mental health interventions in meta-analysis. Evidence-Based Mental Health. 2016; 19(1):16–21.

Melendez-Torres G, Bonell C, Thomas J. Emergent approaches to the meta-analysis of multiple heterogeneous complex interventions. BMC Med Res Methodol. 2015; 15(1):47.

Article   CAS   PubMed   PubMed Central   Google Scholar  

NICE. NICE: Who We Are. https://www.nice.org.uk/about/who-we-are . Accessed 19 Sept 2019.

Kelly M, Morgan A, Ellis S, Younger T, Huntley J, Swann C. Evidence based public health: a review of the experience of the national institute of health and clinical excellence (NICE) of developing public health guidance in England. Soc Sci Med. 2010; 71(6):1056–62.

NICE. Developing NICE Guidelines: The Manual. https://www.nice.org.uk/process/pmg20/chapter/introduction-and-overview . Accessed 19 Sept 2019.

NICE. Public Health Guidance. https://www.nice.org.uk/guidance/published?type=ph . Accessed 19 Sept 2019.

Welton NJ, Caldwell D, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: psychological interventions in coronary heart disease. Am J Epidemiol. 2009; 169(9):1158–65.

Ioannidis JP, Patsopoulos NA, Rothstein HR. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008; 336(7658):1413–5.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002; 21(11):1539–58.

Article   Google Scholar  

Dias S, Sutton A, Welton N, Ades A. NICE DSU Technical Support Document 3: Heterogeneity: Subgroups, Meta-Regression, Bias and Bias-Adjustment: National Institute for Health and Clinical Excellence; 2011, p. 76.

Kendrick D, Ablewhite J, Achana F, et al.Keeping Children Safe: a multicentre programme of research to increase the evidence base for preventing unintentional injuries in the home in the under-fives. Southampton: NIHR Journals Library; 2017.

Google Scholar  

Lunn DJ, Thomas A, Best N, et al.WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility. Stat Comput. 2000; 10:325–37. https://doi.org/10.1023/A:1008929526011 .

Dias S, Caldwell DM. Network meta-analysis explained. Arch Dis Child Fetal Neonatal Ed. 2019; 104(1):8–12. https://doi.org/10.1136/archdischild-2018-315224. http://arxiv.org/abs/https://fn.bmj.com/content/104/1/F8.full.pdf.

Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades A. NICE DSU Technical Support Document 4: Inconsistency in Networks of Evidence Based on Randomised Controlled Trials: National Institute for Health and Clinical Excellence; 2011. (NICE DSU Technical Support Document in Evidence Synthesis; TSD4).

Cipriani A, Higgins JP, Geddes JR, Salanti G. Conceptual and technical challenges in network meta-analysis. Ann Intern Med. 2013; 159(2):130–7.

Riley RD, Steyerberg EW. Meta-analysis of a binary outcome using individual participant data and aggregate data. Res Synth Methods. 2010; 1(1):2–19.

Saramago P, Sutton AJ, Cooper NJ, Manca A. Mixed treatment comparisons using aggregate and individual participant level data. Stat Med. 2012; 31(28):3516–36.

Lambert PC, Sutton AJ, Abrams KR, Jones DR. A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol. 2002; 55(1):86–94.

Article   CAS   PubMed   Google Scholar  

Leahy J, O’Leary A, Afdhal N, Gray E, Milligan S, Wehmeyer MH, Walsh C. The impact of individual patient data in a network meta-analysis: an investigation into parameter estimation and model selection. Res Synth Methods. 2018; 9(3):441–69.

Freeman SC, Scott NW, Powell R, Johnston M, Sutton AJ, Cooper NJ. Component network meta-analysis identifies the most effective components of psychological preparation for adults undergoing surgery under general anesthesia. J Clin Epidemiol. 2018; 98:105–16.

Pompoli A, Furukawa TA, Efthimiou O, Imai H, Tajika A, Salanti G. Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis. Psychol Med. 2018; 48(12):1945–53.

Rücker G, Schmitz S, Schwarzer G. Component network meta-analysis compared to a matching method in a disconnected network: A case study. Biom J. 2020. https://doi.org/10.1002/bimj.201900339 .

Efthimiou O, Debray TP, van Valkenhoef G, Trelle S, Panayidou K, Moons KG, Reitsma JB, Shang A, Salanti G, Group GMR. GetReal in network meta-analysis: a review of the methodology. Res Synth Methods. 2016; 7(3):236–63.

Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS ONE. 2014; 9(7):99682.

Article   CAS   Google Scholar  

Phillippo DM, Dias S, Welton NJ, Caldwell DM, Taske N, Ades A. Threshold analysis as an alternative to grade for assessing confidence in guideline recommendations based on network meta-analyses. Ann Intern Med. 2019; 170(8):538–46.

Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 5: Evidence Synthesis in the Baseline Natural History Model: National Institute for Health and Clinical Excellence; 2011, p. 29. (NICE DSU Technical Support Document in Evidence Synthesis; TSD5).

Achana FA, Cooper NJ, Dias S, Lu G, Rice SJ, Kendrick D, Sutton AJ. Extending methods for investigating the relationship between treatment effect and baseline risk from pairwise meta-analysis to network meta-analysis. Stat Med. 2013; 32(5):752–71.

Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, White IR. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ (Clinical research ed.) 2017; 358:j3932. https://doi.org/10.1136/bmj.j3932 .

Achana FA, Cooper NJ, Bujkiewicz S, Hubbard SJ, Kendrick D, Jones DR, Sutton AJ. Network meta-analysis of multiple outcome measures accounting for borrowing of information across outcomes. BMC Med Res Methodol. 2014; 14(1):92.

Owen RK, Tincello DG, Keith RA. Network meta-analysis: development of a three-level hierarchical modeling approach incorporating dose-related constraints. Value Health. 2015; 18(1):116–26.

Jansen JP. Network meta-analysis of individual and aggregate level data. Res Synth Methods. 2012; 3(2):177–90.

Donegan S, Williamson P, D’Alessandro U, Garner P, Smith CT. Combining individual patient data and aggregate data in mixed treatment comparison meta-analysis: individual patient data may be beneficial if only for a subset of trials. Stat Med. 2013; 32(6):914–30.

Saramago P, Chuang L-H, Soares MO. Network meta-analysis of (individual patient) time to event data alongside (aggregate) count data. BMC Med Res Methodol. 2014; 14(1):105.

Thom HH, Capkun G, Cerulli A, Nixon RM, Howard LS. Network meta-analysis combining individual patient and aggregate data from a mixture of study designs with an application to pulmonary arterial hypertension. BMC Med Res Methodol. 2015; 15(1):34.

Gasparrini A, Armstrong B, Kenward MG. Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat Med. 2012; 31(29):3821–39.

Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in stata. PLoS ONE. 2013; 8(10):76654.

Rücker G, Schwarzer G, Krahn U, König J. netmeta: Network meta-analysis with R. R package version 0.5-0. 2014. R package version 0.5-0. Availiable: http://CRAN.R-project.org/package=netmeta .

van Valkenhoef G, Kuiper J. gemtc: Network Meta-Analysis Using Bayesian Methods. R package version 0.8-2. 2016. Available online at: https://CRAN.R-project.org/package=gemtc .

Lin L, Zhang J, Hodges JS, Chu H. Performing arm-based network meta-analysis in R with the pcnetmeta package. J Stat Softw. 2017; 80(5):1–25. https://doi.org/10.18637/jss.v080.i05 .

Rücker G, Schwarzer G. Automated drawing of network plots in network meta-analysis. Res Synth Methods. 2016; 7(1):94–107.

Welton NJ, Sutton AJ, Cooper N, Abrams KR, Ades A. Evidence Synthesis for Decision Making in Healthcare, vol. 132. UK: Wiley; 2012.

Book   Google Scholar  

Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence synthesis for decision making 1: introduction. Med Decis Making Int J Soc Med Decis Making. 2013; 33(5):597–606. https://doi.org/10.1177/0272989X13487604 .

Sturtz S, Ligges U, Gelman A. R2WinBUGS: a package for running WinBUGS from R. J Stat Softw. 2005; 12(3):1–16.

Béliveau A, Boyne DJ, Slater J, Brenner D, Arora P. Bugsnet: an r package to facilitate the conduct and reporting of bayesian network meta-analyses. BMC Med Res Methodol. 2019; 19(1):196.

Neupane B, Richer D, Bonner AJ, Kibret T, Beyene J. Network meta-analysis using R: a review of currently available automated packages. PLoS ONE. 2014; 9(12):115065.

White IR. Multivariate random-effects meta-analysis. Stata J. 2009; 9(1):40–56.

Chaimani A, Salanti G. Visualizing assumptions and results in network meta-analysis: the network graphs package. Stata J. 2015; 15(4):905–50.

Owen RK, Bradbury N, Xin Y, Cooper N, Sutton A. MetaInsight: An interactive web-based tool for analyzing, interrogating, and visualizing network meta-analyses using R-shiny and netmeta. Res Synth Methods. 2019; 10(4):569–81. https://doi.org/10.1002/jrsm.1373 .

Freeman SC, Kerby CR, Patel A, Cooper NJ, Quinn T, Sutton AJ. Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Med Res Methodol. 2019; 19(1):81.

Nikolakopoulou A, Higgins JPT, Papakonstantinou T, Chaimani A, Del Giovane C, Egger M, Salanti G. CINeMA: An approach for assessing confidence in the results of a network meta-analysis. PLoS Med. 2020; 17(4):e1003082. https://doi.org/10.1371/journal.pmed.1003082 .

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010; 36(3):1–48.

Freeman SC, Carpenter JR. Bayesian one-step ipd network meta-analysis of time-to-event data using royston-parmar models. Res Synth Methods. 2017; 8(4):451–64.

Riley RD, Lambert PC, Staessen JA, Wang J, Gueyffier F, Thijs L, Boutitie F. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Stat Med. 2008; 27(11):1870–93.

Debray TP, Moons KG, van Valkenhoef G, Efthimiou O, Hummel N, Groenwold RH, Reitsma JB, Group GMR. Get real in individual participant data (ipd) meta-analysis: a review of the methodology. Res Synth Methods. 2015; 6(4):293–309.

Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, Rovers M. Individual Participant Data (IPD) Meta-analyses of Randomised Controlled Trials: Guidance on Their Use. PLoS Med. 2015; 12(7):e1001855. https://doi.org/10.1371/journal.pmed.1001855 .

Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the prisma-ipd statement. JAMA. 2015; 313(16):1657–65.

StataCorp. Stata Statistical Software: Release 16. College Station: StataCorp LLC; 2019.

Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A. Comprehensive decision analytical modelling in economic evaluation: a bayesian approach. Health Econ. 2004; 13(3):203–26.

Download references

Acknowledgements

We would like to acknowledge Professor Denise Kendrick as the lead on the NIHR Keeping Children Safe at Home Programme that originally funded the collection of the evidence for the motivating example and some of the analyses illustrated in the paper.

ES is funded by a National Institute for Health Research (NIHR), Doctoral Research Fellow for this research project. This paper presents independent research funded by the National Institute for Health Research (NIHR). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Author information

Authors and affiliations.

Department of Health Sciences, University of Leicester, Lancaster Road, Leicester, UK

Ellesha A. Smith, Nicola J. Cooper, Alex J. Sutton, Keith R. Abrams & Stephanie J. Hubbard

You can also search for this author in PubMed   Google Scholar

Contributions

ES performed the review, analysed the data and wrote the paper. SH supervised the project. SH, KA, NC and AS provided substantial feedback on the manuscript. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Ellesha A. Smith .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

KA is supported by Health Data Research (HDR) UK, the UK National Institute for Health Research (NIHR) Applied Research Collaboration East Midlands (ARC EM), and as a NIHR Senior Investigator Emeritus (NF-SI-0512-10159). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. KA has served as a paid consultant, providing unrelated methodological advice, to; Abbvie, Amaris, Allergan, Astellas, AstraZeneca, Boehringer Ingelheim, Bristol-Meyers Squibb, Creativ-Ceutical, GSK, ICON/Oxford Outcomes, Ipsen, Janssen, Eli Lilly, Merck, NICE, Novartis, NovoNordisk, Pfizer, PRMA, Roche and Takeda, and has received research funding from Association of the British Pharmaceutical Industry (ABPI), European Federation of Pharmaceutical Industries & Associations (EFPIA), Pfizer, Sanofi and Swiss Precision Diagnostics. He is a Partner and Director of Visible Analytics Limited, a healthcare consultancy company.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Key for the Nice public health guideline codes. Available in NICEGuidelinesKey.xlsx .

Additional file 2

NICE public health intervention guideline review flowchart for the inclusion and exclusion of documents. Available in Flowchart.JPG .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Smith, E.A., Cooper, N.J., Sutton, A.J. et al. A review of the quantitative effectiveness evidence synthesis methods used in public health intervention guidelines. BMC Public Health 21 , 278 (2021). https://doi.org/10.1186/s12889-021-10162-8

Download citation

Received : 22 September 2020

Accepted : 04 January 2021

Published : 03 February 2021

DOI : https://doi.org/10.1186/s12889-021-10162-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Meta-analysis
• Systematic review
• Public health
• Decision making
• Evidence synthesis

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

• Systematic review
• Open access
• Published: 19 June 2020

Quantitative measures of health policy implementation determinants and outcomes: a systematic review

• Peg Allen   ORCID: orcid.org/0000-0001-7000-796X 1 ,
• Meagan Pilar 1 ,
• Callie Walsh-Bailey 1 ,
• Cole Hooley 2 ,
• Stephanie Mazzucca 1 ,
• Cara C. Lewis 3 ,
• Kayne D. Mettert 3 ,
• Caitlin N. Dorsey 3 ,
• Jonathan Purtle 4 ,
• Maura M. Kepper 1 ,
• Ana A. Baumann 5 &
• Ross C. Brownson 1 , 6  

Implementation Science volume  15 , Article number:  47 ( 2020 ) Cite this article

20k Accesses

64 Citations

31 Altmetric

Metrics details

Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures.

Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures.

Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported.

Conclusions

Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health.

Registration

Not registered

Peer Review reports

Contributions to the literature

This systematic review identified 70 quantitative measures of implementation outcomes or determinants in health policy studies.

Readiness to implement and organizational climate and culture were commonly assessed determinants, but fewer studies assessed policy actor relationships or implementation outcomes of acceptability, fidelity/compliance, appropriateness, feasibility, or implementation costs.

Study team members rated most identified measures’ pragmatic properties as good, meaning they are straightforward to use, but few studies documented pilot or psychometric testing of measures.

Further development and dissemination of valid and reliable measures of policy implementation outcomes and determinants can facilitate identification, use, and spread of effective policy implementation strategies.

Despite major impacts of policy on population health [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ], there have been relatively few policy studies in dissemination and implementation (D&I) science to inform implementation strategies and evaluate implementation efforts [ 8 ]. While health outcomes of policies are commonly studied, fewer policy studies assess implementation processes and outcomes. Of 146 D&I studies funded by the National Institutes of Health (NIH) through D&I funding announcements from 2007 to 2014, 12 (8.2%) were policy studies that assessed policy content, policy development processes, or health outcomes of policies, representing 10.5% of NIH D&I funding [ 8 ]. Eight of the 12 studies (66.7%) assessed health outcomes, while only five (41.6%) assessed implementation [ 8 ].

Our ability to explore the differential impact of policy implementation determinants and outcomes and disentangle these from health benefits and other societal outcomes requires high quality quantitative measures [ 9 ]. While systematic reviews of measures of implementation of evidence-based interventions (in clinical and community settings) have been conducted in recent years [ 10 , 11 , 12 , 13 ], to our knowledge, no reviews have explored the quality of quantitative measures of determinants and outcomes of policy implementation.

Policy implementation research in political science and the social sciences has been active since at least the 1970s and has much to contribute to the newer field of D&I research [ 1 , 14 ]. Historically, theoretical frameworks and policy research largely emphasized policy development or analysis of the content of policy documents themselves [ 15 ]. For example, Kingdon’s Multiple Streams Framework and its expansions have been widely used in political science and the social sciences more broadly to describe how factors related to sociopolitical climate, attributes of a proposed policy, and policy actors (e.g., organizations, sectors, individuals) contribute to policy change [ 16 , 17 , 18 ]. Policy frameworks can also inform implementation planning and evaluation in D&I research. Although authors have named policy stages since the 1950s [ 19 , 20 ], Sabatier and Mazmanian’s Policy Implementation Process Framework was one of the first such frameworks that gained widespread use in policy implementation research [ 21 ] and later in health promotion [ 22 ]. Yet, available implementation frameworks are not often used to guide implementation strategies or inform why a policy worked in one setting but not another [ 23 ]. Without explicit focus on implementation, the intended benefits of health policies may go unrealized, and the ability may be lost to move the field forward to understand policy implementation (i.e., our collective knowledge building is dampened) [ 24 ].

Differences in perspectives and terminology between D&I and policy research in political science are noteworthy to interpret the present review. For example, Proctor et al. use the term implementation outcomes for what policy researchers call policy outputs [ 14 , 20 , 25 ]. To non-D&I policy researchers, policy implementation outcomes refer to the health outcomes in the target population [ 20 ]. D&I science uses the term fidelity [ 26 ]; policy researchers write about compliance [ 20 ]. While D&I science uses the terms outer setting, outer context, or external context to point to influences outside the implementing organization [ 26 , 27 , 28 ], non-D&I policy research refers to policy fields [ 24 ] which are networks of agencies that carry out policies and programs.

Identification of valid and reliable quantitative measures of health policy implementation processes is needed. These measures are needed to advance from classifying constructs to understanding causality in policy implementation research [ 29 ]. Given limited resources, policy implementers also need to know which aspects of implementation are key to improve policy acceptance, compliance, and sustainability to reap the intended health benefits [ 30 ]. Both pragmatic and psychometrically sound measures are needed to accomplish these objectives [ 10 , 11 , 31 , 32 ], so the field can explore the influence of nuanced determinants and generate reliable and valid findings.

To fill this void in the literature, this systematic review of health policy implementation measures aimed to (1) identify quantitative measures used to assess health policy implementation outcomes (IOF outcomes commonly called policy outputs in policy research) and inner and outer setting determinants, (2) describe and assess pragmatic quality of policy implementation measures, (3) describe and assess the quality of psychometric properties of identified instruments, and (4) elucidate health policy implementation measurement gaps.

The study team used systematic review procedures developed by Lewis and colleagues for reviews of D&I research measures and received detailed guidance from the Lewis team coauthors for each step [ 10 , 11 ]. We followed the PRISMA reporting guidelines as shown in the checklist (Supplemental Table 1 ). We have also provided a publicly available website of measures identified in this review ( https://www.health-policy-measures.org/ ).

For the purposes of this review, policy and policy implementation are defined as follows. We deemed public policy to include legislation at the federal, state/province/regional unit, or local levels; and governmental regulations, whether mandated by national, state/province, or local level governmental agencies or boards of elected officials (e.g., state boards of education in the USA) [ 4 , 20 ]. Here, public policy implementation is defined as the carrying out of a governmental mandate by public or private organizations and groups of organizations [ 20 ].

Two widely used frameworks from the D&I field guide the present review, and a third recently developed framework that bridges policy and D&I research. In the Implementation Outcomes Framework (IOF), Proctor and colleagues identify and define eight implementation outcomes that are differentiated from health outcomes: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability [ 25 ]. In the Consolidated Framework for Implementation Research (CFIR), Damschroder and colleagues articulate determinants of implementation including the domains of intervention characteristics, outer setting, inner setting of an organization, characteristics of individuals within organizations, and process [ 33 ]. Finally, Bullock developed the Policy Implementation Determinants Framework to present a balanced framework that emphasizes both internal setting constructs and external setting constructs including policy actor relationships and networks, political will for implementation, and visibility of policy actors [ 34 ]. The constructs identified in these frameworks were used to guide our list of implementation determinants and outcomes.

Through EBSCO, we searched MEDLINE, PsycInfo, and CINAHL Plus. Through ProQuest, we searched PAIS, Worldwide Political, and ERIC. Due to limited time and staff in the 12-month study, we did not search the grey literature. We used multiple search terms in each of four required levels: health, public policy, implementation, and measurement (Table 1 ). Table 1 shows search terms for each string. Supplemental Tables 2 and 3 show the final search syntax applied in EBSCO and ProQuest.

The authors developed the search strings and terms based on policy implementation framework reviews [ 34 , 35 ], additional policy implementation frameworks [ 21 , 22 ], labels and definitions of the eight implementation outcomes identified by Proctor et al. [ 25 ], CFIR construct labels and definitions [ 9 , 33 ], and additional D&I research and search term sources [ 28 , 36 , 37 , 38 ] (Table 1 ). The full study team provided three rounds of feedback on draft terms, and a library scientist provided additional synonyms and search terms. For each test search, we calculated the percentage of 18 benchmark articles the search captured. We determined a priori 80% as an acceptable level of precision.

Inclusion and exclusion criteria

This review addressed only measures of implementation by organizations mandated to act by governmental units or legislation. Measures of behavior changes by individuals in target populations as a result of legislation or governmental regulations and health status changes were outside the realm of this review.

There were several inclusion criteria: (1) empirical studies of the implementation of public policies already passed or approved that addressed physical or behavioral health, (2) quantitative self-report or archival measurement methods utilized, (3) published in peer-reviewed journals from January 1995 through April 2019, (4) published in the English language, (5) public policy implementation studies from any continent or international governing body, and (6) at least two transferable quantitative self-report or archival items that assessed implementation determinants [ 33 , 34 ] and/or IOF implementation outcomes [ 25 ]. This study sought to identify transferable measures that could be used to assess multiple policies and contexts. Here, a transferable item is defined as one that needed no wording changes or only a change in the referent (e.g., policy title or topic such as tobacco or malaria) to make the item applicable to other policies or settings [ 11 ]. The year 1995 was chosen as a starting year because that is about when web-based quantitative surveying began [ 39 ]. Table 2 provides definitions of the IOF implementation outcomes and the selected determinants of implementation. Broader constructs, such as readiness for implementation, contained multiple categories.

Exclusion criteria in the searches included (1) non-empiric health policy journal articles (e.g., conceptual articles, editorials); (2) narrative and systematic reviews; (3) studies with only qualitative assessment of health policy implementation; (4) empiric studies reported in theses and books; (5) health policy studies that only assessed health outcomes (i.e., target population changes in health behavior or status); (6) bill analyses, stakeholder perceptions assessed to inform policy development, and policy content analyses without implementation assessment; (7) studies of changes made in a private business not encouraged by public policy; and (8) countries with authoritarian regimes. We electronically programmed the searches to exclude policy implementation studies from countries that are not democratically governed due to vast differences in policy environments and implementation factors.

Screening procedures

Citations were downloaded into EndNote version 7.8 and de-duplicated electronically. We conducted dual independent screening of titles and abstracts after two group pilot screening sessions in which we clarified inclusion and exclusion criteria and screening procedures. Abstract screeners used Covidence systematic review software [ 40 ] to code inclusion as yes or no. Articles were included in full-text review if one screener coded it as meeting the inclusion criteria. Full-text screening via dual independent screening was coded in Covidence [ 40 ], with weekly meetings to reach consensus on inclusion/exclusion discrepancies. Screeners also coded one of the pre-identified reasons for exclusion.

Data extraction strategy

Extraction elements included information about (1) measure meta-data (e.g., measure name, total number of items, number of transferable items) and studies (e.g., policy topic, country, setting), (2) development and testing of the measure, (3) implementation outcomes and determinants assessed (Table 2 ), (4) pragmatic characteristics, and (5) psychometric properties. Where needed, authors were emailed to obtain the full measure and measure development information. Two coauthors (MP, CWB) reached consensus on extraction elements. For each included measure, a primary extractor conducted initial entries and coding. Due to time and staff limitations in the 12-month study, we did not search for each empirical use of the measure. A secondary extractor checked the entries, noting any discrepancies for discussion in consensus meetings. Multiple measures in a study were extracted separately.

Quality assessment of measures

To assess the quality of measures, we applied the Psychometric and Pragmatic Evidence Rating Scales (PAPERS) developed by Lewis et al. [ 10 , 11 , 41 , 42 ]. PAPERS includes assessment of five pragmatic instrument characteristics that affect the level of ease or difficulty to use the instrument: brevity (number of items), simplicity of language (readability level), cost (whether it is freely available), training burden (extent of data collection training needed), and analysis burden (ease or difficulty of interpretation of scoring and results). Lewis and colleagues developed the pragmatic domains and rating scales with stakeholder and D&I researchers input [ 11 , 41 , 42 ] and developed the psychometric rating scales in collaboration with D&I researchers [ 10 , 11 , 43 ]. The psychometric rating scale has nine properties (Table 3 ): internal consistency; norms; responsiveness; convergent, discriminant, and known-groups construct validity; predictive and concurrent criterion validity; and structural validity. In both the pragmatic and psychometric scales, reported evidence for each domain is scored from poor (− 1), none/not reported (0), minimal/emerging (1), adequate (2), good (3), or excellent (4). Higher values are indicative of more desirable pragmatic characteristics (e.g., fewer items, freely available, scoring instructions, and interpretations provided) and stronger evidence of psychometric properties (e.g., adequate to excellent reliability and validity) (Supplemental Tables 4 and 5 ).

Data synthesis and presentation

This section describes the synthesis of measure transferability, empiric use study settings and policy topics, and PAPERS scoring. Two coauthors (MP, CWB) consensus coded measures into three categories of item transferability based on quartile item transferability percentages: mostly transferable (≥ 75% of items deemed transferable), partially transferable (25–74% of items deemed transferable), and setting-specific (< 25% of items deemed transferable). Items were deemed transferable if no wording changes or only a change in the referent (e.g., policy title or topic) was needed to make the item applicable to the implementation of other policies or in other settings. Abstractors coded study settings into one of five categories: hospital or outpatient clinics; mental or behavioral health facilities; healthcare cost, access, or quality; schools; community; and multiple. Abstractors also coded policy topics to healthcare cost, access, or quality; mental or behavioral health; infectious or chronic diseases; and other, while retaining documentation of subtopics such as tobacco, physical activity, and nutrition. Pragmatic scores were totaled for the five properties, with possible total scores of − 5 to 20, with higher values indicating greater ease to use the instrument. Psychometric property total scores for the nine properties were also calculated, with possible scores of − 9 to 36, with higher values indicating evidence of multiple types of validity.

The database searches yielded 11,684 articles, of which 3267 were duplicates (Fig. 1 ). Titles and abstracts of the 8417 articles were independently screened by two team members; 870 (10.3%) were selected for full-text screening by at least one screener. Of the 870 studies, 804 were excluded at full-text screening or during extraction attempts with the consensus of two coauthors; 66 studies were included. Two coauthors (MP, CWB) reached consensus on extraction and coding of information on 70 unique quantitative eligible measures identified in the 66 included studies plus measure development articles where obtained. Nine measures were used in more than one included study. Detailed information on identified measures is publicly available at https://www.health-policy-measures.org/ .

PRISMA flow diagram

The most common exclusion reason was lack of transferable items in quantitative measures of policy implementation ( n = 597) (Fig. 1 ). While this review focused on transferable measures across any health issue or setting, researchers addressing specific health policies or settings may find the excluded studies of interest. The frequencies of the remaining exclusion reasons are listed in Fig. 1 .

A variety of health policy topics and settings from over two dozen countries were found in the database searches. For example, the searches identified quantitative and mixed methods implementation studies of legislation (such as tobacco smoking bans), regulations (such as food/menu labeling requirements), governmental policies that mandated specific clinical practices (such as vaccination or access to HIV antiretroviral treatment), school-based interventions (such as government-mandated nutritional content and physical activity), and other public policies.

Among the 70 unique quantitative implementation measures, 15 measures were deemed mostly transferable (at least 75% transferable, Table 4 ). Twenty-three measures were categorized as partially transferable (25 to 74% of items deemed transferable, Table 5 ); 32 measures were setting-specific (< 25% of items deemed transferable, data not shown).

Implementation outcomes

Among the 70 measures, the most commonly assessed implementation outcomes were fidelity/compliance of the policy implementation to the government mandate (26%), acceptability of the policy to implementers (24%), perceived appropriateness of the policy (17%), and feasibility of implementation (17%) (Table 2 ). Fidelity/compliance was sometimes assessed by asking implementers the extent to which they had modified a mandated practice [ 45 ]. Sometimes, detailed checklists were used to assess the extent of compliance with the many mandated policy components, such as school nutrition policies [ 83 ]. Acceptability was assessed by asking staff or healthcare providers in implementing agencies their level of agreement with the provided statements about the policy mandate, scored in Likert scales. Only eight (11%) of the included measures used multiple transferable items to assess adoption, and only eight (11%) assessed penetration.

Twenty-six measures of implementation costs were found during full-text screening (10 in included studies and 14 in excluded studies, data not shown). The cost time horizon varied from 12 months to 21 years, with most cost measures assessed at multiple time points. Ten of the 26 measures addressed direct implementation costs. Nine studies reported cost modeling findings. The implementation cost survey developed by Vogler et al. was extensive [ 53 ]. It asked implementing organizations to note policy impacts in medication pricing, margins, reimbursement rates, and insurance co-pays.

Determinants of implementation

Within the 70 included measures, the most commonly assessed implementation determinants were readiness for implementation (61% assessed any readiness component) and the general organizational culture and climate (39%), followed by the specific policy implementation climate within the implementation organization/s (23%), actor relationships and networks (17%), political will for policy implementation (11%), and visibility of the policy role and policy actors (10%) (Table 2 ). Each component of readiness for implementation was commonly assessed: communication of the policy (31%, 22 of 70 measures), policy awareness and knowledge (26%), resources for policy implementation (non-training resources 27%, training 20%), and leadership commitment to implement the policy (19%).

Only two studies assessed organizational structure as a determinant of health policy implementation. Lavinghouze and colleagues assessed the stability of the organization, defined as whether re-organization happens often or not, within a set of 9-point Likert items on multiple implementation determinants designed for use with state-level public health practitioners, and assessed whether public health departments were stand-alone agencies or embedded within agencies addressing additional services, such as social services [ 69 ]. Schneider and colleagues assessed coalition structure as an implementation determinant, including items on the number of organizations and individuals on the coalition roster, number that regularly attend coalition meetings, and so forth [ 72 ].

Tables of measures

Tables 4 and 5 present the 38 measures of implementation outcomes and/or determinants identified out of the 70 included measures with at least 25% of items transferable (useable in other studies without wording changes or by changing only the policy name or other referent). Table 4 shows 15 mostly transferable measures (at least 75% transferable). Table 5 shows 23 partially transferable measures (25–74% of items deemed transferable). Separate measure development articles were found for 20 of the 38 measures; the remaining measures seemed to be developed for one-time, study-specific use by the empirical study authors cited in the tables. Studies listed in Tables 4 and 5 were conducted most commonly in the USA ( n = 19) or Europe ( n = 11). A few measures were used elsewhere: Africa ( n = 3), Australia ( n = 1), Canada ( n = 1), Middle East ( n = 1), Southeast Asia ( n = 1), or across multiple continents ( n = 1).

Quality of identified measures

Figure 2 shows the median pragmatic quality ratings across the 38 measures with at least 25% transferable items shown in Tables 4 and 5 . Higher scores are desirable and indicate the measures are easier to use (Table 3 ). Overall, the measures were freely available in the public domain (median score = 4), brief with a median of 11–50 items (median score = 3), and had good readability, with a median reading level between 8th and 12th grade (median score = 3). However, instructions on how to score and interpret item scores were lacking, with a median score of 1, indicating the measures did not include suggestions for interpreting score ranges, clear cutoff scores, and instructions for handling missing data. In general, information on training requirements or availability of self-training manuals on how to use the measures was not reported in the included study or measure development article/s (median score = 0, not reported). Total pragmatic rating scores among the 38 measures with at least 25% of items transferable ranged from 7 to 17 (Tables 4 and 5 ), with a median total score of 12 out of a possible total score of 20. Median scores for each pragmatic characteristic were the same across all measures as for the 38 mostly or partially transferable measures, with a median total score of 11 across all measures.

Pragmatic rating scale results across identified measures. Footnote: pragmatic criteria scores from Psychometric and Pragmatic Evidence Rating Scale (PAPERS) (Lewis et al. [ 11 ], Stanick et al. [ 42 ]). Total possible score = 20, total median score across 38 measures = 11. Scores ranged from 0 to 18. Rating scales for each domain are provided in Supplemental Table 4

Few psychometric properties were reported. The study team found few reports of pilot testing and measure refinement as well. Among the 38 measures with at least 25% transferable items, the psychometric properties from the PAPERS rating scale total scores ranged from − 1 to 17 (Tables 4 and 5 ), with a median total score of 5 out of a possible total score of 36. Higher scores indicate more types of validity and reliability were reported with high quality. The 32 measures with calculable norms had a median norms PAPERS score of 3 (good), indicating appropriate sample size and distribution. The nine measures with reported internal consistency mostly showed Cronbach’s alphas in the adequate (0.70 to 0.79) to excellent (≥ 90) range, with a median of 0.78 (PAPERS score of 2, adequate) indicating adequate internal consistency. The five measures with reported structural validity had a median PAPERS score of 2, adequate (range 1 to 3, poor to good), indicating the sample size was sufficient and the factor analysis goodness of fit was reasonable. Among the 38 measures, no reports were found for responsiveness, convergent validity, discriminant validity, known-groups construct validity, or predictive or concurrent criterion validity.

In this systematic review, we sought to identify quantitative measures used to assess health policy implementation outcomes and determinants, rate the pragmatic and psychometric quality of identified measures, and point to future directions to address measurement gaps. In general, the identified measures are easy to use and freely available, but we found little data on validity and reliability. We found more quantitative measures of intra-organizational determinants of policy implementation than measures of the relationships and interactions between organizations that influence policy implementation. We found a limited number of measures that had been developed for or used to assess one of the eight IOF policy implementation outcomes that can be applied to other policies or settings, which may speak more to differences in terms used by policy researchers and D&I researchers than to differences in conceptualizations of policy implementation. Authors used a variety of terms and rarely provided definitions of the constructs the items assessed. Input from experts in policy implementation is needed to better understand and define policy implementation constructs for use across multiple fields involved in policy-related research.

We found several researchers had used well-tested measures of implementation determinants from D&I research or from organizational behavior and management literature (Tables 4 and 5 ). For internal setting of implementing organizations, whether mandated through public policy or not, well-developed and tested measures are available. However, a number of authors crafted their own items, with or without pilot testing, and used a variety of terms to describe what the items assessed. Further dissemination of the availability of well-tested measures to policy researchers is warranted [ 9 , 13 ].

What appears to be a larger gap involves the availability of well-developed and tested quantitative measures of the external context affecting policy implementation that can be used across multiple policy settings and topics [ 9 ]. Lack of attention to how a policy initiative fits with the external implementation context during policymaking and lack of policymaker commitment of adequate resources for implementation contribute to this gap [ 23 , 93 ]. Recent calls and initiatives to integrate health policies during policymaking and implementation planning will bring more attention to external contexts affecting not only policy development but implementation as well [ 93 , 94 , 95 , 96 , 97 , 98 , 99 ]. At the present time, it is not well-known which internal and external determinants are most essential to guide and achieve sustainable policy implementation [ 100 ]. Identification and dissemination of measures that assess factors that facilitate the spread of evidence-informed policy implementation (e.g., relative advantage, flexibility) will also help move policy implementation research forward [ 1 , 9 ].

Given the high potential population health impact of evidence-informed policies, much more attention to policy implementation is needed in D&I research. Few studies from non-D&I researchers reported policy implementation measure development procedures, pilot testing, scoring procedures and interpretation, training of data collectors, or data analysis procedures. Policy implementation research could benefit from the rigor of D&I quantitative research methods. And D&I researchers have much to learn about the contexts and practical aspects of policy implementation and can look to the rich depth of information in qualitative and mixed methods studies from other fields to inform quantitative measure development and testing [ 101 , 102 , 103 ].

Limitations

This systematic review has several limitations. First, the four levels of the search string and multiple search terms in each level were applied only to the title, abstract, and subject headings, due to limitations of the search engines, so we likely missed pertinent studies. Second, a systematic approach with stakeholder input is needed to expand the definitions of IOF implementation outcomes for policy implementation. Third, although the authors value intra-organizational policymaking and implementation, the study team restricted the search to governmental policies due to limited time and staffing in the 12-month study. Fourth, by excluding tools with only policy-specific implementation measures, we excluded some well-developed and tested instruments in abstract and full-text screening. Since only 12 measures had 100% transferable items, researchers may need to pilot test wording modifications of other items. And finally, due to limited time and staffing, we only searched online for measures and measures development articles and may have missed separately developed pragmatic information, such as training and scoring materials not reported in a manuscript.

Despite the limitations, several recommendations for measure development follow from the findings and related literature [ 1 , 11 , 20 , 35 , 41 , 104 ], including the need to (1) conduct systematic, mixed-methods procedures (concept mapping, expert panels) to refine policy implementation outcomes, (2) expand and more fully specify external context domains for policy implementation research and evaluation, (3) identify and disseminate well-developed measures for specific policy topics and settings, (4) ensure that policy implementation improves equity rather than exacerbating disparities [ 105 ], and (5) develop evidence-informed policy implementation guidelines.

Easy-to-use, reliable, and valid quantitative measures of policy implementation can further our understanding of policy implementation processes, determinants, and outcomes. Due to the wide array of health policy topics and implementation settings, sound quantitative measures that can be applied across topics and settings will help speed learnings from individual studies and aid in the transfer from research to practice. Quantitative measures can inform the implementation of evidence-informed policies to further the spread and effective implementation of policies to ultimately reap greater population health benefit. This systematic review of measures is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and predictors to help practitioners and researchers spread evidence-informed policies to improve population health and reduce inequities.

Availability of data and materials

A compendium of identified measures is available for dissemination at https://www.health-policy-measures.org/ . A link will be provided on the website of the Prevention Research Center, Brown School, Washington University in St. Louis, at https://prcstl.wustl.edu/ . The authors invite interested organizations to provide a link to the compendium. Citations and abstracts of excluded policy-specific measures are available on request.

Abbreviations

Consolidated Framework for Implementation Research

Cumulative Index of Nursing and Allied Health Literature

Dissemination and implementation science

Elton B. Stephens Company

Education Resources Information Center

Implementation Outcomes Framework

Psychometric and Pragmatic Evidence Rating Scale

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Purtle J, Dodson EA, Brownson RC. Policy dissemination research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice, Second Edition. New York: Oxford University Press; 2018.

Google Scholar  

Brownson RC, Baker EA, Deshpande AD, Gillespie KN. Evidence-based public health. Third ed. New York, NY: Oxford University Press; 2018.

Guide to Community Preventive Services. About the community guide.: community preventive services task force; 2020 [updated October 03, 2019; cited 2020. Available from: https://www.thecommunityguide.org/ .

Eyler AA, Chriqui JF, Moreland-Russell S, Brownson RC, editors. Prevention, policy, and public health, first edition. New York, NY: Oxford University Press; 2016.

Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ, et al. Vaccination greatly reduces disease, disability, death, and inequity worldwide. Geneva, Switzerland: World Health Organization; 2008 February 2008. Contract No.: 07-040089.

Cheng JJ, Schuster-Wallace CJ, Watt S, Newbold BK, Mente A. An ecological quantification of the relationships between water, sanitation and infant, child, and maternal mortality. Environ Health. 2012;11:4.

PubMed   PubMed Central   Google Scholar  

Levy DT, Li Y, Yuan Z. Impact of nations meeting the MPOWER targets between 2014 and 2016: an update. Tob Control. 2019.

Purtle J, Peters R, Brownson RC. A review of policy dissemination and implementation research funded by the National Institutes of Health, 2007-2014. Implement Sci. 2016;11:1.

Lewis CC, Proctor EK, Brownson RC. Measurement issues in dissemination and implementation research. In: Brownson RC, Ga C, Proctor EK, editors. Disssemination and Implementation Research in Health: Translating Science to Practice, Second Edition. New York: Oxford University Press; 2018.

Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155.

Lewis CC, Mettert KD, Dorsey CN, Martinez RG, Weiner BJ, Nolen E, et al. An updated protocol for a systematic review of implementation-related measures. Syst Rev. 2018;7(1):66.

Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22.

Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, et al. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016;11:42.

Nilsen P, Stahl C, Roback K, Cairney P. Never the twain shall meet?--a comparison of implementation science and policy implementation research. Implement Sci. 2013;8:63.

Sabatier PA, editor. Theories of the Policy Process. New York, NY: Routledge; 2019.

Kingdon J. Agendas, alternatives, and public policies, second edition. Second ed. New York: Longman; 1995.

Jones MD, Peterson HL, Pierce JJ, Herweg N, Bernal A, Lamberta Raney H, et al. A river runs through it: a multiple streams meta-review. Policy Stud J. 2016;44(1):13–36.

Fowler L. Using the multiple streams framework to connect policy adoption to implementation. Policy Studies Journal. 2020 (11 Feb).

Howlett M, Mukherjee I, Woo JJ. From tools to toolkits in policy design studies: the new design orientation towards policy formulation research. Policy Polit. 2015;43(2):291–311.

Natesan SD, Marathe RR. Literature review of public policy implementation. Int J Public Policy. 2015;11(4):219–38.

Sabatier PA, Mazmanian. Implementation of public policy: a framework of analysis. Policy Studies Journal. 1980 (January).

Sabatier PA. Theories of the Policy Process. Westview; 2007.

Tomm-Bonde L, Schreiber RS, Allan DE, MacDonald M, Pauly B, Hancock T, et al. Fading vision: knowledge translation in the implementation of a public health policy intervention. Implement Sci. 2013;8:59.

Roll S, Moulton S, Sandfort J. A comparative analysis of two streams of implementation research. Journal of Public and Nonprofit Affairs. 2017;3(1):3–22.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76.

Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice, second edition. New York: Oxford University Press; 2018.

Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–50.

Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–23.

PubMed   Google Scholar  

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

Boyd MR, Powell BJ, Endicott D, Lewis CC. A method for tracking implementation strategies: an exemplar implementing measurement-based care in community behavioral health clinics. Behav Ther. 2018;49(4):525–37.

Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013;40(3):257–65.

Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45(2):237–43.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Bullock HL. Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis in: How do systems achieve their goals? the role of implementation in mental health systems improvement [Dissertation]. Hamilton, Ontario: McMaster University; 2019.

Watson DP, Adams EL, Shue S, Coates H, McGuire A, Chesher J, et al. Defining the external implementation context: an integrative systematic literature review. BMC Health Serv Res. 2018;18(1):209.

McKibbon KA, Lokker C, Wilczynski NL, Ciliska D, Dobbins M, Davis DA, et al. A cross-sectional study of the number and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel? Implement Sci. 2010;5:16.

Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009;18(8):1115–23.

Egan M, Maclean A, Sweeting H, Hunt K. Comparing the effectiveness of using generic and specific search terms in electronic databases to identify health outcomes for a systematic review: a prospective comparative study of literature search method. BMJ Open. 2012;2:3.

Dillman DA, Smyth JD, Christian LM. Internet, mail, and mixed-mode surveys: the tailored design method. Hoboken, NJ: John Wiley & Sons; 2009.

Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation. https://www.covidence.org . Accessed Mar 2019.

Powell BJ, Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Barwick MA, et al. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118.

Stanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2019.

Henrikson NB, Blasi PR, Dorsey CN, Mettert KD, Nguyen MB, Walsh-Bailey C, et al. Psychometric and pragmatic properties of social risk screening tools: a systematic review. Am J Prev Med. 2019;57(6S1):S13–24.

Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65.

Lau AS, Brookman-Frazee L. The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implement Sci. 2016;11:1–8.

Ekvall G. Organizational climate for creativity and innovation. European J Work Organizational Psychology. 1996;5(1):105–23.

Lövgren G, Eriksson S, Sandman PO. Effects of an implemented care policy on patient and personnel experiences of care. Scand J Caring Sci. 2002;16(1):3–11.

Dwyer DJ, Ganster DC. The effects of job demands and control on employee attendance and satisfaction. J Organ Behav. 1991;12:595–608.

Condon-Paoloni D, Yeatman HR, Grigonis-Deane E. Health-related claims on food labels in Australia: understanding environmental health officers’ roles and implications for policy. Public Health Nutr. 2015;18(1):81–8.

Patterson MG, West MA, Shackleton VJ, Dawson JF, Lawthom R, Maitlis S, et al. Validating the organizational climate measure: links to managerial practices, productivity and innovation. J Organ Behav. 2005;26:279–408.

Glisson C, Green P, Williams NJ. Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child Abuse Negl. 2012;36(9):621–32.

Beidas RS, Aarons G, Barg F, Evans A, Hadley T, Hoagwood K, et al. Policy to implementation: evidence-based practice in community mental health--study protocol. Implement Sci. 2013;8(1):38.

Eisenberger R, Cummings J, Armeli S, Lynch P. Perceived organizational support, discretionary treatment, and job satisfaction. J Appl Psychol. 1997;82:812–20.

CAS   PubMed   Google Scholar  

Eby L, George K, Brown BL. Going tobacco-free: predictors of clinician reactions and outcomes of the NY state office of alcoholism and substance abuse services tobacco-free regulation. J Subst Abus Treat. 2013;44(3):280–7.

Vogler S, Zimmermann N, de Joncheere K. Policy interventions related to medicines: survey of measures taken in European countries during 2010-2015. Health Policy. 2016;120(12):1363–77.

Wanberg CRB, Banas JT. Predictors and outcomes of openness to change in a reorganizing workplace. J Applied Psychology. 2000;85:132–42.

CAS   Google Scholar  

Hardy LJ, Wertheim P, Bohan K, Quezada JC, Henley E. A model for evaluating the activities of a coalition-based policy action group: the case of Hermosa Vida. Health Promot Pract. 2013;14(4):514–23.

Gavriilidis G, Östergren P-O. Evaluating a traditional medicine policy in South Africa: phase 1 development of a policy assessment tool. Glob Health Action. 2012;5:17271.

Hongoro C, Rutebemberwa E, Twalo T, Mwendera C, Douglas M, Mukuru M, et al. Analysis of selected policies towards universal health coverage in Uganda: the policy implementation barometer protocol. Archives Public Health. 2018;76:12.

Roeseler A, Solomon M, Beatty C, Sipler AM. The tobacco control network’s policy readiness and stage of change assessment: what the results suggest for moving tobacco control efforts forward at the state and territorial levels. J Public Health Manag Pract. 2016;22(1):9–19.

Brämberg EB, Klinga C, Jensen I, Busch H, Bergström G, Brommels M, et al. Implementation of evidence-based rehabilitation for non-specific back pain and common mental health problems: a process evaluation of a nationwide initiative. BMC Health Serv Res. 2015;15(1):79.

Rütten A, Lüschen G, von Lengerke T, Abel T, Kannas L, Rodríguez Diaz JA, et al. Determinants of health policy impact: comparative results of a European policymaker study. Sozial-Und Praventivmedizin. 2003;48(6):379–91.

Smith SN, Lai Z, Almirall D, Goodrich DE, Abraham KM, Nord KM, et al. Implementing effective policy in a national mental health reengagement program for veterans. J Nerv Ment Dis. 2017;205(2):161–70.

Carasso BS, Lagarde M, Cheelo C, Chansa C, Palmer N. Health worker perspectives on user fee removal in Zambia. Hum Resour Health. 2012;10:40.

Goldsmith REH, C.F. Measuring consumer innovativeness. J Acad Mark Sci. 1991;19(3):209–21.

Webster CA, Caputi P, Perreault M, Doan R, Doutis P, Weaver RG. Elementary classroom teachers’ adoption of physical activity promotion in the context of a statewide policy: an innovation diffusion and socio-ecologic perspective. J Teach Phys Educ. 2013;32(4):419–40.

Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychol Assess. 2010;22(2):356–65.

Gill KJ, Campbell E, Gauthier G, Xenocostas S, Charney D, Macaulay AC. From policy to practice: implementing frontline community health services for substance dependence--study protocol. Implement Sci. 2014;9:108.

Lavinghouze SR, Price AW, Parsons B. The environmental assessment instrument: harnessing the environment for programmatic success. Health Promot Pract. 2009;10(2):176–85.

Bull FC, Milton K, Kahlmeier S. National policy on physical activity: the development of a policy audit tool. J Phys Act Health. 2014;11(2):233–40.

Bull F, Milton K, Kahlmeier S, Arlotti A, Juričan AB, Belander O, et al. Turning the tide: national policy approaches to increasing physical activity in seven European countries. British J Sports Med. 2015;49(11):749–56.

Schneider EC, Smith ML, Ory MG, Altpeter M, Beattie BL, Scheirer MA, et al. State fall prevention coalitions as systems change agents: an emphasis on policy. Health Promot Pract. 2016;17(2):244–53.

Helfrich CD, Savitz LA, Swiger KD, Weiner BJ. Adoption and implementation of mandated diabetes registries by community health centers. Am J Prev Med. 2007;33(1,Suppl):S50-S65.

Donchin M, Shemesh AA, Horowitz P, Daoud N. Implementation of the Healthy Cities’ principles and strategies: an evaluation of the Israel Healthy Cities network. Health Promot Int. 2006;21(4):266–73.

Were MC, Emenyonu N, Achieng M, Shen C, Ssali J, Masaba JP, et al. Evaluating a scalable model for implementing electronic health records in resource-limited settings. J Am Med Inform Assoc. 2010;17(3):237–44.

Konduri N, Sawyer K, Nizova N. User experience analysis of e-TB Manager, a nationwide electronic tuberculosis recording and reporting system in Ukraine. ERJ Open Research. 2017;3:2.

McDonnell E, Probart C. School wellness policies: employee participation in the development process and perceptions of the policies. J Child Nutr Manag. 2008;32:1.

Mersini E, Hyska J, Burazeri G. Evaluation of national food and nutrition policy in Albania. Zdravstveno Varstvo. 2017;56(2):115–23.

Cavagnero E, Daelmans B, Gupta N, Scherpbier R, Shankar A. Assessment of the health system and policy environment as a critical complement to tracking intervention coverage for maternal, newborn, and child health. Lancet. 2008;371 North American Edition(9620):1284-93.

Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abus Treat. 2002;22(4):197–209.

Pankratz M, Hallfors D, Cho H. Measuring perceptions of innovation adoption: the diffusion of a federal drug prevention policy. Health Educ Res. 2002;17(3):315–26.

Cook JM, Thompson R, Schnurr PP. Perceived characteristics of intervention scale: development and psychometric properties. Assessment. 2015;22(6):704–14.

Probart C, McDonnell ET, Jomaa L, Fekete V. Lessons from Pennsylvania’s mixed response to federal school wellness law. Health Aff. 2010;29(3):447–53.

Probart C, McDonnell E, Weirich JE, Schilling L, Fekete V. Statewide assessment of local wellness policies in Pennsylvania public school districts. J Am Diet Assoc. 2008;108(9):1497–502.

Rakic S, Novakovic B, Stevic S, Niskanovic J. Introduction of safety and quality standards for private health care providers: a case-study from the Republic of Srpska, Bosnia and Herzegovina. Int J Equity Health. 2018;17(1):92.

Rozema AD, Mathijssen JJP, Jansen MWJ, van Oers JAM. Sustainability of outdoor school ground smoking bans at secondary schools: a mixed-method study. Eur J Pub Health. 2018;28(1):43–9.

Barbero C, Moreland-Russell S, Bach LE, Cyr J. An evaluation of public school district tobacco policies in St. Louis County, Missouri. J Sch Health. 2013;83(8):525–32.

Williams KM, Kirsh S, Aron D, Au D, Helfrich C, Lambert-Kerzner A, et al. Evaluation of the Veterans Health Administration’s specialty care transformational initiatives to promote patient-centered delivery of specialty care: a mixed-methods approach. Telemed J E-Health. 2017;23(7):577–89.

Spencer E, Walshe K. National quality improvement policies and strategies in European healthcare systems. Quality Safety Health Care. 2009;18(Suppl 1):i22–i7.

Assunta M, Dorotheo EU. SEATCA Tobacco Industry Interference Index: a tool for measuring implementation of WHO Framework Convention on Tobacco Control Article 5.3. Tob Control. 2016;25(3):313–8.

Tummers L. Policy alienation of public professionals: the construct and its measurement. Public Adm Rev. 2012;72(4):516–25.

Tummers L, Bekkers V. Policy implementation, street-level bureaucracy, and the importance of discretion. Public Manag Rev. 2014;16(4):527–47.

Raghavan R, Bright CL, Shadoin AL. Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implement Sci. 2008;3:26.

Peters D, Harting J, van Oers H, Schuit J, de Vries N, Stronks K. Manifestations of integrated public health policy in Dutch municipalities. Health Promot Int. 2016;31(2):290–302.

Tosun J, Lang A. Policy integration: mapping the different concepts. Policy Studies. 2017;38(6):553–70.

Tubbing L, Harting J, Stronks K. Unravelling the concept of integrated public health policy: concept mapping with Dutch experts from science, policy, and practice. Health Policy. 2015;119(6):749–59.

Donkin A, Goldblatt P, Allen J, Nathanson V, Marmot M. Global action on the social determinants of health. BMJ Glob Health. 2017;3(Suppl 1):e000603-e.

Baum F, Friel S. Politics, policies and processes: a multidisciplinary and multimethods research programme on policies on the social determinants of health inequity in Australia. BMJ Open. 2017;7(12):e017772-e.

Delany T, Lawless A, Baum F, Popay J, Jones L, McDermott D, et al. Health in All Policies in South Australia: what has supported early implementation? Health Promot Int. 2016;31(4):888–98.

Valaitis R, MacDonald M, Kothari A, O'Mara L, Regan S, Garcia J, et al. Moving towards a new vision: implementation of a public health policy intervention. BMC Public Health. 2016;16:412.

Bennett LM, Gadlin H, Marchand, C. Collaboration team science: a field guide. Bethesda, MD: National Cancer Institute, National Institutes of Health; 2018. Contract No.: NIH Publication No. 18-7660.

Mazumdar M, Messinger S, Finkelstein DM, Goldberg JD, Lindsell CJ, Morton SC, et al. Evaluating academic scientists collaborating in team-based research: a proposed framework. Acad Med. 2015;90(10):1302–8.

Brownson RC, Fielding JE, Green LW. Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research. Annu Rev Public Health. 2018;39:27–53.

Brownson RC, Colditz GA, Proctor EK. Future issues in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. Second Edition ed. New York: Oxford University Press; 2018.

Thomson K, Hillier-Brown F, Todd A, McNamara C, Huijts T, Bambra C. The effects of public health policies on health inequalities in high-income countries: an umbrella review. BMC Public Health. 2018;18(1):869.

Download references

Acknowledgements

The authors are grateful for the policy expertise and guidance of Alexandra Morshed and the administrative support of Mary Adams, Linda Dix, and Cheryl Valko at the Prevention Research Center, Brown School, Washington University in St. Louis. We thank Lori Siegel, librarian, Brown School, Washington University in St. Louis, for assistance with search terms and procedures. We appreciate the D&I contributions of Enola Proctor and Byron Powell at the Brown School, Washington University in St. Louis, that informed this review. We thank Russell Glasgow, University of Colorado Denver, for guidance on the overall review and pragmatic measure criteria.

This project was funded March 2019 through February 2020 by the Foundation for Barnes-Jewish Hospital, with support from the Washington University in St. Louis Institute of Clinical and Translational Science Pilot Program, NIH/National Center for Advancing Translational Sciences (NCATS) grant UL1 TR002345. The project was also supported by the National Cancer Institute P50CA244431, Cooperative Agreement number U48DP006395-01-00 from the Centers for Disease Control and Prevention, R01MH106510 from the National Institute of Mental Health, and the National Institute of Diabetes and Digestive and Kidney Diseases award number P30DK020579. The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the Foundation for Barnes-Jewish Hospital, Washington University in St. Louis Institute of Clinical and Translational Science, National Institutes of Health, or the Centers for Disease Control and Prevention.

Author information

Authors and affiliations.

Prevention Research Center, Brown School, Washington University in St. Louis, One Brookings Drive, Campus Box 1196, St. Louis, MO, 63130, USA

Peg Allen, Meagan Pilar, Callie Walsh-Bailey, Stephanie Mazzucca, Maura M. Kepper & Ross C. Brownson

School of Social Work, Brigham Young University, 2190 FJSB, Provo, UT, 84602, USA

Cole Hooley

Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave, Seattle, WA, 98101, USA

Cara C. Lewis, Kayne D. Mettert & Caitlin N. Dorsey

Department of Health Management & Policy, Drexel University Dornsife School of Public Health, Nesbitt Hall, 3215 Market St, Philadelphia, PA, 19104, USA

Jonathan Purtle

Brown School, Washington University in St. Louis, One Brookings Drive, Campus Box 1196, St. Louis, MO, 63130, USA

Ana A. Baumann

Department of Surgery (Division of Public Health Sciences) and Alvin J. Siteman Cancer Center, Washington University School of Medicine, 4921 Parkview Place, Saint Louis, MO, 63110, USA

Ross C. Brownson

You can also search for this author in PubMed   Google Scholar

Contributions

Review methodology and quality assessment scale: CCL, KDM, CND. Eligibility criteria: PA, RCB, CND, KDM, SM, MP, JP. Search strings and terms: CH, PA, MP with review by AB, RCB, CND, CCL, MMK, SM, KDM. Framework selection: PA, AB, CH, MP. Abstract screening: PA, CH, MMK, SM, MP. Full-text screening: PA, CH, MP. Pilot extraction: PA, DNC, CH, KDM, SM, MP. Data extraction: MP, CWB. Data aggregation: MP, CWB. Writing: PA, RCB, JP. Editing: RCB, JP, SM, AB, CD, CH, MMK, CCL, KM, MP, CWB. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Peg Allen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

The authors declare they have no conflicting interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: table s1.

. PRISMA checklist. Table S2 . Electronic search terms for databases searched through EBSCO. Table S3 . Electronic search terms for searches conducted through PROQUEST. Table S4: PAPERS Pragmatic rating scales. Table S5 . PAPERS Psychometric rating scales.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Allen, P., Pilar, M., Walsh-Bailey, C. et al. Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implementation Sci 15 , 47 (2020). https://doi.org/10.1186/s13012-020-01007-w

Download citation

Received : 24 March 2020

Accepted : 05 June 2020

Published : 19 June 2020

DOI : https://doi.org/10.1186/s13012-020-01007-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation science
• Health policy
• Policy implementation
• Implementation
• Public policy
• Psychometric

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Quantitative Results of a National Intervention to Prevent Hospital-Acquired Catheter-Associated Urinary Tract Infection: A Pre-Post Observational Study

Affiliations.

• 1 University of Michigan Medical School and Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan (J.M., M.T.G., S.S.).
• 2 University of Michigan School of Nursing, Ann Arbor, Michigan (M.M.).
• 3 University of Michigan Medical School, Ann Arbor, Michigan (J.M.A., A.S.).
• 4 Integrated Clinical Services Team, Trinity Health, Livonia, Michigan (R.N.O.).
• 5 Health Research & Educational Trust, American Hospital Association, Chicago, Illinois (A.J.R.).
• 6 Center for Clinical Management Research, Veterans Affairs Ann Arbor Healthcare System, Ann Arbor, Michigan (D.R.).
• PMID: 31569231
• DOI: 10.7326/M18-3534

Background: Many hospitals struggle to prevent catheter-associated urinary tract infection (CAUTI).

Objective: To evaluate the effect of a multimodal initiative on CAUTI in hospitals with high burden of health care-associated infection (HAI).

Design: Prospective, national, nonrandomized, clustered, externally facilitated, pre-post observational quality improvement initiative, for 3 cohorts active between November 2016 and May 2018.

Setting: Acute care, long-term acute care, and critical access hospitals, including intensive care and non-intensive care wards.

Participants: Target hospitals had a high burden of Clostridioides difficile infection plus central line-associated bloodstream infection, CAUTI, or hospital-onset methicillin-resistant Staphylococcus aureus bloodstream infection, defined as cumulative attributable differences above the first tertile in the Targeted Assessment for Prevention (TAP) strategy. Some additional nonrecruited hospitals also joined.

Intervention: Multimodal intervention, including Practice Change Assessment tool to identify infection prevention and control (IPC) and HAI prevention gaps; Web-based, on-demand modules involving onboarding, foundational IPC practices, HAI-specific 2-tiered approach to prioritize and implement interventions, and TAP resources; monthly webinars; state partner-led in-person meetings; and feedback. State partners made site visits to at least 50% of their enrolled hospitals, to support self-assessments and coach.

Measurements: Rates of CAUTI and urinary catheter device utilization ratio.

Results: Of 387 participating hospitals from 23 states and the District of Columbia, 361 provided CAUTI data. Over the study period, the unadjusted CAUTI rate was low and relatively stable, decreasing slightly from 1.12 to 1.04 CAUTIs per 1000 catheter-days. Catheter utilization decreased from 21.46 to 19.83 catheter-days per 100 patient-days from the pre- to the postintervention period.

Limitations: The intervention period was brief, with no assessment of fidelity. Baseline CAUTI rates were low. Patient characteristics were not assessed.

Conclusion: This multimodal intervention yielded no substantial improvements in CAUTI or urinary catheter utilization.

Primary funding source: Centers for Disease Control and Prevention.

PubMed Disclaimer

Similar articles

• Quantitative Results of a National Intervention to Prevent Central Line-Associated Bloodstream Infection: A Pre-Post Observational Study. Patel PK, Greene MT, Jones K, Rolle AJ, Ratz D, Snyder A, Saint S, Chopra V. Patel PK, et al. Ann Intern Med. 2019 Oct 1;171(7_Suppl):S23-S29. doi: 10.7326/M18-3533. Ann Intern Med. 2019. PMID: 31569230
• Quantitative Results of a National Intervention to Prevent Clostridioides difficile Infection: A Pre-Post Observational Study. Dubberke ER, Rohde JM, Saint S, Jones K, Snyder A, Rolle AJ, Chopra V. Dubberke ER, et al. Ann Intern Med. 2019 Oct 1;171(7_Suppl):S52-S58. doi: 10.7326/M18-3545. Ann Intern Med. 2019. PMID: 31569233
• Quantitative Results of a National Intervention to Prevent Hospital-Onset Methicillin-Resistant Staphylococcus aureus Bloodstream Infection: A Pre-Post Observational Study. Calfee DP, Davila S, Chopra V, Patel PK, Snyder A, Ratz D, Rolle AJ, Olmsted RN, Popovich KJ. Calfee DP, et al. Ann Intern Med. 2019 Oct 1;171(7_Suppl):S66-S72. doi: 10.7326/M18-3535. Ann Intern Med. 2019. PMID: 31569232
• Nursing interventions to reduce the risk of catheter-associated urinary tract infection: part 2: staff education, monitoring, and care techniques. Willson M, Wilde M, Webb ML, Thompson D, Parker D, Harwood J, Callan L, Gray M. Willson M, et al. J Wound Ostomy Continence Nurs. 2009 Mar-Apr;36(2):137-54. doi: 10.1097/01.WON.0000347655.56851.04. J Wound Ostomy Continence Nurs. 2009. PMID: 19287262 Review.
• The evolution of catheter-associated urinary tract infection (CAUTI): Is it time for more inclusive metrics? Advani SD, Fakih MG. Advani SD, et al. Infect Control Hosp Epidemiol. 2019 Jun;40(6):681-685. doi: 10.1017/ice.2019.43. Epub 2019 Mar 27. Infect Control Hosp Epidemiol. 2019. PMID: 30915925 Review.
• Systematic review of contemporary interventions for improving discharge support and transitions of care from the patient experience perspective. Jesus TS, Stern BZ, Lee D, Zhang M, Struhar J, Heinemann AW, Jordan N, Deutsch A. Jesus TS, et al. PLoS One. 2024 May 21;19(5):e0299176. doi: 10.1371/journal.pone.0299176. eCollection 2024. PLoS One. 2024. PMID: 38771768 Free PMC article.
• Strategies to prevent catheter-associated urinary tract infections in acute-care hospitals: 2022 Update. Patel PK, Advani SD, Kofman AD, Lo E, Maragakis LL, Pegues DA, Pettis AM, Saint S, Trautner B, Yokoe DS, Meddings J. Patel PK, et al. Infect Control Hosp Epidemiol. 2023 Aug;44(8):1209-1231. doi: 10.1017/ice.2023.137. Epub 2023 Aug 25. Infect Control Hosp Epidemiol. 2023. PMID: 37620117 Free PMC article.
• Implementing strategies to prevent infections in acute-care settings. Trivedi KK, Schaffzin JK, Deloney VM, Aureden K, Carrico R, Garcia-Houchins S, Garrett JH Jr, Glowicz J, Lee GM, Maragakis LL, Moody J, Pettis AM, Saint S, Schweizer ML, Yokoe DS, Berenholtz S. Trivedi KK, et al. Infect Control Hosp Epidemiol. 2023 Aug;44(8):1232-1246. doi: 10.1017/ice.2023.103. Epub 2023 Jul 11. Infect Control Hosp Epidemiol. 2023. PMID: 37431239 Free PMC article.
• Effect of Time of Daily Data Collection on the Calculation of Catheter-associated Urinary Tract Infection Rates. Donnelly LF, Wood M, Loh L, Tekic N, Shin AY, Scheinker D. Donnelly LF, et al. Pediatr Qual Saf. 2021 Aug 26;6(5):e466. doi: 10.1097/pq9.0000000000000466. eCollection 2021 Sep-Oct. Pediatr Qual Saf. 2021. PMID: 34476317 Free PMC article.

Publication types

• Search in MeSH

Related information

Linkout - more resources, full text sources.

• Ovid Technologies, Inc.
• MedlinePlus Health Information

Miscellaneous

• NCI CPTAC Assay Portal

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Advertisement

• Previous Issue
• Previous Article
• Next Article

Clarifying the Research Purpose

Methodology, measurement, data analysis and interpretation, tools for evaluating the quality of medical education research, research support, competing interests, quantitative research methods in medical education.

Submitted for publication January 8, 2018. Accepted for publication November 29, 2018.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Open the PDF for in another window
• Cite Icon Cite
• Get Permissions
• Search Site

John T. Ratelle , Adam P. Sawatsky , Thomas J. Beckman; Quantitative Research Methods in Medical Education. Anesthesiology 2019; 131:23–35 doi: https://doi.org/10.1097/ALN.0000000000002727

Download citation file:

• Ris (Zotero)
• Reference Manager

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Image: J. P. Rathmell and Terri Navarette.

There has been an explosion of research in the field of medical education. A search of PubMed demonstrates that more than 40,000 articles have been indexed under the medical subject heading “Medical Education” since 2010, which is more than the total number of articles indexed under this heading in the 1980s and 1990s combined. Keeping up to date requires that practicing clinicians have the skills to interpret and appraise the quality of research articles, especially when serving as editors, reviewers, and consumers of the literature.

While medical education shares many characteristics with other biomedical fields, substantial particularities exist. We recognize that practicing clinicians may not be familiar with the nuances of education research and how to assess its quality. Therefore, our purpose is to provide a review of quantitative research methodologies in medical education. Specifically, we describe a structure that can be used when conducting or evaluating medical education research articles.

Clarifying the research purpose is an essential first step when reading or conducting scholarship in medical education. 1   Medical education research can serve a variety of purposes, from advancing the science of learning to improving the outcomes of medical trainees and the patients they care for. However, a well-designed study has limited value if it addresses vague, redundant, or unimportant medical education research questions.

What is the research topic and why is it important? What is unknown about the research topic? Why is further research necessary?

What is the conceptual framework being used to approach the study?

What is the statement of study intent?

What are the research methodology and study design? Are they appropriate for the study objective(s)?

Which threats to internal validity are most relevant for the study?

What is the outcome and how was it measured?

Can the results be trusted? What is the validity and reliability of the measurements?

How were research subjects selected? Is the research sample representative of the target population?

Was the data analysis appropriate for the study design and type of data?

What is the effect size? Do the results have educational significance?

Fortunately, there are steps to ensure that the purpose of a research study is clear and logical. Table 1   2–5   outlines these steps, which will be described in detail in the following sections. We describe these elements not as a simple “checklist,” but as an advanced organizer that can be used to understand a medical education research study. These steps can also be used by clinician educators who are new to the field of education research and who wish to conduct scholarship in medical education.

Steps in Clarifying the Purpose of a Research Study in Medical Education

Literature Review and Problem Statement

A literature review is the first step in clarifying the purpose of a medical education research article. 2 , 5 , 6   When conducting scholarship in medical education, a literature review helps researchers develop an understanding of their topic of interest. This understanding includes both existing knowledge about the topic as well as key gaps in the literature, which aids the researcher in refining their study question. Additionally, a literature review helps researchers identify conceptual frameworks that have been used to approach the research topic. 2  

When reading scholarship in medical education, a successful literature review provides background information so that even someone unfamiliar with the research topic can understand the rationale for the study. Located in the introduction of the manuscript, the literature review guides the reader through what is already known in a manner that highlights the importance of the research topic. The literature review should also identify key gaps in the literature so the reader can understand the need for further research. This gap description includes an explicit problem statement that summarizes the important issues and provides a reason for the study. 2 , 4   The following is one example of a problem statement:

“Identifying gaps in the competency of anesthesia residents in time for intervention is critical to patient safety and an effective learning system… [However], few available instruments relate to complex behavioral performance or provide descriptors…that could inform subsequent feedback, individualized teaching, remediation, and curriculum revision.” 7  

This problem statement articulates the research topic (identifying resident performance gaps), why it is important (to intervene for the sake of learning and patient safety), and current gaps in the literature (few tools are available to assess resident performance). The researchers have now underscored why further research is needed and have helped readers anticipate the overarching goals of their study (to develop an instrument to measure anesthesiology resident performance). 4  

The Conceptual Framework

Following the literature review and articulation of the problem statement, the next step in clarifying the research purpose is to select a conceptual framework that can be applied to the research topic. Conceptual frameworks are “ways of thinking about a problem or a study, or ways of representing how complex things work.” 3   Just as clinical trials are informed by basic science research in the laboratory, conceptual frameworks often serve as the “basic science” that informs scholarship in medical education. At a fundamental level, conceptual frameworks provide a structured approach to solving the problem identified in the problem statement.

Conceptual frameworks may take the form of theories, principles, or models that help to explain the research problem by identifying its essential variables or elements. Alternatively, conceptual frameworks may represent evidence-based best practices that researchers can apply to an issue identified in the problem statement. 3   Importantly, there is no single best conceptual framework for a particular research topic, although the choice of a conceptual framework is often informed by the literature review and knowing which conceptual frameworks have been used in similar research. 8   For further information on selecting a conceptual framework for research in medical education, we direct readers to the work of Bordage 3   and Irby et al. 9  

To illustrate how different conceptual frameworks can be applied to a research problem, suppose you encounter a study to reduce the frequency of communication errors among anesthesiology residents during day-to-night handoff. Table 2 10 , 11   identifies two different conceptual frameworks researchers might use to approach the task. The first framework, cognitive load theory, has been proposed as a conceptual framework to identify potential variables that may lead to handoff errors. 12   Specifically, cognitive load theory identifies the three factors that affect short-term memory and thus may lead to communication errors:

Conceptual Frameworks to Address the Issue of Handoff Errors in the Intensive Care Unit

Intrinsic load: Inherent complexity or difficulty of the information the resident is trying to learn ( e.g. , complex patients).

Extraneous load: Distractions or demands on short-term memory that are not related to the information the resident is trying to learn ( e.g. , background noise, interruptions).

Germane load: Effort or mental strategies used by the resident to organize and understand the information he/she is trying to learn ( e.g. , teach back, note taking).

Using cognitive load theory as a conceptual framework, researchers may design an intervention to reduce extraneous load and help the resident remember the overnight to-do’s. An example might be dedicated, pager-free handoff times where distractions are minimized.

The second framework identified in table 2 , the I-PASS (Illness severity, Patient summary, Action list, Situational awareness and contingency planning, and Synthesis by receiver) handoff mnemonic, 11   is an evidence-based best practice that, when incorporated as part of a handoff bundle, has been shown to reduce handoff errors on pediatric wards. 13   Researchers choosing this conceptual framework may adapt some or all of the I-PASS elements for resident handoffs in the intensive care unit.

Note that both of the conceptual frameworks outlined above provide researchers with a structured approach to addressing the issue of handoff errors; one is not necessarily better than the other. Indeed, it is possible for researchers to use both frameworks when designing their study. Ultimately, we provide this example to demonstrate the necessity of selecting conceptual frameworks to clarify the research purpose. 3 , 8   Readers should look for conceptual frameworks in the introduction section and should be wary of their omission, as commonly seen in less well-developed medical education research articles. 14  

Statement of Study Intent

After reviewing the literature, articulating the problem statement, and selecting a conceptual framework to address the research topic, the final step in clarifying the research purpose is the statement of study intent. The statement of study intent is arguably the most important element of framing the study because it makes the research purpose explicit. 2   Consider the following example:

This study aimed to test the hypothesis that the introduction of the BASIC Examination was associated with an accelerated knowledge acquisition during residency training, as measured by increments in annual ITE scores. 15  

This statement of study intent succinctly identifies several key study elements including the population (anesthesiology residents), the intervention/independent variable (introduction of the BASIC Examination), the outcome/dependent variable (knowledge acquisition, as measure by in In-training Examination [ITE] scores), and the hypothesized relationship between the independent and dependent variable (the authors hypothesize a positive correlation between the BASIC examination and the speed of knowledge acquisition). 6 , 14  

The statement of study intent will sometimes manifest as a research objective, rather than hypothesis or question. In such instances there may not be explicit independent and dependent variables, but the study population and research aim should be clearly identified. The following is an example:

“In this report, we present the results of 3 [years] of course data with respect to the practice improvements proposed by participating anesthesiologists and their success in implementing those plans. Specifically, our primary aim is to assess the frequency and type of improvements that were completed and any factors that influence completion.” 16  

The statement of study intent is the logical culmination of the literature review, problem statement, and conceptual framework, and is a transition point between the Introduction and Methods sections of a medical education research report. Nonetheless, a systematic review of experimental research in medical education demonstrated that statements of study intent are absent in the majority of articles. 14   When reading a medical education research article where the statement of study intent is absent, it may be necessary to infer the research aim by gathering information from the Introduction and Methods sections. In these cases, it can be useful to identify the following key elements 6 , 14 , 17   :

Population of interest/type of learner ( e.g. , pain medicine fellow or anesthesiology residents)

Independent/predictor variable ( e.g. , educational intervention or characteristic of the learners)

Dependent/outcome variable ( e.g. , intubation skills or knowledge of anesthetic agents)

Relationship between the variables ( e.g. , “improve” or “mitigate”)

Occasionally, it may be difficult to differentiate the independent study variable from the dependent study variable. 17   For example, consider a study aiming to measure the relationship between burnout and personal debt among anesthesiology residents. Do the researchers believe burnout might lead to high personal debt, or that high personal debt may lead to burnout? This “chicken or egg” conundrum reinforces the importance of the conceptual framework which, if present, should serve as an explanation or rationale for the predicted relationship between study variables.

Research methodology is the “…design or plan that shapes the methods to be used in a study.” 1   Essentially, methodology is the general strategy for answering a research question, whereas methods are the specific steps and techniques that are used to collect data and implement the strategy. Our objective here is to provide an overview of quantitative methodologies ( i.e. , approaches) in medical education research.

The choice of research methodology is made by balancing the approach that best answers the research question against the feasibility of completing the study. There is no perfect methodology because each has its own potential caveats, flaws and/or sources of bias. Before delving into an overview of the methodologies, it is important to highlight common sources of bias in education research. We use the term internal validity to describe the degree to which the findings of a research study represent “the truth,” as opposed to some alternative hypothesis or variables. 18   Table 3   18–20   provides a list of common threats to internal validity in medical education research, along with tactics to mitigate these threats.

Threats to Internal Validity and Strategies to Mitigate Their Effects

Experimental Research

The fundamental tenet of experimental research is the manipulation of an independent or experimental variable to measure its effect on a dependent or outcome variable.

True Experiment

True experimental study designs minimize threats to internal validity by randomizing study subjects to experimental and control groups. Through ensuring that differences between groups are—beyond the intervention/variable of interest—purely due to chance, researchers reduce the internal validity threats related to subject characteristics, time-related maturation, and regression to the mean. 18 , 19  

Quasi-experiment

There are many instances in medical education where randomization may not be feasible or ethical. For instance, researchers wanting to test the effect of a new curriculum among medical students may not be able to randomize learners due to competing curricular obligations and schedules. In these cases, researchers may be forced to assign subjects to experimental and control groups based upon some other criterion beyond randomization, such as different classrooms or different sections of the same course. This process, called quasi-randomization, does not inherently lead to internal validity threats, as long as research investigators are mindful of measuring and controlling for extraneous variables between study groups. 19  

Single-group Methodologies

All experimental study designs compare two or more groups: experimental and control. A common experimental study design in medical education research is the single-group pretest–posttest design, which compares a group of learners before and after the implementation of an intervention. 21   In essence, a single-group pre–post design compares an experimental group ( i.e. , postintervention) to a “no-intervention” control group ( i.e. , preintervention). 19   This study design is problematic for several reasons. Consider the following hypothetical example: A research article reports the effects of a year-long intubation curriculum for first-year anesthesiology residents. All residents participate in monthly, half-day workshops over the course of an academic year. The article reports a positive effect on residents’ skills as demonstrated by a significant improvement in intubation success rates at the end of the year when compared to the beginning.

This study does little to advance the science of learning among anesthesiology residents. While this hypothetical report demonstrates an improvement in residents’ intubation success before versus after the intervention, it does not tell why the workshop worked, how it compares to other educational interventions, or how it fits in to the broader picture of anesthesia training.

Single-group pre–post study designs open themselves to a myriad of threats to internal validity. 20   In our hypothetical example, the improvement in residents’ intubation skills may have been due to other educational experience(s) ( i.e. , implementation threat) and/or improvement in manual dexterity that occurred naturally with time ( i.e. , maturation threat), rather than the airway curriculum. Consequently, single-group pre–post studies should be interpreted with caution. 18  

Repeated testing, before and after the intervention, is one strategy that can be used to reduce the some of the inherent limitations of the single-group study design. Repeated pretesting can mitigate the effect of regression toward the mean, a statistical phenomenon whereby low pretest scores tend to move closer to the mean on subsequent testing (regardless of intervention). 20   Likewise, repeated posttesting at multiple time intervals can provide potentially useful information about the short- and long-term effects of an intervention ( e.g. , the “durability” of the gain in knowledge, skill, or attitude).

Observational Research

Unlike experimental studies, observational research does not involve manipulation of any variables. These studies often involve measuring associations, developing psychometric instruments, or conducting surveys.

Association Research

Association research seeks to identify relationships between two or more variables within a group or groups (correlational research), or similarities/differences between two or more existing groups (causal–comparative research). For example, correlational research might seek to measure the relationship between burnout and educational debt among anesthesiology residents, while causal–comparative research may seek to measure differences in educational debt and/or burnout between anesthesiology and surgery residents. Notably, association research may identify relationships between variables, but does not necessarily support a causal relationship between them.

Psychometric and Survey Research

Psychometric instruments measure a psychologic or cognitive construct such as knowledge, satisfaction, beliefs, and symptoms. Surveys are one type of psychometric instrument, but many other types exist, such as evaluations of direct observation, written examinations, or screening tools. 22   Psychometric instruments are ubiquitous in medical education research and can be used to describe a trait within a study population ( e.g. , rates of depression among medical students) or to measure associations between study variables ( e.g. , association between depression and board scores among medical students).

Psychometric and survey research studies are prone to the internal validity threats listed in table 3 , particularly those relating to mortality, location, and instrumentation. 18   Additionally, readers must ensure that the instrument scores can be trusted to truly represent the construct being measured. For example, suppose you encounter a research article demonstrating a positive association between attending physician teaching effectiveness as measured by a survey of medical students, and the frequency with which the attending physician provides coffee and doughnuts on rounds. Can we be confident that this survey administered to medical students is truly measuring teaching effectiveness? Or is it simply measuring the attending physician’s “likability”? Issues related to measurement and the trustworthiness of data are described in detail in the following section on measurement and the related issues of validity and reliability.

Measurement refers to “the assigning of numbers to individuals in a systematic way as a means of representing properties of the individuals.” 23   Research data can only be trusted insofar as we trust the measurement used to obtain the data. Measurement is of particular importance in medical education research because many of the constructs being measured ( e.g. , knowledge, skill, attitudes) are abstract and subject to measurement error. 24   This section highlights two specific issues related to the trustworthiness of data: the validity and reliability of measurements.

Validity regarding the scores of a measurement instrument “refers to the degree to which evidence and theory support the interpretations of the [instrument’s results] for the proposed use of the [instrument].” 25   In essence, do we believe the results obtained from a measurement really represent what we were trying to measure? Note that validity evidence for the scores of a measurement instrument is separate from the internal validity of a research study. Several frameworks for validity evidence exist. Table 4 2 , 22 , 26   represents the most commonly used framework, developed by Messick, 27   which identifies sources of validity evidence—to support the target construct—from five main categories: content, response process, internal structure, relations to other variables, and consequences.

Sources of Validity Evidence for Measurement Instruments

Reliability

Reliability refers to the consistency of scores for a measurement instrument. 22 , 25 , 28   For an instrument to be reliable, we would anticipate that two individuals rating the same object of measurement in a specific context would provide the same scores. 25   Further, if the scores for an instrument are reliable between raters of the same object of measurement, then we can extrapolate that any difference in scores between two objects represents a true difference across the sample, and is not due to random variation in measurement. 29   Reliability can be demonstrated through a variety of methods such as internal consistency ( e.g. , Cronbach’s alpha), temporal stability ( e.g. , test–retest reliability), interrater agreement ( e.g. , intraclass correlation coefficient), and generalizability theory (generalizability coefficient). 22 , 29  

Example of a Validity and Reliability Argument

This section provides an illustration of validity and reliability in medical education. We use the signaling questions outlined in table 4 to make a validity and reliability argument for the Harvard Assessment of Anesthesia Resident Performance (HARP) instrument. 7   The HARP was developed by Blum et al. to measure the performance of anesthesia trainees that is required to provide safe anesthetic care to patients. According to the authors, the HARP is designed to be used “…as part of a multiscenario, simulation-based assessment” of resident performance. 7  

Content Validity: Does the Instrument’s Content Represent the Construct Being Measured?

To demonstrate content validity, instrument developers should describe the construct being measured and how the instrument was developed, and justify their approach. 25   The HARP is intended to measure resident performance in the critical domains required to provide safe anesthetic care. As such, investigators note that the HARP items were created through a two-step process. First, the instrument’s developers interviewed anesthesiologists with experience in resident education to identify the key traits needed for successful completion of anesthesia residency training. Second, the authors used a modified Delphi process to synthesize the responses into five key behaviors: (1) formulate a clear anesthetic plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize one’s limits. 7 , 30  

Response Process Validity: Are Raters Interpreting the Instrument Items as Intended?

In the case of the HARP, the developers included a scoring rubric with behavioral anchors to ensure that faculty raters could clearly identify how resident performance in each domain should be scored. 7  

Internal Structure Validity: Do Instrument Items Measuring Similar Constructs Yield Homogenous Results? Do Instrument Items Measuring Different Constructs Yield Heterogeneous Results?

Item-correlation for the HARP demonstrated a high degree of correlation between some items ( e.g. , formulating a plan and modifying the plan under changing conditions) and a lower degree of correlation between other items ( e.g. , formulating a plan and identifying performance improvement opportunities). 30   This finding is expected since the items within the HARP are designed to assess separate performance domains, and we would expect residents’ functioning to vary across domains.

Relationship to Other Variables’ Validity: Do Instrument Scores Correlate with Other Measures of Similar or Different Constructs as Expected?

As it applies to the HARP, one would expect that the performance of anesthesia residents will improve over the course of training. Indeed, HARP scores were found to be generally higher among third-year residents compared to first-year residents. 30  

Consequence Validity: Are Instrument Results Being Used as Intended? Are There Unintended or Negative Uses of the Instrument Results?

While investigators did not intentionally seek out consequence validity evidence for the HARP, unanticipated consequences of HARP scores were identified by the authors as follows:

“Data indicated that CA-3s had a lower percentage of worrisome scores (rating 2 or lower) than CA-1s… However, it is concerning that any CA-3s had any worrisome scores…low performance of some CA-3 residents, albeit in the simulated environment, suggests opportunities for training improvement.” 30  

That is, using the HARP to measure the performance of CA-3 anesthesia residents had the unintended consequence of identifying the need for improvement in resident training.

Reliability: Are the Instrument’s Scores Reproducible and Consistent between Raters?

The HARP was applied by two raters for every resident in the study across seven different simulation scenarios. The investigators conducted a generalizability study of HARP scores to estimate the variance in assessment scores that was due to the resident, the rater, and the scenario. They found little variance was due to the rater ( i.e. , scores were consistent between raters), indicating a high level of reliability. 7  

Sampling refers to the selection of research subjects ( i.e. , the sample) from a larger group of eligible individuals ( i.e. , the population). 31   Effective sampling leads to the inclusion of research subjects who represent the larger population of interest. Alternatively, ineffective sampling may lead to the selection of research subjects who are significantly different from the target population. Imagine that researchers want to explore the relationship between burnout and educational debt among pain medicine specialists. The researchers distribute a survey to 1,000 pain medicine specialists (the population), but only 300 individuals complete the survey (the sample). This result is problematic because the characteristics of those individuals who completed the survey and the entire population of pain medicine specialists may be fundamentally different. It is possible that the 300 study subjects may be experiencing more burnout and/or debt, and thus, were more motivated to complete the survey. Alternatively, the 700 nonresponders might have been too busy to respond and even more burned out than the 300 responders, which would suggest that the study findings were even more amplified than actually observed.

When evaluating a medical education research article, it is important to identify the sampling technique the researchers employed, how it might have influenced the results, and whether the results apply to the target population. 24  

Sampling Techniques

Sampling techniques generally fall into two categories: probability- or nonprobability-based. Probability-based sampling ensures that each individual within the target population has an equal opportunity of being selected as a research subject. Most commonly, this is done through random sampling, which should lead to a sample of research subjects that is similar to the target population. If significant differences between sample and population exist, those differences should be due to random chance, rather than systematic bias. The difference between data from a random sample and that from the population is referred to as sampling error. 24  

Nonprobability-based sampling involves selecting research participants such that inclusion of some individuals may be more likely than the inclusion of others. 31   Convenience sampling is one such example and involves selection of research subjects based upon ease or opportuneness. Convenience sampling is common in medical education research, but, as outlined in the example at the beginning of this section, it can lead to sampling bias. 24   When evaluating an article that uses nonprobability-based sampling, it is important to look for participation/response rate. In general, a participation rate of less than 75% should be viewed with skepticism. 21   Additionally, it is important to determine whether characteristics of participants and nonparticipants were reported and if significant differences between the two groups exist.

Interpreting medical education research requires a basic understanding of common ways in which quantitative data are analyzed and displayed. In this section, we highlight two broad topics that are of particular importance when evaluating research articles.

The Nature of the Measurement Variable

Measurement variables in quantitative research generally fall into three categories: nominal, ordinal, or interval. 24   Nominal variables (sometimes called categorical variables) involve data that can be placed into discrete categories without a specific order or structure. Examples include sex (male or female) and professional degree (M.D., D.O., M.B.B.S., etc .) where there is no clear hierarchical order to the categories. Ordinal variables can be ranked according to some criterion, but the spacing between categories may not be equal. Examples of ordinal variables may include measurements of satisfaction (satisfied vs . unsatisfied), agreement (disagree vs . agree), and educational experience (medical student, resident, fellow). As it applies to educational experience, it is noteworthy that even though education can be quantified in years, the spacing between years ( i.e. , educational “growth”) remains unequal. For instance, the difference in performance between second- and third-year medical students is dramatically different than third- and fourth-year medical students. Interval variables can also be ranked according to some criteria, but, unlike ordinal variables, the spacing between variable categories is equal. Examples of interval variables include test scores and salary. However, the conceptual boundaries between these measurement variables are not always clear, as in the case where ordinal scales can be assumed to have the properties of an interval scale, so long as the data’s distribution is not substantially skewed. 32  

Understanding the nature of the measurement variable is important when evaluating how the data are analyzed and reported. Medical education research commonly uses measurement instruments with items that are rated on Likert-type scales, whereby the respondent is asked to assess their level of agreement with a given statement. The response is often translated into a corresponding number ( e.g. , 1 = strongly disagree, 3 = neutral, 5 = strongly agree). It is remarkable that scores from Likert-type scales are sometimes not normally distributed ( i.e. , are skewed toward one end of the scale), indicating that the spacing between scores is unequal and the variable is ordinal in nature. In these cases, it is recommended to report results as frequencies or medians, rather than means and SDs. 33  

Consider an article evaluating medical students’ satisfaction with a new curriculum. Researchers measure satisfaction using a Likert-type scale (1 = very unsatisfied, 2 = unsatisfied, 3 = neutral, 4 = satisfied, 5 = very satisfied). A total of 20 medical students evaluate the curriculum, 10 of whom rate their satisfaction as “satisfied,” and 10 of whom rate it as “very satisfied.” In this case, it does not make much sense to report an average score of 4.5; it makes more sense to report results in terms of frequency ( e.g. , half of the students were “very satisfied” with the curriculum, and half were not).

Effect Size and CIs

In medical education, as in other research disciplines, it is common to report statistically significant results ( i.e. , small P values) in order to increase the likelihood of publication. 34 , 35   However, a significant P value in itself does necessarily represent the educational impact of the study results. A statement like “Intervention x was associated with a significant improvement in learners’ intubation skill compared to education intervention y ( P < 0.05)” tells us that there was a less than 5% chance that the difference in improvement between interventions x and y was due to chance. Yet that does not mean that the study intervention necessarily caused the nonchance results, or indicate whether the between-group difference is educationally significant. Therefore, readers should consider looking beyond the P value to effect size and/or CI when interpreting the study results. 36 , 37  

Effect size is “the magnitude of the difference between two groups,” which helps to quantify the educational significance of the research results. 37   Common measures of effect size include Cohen’s d (standardized difference between two means), risk ratio (compares binary outcomes between two groups), and Pearson’s r correlation (linear relationship between two continuous variables). 37   CIs represent “a range of values around a sample mean or proportion” and are a measure of precision. 31   While effect size and CI give more useful information than simple statistical significance, they are commonly omitted from medical education research articles. 35   In such instances, readers should be wary of overinterpreting a P value in isolation. For further information effect size and CI, we direct readers the work of Sullivan and Feinn 37   and Hulley et al. 31  

In this final section, we identify instruments that can be used to evaluate the quality of quantitative medical education research articles. To this point, we have focused on framing the study and research methodologies and identifying potential pitfalls to consider when appraising a specific article. This is important because how a study is framed and the choice of methodology require some subjective interpretation. Fortunately, there are several instruments available for evaluating medical education research methods and providing a structured approach to the evaluation process.

The Medical Education Research Study Quality Instrument (MERSQI) 21   and the Newcastle Ottawa Scale-Education (NOS-E) 38   are two commonly used instruments, both of which have an extensive body of validity evidence to support the interpretation of their scores. Table 5 21 , 39   provides more detail regarding the MERSQI, which includes evaluation of study design, sampling, data type, validity, data analysis, and outcomes. We have found that applying the MERSQI to manuscripts, articles, and protocols has intrinsic educational value, because this practice of application familiarizes MERSQI users with fundamental principles of medical education research. One aspect of the MERSQI that deserves special mention is the section on evaluating outcomes based on Kirkpatrick’s widely recognized hierarchy of reaction, learning, behavior, and results ( table 5 ; fig .). 40   Validity evidence for the scores of the MERSQI include its operational definitions to improve response process, excellent reliability, and internal consistency, as well as high correlation with other measures of study quality, likelihood of publication, citation rate, and an association between MERSQI score and the likelihood of study funding. 21 , 41   Additionally, consequence validity for the MERSQI scores has been demonstrated by its utility for identifying and disseminating high-quality research in medical education. 42  

Kirkpatrick’s hierarchy of outcomes as applied to education research. Reaction = Level 1, Learning = Level 2, Behavior = Level 3, Results = Level 4. Outcomes become more meaningful, yet more difficult to achieve, when progressing from Level 1 through Level 4. Adapted with permission from Beckman and Cook, 2007. 2  

The Medical Education Research Study Quality Instrument for Evaluating the Quality of Medical Education Research

The NOS-E is a newer tool to evaluate the quality of medication education research. It was developed as a modification of the Newcastle-Ottawa Scale 43   for appraising the quality of nonrandomized studies. The NOS-E includes items focusing on the representativeness of the experimental group, selection and compatibility of the control group, missing data/study retention, and blinding of outcome assessors. 38 , 39   Additional validity evidence for NOS-E scores includes operational definitions to improve response process, excellent reliability and internal consistency, and its correlation with other measures of study quality. 39   Notably, the complete NOS-E, along with its scoring rubric, can found in the article by Cook and Reed. 39  

A recent comparison of the MERSQI and NOS-E found acceptable interrater reliability and good correlation between the two instruments 39   However, noted differences exist between the MERSQI and NOS-E. Specifically, the MERSQI may be applied to a broad range of study designs, including experimental and cross-sectional research. Additionally, the MERSQI addresses issues related to measurement validity and data analysis, and places emphasis on educational outcomes. On the other hand, the NOS-E focuses specifically on experimental study designs, and on issues related to sampling techniques and outcome assessment. 39   Ultimately, the MERSQI and NOS-E are complementary tools that may be used together when evaluating the quality of medical education research.

Conclusions

This article provides an overview of quantitative research in medical education, underscores the main components of education research, and provides a general framework for evaluating research quality. We highlighted the importance of framing a study with respect to purpose, conceptual framework, and statement of study intent. We reviewed the most common research methodologies, along with threats to the validity of a study and its measurement instruments. Finally, we identified two complementary instruments, the MERSQI and NOS-E, for evaluating the quality of a medical education research study.

Bordage G: Conceptual frameworks to illuminate and magnify. Medical education. 2009; 43(4):312–9.

Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: Theory and application. The American journal of medicine. 2006; 119(2):166. e7–166. e116.

Franenkel JR, Wallen NE, Hyun HH: How to Design and Evaluate Research in Education. 9th edition. New York, McGraw-Hill Education, 2015.

Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB: Designing clinical research. 4th edition. Philadelphia, Lippincott Williams & Wilkins, 2011.

Irby BJ, Brown G, Lara-Alecio R, Jackson S: The Handbook of Educational Theories. Charlotte, NC, Information Age Publishing, Inc., 2015

Standards for Educational and Psychological Testing (American Educational Research Association & American Psychological Association, 2014)

Swanwick T: Understanding medical education: Evidence, theory and practice, 2nd edition. Wiley-Blackwell, 2013.

Sullivan GM, Artino Jr AR: Analyzing and interpreting data from Likert-type scales. Journal of graduate medical education. 2013; 5(4):541–2.

Sullivan GM, Feinn R: Using effect size—or why the P value is not enough. Journal of graduate medical education. 2012; 4(3):279–82.

Tavakol M, Sandars J: Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical teacher. 2014; 36(10):838–48.

Support was provided solely from institutional and/or departmental sources.

The authors declare no competing interests.

Citing articles via

Most viewed, email alerts, related articles, social media, affiliations.

• ASA Practice Parameters
• Online First
• Author Resource Center
• About the Journal
• Editorial Board
• Rights & Permissions
• Online ISSN 1528-1175
• Print ISSN 0003-3022
• Anesthesiology
• ASA Monitor

• Terms & Conditions Privacy Policy
• Manage Cookie Preferences
• © Copyright 2024 American Society of Anesthesiologists

This Feature Is Available To Subscribers Only

Sign In or Create an Account

• Research article
• Open access
• Published: 01 December 2006

Using quantitative and qualitative data in health services research – what happens when mixed method findings conflict? [ISRCTN61522618]

• Suzanne Moffatt 1 ,
• Martin White 1 ,
• Joan Mackintosh 1 &
• Denise Howel 1  

BMC Health Services Research volume  6 , Article number:  28 ( 2006 ) Cite this article

85k Accesses

128 Citations

16 Altmetric

Metrics details

In this methodological paper we document the interpretation of a mixed methods study and outline an approach to dealing with apparent discrepancies between qualitative and quantitative research data in a pilot study evaluating whether welfare rights advice has an impact on health and social outcomes among a population aged 60 and over.

Quantitative and qualitative data were collected contemporaneously. Quantitative data were collected from 126 men and women aged over 60 within a randomised controlled trial. Participants received a full welfare benefits assessment which successfully identified additional financial and non-financial resources for 60% of them. A range of demographic, health and social outcome measures were assessed at baseline, 6, 12 and 24 month follow up. Qualitative data were collected from a sub-sample of 25 participants purposively selected to take part in individual interviews to examine the perceived impact of welfare rights advice.

Separate analysis of the quantitative and qualitative data revealed discrepant findings. The quantitative data showed little evidence of significant differences of a size that would be of practical or clinical interest, suggesting that the intervention had no impact on these outcome measures. The qualitative data suggested wide-ranging impacts, indicating that the intervention had a positive effect. Six ways of further exploring these data were considered: (i) treating the methods as fundamentally different; (ii) exploring the methodological rigour of each component; (iii) exploring dataset comparability; (iv) collecting further data and making further comparisons; (v) exploring the process of the intervention; and (vi) exploring whether the outcomes of the two components match.

The study demonstrates how using mixed methods can lead to different and sometimes conflicting accounts and, using this six step approach, how such discrepancies can be harnessed to interrogate each dataset more fully. Not only does this enhance the robustness of the study, it may lead to different conclusions from those that would have been drawn through relying on one method alone and demonstrates the value of collecting both types of data within a single study. More widespread use of mixed methods in trials of complex interventions is likely to enhance the overall quality of the evidence base.

Combining quantitative and qualitative methods in a single study is not uncommon in social research, although, 'traditionally a gulf is seen to exist between qualitative and quantitative research with each belonging to distinctively different paradigms'. [ 1 ] Within health research there has, more recently, been an upsurge of interest in the combined use of qualitative and quantitative methods, sometimes termed mixed methods research [ 2 ] although the terminology can vary. [ 3 ] Greater interest in qualitative research has come about for a number of reasons: the numerous contributions made by qualitative research to the study of health and illness [ 4 – 6 ]; increased methodological rigor [ 7 ] within the qualitative paradigm, which has made it more acceptable to researchers or practitioners trained within a predominantly quantitative paradigm [ 8 ]; and, because combining quantitative and qualitative methods may generate deeper insights than either method alone. [ 9 ] It is now widely recognised that public health problems are embedded within a range of social, political and economic contexts. [ 10 ] Consequently, a range of epidemiological and social science methods are employed to research these complex issues. [ 11 ] Further legitimacy for the use of qualitative methods alongside quantitative has resulted from the recognition that qualitative methods can make an important contribution to randomised controlled trials (RCTs) evaluating complex health service interventions. There is published work on the various ways that qualitative methods are being used in RCTs (e.g. [ 12 , 13 ] but little on how they can optimally enhance the usefulness and policy relevance of trial findings. [ 14 , 15 ]

A number of mixed methods publications outline the various ways in which qualitative and quantitative methods can be combined. [ 1 , 2 , 9 , 16 ] For the purposes of this paper with its focus on mixed methods in the context of a pilot RCT, the significant aspects of mixed methods appear to be: purpose, process and, analysis and interpretation. In terms of purpose, qualitative research may be used to help identify the relevant variables for study [ 17 ], develop an instrument for quantitative research [ 18 ], to examine different questions (such as acceptability of the intervention, rather than its outcome) [ 19 ]; and to examine the same question with different methods (using, for example participant observation or in depth interviews [ 1 ]). Process includes the priority accorded to each method and ordering of both methods which may be concurrent, sequential or iterative. [ 20 ] Bryman [ 9 ] points out that, 'most researchers rely primarily on a method associated with either quantitative or qualitative methods and then buttress their findings with a method associated with the other tradition' (p128). Both datasets may be brought together at the 'analysis/interpretation' phase, often known as 'triangulation' [ 21 ]. Brannen [ 1 ] suggests that most researchers have taken this to mean more than one type of data, but she stresses that Denzin's original conceptualisation involved methods, data, investigators or theories. Bringing different methods together almost inevitably raises discrepancies in findings and their interpretation. However, the investigation of such differences may be as illuminating as their points of similarity. [ 1 , 9 ]

Although mixed methods are now widespread in health research, quantitative and qualitative methods and results are often published separately. [ 22 , 23 ] It is relatively rare to see an account of the methodological implications of the strategy and the way in which both methods are combined when interpreting the data within a particular study. [ 1 ] A notable exception is a study showing divergence between qualitative and quantitative findings of cancer patients' quality of life using a detailed case study approach to the data. [ 13 ]

By presenting quantitative and qualitative data collected within a pilot RCT together, this paper has three main aims: firstly, to demonstrate how divergent quantitative and qualitative data led us to interrogate each dataset more fully and assisted in the interpretation process, producing a greater research yield from each dataset; secondly, to demonstrate how combining both types of data at the analysis stage produces 'more than the sum of its parts'; and thirdly, to emphasise the complementary nature of qualitative and quantitative methods in RCTs of complex interventions. In doing so, we demonstrate how the combination of quantitative and qualitative data led us to conclusions different from those that would have been drawn through relying on one or other method alone.

The study that forms the basis of this paper, a pilot RCT to examine the impact of welfare rights advice in primary care, was funded under the UK Department of Health's Policy Research Programme on tackling health inequalities, and focused on older people. To date, little research has been able to demonstrate how health inequalities can be tackled by interventions within and outside the health sector. Although living standards have risen among older people, a common experience of growing old is worsening material circumstances. [ 24 ] In 2000–01 there were 2.3 million UK pensioners living in households with below 60 per cent of median household income, after housing costs. [ 25 ] Older people in the UK may be eligible for a number of income- or disability-related benefits (the latter could be non-financial such as parking permits or adaptations to the home), but it has been estimated that approximately one in four (about one million) UK pensioner households do not claim the support to which they are entitled. [ 26 ] Action to facilitate access to and uptake of welfare benefits has taken place outside the UK health sector for many years and, more recently, has been introduced within parts of the health service, but its potential to benefit health has not been rigorously evaluated. [ 27 – 29 ]

There are a number of models of mixed methods research. [ 2 , 16 , 30 ] We adopted a model which relies of the principle of complementarity, using the strengths of one method to enhance the other. [ 30 ] We explicitly recognised that each method was appropriate for different research questions. We undertook a pragmatic RCT which aimed to evaluate the health effects of welfare rights advice in primary care among people aged over 60. Quantitative data included standardised outcome measures of health and well-being, health related behaviour, psycho-social interaction and socio-economic status ; qualitative data used semi-structured interviews to explore participants' views about the intervention, its outcome, and the acceptability of the research process.

Following an earlier qualitative pilot study to inform the selection of appropriate outcome measures [ 31 ], contemporaneous quantitative and qualitative data were collected. Both datasets were analysed separately and neither compared until both analyses were complete. The sampling strategy mirrored the embedded design; probability sampling for the quantitative study and theoretical sampling for the qualitative study, done on the basis of factors identified in the quantitative study.

Approval for the study was obtained from Newcastle and North Tyneside Joint Local Research Ethics Committee and from Newcastle Primary Care Trust.

The intervention

The intervention was delivered by a welfare rights officer from Newcastle City Council Welfare Rights Service in participants' own homes and comprised a structured assessment of current welfare status and benefits entitlement, together with active assistance in making claims where appropriate over the following six months, together with necessary follow-up for unresolved claims.

Quantitative study

The design presented ethical dilemmas as it was felt problematic to deprive the control group of welfare rights advice, since there is adequate evidence to show that it leads to significant financial gains. [ 32 ] To circumvent this dilemma, we delivered welfare rights advice to the control group six months after the intervention group. A single-blinded RCT with allocation of individuals to intervention (receipt of welfare rights consultation immediately) and control condition (welfare rights consultation six months after entry into the trial) was undertaken.

Four general practices located at five surgeries across Newcastle upon Tyne took part. Three of the practices were located in the top ten per cent of most deprived wards in England using the Index of Multiple Deprivation (two in the top one percent – ranked 30 th and 36 th most deprived); the other practice was ranked 3,774 out of a total of 8,414 in England. [ 33 ]

Using practice databases, a random sample of 100 patients aged 60 years or over from each of four participating practices was invited to take part in the study. Only one individual per household was allowed to participate in the trial, but if a partner or other adult household member was also eligible for benefits, they also received welfare rights advice. Patients were excluded if they were permanently hospitalised or living in residential or nursing care homes.

Written informed consent was obtained at the baseline interview. Structured face to face interviews were carried out at baseline, six, 12 and 24 months using standard scales covering the areas of demographics, mental and physical health (SF36) [ 34 ], Hospital Anxiety and Depression Scale (HADS) [ 35 ], psychosocial descriptors (e.g. Social Support Questionnaire [ 36 ] and the Self-Esteem Inventory, [ 37 ], and socioeconomic indicators (e.g. affordability and financial vulnerability). [ 38 ] Additionally, a short semi-structured interview was undertaken at 24 months to ascertain the perceived impact of additional resources for those who received them.

All health and welfare assessment data were entered onto customised MS Access databases and checked for quality and completeness. Data were transferred to the Statistical Package for the Social Sciences (SPSS) v11.0 [ 39 ] and STATA v8.0 for analysis. [ 40 ]

Qualitative study

The qualitative findings presented in this paper focus on the impact of the intervention. The sampling frame was formed by those (n = 96) who gave their consent to be contacted during their baseline interview for the RCT. The study sample comprised respondents from intervention and control groups purposively selected to include those eligible for the following resources: financial only; non-financial only; both financial and non financial; and, none. Sampling continued until no new themes emerged from the interviews; until data 'saturation' was reached. [ 21 ]

Initial interviews took place between April and December 2003 in participants' homes after their welfare rights assessment; follow-up interviews were undertaken in January and February 2005. The semi-structured interview schedule covered perceptions of: impact of material and/or financial benefits; impact on mental and/or physical health; impact on health related behaviours; social benefits; and views about the link between material resources and health. All participants agreed to the interview being audio-recorded. Immediately afterwards, observational field notes were made. Interviews were transcribed in full.

Data analysis largely followed the framework approach. [ 41 ] Data were coded, indexed and charted systematically; and resulting typologies discussed with other members of the research team, 'a pragmatic version of double coding'. [ 42 ] Constant comparison [ 43 ] and deviant case analysis [ 44 ] were used since both methods are important for internal validation. [ 7 , 42 ] Finally, sets of categories at a higher level of abstraction were developed.

A brief semi-structured interview was undertaken (by JM) with all participants who received additional resources. These interview data explored the impact data of additional resources on all of those who received them, not just the qualitative sub-sample. The data were independently coded by JM and SM using the same coding frame. Discrepant codes were examined by both researchers and a final code agreed.

One hundred and twenty six people were recruited into the study; there were 117 at 12 month follow-up and 109 at 24 months (five deaths, one moved, the remainder declined).

Table 1 shows the distribution of financial and non-financial benefits awarded as a result of the welfare assessments. Sixty percent of participants were awarded some form of welfare benefit, and just over 40% received a financial benefit. Some households received more than one type of benefit.

Table 2 compares the quantitative and qualitative sub-samples on a number of personal, economic, health and lifestyle factors at baseline. Intervention and control groups were comparable.

Table 3 compares outcome measures by award group, i.e. no award, non-financial and financial and shows only small differences between the mean changes across each group, none of which were statistically significant. Other analyses of the quantitative data compared the changes seen between baseline and six months (by which time the intervention group had received the welfare rights advice but the control group had not) and found little evidence of differences between the intervention and control groups of any practical importance. The only statistically significant difference between the groups was a small decrease in financial vulnerability in the intervention group after six months. [ 45 ]

There was little evidence for differences in health and social outcomes measures as a result of the receipt of welfare advice of a size that would be of major practical or clinical interest. However, this was a pilot study, with only the power to detect large differences if they were present. One reason for a lack of difference may be that the scales were less appropriate for older people and did not capture all relevant outcomes. Another reason for the lack of differences may be that insufficient numbers of people had received their benefits for long enough to allow any health outcomes to have changed when comparisons were made. Fourteen per cent of participants found to be eligible for financial benefits had not started receiving their benefits by the time of the first follow-up interview after their benefit assessment (six months for intervention, 12 months for control); and those who had, had only received them for an average of 2 months. This is likely to have diluted any impact of the intervention effect, and might account, to some extent, for the lack of observed effect.

Twenty five interviews were completed, fourteen of whom were from the intervention group. Ten participants were interviewed with partners who made active contributions. Twenty two follow-up interviews were undertaken between twelve and eighteen months later (three individuals were too ill to take part).

Table 1 (fifth column) shows that 14 of the participants in the qualitative study received some financial award. The median income gain was (€84, $101) (range £10 (€15, $18) -£100 (€148, $178)) representing a 4%-55% increase in weekly income. 18 participants were in receipt of benefit, either as a result of the current intervention or because of claims made prior to this study.

By the follow-up (FU) interviews all but one participant had been receiving their benefits for between 17 and 31 months. The intervention was viewed positively by all interviewees irrespective of outcome. However, for the fourteen participants who received additional financial resources the impact was considerable and accounts revealed a wide range of uses for the extra money. Participants' accounts revealed four linked categories, summarised on Table 4 . Firstly, increased affordability of necessities , without which maintaining independence and participating in daily life was difficult. This included accessing transport, maintaining social networks and social activities, buying better quality food, stocking up on food, paying bills, preventing debt and affording paid help for household activities. Secondly, occasional expenses such as clothes, household equipment, furniture and holidays were more affordable. Thirdly, extra income was used to act as a cushion against potential emergencies and to increase savings . Fourthly, all participants described the easing of financial worries as bringing ' peace of mind' .

Without exception, participants were of the view that extra money or resources would not improve existing health problems. The reasons behind these strongly held views about individual health conditions was generally that their poor health was attributed to specific health conditions and a combination of family history or fate, which were immune to the effects of money. Most participants had more than one chronic condition and felt that because of these conditions, plus their age, additional money would have no effect.

However, a number of participants linked the impact of the intervention with improved ways of coping with their conditions because of what the extra resources enabled them to do:

Mrs T: Having money is not going to improve his health, we could win the lottery and he would still have his health problems.

Mr T: No, but we don't need to worry if I wanted .... Well I mean I eat a lot of honey and I think it's very good, very healthful for you ... at one time we couldn't have afforded to buy these things. Now we can go and buy them if I fancy something, just go and get it where we couldn't before .

Mrs T: Although the Attendance Allowance is actually his [partners], it's made me relax a bit more ... I definitely worry less now (N15, female, 62 and partner)

Despite the fact that no-one expected their own health conditions to improve, most people believed that there was a link between resources and health in a more abstract sense, either because they experienced problems affording necessities such as healthy food or maintaining adequate heat in their homes, or because they empathised with those who lacked money. Participants linked adequate resources to maintaining health and contributing to a sense of well-being.

Money does have a lot to do with health if you are poor. It would have a lot to do with your health ... I don't buy loads and loads of luxuries, but I know I can go out and get the food we need and that sort of thing. I think that money is a big part of how a house, or how people in that house are . (N13, female, 72)

Comparing the results from the two datasets

When the separate analyses of the quantitative and qualitative datasets after the 12 month follow-up structured interviews were completed, the discrepancy in the findings became apparent. The quantitative study showed little evidence of a size that would be of practical or clinical interest, suggesting that the intervention had no impact on these outcome measures. The qualitative study found a wide-ranging impact, indicating that the intervention had a positive effect. The presence of such inter-method discrepancy led to a great deal of discussion and debate, as a result of which we devised six ways of further exploring these data.

(i) Treating the methods as fundamentally different

This process of simultaneous qualitative and quantitative dataset interrogation enables a deeper level of analysis and interpretation than would be possible with one or other alone and demonstrates how mixed methods research produces more than the sum of its parts. It is worth emphasising however, that it is not wholly surprising that each method comes up with divergent findings since each asked different, but related questions, and both are based on fundamentally different theoretical paradigms. Brannen [ 1 ] and Bryman [ 9 ] argue that it is essential to take account of these theoretical differences and caution against taking a purely technical approach to the use of mixed methods, a simple 'bolting together' of techniques. [ 17 ] Combining the two methods for crossvalidation (triangulation) purposes is not a viable option because it rests on the premise that both methods are examining the same research problem. [ 1 ] We have approached the divergent findings as indicative of different aspects of the phenomena in question and searched for reasons which might explain these inconsistencies. In the approach that follows, we have treated the datasets as complementary, rather than attempt to integrate them, since each approach reflects a different view on how social reality ought to be studied.

(ii) Exploring the methodological rigour of each component

It is standard practice at the data analysis and interpretation phases of any study to scrutinise methodological rigour. However, in this case, we had another dataset to use as a yardstick for comparison and it became clear that our interrogation of each dataset was informed to some extent by the findings of the other. It was not the case that we expected to obtain the same results, but clearly the divergence of our findings was of great interest and made us more circumspect about each dataset. We began by examining possible reasons why there might be problems with each dataset individually, but found ourselves continually referring to the results of the other study as a benchmark for comparison.

With regard to the quantitative study, it was a pilot, of modest sample size, and thus not powered to detect small differences in the key outcome measures. In addition there were three important sources of dilution effects: firstly, only 63% of intervention group participants received some type of financial award; secondly, we found that 14% of those in the trial eligible for financial benefits did not receive their money until after the follow up assessments had been carried out; and thirdly, many had received their benefits for only a short period, reducing the possibility of detecting any measurable effects at the time of follow-up. All of these factors provide some explanation for the lack of a measurable effect between intervention and control group and between those who did and did not receive additional financial resources.

The number of participants in the qualitative study who received additional financial resources as a result of this intervention was small (n = 14). We would argue that the fieldwork, analysis and interpretation [ 46 ] were sufficiently transparent to warrant the degree of methodological rigour advocated by Barbour [ 7 , 17 ] and that the findings were therefore an accurate reflection of what was being studied. However, there still remained the possibility that a reason for the discrepant findings was due to differences between the qualitative sub-sample and the parent sample, which led us to step three.

(iii) Exploring dataset comparability

We compared the qualitative and quantitative samples on a number of social and economic factors (Table 2 ). In comparison to the parent sample, the qualitative sub-sample was slightly older, had fewer men, a higher proportion with long-term limiting illness, but fewer current smokers. However, there was nothing to indicate that such small differences would account for the discrepancies. There were negligible differences in SF-36 (Physical and Mental) and HAD (Anxiety and Depression) scores between the groups at baseline, which led us to discount the possibility that those in the quantitative sub sample were markedly different to the quantitative sample on these outcome measures.

(iv) Collection of additional data and making further comparisons

The divergent findings led us to seek further funding to undertake collection of additional quantitative and qualitative data at 24 months. The quantitative and qualitative follow-up data verified the initial findings of each study. [ 45 ] We also collected a limited amount of qualitative data on the perceived impact of resources, from all participants who had received additional resources. These data are presented in figure 1 which shows the uses of additional resources at 24 month follow-up for 35 participants (N = 35, 21 previously in quantitative study only, 14 in both). This dataset demonstrates that similar issues emerged for both qualitative and quantitative datasets: transport, savings and 'peace of mind' emerged as key issues, but the data also showed that the additional money was used on a wide range of items. This follow-up confirmed the initial findings of each study and further, indicated that the perceived impact of the additional resources was the same for a larger sample than the original qualitative sub-sample, further confirming our view that the positive findings extended beyond the fourteen participants in the qualitative sub-sample, to all those receiving additional resources.

Use of additional resources at 2 year follow up (N = 35)*.

(v) Exploring whether the intervention under study worked as expected

The qualitative study revealed that many participants had received welfare benefits via other services prior to this study, revealing the lack of a 'clean slate' with regard to the receipt of benefits, which we had not anticipated. We investigated this further in the quantitative dataset and found that 75 people (59.5%) had received benefits prior to the study; if the first benefit was on health grounds, a later one may have been because their health had deteriorated further.

(vi) Exploring whether the outcomes of the quantitative and qualitative components match

'Probing certain issues in greater depth' as advocated by Bryman (p134) [ 1 ] focussed our attention on the outcome measures used in the quantitative part of the study and revealed several challenges. Firstly, the qualitative study revealed a number of dimensions not measured by the quantitative study, such as, 'maintaining independence' which included affording paid help, increasing and improving access to facilities and managing better within the home. Secondly, some of the measures used with the intention of capturing dimensions of mental health did not adequately encapsulate participants' accounts of feeling 'less stressed' and 'less depressed' by financial worries. Probing both datasets also revealed congruence along the dimension of physical health. No differences were found on the SF36 physical scale and participants themselves did not expect an improvement in physical health (for reasons of age and chronic health problems). The real issue would appear to be measuring ways in which older people are better able to cope with existing health problems and maintain their independence and quality of life, despite these conditions.

Qualitative study results also led us to look more carefully at the quantitative measures we used. Some of the standardised measures were not wholly applicable to a population of older people. Mallinson [ 47 ] also found this with the SF36 when she demonstrated some of its limitations with this age group, as well as how easy it is to, 'fall into the trap of using questionnaires like a form of laboratory equipment and forget that ... they are open to interpretation'. The data presented here demonstrate the difficulties of trying to capture complex phenomena quantitatively. However, they also demonstrate the usefulness of having alternative data forms on which to draw whether complementary (where they differ but together generate insights) or contradictory (where the findings conflict). [ 30 ] In this study, the complementary and contradictory findings of the two datasets proved useful in making recommendations for the design of a definitive study.

Many researchers understand the importance, indeed the necessity, of combining methods to investigate complex health and social issues. Although quantitative research remains the dominant paradigm in health services research, qualitative research has greater prominence than before and is no longer, as Barbour [ 42 ] points out regarded as the 'poor relation to quantitative research that it has been in the past' (p1019). Brannen [ 48 ] argues that, despite epistemological differences there are 'more overlaps than differences'. Despite this, there is continued debate about the authority of each individual mode of research which is not surprising since these different styles, 'take institutional forms, in relation to cultures of and markets for knowledge' (p168). [ 49 ] Devers [ 50 ] points out that the dominance of positivism, especially within the RCT method, has had an overriding influence on the criteria used to assess research which has had the inevitable result of viewing qualitative studies unfavourably. We advocate treating qualitative and quantitative datasets as complementary rather than in competition for identifying the true version of events. This, we argue, leads to a position which exploits the strengths of each method and at the same time counters the limitations of each. The process of interpreting the meaning of these divergent findings has led us to conclude that much can be learned from scientific realism [ 51 ]which has 'sought to position itself as a model of scientific explanation which avoids the traditional epistemological poles of positivism and relativism' (p64). This stance enables investigators to take account of the complexity inherent in social interventions and reinforces, at a theoretical level, the problems of attempting to measure the impact of a social intervention via experimental means. However, the current focus on evidence based health care [ 52 ] now includes public health [ 53 , 54 ] and there is increased attention paid to the results of trials of public health interventions, attempting as they do, to capture complex social phenomena using standardised measurement tools. We would argue that at the very least, the inclusion of both qualitative and quantitative elements in such studies, is essential and ultimately more cost-effective, increasing the likelihood of arriving at a more thoroughly researched and better understood set of results.

The findings of this study demonstrate how the use of mixed methods can lead to different and sometimes conflicting accounts. This, we argue, is largely due to the outcome measures in the RCT not matching the outcomes emerging from the qualitative arm of the study. Instead of making assumptions about the correct version, we have reported the results of both datasets together rather than separately, and advocate six steps to interrogate each dataset more fully. The methodological strategy advocated by this approach involves contemporaneous qualitative and quantitative data collection, analysis and reciprocal interrogation to inform interpretation in trials of complex interventions. This approach also indicates the need for a realistic appraisal of quantitative tools. More widespread use of mixed methods in trials of complex interventions is likely to enhance the overall quality of the evidence base.

Brannen J: Mixing Methods: qualitative and quantitative research. 1992, Aldershot, Ashgate

Google Scholar  

Tashakkori A, Teddlie C: Handbook of Mixed Methods in Social and Behavioural Research. 2003, London, Sage

Morgan DL: Triangulation and it's discontents: Developing pragmatism as an alternative justification for combining qualitative and quantitative methods. Cambridge, 11-12 July.. 2005.

Pill R, Stott NCH: Concepts of illness causation and responsibility: some preliminary data from a sample of working class mothers. Social Science and Medicine. 1982, 16: 43-52. 10.1016/0277-9536(82)90422-1.

Article   CAS   PubMed   Google Scholar  

Scambler G, Hopkins A: Generating a model of epileptic stigma: the role of qualitative analysis. Social Science and Medicine. 1990, 30: 1187-1194. 10.1016/0277-9536(90)90258-T.

Townsend A, Hunt K, Wyke S: Managing multiple morbidity in mid-life: a qualitative study of attitudes to drug use. BMJ. 2003, 327: 837-841. 10.1136/bmj.327.7419.837.

Article   PubMed   PubMed Central   Google Scholar  

Barbour RS: Checklists for improving rigour in qualitative research: the case of the tail wagging the dog?. British Medical Journal. 2001, 322: 1115-1117.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Pope C, Mays N: Qualitative Research in Health Care. 2000, London, BMJ Books

Bryman A: Quantity and Quality in Social Research. 1995, London, Routledge

Ashton J: Healthy cities. 1991, Milton Keynes, Open University Press

Baum F: Researching Public Health: Behind the Qualitative-Quantitative Methodological Debate. Social Science and Medicine. 1995, 40: 459-468. 10.1016/0277-9536(94)E0103-Y.

Donovan J, Mills N, Smith M, Brindle L, Jacoby A, Peters T, Frankel S, Neal D, Hamdy F: Improving design and conduct of randomised trials by embedding them in qualitative research: (ProtecT) study. British Medical Journal. 2002, 325: 766-769.

Cox K: Assessing the quality of life of patients in phase I and II anti-cancer drug trials: interviews versus questionnaires. Social Science and Medicine. 2003, 56: 921-934. 10.1016/S0277-9536(02)00100-4.

Article   PubMed   Google Scholar  

Lewin S: Mixing methods in complex health service randomised controlled trials: research 'best practice'. Mixed Methods in Health Services Research Conference 23rd November 2004, Sheffield University.

Cresswell JW: Mixed methods research and applications in intervention studies.: ; Cambridge, July 11-12. 2005.

Cresswell JW: Research Design. Qualitative, Quantitative and Mixed Methods Approaches. 2003, London, Sage

Barbour RS: The case for combining qualitative and quanitative approaches in health services research. Journal of Health Services Research and Policy. 1999, 4: 39-43.

Gabriel Z, Bowling A: Quality of life from the perspectives of older people. Ageing & Society. 2004, 24: 675-691. 10.1017/S0144686X03001582.

Article   Google Scholar  

Koops L, Lindley RL: Thrombolysis for acute ischaemic stroke: consumer involvement in design of new randomised controlled trial. BMJ. 2002, 325: 415-418. 10.1136/bmj.325.7361.415.

O'Cathain A, Nicholl J, Murphy E: Making the most of mixed methods. Mixed Methods in Health Services Research Conference 23rd November 2004, Sheffield University.

Denzin NK: The Research Act. 1978, New York, McGraw-Hill Book Company

Roberts H, Curtis K, Liabo K, Rowland D, DiGuiseppi C, Roberts I: Putting public health evidence into practice: increasing the prevalance of working smoke alarms in disadvantaged inner city housing. Journal of Epidemiology & Community Health. 2004, 58: 280-285. 10.1136/jech.2003.007948.

Article   CAS   Google Scholar  

Rowland D, DiGuiseppi C, Roberts I, Curtis K, Roberts H, Ginnelly L, Sculpher M, Wade A: Prevalence of working smoke alarms in local authority inner city housing: randomised controlled trial. British Medical Journal. 2002, 325: 998-1001.

Vincent J: Old Age. 2003, London, Routledge

Chapter   Google Scholar  

Department for Work and Pensions: Households below average income statistics 2000/01. 2002, London, Department for Work and Pensions

National Audit Office: Tackling Pensioner Pverty: Encouraging take-up of entitlements. 2002, London, National Audit Office

Paris JAG, Player D: Citizens advice in general practice. British Medical Journal. 1993, 306: 1518-1520.

Abbott S: Prescribing welfare benefits advice in primary care: is it a health intervention, and if so, what sort?. Journal of Public Health Medicine. 2002, 24: 307-312. 10.1093/pubmed/24.4.307.

Harding R, Sherr L, Sherr A, Moorhead R, Singh S: Welfare rights advice in primary care: prevalence, processes and specialist provision. Family Practice. 2003, 20: 48-53. 10.1093/fampra/20.1.48.

Morgan DL: Practical strategies for combining qualitative and quantitative methods: applications to health research. Qualitative Health Research. 1998, 8: 362-376.

Moffatt S, White M, Stacy R, Downey D, Hudson E: The impact of welfare advice in primary care: a qualitative study. Critical Public Health. 2004, 14: 295-309. 10.1080/09581590400007959.

Thomson H, Hoskins R, Petticrew M, Ogilvie D, Craig N, Quinn T, Lindsey G: Evaluating the health effects of social interventions. British Medical Journal. 2004, 328: 282-285.

Department for the Environment TR: Measuring Multiple Deprivation at the Small Area Level: The Indices of Deprivation. 2000, London, Department for the Envrionment, Trransport and the Regions

Ware JE, Sherbourne CD: The MOS 36 item short form health survey (SF-36). Conceptual framework and item selection. Medical Care. 1992, 30: 473-481.

Snaith RP, Zigmond AS: The hospital anxiety and depression scale. Acta Psychiatrica Scandinivica. 1983, 67: 361-370.

Sarason I, Carroll C, Maton K: Assessing social support: the social support questionnaire. Journal of Personality and Social Psychology. 1983, 44: 127-139. 10.1037//0022-3514.44.1.127.

Ward R: The impact of subjective age and stigma on older persons. Journal of Gerontology. 1977, 32: 227-232.

Ford G, Ecob R, Hunt K, Macintyre S, West P: Patterns of class inequality in health through the lifespan: class gradients at 15, 35 and 55 years in the west of Scotland. Social Science and Medicine. 1994, 39: 1037-1050. 10.1016/0277-9536(94)90375-1.

SPSS.: v. 11.0 for Windows [program]. 2003, Chicago, Illinois.

STATA.: Statistical Software [program]. 8.0 version. 2003, Texas, College Station

Ritchie J, Lewis J: Qualitative Research Practice. A Guide for Social Scientists. 2003, London, Sage

Barbour RS: The Newfound Credibility of Qualitative Research? Tales of Technical Essentialism and Co-Option. Qualitative Health Research. 2003, 13: 1019-1027. 10.1177/1049732303253331.

Silverman D: Doing qualitative research. 2000, London, Sage

Clayman SE, Maynard DW: Ethnomethodology and conversation analysis. Situated Order: Studies in the Social Organisation of Talk and Embodied Activities. Edited by: Have PT and Psathas G. 1994, Washington, D.C., University Press of America

White M, Moffatt S, Mackintosh J, Howel D, Sandell A, Chadwick T, Deverill M: Randomised controlled trial to evaluate the health effects of welfare rights advice in primary health care: a pilot study. Report to the Department of Health, Policy Research Programme. 2005, Newcastle upon Tyne, University of Newcastle upon Tyne

Moffatt S: "All the difference in the world". A qualitative study of the perceived impact of a welfare rights service provided in primary care. 2004, , University College London

Mallinson S: Listening to reposndents: a qualitative assessment of the Short-Form 36 Health Status Questionnaire. Social Science and Medicine. 2002, 54: 11-21. 10.1016/S0277-9536(01)00003-X.

Brannen J: Mixing Methods: The Entry of Qualitative and Quantitative Approaches into the Research Process. International Journal of Social Research Methodology. 2005, 8: 173-184. 10.1080/13645570500154642.

Green A, Preston J: Editorial: Speaking in Tongues- Diversity in Mixed Methods Research. International Journal of Social Research Methodology. 2005, 8: 167-171. 10.1080/13645570500154626.

Devers KJ: How will we know "good" qualitative research when we see it? Beginning the dialogue in Health Services Research. Health Services Research. 1999, 34: 1153-1188.

CAS   PubMed   PubMed Central   Google Scholar  

Pawson R, Tilley N: Realistic Evaluation. 2004, London, Sage

Miles A, Grey JE, Polychronis A, Price N, Melchiorri C: Current thinking in the evidence-based health care debate. Journal of Evaluation in Clinical Practice. 2003, 9: 95-109. 10.1046/j.1365-2753.2003.00438.x.

Pencheon D, Guest C, Melzer D, Gray JAM: Oxford Handbook of Public Health Practice. 2001, Oxford, Oxford University Press

Wanless D: Securing Good Health for the Whole Population. 2004, London, HMSO

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1472-6963/6/28/prepub

Download references

Acknowledgements

We wish to thank: Rosemary Bell, Jenny Dover and Nick Whitton from Newcastle upon Tyne City Council Welfare Rights Service; all the participants and general practice staff who took part; and for their extremely helpful comments on earlier drafts of this paper, Adam Sandell, Graham Scambler, Rachel Baker, Carl May and John Bond. We are grateful to referees Alicia O'Cathain and Sally Wyke for their insightful comments. The views expressed in this paper are those of the authors and not necessarily those of the Department of Health.

Author information

Authors and affiliations.

Public Health Research Group, School of Population & Health Sciences, Faculty of Medical Sciences, William Leech Building, Framlington Place, Newcastle upon Tyne, NE2 4HH, UK

Suzanne Moffatt, Martin White, Joan Mackintosh & Denise Howel

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Suzanne Moffatt .

Additional information

Competing interests.

The author(s) declare that they have no competing interests.

Authors' contributions

SM and MW had the original idea for the study, and with the help of DH, Adam Sandell and Nick Whitton developed the proposal and gained funding. JM collected the data for the quantitative study, SM designed and collected data for the qualitative study. JM, DH and MW analysed the quantitative data, SM analysed the qualitative data. All authors contributed to interpretation of both datasets. SM wrote the first draft of the paper, JM, MW and DH commented on subsequent drafts. All authors have read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions.

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Moffatt, S., White, M., Mackintosh, J. et al. Using quantitative and qualitative data in health services research – what happens when mixed method findings conflict? [ISRCTN61522618]. BMC Health Serv Res 6 , 28 (2006). https://doi.org/10.1186/1472-6963-6-28

Download citation

Received : 29 September 2005

Accepted : 08 March 2006

Published : 01 December 2006

DOI : https://doi.org/10.1186/1472-6963-6-28

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mixed Method
• Welfare Benefit
• Mixed Method Research
• Divergent Finding
• Financial Vulnerability

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 25 August 2022

A quantitative study of nurses perception to advance directive in selected private and public secondary healthcare facilities in Ibadan, Nigeria

• Oluwaseyi Emiola Ojedoyin 1 &
• Ayodele Samuel Jegede 1  

BMC Medical Ethics volume  23 , Article number:  87 ( 2022 ) Cite this article

9618 Accesses

1 Citations

3 Altmetric

Metrics details

The study evaluated nurses’ perceptions on the benefits, drawbacks, and their roles in initiating and implementing advance directives (AD) at private and public secondary healthcare units.

The study adopted a cross-sectional, comparative-descriptive research design and was anchored on the structural functional theory. A total of 401 nurses (131 private and 270 public) were chosen on purpose. The data was collected between January and March 2018 among nurses at the selected hospitals. Analysis was done via SPSSv28.0.1.0.

Compared to nurses working in private healthcare facilities (72.5%), the majority of nurses at the public healthcare facilities (75.2%) indicated a more favorable opinion of AD’s benefits and (61.9%) felt they had a substantial involvement in the development and execution of AD than their private counterpart (56.5%). Similarly, 60.7% of nurses employed by the government agreed that AD has some disadvantages compared to those employed by the private sector (58.8%). Significantly, Christian nurses are 0.53 times less likely than Muslims to contest AD’s benefits; 0.78 times less likely than Muslim to disagree that AD has flaws; and 1.30 times more likely than Muslim nurses to deny they contributed to the development and execution of AD, though not significant.

Making decisions at the end-of-life can be challenging, thus AD should be supported across the board in the healthcare industry. Nurses should be trained on their role in developing and implementing AD, as well as on its advantages and how to deal with its challenges.

Peer Review reports

Introduction

Humans are born with the fundamental right to life. Due to this, many view death as undesirable, and even healthcare professionals avoid the discussion [ 1 , 2 ]. However, death is an inevitable, natural occurrence that all patients with life limiting illnesses should be prepared for in order to minimize distress at the end stage of life. Advance care planning (APC) is a method of communicating intentions that allows patients to let their loved ones and healthcare providers know in advance how they would like to be treated. One strategy in APC that aid readiness for future illness-related incapacitation, patients’ autonomy and dignity is advance directive (AD). AD is a written document or spoken declaration that enables competent people to make and document their healthcare decisions in advance [ 3 , 4 , 5 , 6 ]. Although patient’s “written directives” is a helpful tool for determining their preferences, tradition still dominates in most Africa countries. AD is yet to be legalised in Nigeria [ 7 ]. However, patients verbally expressed their preferences of care to healthcare professionals, and some even name individuals to make treatment decisions on their behalf when they are incapacitated [ 7 ]. These do not only promote patient participation in EOL discussion but also mitigate the paternalistic aspect of Nigeria’s healthcare system [ 8 , 9 ].

The Nigeria healthcare unit is divided into 3—primary, secondary and tertiary. Healthcare facilities at each unit can be privately owned or publicly owned. The difference between the two hospitals are found in their governance—the former are owned and run by an individual or group of individuals while the later are managed and funded by the government. The secondary healthcare facilities—which was the focus in this study manage advanced medical conditions [ 10 ] and it had been shown that, private hospitals are mostly used by Nigerians [ 11 ]. Nurses at these two facilities play significant roles in patients’ care. They provide medical, emotional, educational, patient-centered care and also serve as mediator between patients and doctors [ 12 , 13 ]. These put them in the best position to help in advance care planning—a procedure for communicating patients’ intentions [ 14 ]. Therefore, comparing the viewpoints of these nurses regarding AD will help to determine how end-of-life care is provided at this healthcare unit. There is a paucity of data on nurses’ perceptions of AD in Nigeria, and no study has described nurses’ perceptions at both private and public secondary healthcare facilities to the best of our knowledge. Previous researches focused on patient perception of AD and advocacy for AD inclusion in the country’s healthcare system [ 5 , 7 , 15 ]. This study therefore compared perceptions of nurses at the private and public secondary healthcare facilities on the advantages, roles and shortcoming of ADs in Ibadan, Oyo state, Nigeria.

Theoretical orientation

Structural–functional model.

A sociological theory known as functionalism views society as an organism of several elements (social institutions) that work together to maintain and reproduce the society [ 16 ]. These social institutions are typical means by which a society can attend to and satisfy both its social and individual needs. For instance, hospital is a social institution with many healthcare professionals collaborating to provide the best possible healthcare services to the community. Social institutions are also examined by functionalists in terms of the roles they played. Hence, to comprehend every part of society (e.g. doctor, nurse, teacher, AD, etc.) and how they affect social cohesion, reproduction, or the effective operation of a larger community, the functions of such institutions, beliefs, or ideologies are taken into considerations.

Merton however proposed that not all structure, custom, religion, ideology etc., serves positive purposes because they may serve both manifest and latent functions [ 17 ]. The latent functions are elements of behaviour or functions that are not openly declared, recognised, desired or intended. While the manifest functions are elements of conduct or functions that are conscious and purposefully [ 17 ]. Both the latent and manifest functions of AD was examined in the present study.

Research design

The study was a cross-sectional comparative-descriptive research design.

Participants

Nurses working in government-owned (public) and privately-owned secondary healthcare institutions as well as nursing students at the chosen hospitals participated in the survey.

Study location

The study was carried out in Ibadan, the Oyo state capital of Nigeria. Ibadan was deliberately chosen because it is Nigeria’s third-most populous city after Lagos and Kano, and because the region has historically had limited access to health care services [ 18 ]. Six out of eleven local government areas (LGAs) in Ibadan were chosen for this study—Ibadan Northeast, Ibadan Southwest, Ibadan Southeast, Ibadan North, and Egbeda. The high number of secondary health care facilities in these LGA coupled with the fact that no study on AD has been carried out among nurses in these locations were a deciding factors.

Sampling technique

A convenient non-probability sampling method was used to select nurses. This was employed due to the low staff strength, heavy workload and burnout on available staff. Five general hospitals and ten private secondary hospitals were included in the study—because of the high proportion of private secondary health facilities to public secondary health care facilities in the location and Nigeria as a whole [ 19 ]. A total of four hundred and one (401) nurses—270 nurses from public and 131 nurses from private hospitals—participated in the study.

Research instrument

Questionnaire was used to elicit information from respondents. Data was gathered in 2018 between January and March. The surveys were distributed to all nurses on-duty at their offices. A total of 430 survey was distributed out of which 401 was returned, making a 93% response rate. A total of 7% of the data was missing because several nurses worked night shifts, took the survey home, went on leave, and neglected to return the questionnaire.

The survey questions were developed after careful examination of literature from various countries [ 20 , 21 , 22 , 23 ]. Additionally, the opinions of three experts on prospective contents that required evaluation were sought. The questions’ ambiguity, relevance, clarity, and comprehensiveness were also evaluated. They assessed the questionnaire’s validity in terms of both face and content. The comments was examined, and the changes were added in the final survey. However, pilot survey was not conducted.

The questionnaire comprises two sections. The first section was on respondents’ socio-demographical characteristics. The second section was on perception and comprises 13 items—4 questions on benefits of AD, 5 questions on nurses’ roles in the initiation and implementation of AD, and 4 questions on shortcomings of AD. A 5-point likert scale was used to grade the responses of the participants ranging from strongly agree (5) to strongly disagree (0).

Data analysis

Data entry, cleaning, and analysis were performed using SPSS 28.0.1.0. Descriptive statistics was calculated for the socio-demographic and perception of nurses to AD. For questions on benefit of and nurses role in AD initiation and implementation, strongly agree and agree responses were merged to form correct perception to AD while, neutral, disagree and strongly disagree was merged as incorrect response. For questions on shortcomings of AD, strongly agree, agree and neutral responses were merged to form incorrect perception to AD while, disagree and strongly disagree was merged as correct response. The score for a correct response was two, while the score for an incorrect response was zero. The mean was calculated and response below the mean was considered as negative perceptions and those above or within the mean as positive perception.

On both the total benefits and drawbacks questions, 75% percentile (scoring three or more out of the four questions) was defined as positive perception, while 25% percentile (scored one out of the four questions) was labeled as negative view. The percentiles for the role of nurses in the initiation and implementation of AD were 60% (scoring 3 or more out of the 5 questions) and 40% (scored 2 or fewer out of the 5 questions). Differences between public and private nurses and nurses religion was examine using the odd ratios.

Reliability assessment of the questionnaire was conducted using Cronbach’s alpha coefficient based on Heden scale as cited in Peicus et al. [ 21 ] internal reliability assessment and recommendation. It stated that, a scale is reliable if the Cronbach’s alpha is > 5. The Cronbach alpha for the study is (0.62).

Ethical consideration

The Oyo State Research Ethics Review Committee, with reference number AD13/479/837, as well as administrative officers from each of the chosen hospitals and each participant, gave their approval before the data collection began.

Characteristic and representative of nurses in the study

The complete list of participants characteristics is shown in Table 1 below. The majority of respondents are women (88.9% public and 96.9% private). The majority (56.7%) of staff members at public hospitals hold diploma degrees, with one (0.4%) PhD degree holder. In contrast to the government hospitals, where 44.1% of participants had more than ten years of work experience, more than half (55.7%) of the private participants are within 1–5 years of work experience group. Predominant group are Yoruba (94%), Christians (79.3%) and more respondents from the public hospital (67.3%).

Distribution of nurses perception of benefits of advance directive

As shown in Table 2 below, most of the nurses agreed the AD is helpful when deciding how to treat patients (public-94.5% and private-93.1%); makes decision easier (public-88.2%, private-93.1%), minimize family conflict (public-85.9%, private-80.1%) and majority felt it reduced wasteful spending (public 77%, private 77%);

Perceived nurses role in advance directives

Majority agreed that nurses are crucial in educating about AD (public-78.1%, private-73.3%); in best position to access the appropriate time for end-of-life discussions (public-84.4%; private-78.6%) and are responsible to initiate end-of-life discussion (public-76%, private-57.3%). More participants in the private facilities than those at the public agreed that nurse can transfer a patient to another nurse when not comfortable with the directives.

Perceived shortcomings of advance directive

More participants in public (42.2%) than private (35.9%) disagreed that interpreting AD can be challenging. Two-thirds of private nurses (65.7%) and 55.5% of nurses in the public hospital agreed that AD can lead to requests for care not in the patient’s best interests. The little more than half of the participants felt AD might not accurately reflect patient’s current preferences (public-57.8%; private-52.7%) and uncertain (public-51.8%, private 39.7).

Classification of nurses responses into positive and negative perception

Table 3 shows how nurses generally perceived the benefits of AD, their involvement in its initiation and execution, and its perceived drawbacks. Majority of nurses in the public sector (75.2%) and private sector (72.5%) agreed AD is beneficial to patients, their families, and healthcare providers. More participants in the public sector (61.9%) than private (56.5%) thought they played a critical role in the development and implementation of AD. More nurses (60.7%) in the public sector concurred that AD had drawbacks than its private counterpart (58.8%).

Differences on nurses perception to advance directive

Table 4 below displays how Muslim nurses and Christian nurses perceive AD using odd ratios. Significantly, Christian nurses are 0.53 times less likely than Muslims to contest AD’s benefits; are 0.78 times less likely than Muslim to disagree that AD has flaws but are 1.30 times more likely than Muslim nurses to deny they contributed to the development and execution of AD, albeit, these differences are not statistically significant.

This study focused on nurses’ perceptions on the benefits, the role of nurses, and the negative aspects of AD at public and private secondary healthcare units in Ibadan, Nigeria. Positive perception regarding AD advantages was found among nurses at both public and private secondary healthcare units. This supported previous reported finding in Australia and Korea. According to these researches, AD guarantee patient autonomy, improve end-of-life care, and give patients a chance to reflect on their own dying stage and demise [ 22 , 23 , 24 ]. The study findings also agreed with prior researches where it was reported that the enforcement of ADs relieved families and patients’ financial, emotional weariness and disagreement [ 24 , 25 ] as we found that, participants agreed that AD can reduced needless stress, excessive spending and prevented or resolved conflict among healthcare practitioners, patients and patients relatives.

Nurses are more available at hospital and are closer to patients than any other healthcare practitioners. As a result, they agreed they are the best resource for patients and their families seeking information about AD. This support earlier researches in Portugal, Korea, New Zealand, and Australia [ 3 , 12 , 22 , 24 , 26 ]. The disparity reported on who is proficient in figuring out the appropriate time to initiate AD among the two group of nurses could be attributed to the quantity and quality of training enjoyed by these nurses. While more trainings are planned for nurses at the public sector, little of such training is available for nurses at the private sector in Nigeria. Davidson et al. also reported that nurses are in the best position to initiate AD [ 12 ]. On who should start the end-of-life conversation with a patient, the nurses at the two healthcare facilities had contrasting opinions. Nurses at private facilities saw it as the doctors’ obligation to begin and record the decision while they made the document readily available when needed, in contrast to nurses at public hospitals who saw it as their role. These was similar to findings in South Africa, Korea and Australia by Bull and Mash, Son et al., and Hobden et al. [ 24 , 27 , 28 ], where nurses saw themselves as the custodians of AD document rather than its initiators and/or implementers. The findings demonstrated that nurses in the private sector are more likely to refer patients whose orders they find objectionable to another nurse or facility. These both supports Siamak’s [ 23 ] findings that nurses have the autonomy to decline participation in the withdrawing or withholding of treatment if such a decision contradicts their personal and/or professional convictions and Hobden’s [ 27 ] findings where 60% of their study participants showed neutrality or disagreement that ADs will still be adhered to even if the medical team does not agree with them. Fear of litigation and the fact that nurses at the public sector enjoyed more autonomy, employment security, and public reputation than those in the private sector are some contributing factors to this [ 26 ]. Making known and reporting violation of patients’ directives were found to be nurses’ responsibilities in the present study. This was in consistent with Hobden et al. [ 27 ] that found nurses play a key role in ensuring that patients’ preferences are honored throughout end-of-life care.

Regarding AD’s shortcomings, the study demonstrates consensus that AD has some degree of negativity, but to various degrees. Over 50% of the study participants in the two sectors agreed and are neutral on the statement that many ambiguous terms are frequently used in AD without enough context or justification thereby making it difficult to interpret. Previous researchers have also noted that unclear instructions and the use of ambiguous language could lead to misreading of patients’ preferences [ 20 , 21 , 27 , 29 ]. The fact that patients’ mostly give their directives verbally when they are critically ill and sometime by their relatives in Nigeria can also contribute to the misapprehension of the directives [ 7 , 30 ]. More public sector nurses thought it was challenging to prove that AD is certain and accurately reflect patients’ current preferences and this made its implementation challenging. Reasons could be because, patients’ decisions regarding their treatment preference evolved over the illness episode due to factors like finance, relative decision, religious beliefs among others. However, these changes may not have reflected in the patient AD or known to the patient proxy. These contributed to the controversy in its implementation. Thus, decisional conflict that results from translating a written order into practice has previously been identified as an obstacle to the application of AD [ 26 , 27 , 28 ]. Ernestina et al. [ 3 ] showed that AD can fail in practice if changes in patient personal value fail to reflect in the directive. Therefore, AD should be periodically addressed and revisited for timely updates [ 21 ]. More than half of the study participants agreed that, there are chances that patients will asked for treatment that is not in their best interests in their AD. This finding supported researches conducted in Queensland, Australia, and Korea where it was reported that AD inhibited medical personnel from providing ethically and medically appropriate treatment to patient [ 20 , 26 , 29 ]. Inadequate knowledge and wrong cultural preconceptions about health, illness and treatment among patients could contributed to this perception. This study has been able to support existing knowledge that religion affiliation influence perception to end-of-life care [ 31 , 32 ]. While more Christian nurses are optimistic on the benefits of AD than Muslim nurses, more Muslim nurses believed they have a role to play in its initiation and execution than their Christians counterpart and thought AD had lesser flaws than the Christians. One of the tenets of Islam is to work for this life as if you were going to live forever and strive for the afterlife as if you were going to die tomorrow [ 33 ]. The Holy Qur’an also instructs Muslims to prepare and strategize their affairs. These may have influenced their perception that they have a greater role to play in the planning and implementation of the patient’s AD and support for AD. The Christian religion also supports AD as useful because it aids patients to avoid unbeneficial treatment [ 34 ].

In line with the theoretical explanation, the study had demonstrated that although AD has some benefits, such as quick decision-making, conflict resolution, and the prevention of wasteful spending; nurses as members of the healthcare team have a role to play in its initiation and implementation of AD. However, AD does have certain unintended consequences, which are its drawbacks [ 17 ].

This study has added to the corpus of research by identifying the perception of AD at the secondary healthcare facility in Nigeria and the chance that it will be adopted by nurses, who make up the majority of healthcare professionals. The study is limited by the use of the Likert scale to score nurses’ perceptions, which might have inhibited participants from fully expressing their perspectives on the matter. Further research should look into the acceptance of AD among terminally ill patients and their families as well as the amount of abuse or improper inducement of AD among healthcare professionals in secondary and tertiary healthcare facilities in Nigeria.

Making decisions in the final stages of life might be challenging, however AD may make these challenges easier. As a result, AD should be acknowledged in all healthcare sectors as a tool capable of granting patients’ liberty and dignity. Both in the public secondary healthcare unit and the private unit, nurses play a vital role as care providers in the development and execution of patient Ads. However, some of the difficulties in implementing AD that have been identified in this study should be addressed by stakeholders, and nurses at both sectors should be provided with necessary training on how to avoid these difficulties.

Availability of data and materials

Due to confidentiality rules, the datasets created and/or analyzed for the current work are not publically accessible, but they are available from the corresponding author upon justifiable request.

Teixeira C, Ribeiro O, Fonseca A, Carvalho AS. Ethical decision making in intensive care units: A burnout risk factor? Results from a multicenter study conducted with physicians and nurses. J Med Ethics. 2014;40:97–103.

Article   Google Scholar  

Dijxhoorn AF, Brom L, Van der Linden Y, Leget C, Raijmakers N. Prevalence of burnout in healthcare professionals providing palliative care and the effect of interventions to reduce symptoms: a systematic literature review. Palliat Med. 2021;35(1):6–26.

Ernestina S, Maria N, Daniel S. Nurses’ perceptions of advance directives. researchgate.net/publication/ 2014;269:187–357.

Izumi S. Advance care planning: the nurse’s role. Am J Nurs. 2017;117(6):56–61.

Ekore RI, Lanre-Abass B. African cultural concept of death and the idea of advance care directives. Indian J Palliat Care. 2016;22(4):369–72.

American Cancer Society (ACS). Types of advance directives. 2015. Retrieved from http://www.cancer.org/treatment/findingandpayingfortreatment/understandingfinancialandlegalmatters/advancedirectives/advance-directives-types-of-advancehealth-care-directives

Jegede AS, Adegoke OO. Advance directive in end-of-life decision-making among the Yoruba of South-Western Nigeria. BEOnline J West Afr Bioethics Train Prog. 2016;3(3):41–67.

Google Scholar  

Hong JH, Kwon JH, Kim IK, Ko JH, Kang YJ, Kim HK. Adopting advance directives reinforces patient participation in end-of-life care discussion. Cancer Res Treat. 2016;48(2):753–8. https://doi.org/10.4143/crt.2015.281 .

Innocent EO, Uche OA, Uche IB. Building a solid health care system in Nigeria: challenges and prospects. Acad J Interdiscip Stud. 2014;3(6):501–10. https://doi.org/10.5901/ajis.2014.v3n6p501 .

Koce F, Randhawa G, Ochieng B. Understanding healthcare self-referral in Nigeria from the service users’ perspective: a qualitative study of Niger state. BMC Health Serv Res. 2019;19:209.

Ofoli JNT, Ashau-Oladipo T, Hati SS, et al. Preventive healthcare uptake in private hospitals in Nigeria: a cross-sectional survey (Nisa premier hospital). BMC Health Serv Res. 2020;20:273.

Davidson R, Banister E, de Vries K. Primary healthcare NZ nurses’ experiences of advance directives: understanding their potential role. Nurs Prax N Z. 2013;29(2):26–33.

Oldland E, Botti M, Hutchinson AM, Redley B. A framework of nurses’ responsibilities for quality healthcare—exploration of content validity. Collegian. 2020;27(2):150–63.

Ke LS, Huang X, O’Connor M, Lee S. Nurses’ views regarding implementing advance care planning for older people: a systematic review and synthesis of qualitative studies. J Clin Nurs. 2015;24(15–16):2057–73.

Nguyen KT. Exploring patient perspective at end of life: qualitative interviews with terminally Ill patients. Yale; 2012.

Amboro JL, Wahyuningsih N, Prameswari NS. A structural functionalism point of view: ritual ceremony philosophy of Grebeg Maulud at Kasunanan Palace, Surakarta. J Adv Humanit Soc Sci. 2020;6(2):71–7.

Longhofer W, Winchester D. Social theory re-wired: new connections to classical and contemporary perspective. 2nd ed. New York: Routledge; 2016.

Book   Google Scholar  

C. A. Wojuade and S. O. Fadare. Research on Humanities and Social Sciences. 2014;4(6). www.iiste.org ISSN (Paper)2224-5766 ISSN (Online)2225-0484 (Online).

Makinde OA, Sule A, Ayankogbe O, Boone D. Distribution of health facilities in Nigeria: implications and options for Universal Health Coverage. Int J Health Plan Manag. 2018;33(4):e1179–92. https://doi.org/10.1002/hpm.2603 .

Lindy W, Ben W, Cheryl T, Jill W, Kelly P. Advance health directives: competing perceptions, intentions and use by patients and doctors in Queensland. QUT Law Rev. 2013;13(1):520–32.

Peicius E, Blazeviciene A, Kaminskas R. are advance directive helpful for good end of life decision making: a cross sectional survey of health professionals. BMC Med Ethnics. 2017;18(1):40.

Johnson C, Singer R, Masso M, Sellars M, Silvester W. Palliative care health professionals’ experiences of caring for patients with advance care directives. Aust Health. 2015;39(2):154–9.

Siamak NN. Advance medical directives (living will, power of attorney and health-care proxy). Medicine Net. 2016

Son Y, Choib J, Ahn J. Nurses’ perspectives on advance directives before the establishment of the new well-dying law in Korea: a mixed methods study. Appl Nurs Res. 2019;51:151187.

Donnelly LK. Increasing advance directive knowledge among healthcare professionals. Family Medicine Block Clerkship, Student Projects. 2017;238.

Poveda-Mora S, Rodriguez-Martin D, Codern-Bove N, Jose-Maria P, Sanchez-Valero P, Pomares-Quintana N, et al. Managing ethical aspects of advance directives in emergency care services. Nurs Ethics. 2021;28(1):91–105. https://doi.org/10.1177/0969733020952112 .

Hobden B, Bryant J, Waller A, Shepherd J, Sanson-Fisher R. Oncology nurses’ perceptions of advance directives for patients with cancer. Nurs Health Sci. 2021;23:1–8.

Bull A, Mash B. Advance Directives or Living Wills: Reflections of general practitioners and frail care coordinators in a small town in KwaZulu-Natal. S Afr Acad Fam Pract Prim Care. 2014;54(6):507–12.

Moore N, Detering KM, Low T, Nolte L, Fraser S, Sellars M. Doctors’ perspectives on adhering to advance care directives when making medical decisions for patients: an Australian interview study. BMJ Open. 2019;9:e032638.

Fadare JO, Obimakinde AM, Olaogun DO, Afolayan JM, Olatunya O, Ogundipe KO. Perception of nurses about palliative care: Experience from South-West Nigeria. Ann Med Health Sci Res. 2014;4(5):723. https://doi.org/10.4103/2141-9248.141532 .

Abudari G, Hazeim H, Ginete A. Caring for terminally ill Muslim patients: LIVED experiences of non-Muslim nurses. Palliat Support Care. 2016;14:599–611. https://doi.org/10.1017/S1478951516000249 .

Seale C. The role of doctors’ religious faith and ethnicity in taking ethically controversial decisions during end-of-life care. J Med Ethics. 2010;36:677–82. https://doi.org/10.1136/jme.2010.036194 .

Khan FA. Khan. 2022. Religious teachings and reflections on advance directive—religious values and legal dilemmas in bioethics: an Islamic perspective, 30 Fordham Urb. L.J. 267. 2002. Available at: https://ir.lawnet.fordham.edu/ulj/vol30/iss1/1 .

Kaplan RL. Religion and advance medical directives: formulation and enforcement implications. University Of Illinois Law Review; 2016. Accessed from: https://polstfl.org/wp-content/uploads/2017/12/Religion-and-Advance-Directives.pd .

Download references

Acknowledgements

The researchers appreciate the effort and interest of all the nurses who took part in the study, as well as the thoughtful criticism provided by the anonymous reviewers.

The article was not funded in any way.

Author information

Authors and affiliations.

University of Ibadan, Ibadan, Nigeria

Oluwaseyi Emiola Ojedoyin & Ayodele Samuel Jegede

You can also search for this author in PubMed   Google Scholar

Contributions

OO made contributions to the concept, literature review, design, data collection, analysis, and findings discussion. AJ oversaw the study, contributed to the concept and design, and critically examined the report for key intellectual content. The final manuscript was read and approved by all writers.

Corresponding author

Correspondence to Oluwaseyi Emiola Ojedoyin .

Ethics declarations

Ethics approval and consent to participate.

The Oyo State Research Ethics Review Committee, with reference number AD13/479/837, as well as the administrative head in each of the chosen hospitals and each participant, all approved this study. Every approach used in the study complied with the rules and regulations established by the institutional Research Committee for research involving people. All individuals participated in the study provided their written, informed consent.

Consent for publication

Not applicable.

Competing interests

No potential conflicts of interest were disclosed by the authors with regard to the research, writing, or publication of this paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ojedoyin, O.E., Jegede, A.S. A quantitative study of nurses perception to advance directive in selected private and public secondary healthcare facilities in Ibadan, Nigeria. BMC Med Ethics 23 , 87 (2022). https://doi.org/10.1186/s12910-022-00825-5

Download citation

Received : 07 February 2022

Accepted : 16 August 2022

Published : 25 August 2022

DOI : https://doi.org/10.1186/s12910-022-00825-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Treatment preference
• End-of-life care
• Living will
• General hospital
• Private hospital

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals

You are here

• Volume 59, Issue 5
• Quantitative health impact assessment: current practice and future directions
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• J L Veerman ,
• J J Barendregt ,
• J P Mackenbach
• Department of Public Health, Erasmus MC-University Medical Centre Rotterdam, Netherlands
• Correspondence to:
 Mr J L Veerman
 Department of Public Health, Erasmus MC, PO Box 1738, 3000 DR Rotterdam, Netherlands; j.veermanerasmusmc.nl

Study objective: To assess what methods are used in quantitative health impact assessment (HIA), and to identify areas for future research and development.

Design: HIA reports were assessed for (1) methods used to quantify effects of policy on determinants of health (exposure impact assessment) and (2) methods used to quantify health outcomes resulting from changes in exposure to determinants (outcome assessment).

Main results: Of 98 prospective HIA studies, 17 reported quantitative estimates of change in exposure to determinants, and 16 gave quantified health outcomes. Eleven (categories of) determinants were quantified up to the level of health outcomes. Methods for exposure impact assessment were: estimation on the basis of routine data and measurements, and various kinds of modelling of traffic related and environmental factors, supplemented with experts’ estimates and author’s assumptions. Some studies used estimates from other documents pertaining to the policy. For the calculation of health outcomes, variants of epidemiological and toxicological risk assessment were used, in some cases in mathematical models.

Conclusions: Quantification is comparatively rare in HIA. Methods are available in the areas of environmental health and, to a lesser extent, traffic accidents, infectious diseases, and behavioural factors. The methods are diverse and their reliability and validity are uncertain. Research and development in the following areas could benefit quantitative HIA: methods to quantify the effect of socioeconomic and behavioural determinants; user friendly simulation models; the use of summary measures of public health, expert opinion and scenario building; and empirical research into validity and reliability.

• health impact assessment
• quantitative methods
• determinants

https://doi.org/10.1136/jech.2004.026039

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Funding: this work was sponsored by ZonMw, the Netherlands Institute of Health Sciences and the Foundation Vereniging Trustfonds Erasmus Universiteit Rotterdam.

Competing interests: none declared.

Linked Articles

• In this issue Common sense, the least common sense? Carlos Alvarez-Dardet John R Ashton Journal of Epidemiology & Community Health 2005; 59 341-341 Published Online First: 14 Apr 2005.

Read the full text or download the PDF:

• Open access
• Published: 03 September 2024

Enhanced train-the-trainer program for registered nurses and social workers to apply the founding principles of primary care in their practice: a pre-post study

• Marie-Eve Poitras 1 ,
• Yves Couturier 2 ,
• Emmauelle Doucet 1 ,
• Vanessa T. Vaillancourt 1 ,
• Gilles Gauthier 1 ,
• Marie-Dominique Poirier 1 ,
• Sylvie Massé 3 ,
• Catherine Hudon 1 ,
• Nathalie Delli-Colli 2 ,
• Dominique Gagnon 4 ,
• Emmanuelle Careau 5 ,
• Arnaud Duhoux 6 ,
• Isabelle Gaboury 1 ,
• Djamal Berbiche 7 ,
• Ali Ben Charif 8 ,
• Rachelle Ashcroft 9 ,
• Julia Lukewich 10 ,
• Aline Ramond-Roquin 11 ,
• Priscilla Beaupré 1 &
• Anaëlle Morin 1  

BMC Primary Care volume  25 , Article number:  322 ( 2024 ) Cite this article

Metrics details

A train-the-trainer approach can effectively support the integration of new practice standards for health and social services professionals. This study describes the effects of an enhanced train-the-trainer program to support registered nurses and social workers working in primary care clinics in their understanding of the fundamental principles of primary care.

We implemented an enhanced train-the-trainer program for registered nurses and social workers in six primary care clinics. We conducted a pre-post study using quantitative and qualitative data to assess trainers’ and trainees’ intention, commitment, and confidence in applying acquired knowledge.

We trained 11 trainers and 33 trainees. All the trainers and trainees were satisfied with the program. Trainers were less confident in their abilities as trainers following the training, especially regarding tailored coaching ( p  = 0.03). Trainees’ commitment to becoming familiar with the functioning of their clinic ( p  = 0.05) and becoming part of the team increased significantly ( p  = 0.01); however, their intention to use their knowledge decreased ( p  = 0.02). Trainers and trainees identified organizational and professional barriers that may explain the observed decrease.

An enhanced train-the-trainer program positively impacted registered nurses’ and social workers’ assimilation of the fundamental principles of primary care. Further research is needed to understand the long-term effects of train-the-trainer programs on primary care trainees and how these effects translate into patient care.

Peer Review reports

It is widely recognized that primary healthcare is the foundation of the healthcare system [ 1 ] and that its performance should be closely monitored. In Canada, most primary care clinics (PCCs) are based on the Patient Medical Home model and must offer comprehensive, interdisciplinary care [ 2 ]. PCCs are medical clinics grouping physicians collaborating with other healthcare professionals to improve access and quality of care [ 1 ]. Despite this interdisciplinary structure and the apparent direction of comprehensive care promoted by this new structure, the Canada Foundation for Innovation reported that, in Canada, access to care, integration and coordination of services, interprofessional collaboration and patient engagement [ 1 ] are not meeting targets. Failure to implement comprehensive and interdisciplinary care in primary care results from various barriers, including a lack of understanding of professionals’ roles, which limits the scope of practice, suboptimal work in interdisciplinary teams, and low application of best practices related to patient engagement. Furthermore, many professionals working in primary care clinics need to be trained in an interdisciplinary and patient-centred approach including close collaboration with family physicians. Registered nurses and social workers are professionals who lack training in a PCC context, which needs to be improved and included in their initial training [ 3 , 4 , 5 , 6 ].

Several Canadian provinces, professional associations, and primary care networks have tried to support integrating comprehensive and interprofessional care in PCCs. For example, guidelines and recommendations were published and distributed to primary care professionals [ 7 , 8 , 9 ]. In 2019, the Quebec Ministry of Health and Social Services collaborated with various experts to develop guidelines [ 10 , 11 ], one for registered nurses and one for social workers working with family physicians in PCCs. Those guidelines outlined the fundamental principles of comprehensive care in primary care: interdisciplinary collaboration, patient engagement, and the importance of practising a full scope to provide high-quality care. Those guidelines are for nursing and social services professionals new to primary care or wanting to improve their practice in line with primary care principles.

Guidelines are an effective knowledge transfer tool for disseminating information to various professionals. Guidelines ensure a common understanding of the vision and approach that needs to be operationalized in clinical practice. However, various implementation strategies, including training, are required to optimize guidelines’ reach and use. These strategies should also consider the specific contexts of PCCs, which are often geographically delocalized entities, including rural and remote ones, with an interprofessional dynamic based on patient needs [ 12 , 13 , 14 , 15 , 16 , 17 , 18 ]. The spread of clinical sites requires a teaching approach that can serve several professionals simultaneously [ 18 ]. Furthermore, including patient partners in training courses, mainly when intended for improving patient care practices, is also essential [ 19 ].

Among the educational interventions that enable several professionals to be trained on the same topic, including guidelines content, train-the-trainer (TTT) programs have shown effectiveness [ 20 , 21 , 22 , 23 ]. Pearce and collaborators (2012) [ 24 ] showed that TTT programs that combine different andragogical strategies promote knowledge acquisition and clinicians learning. However, how this approach must be used to support the appropriation of knowledge by registered nurses and social workers working in PCCs is still being determined [ 25 ]. It also needs to be clarified how sustained and high-intensity coaching during training affects knowledge acquisition. Additionally, patient engagement is a fundamental aspect of comprehensive care [ 26 ], but it needs to be adequately covered in the continuing education of primary care professionals [ 27 , 28 ].

Current literature fails to document the development, implementation, and effects of enhanced training for PCC professionals, which integrates patients as trainers training alongside clinical trainers and is not limited to a testimonial role. Therefore, evaluating a TTT program that incorporates patient engagement is crucial [ 12 , 16 , 20 , 21 , 29 ]. This article evaluates the effect of an enhanced TTT program promoting primary care’s founding principles on the knowledge, intention, commitment, and confidence of registered nurses and social workers working in PCCs.

We implemented an enhanced TTT program for registered nurses and social workers in six PCCs located in three various regions. Two PCCs were in an urban area, two in a suburban area, and two in a remote area. We conducted a pre-post study [ 30 ] to assess trainers’ and trainees’ intention, commitment and confidence in applying acquired knowledge. We used a qualitative approach to describe how the enhanced train-the-trainer program impacted trainees’ disciplinary, interprofessional, and patient engagement. The study’s design and protocol were co-created with stakeholders based on an integrated knowledge approach and are published elsewhere [ 31 ].

Trainers and trainees

Through each area’s governance representatives, we recruited at least one social worker and one registered nurse with a clinical coaching position. Three social workers and four registered nurses were recruited, hereafter referred to as clinical trainers. To avoid a power imbalance between clinical and patient trainers, we recruited two patient trainers for each area ( n  = 6) through healthcare organizations’ patient partnership offices or patient associations. Through each PCC’s governance representatives, we recruited 25 registered nurses and eight social workers to be trained by the clinical trainers, hereafter referred to as trainees.

Enhanced train-the-trainer intervention

To optimize the success of our intervention, we used the Knowledge-to-Action framework [ 32 ], which recommends involving relevant stakeholders in the creation process, adapting to the local context, and implementing a tailored intervention. A training development committee consisting of four principal investigators (two researchers (YC, MEP) and two patient partners (GG, MDP)) and three content experts (VTV, ED, LP) co-created the enhanced TTT program through an iterative process with stakeholders (patient partners, clinicians, decision-makers). The committee developed training content to support the development of knowledge of the four founding principles of primary care found in the guidelines: Role of PCCs in primary care trajectories, interdisciplinary collaboration, patient engagement, and the importance of occupying the full scope of practice to provide high-quality care. They also developed additional training content about andragogy and tailored coaching to support clinical and patient trainers in training trainees.

The development committee provided the trainers with two days (14 h) of in-person training that covered the role of PCCs in primary care trajectories, registered nurses’ scope of practice, social workers’ scope of practice, interprofessional collaboration in PCC, and patient engagement. Each trainer received a toolkit containing educational activities and a copy of both registered nurses’ and social workers’ guidelines as a reference during training. The development committee used several andragogical strategies during training: lectures, discussions, interactive quizzes using Poll Everywere software, reflexive exercise, association games, clinical cases, unguided/guided group discussions, myth buster game, testimonials, hands-on session videos of clinical vignettes, and role playing. Table  1 overviews the interdisciplinary and patient-oriented enhanced TTT program and post-training coaching [ 22 ].

The development committee organized monthly and on-demand professional co-development sessions to support trainers in developing and mastering their roles. For example, clinical trainers received additional customized training related to partnering with patients to better understand patients’ roles and the value of their experiential knowledge. The patient partners who co-lead this study provided one-on-one support to some patient trainers who had faced challenges fulfilling their roles. The development committee also met with trainers from each area one-on-one to support them with their preparation to train trainees. Finally, trainers continuously communicated with each other and research team members via an online community, email, or telephone.

After being trained by the development committee, the trainers from each area trained the recruited trainees for 4 to 6 h, according to the needs and characteristics of each PCC. To do so, the trainers from each area mapped out characteristics, services, and staff for each participating PCC. They also identified specific training needs through discussions with managers and clinicians. The trainers could tailor training duration, emphasize certain parts, and select appropriate andragogic strategies. However, they were not allowed to modify training content. Following training, the teams of trainers conducted tailored clinical coaching activities over a 6-month period to support the trainees in assimilating the clinical practice guidelines and implementing expected practices.

Training evaluation

We used the New World Kirkpatrick Model (Fig.  1 ) to evaluate the training program [ 33 ]. This model, composed of four levels of training measures (Reaction, Learning, Behavior and Results), is widely used to assess training programs and to maximize the transfer of learnings into behaviours and subsequent organizational or patient-oriented results [ 23 , 34 , 35 ]. For this study, we evaluated items related to level 1-Reaction, which refers to the degree to which participants find the training favorable, engaging, and relevant to their job, and level 2-Learning, which refers to the degree to which participants acquire the intended skills, confidence and commitment [ 33 ]. We also measured intention because it is a strong predictor of behaviour [ 36 , 37 ]. Intention corresponds to the degree to which a person has formulated conscious plans to perform or not perform a specific future behaviour [ 38 ]. Four methods of data collection were used to inform the evaluation.

Four levels of Kirkpatrick’s training evaluation model and data collection tools according to participant type

First, we collected quantitative pre and post-training data with Survey Monkey (California, United States of America) self-administrated questionnaires composed of modified items from Kirkpatrick [ 39 , 40 ] (available in Additional file 1 ). Trainers answered multiple choice questions describing their satisfaction regarding the enhanced training with 16 items formatted as a 5-point Likert scale (Level 1Reaction). They also expressed their confidence level (Level 2-Learning) for 25 items related to implementing the enhanced TTT program or the research project, training and coaching of trainees, and communication with stakeholders by completing a 5-point Likert scale. Trainees assessed their satisfaction with the training with a 16-item 5-point Likert scale (Level 1-Reaction), their confidence and commitment level (Level 2-Learning) for ten items related to their scope of practice, interprofessional collaboration, and patient engagement. Trainers’ and trainees’ intention to apply knowledge was assessed with a 10-point scale item [ 33 , 41 , 42 ].

Second, we collected qualitative data. Each trainer and trainee answered post-training open-ended questions to identify training strengths, opportunities for improvement, and elements that could impede the application of training knowledge. They also completed a sociodemographic questionnaire.

Third, we conducted post-training focus groups with trainees to gain an in-depth understanding of Level 2-Learning and how the enhanced TTT program has affected their intention, confidence, and commitment to integrate the content of the clinical practice guidelines in their practice, especially interprofessional collaboration and patient engagement.

Lastly, we documented the development of the enhanced TTT program and its effects by collecting qualitative data through logbooks. Trainers and development committee members noted their observations and thoughts during the project (e.g., the impact of the training and elements that enable or restrain assimilation of the trainer’s role).

Data analysis

First, we used descriptive statistics to present socio-demographic data. To describe the effects of the educational intervention, continuous dependent variables were analyzed using linear Mixed Models with SAS’s PROC MIXED, a generalization of a paired data model or more like a repeated measures ANOVA. Quantitative data analyses were conducted with SAS version 9.4. Items were considered as significant if ≤ 0.05.

Second, we used the NVivo software to manage qualitative data (logbooks, verbatim transcriptions from interviews). Three concurrent streams of qualitative analysis were used: condensation (e.g., transformation of raw data), presentation (e.g., narrative text) and verification of conclusions (e.g., going back to field notes) [ 43 ]. Both principal investigators, patient co-leads and two research agents carried out data analysis by exploring themes related to (1) Level 2 skills, confidence and commitment to use clinical practice guidelines, interprofessional collaboration and patient engagement and elements that enable or restrain it and (2) the perceived impacts of the education intervention on trainers and trainees. Finally, a sixth team member validated the emerging themes and final propositions.

Mixed data integration

The principal investigators (researchers and patient partners) presented the qualitative and quantitative data to the co-investigators of this study. We used qualitative data to interpret quantitative data. We also identified discrepancies or convergences between the data sources to understand the impacts of the enhanced TTT program on trainers and trainees.

Of the 13 trainers recruited, 11 were trained in October 2019 by the development committee. Two trainers did not receive the enhanced training and were removed from the study; one was given only three hours of training, while the other could not attend due to a health condition. Four men and nine women aged 51 ± 13 years were trained, including four registered nurses, three social workers, and six patient trainers. Trainers trained a total of 33 trainees between November 2019 and January 2020. Twenty-five registered nurses and eight social workers were trained. Table  2 shows the sociodemographic characteristics of the 11 trainers.

Level 1 – reaction

Table  3 presents the 16 items used to assess Level 1-Reaction (did the participant enjoy the training). Trainers were highly satisfied with the training (4.27/5 ± 0.79 (mean score)). They positively noted every item related to development committee skills (range of 4.27 to 4.64/5); the highest-rated item was their level of knowledge (4.64/5 ± 0.50). The qualitative data validated this as trainers identified dynamic delivery and ability to communicate as a strength of the training program as expressed by some:

The trainers are dynamic and friendly. While mastering their subject very well, they remain humble and listen to group participants (Patient trainer 2).

[referring to what he liked] Diversity in the delivery formats and ways of dealing with the [training] content. The dynamism of the trainers. Exchanges between the participants. Equipment planning (including accommodation and others) (Social worker 2).

The lowest rated item was « I feel able to apply what I have learned » (3.91/5 ± 0.51), and trainers did identify some potential improvements. For example, many felt that the training was too short:

[We would have needed] more time for group discussions… (Patient trainer 1).

The section on interprofessional collaboration is too heavy for the time available; the content could have been more focused and presented dynamically and interactively to provide stronger anchors. (Social worker 1)

The trainees were satisfied with both the training (3.14/5 ± 0.95 (mean rating)) and the competencies of the trainers (range of 3.66 to 3.9/5). Trainees appreciated the ability of trainers to keep the training interactive (3.90 ± 0.77) and to communicate (mean rating 3.90 ± 0.77). Trainees also enjoyed the multiple andragogical approaches used.

[One of the strong points was] the diversity of teaching methods making the training more dynamic (Nurse 18).

Several trainees appreciated the richness of the discussions and the sharing with other trainees and trainers. Regarding the module on primary care and the roles of PCCs in the care service trajectories, one trainee also appreciated learning about the realities of other professionals from different backgrounds and regions.

It was interesting to get together and learn about what is being done elsewhere in Quebec (Nurse 5).

The qualitative data indicated that trainees felt the training did not focus enough on the clinical practice guidelines and how to use them to support their practice development. Some participants reported that these guidelines introduced new concepts, and they would have benefited if trainers had referred to the guide more often.

There was much time [during the training] to introduce what a PCC is, much time for the nurses, and there wasn’t enough time at the end to discuss the points I had read in the [practice] guidelines. I would have liked more hands-on time on the action plan than on things we’re already doing that are not new concepts to us. (Nurse 10)

Finally, some trainees reported that the enhanced TTT program met a need for interprofessional team support to resolve specific issues already known in the PCCs, as explained by one of them:

We have key resources to help us address issues we’ve been trying to work on for years. The fact that it gives us a common language. (Nurse 14)

Level 2 – learning (intention, confidence, and commitment)

We used 26 items to assess Level 2-Learning (did trainers acquire the intended intention, confidence, and commitment) of the New World Kirkpatrick Model.

Overall, analyses revealed that trainers were less confident in their abilities as trainers for almost every item related to implementing the enhanced TTT program or the research project itself, training and coaching of trainees, and communication with stakeholders (Table  4 ). Indeed, only one significant difference between pre- and post-training was observed, and it was related to their ability to tailor coaching ( p  = 0.03). The qualitative data allowed us to identify elements that explain this result. Organizational challenges concerning the application of the training content persist despite the training and the various appropriation activities, as expressed by some trainers:

Lack of openness and reluctance to change operations to adhere to best practices (Trainer-Social worker 1).

[in terms of his ability to train] It all depends on the physicians’ requests [their willingness to collaborate on the project] (Trainer-Nurse 4).

I am concerned that I will not have enough time [to fulfill my role as a trainer within my current tasks] (Trainer-Nurse 2).

As reported in Table  4 , an increase for every item related to trainers’ level of confidence to train nurses and social workers in PCCs was observed post-training.

Table  5 shows the pre-post evaluation of trainees’ confidence and demonstrated a general increasing trend for each item measured except regarding management modalities of PCCs, which decreased ( p  = 0.03). The most significant trends were observed for the following items: integrating into the team ( p  = 0.08), exercising collaborative leadership ( p  = 0.06), and actively participating in analysis and problem-solving regarding the application of the guidelines ( p  = 0.09). Qualitative data highlighted some elements that hindered the improvement of their confidence. Four trainees justified their decreased level of confidence by the misunderstanding of their role, while others raised that suboptimal collaboration with governance hinders their confidence:

We need support [from managers] to bring changes in the vision of medical delegation about the fields of practice of nurses (Nurse 4).

Some collaborative relationships will need to be improved to implement the strategies [promoted through training] (Social Worker 2).

Every item evaluated in regard to trainees’ commitment to applying knowledge increased (Table  5 ). Trainees demonstrated a significant increase in their commitment to familiarizing themselves with the processes in their PCCs ( p  = 0.05) and integrating themselves into the team ( p  = 0.01). Trends also showed that they were more committed to taking on their role in the PCCs ( p  = 0.07) and actively participating in analyzing and resolving problems related to applying the new clinical practice guidelines ( p  = 0.08). Trainees identified some barriers to committing to applying knowledge, such as prioritization of clinical activities:

[My commitment] depends on my workload and the support [of my manager] to do it. (Nurse 4)

Qualitative data were inconsistent regarding trainees’ commitment in interprofessional collaborative practices, as evidenced by these excerpts:

To our surprise, the PCC nurses regularly meet to discuss various topics, including more complex patient cases. They spontaneously included the social worker and perhaps will include the nutritionist in their next meetings. This reaction demonstrates that they understand the principle of putting the patient at the center and working in an interdisciplinary manner. So, for me, it’s mission accomplished! (Patient trainer 2)

The process of referring patients to nurses and social workers is an issue. Physicians and nurse practitioners do not know when or why to refer patients to nurses and social workers (Patient trainers).

Despite their high commitment, trainees reported decreased intention to apply the knowledge learned in the enhanced TTT program ( p  = 0.02) (Table  5 ). Two trainees reported priorities other than enhancing their professional practice, collaboration, and patient engagement, which influenced their intention.

This study, combining quantitative and qualitative pre- and post-data, aimed to evaluate the effect of an enhanced TTT program to increase the knowledge, intention, commitment, and confidence of trainers and trainees (registered nurses and social workers) working in PCCs in applying primary care founding principles in their practice. To our knowledge, this study is one of the first to describe the effects of an enhanced TTT program on registered nurses and social workers in primary care. The data presented shows that the enhanced TTT program is an effective way to improve knowledge but, according to Kirkpatrick’s level of learning, has more mixed effects on some items related to the intention and confidence of clinical trainers. The inclusion of patients as trainers, although essential, may be perceived at least as a barrier to be anticipated. These results lead us to the following observations.

First, the training seemed to improve future clinical trainers’ knowledge of working in primary care with registered nurses and social workers in PCCs, which is consistent with the literature [ 23 , 44 ]. Indeed, items related to the confidence level about the topics presented in the training increased (even if not significantly). However, the study described that evaluated items that decreased are related to the operationalization of the training and coaching or the research project itself.

We found that two elements could negatively influence trainers’ confidence in their roles. The first element is the discomfort of meeting medical and clinical managers to explain their role and the enhanced TTT program. This discomfort may be explained by the fact that this was a task that the clinical trainers and patient trainers had never done. The silo-based and medico-administrative views of the clinical support structure in PCCs may have influenced trainers’ perception of the importance of meeting with decision-making actors. However, these meetings are essential to promote adequate knowledge transfer and effectively initiate change using an integrated approach [ 32 , 45 , 46 ]. The innovators can anticipate this reluctance by offering direct support for these meetings and making them a specific training focus (how to deal with medical decision-makers). It remains essential, however, that trainers [ 47 ] become these vectors of change and promoters of innovation [ 44 ] and become visible to the clinicians in the settings in which they intervene. This presence legitimizes the role they play and facilitates the management of change, especially in geographically-dispersed or large-scale organizations [ 18 ].

The second element that could have negatively impacted the effectiveness of training is the inability of trainers to adequately tailor the intervention to the needs of the trainees and PCCs. Trainers must be aware of the unique context of each PCC and refrain from offering generic training, which would remove the innovative power of the intervention. Because we underestimated the skills needed to accomplish this task, the training offered did not adequately prepare trainers to adapt the training to variable contexts across PCCs [ 5 , 48 , 49 , 50 , 51 , 52 , 53 ] in an effective and tailored way. This tailoring requires a high level of competence and an excellent knowledge of the clinical settings [ 35 , 54 ], which may vary across trainers [ 35 ]. This could also explain the discrepancy between PCCs regarding trainee’s engagement. To address this issue, any primary care TTT program must include longitudinal activities to foster the development of trainers’ knowledge and confidence in applying their knowledge. These activities also prepare trainers to exercise leadership with clinical and decision-making stakeholders to improve learning outcomes [ 55 ]. We then could conclude that the training for trainers was appreciated and was appropriate for the content related to the four primary care founding principles but might not be appropriate for the one related to the implementation of the enhanced TTT program or the research project itself.

Second, at the beginning of this project, the clinical trainers proved that they needed to familiarize themselves with the role of patient trainers. Indeed, the active involvement of patients as trainers, and not just as witnesses of their life experiences, is infrequent and represents an innovation [ 56 ]. This ambitious collaborative vision requires an openness and a deep understanding of experiential knowledge [ 19 ], which characterizes a real integration of patient expertise as its knowledge. It positions the patient not as a recipient of care nor as a witness but as a full partner who contributes to improving the skills of professionals in clinical settings. Some authors also point out that clinical and patient trainers need support to integrate the patient trainers adequately to assume their full role [ 57 ]. Data collected from the clinical trainers, however, describe that their sense of discomfort faded when they observed the added value of the patient’s presence during their training and the training of trainees. However, despite the perceived added value of the patient trainer’s role in training clinicians, we found this alliance challenging to operationalize in PCCs. Combining clinical trainers’ schedules with patients’ schedules is a challenge, as the pace of work and availability are different. Including a new trainer in a team requires time to get to know and recognize each other [ 58 , 59 , 60 ], a key principle of interprofessional collaboration. Moreover, including a trainer from outside healthcare institutions may represent a culture shock for some clinical trainers. Therefore, it seems essential to prepare clinical trainers to collaborate with patient trainers and provide them with strategies to overcome the potential barriers.

Thirdly, for trainees, all items increased significantly (except for one) or with a trend. Trainees’ training appeared to be adequate despite being less appreciated. Although trainees obtained acceptable scores at levels 1 and 2 of Kirkpatrick’s model, these results do not indicate a long-term change in practice. Indeed, many studies show favorable results following TTT programs without observing sustainable changes in practice [ 22 , 23 ]. This is partly because changes are operationalized longitudinally, and ongoing support that extends beyond the training is needed. As training alone is not enough, trainers must be able to support trainees in making the training content their own. This support is more important in PCCs, where professionals feel professionally isolated from their peers for various reasons [ 18 , 44 , 61 ].

Strengths and limitations

This study is one of the first to include an enhanced TTT program provided by a team composed of clinicians and patients to support integrating new practice standards for health and social services professionals. Although the COVID-19 period delayed data collection up to six months after the intervention as planned, the presence of both quantitative and qualitative data allows us to make some suggestions. The triangulation of data sources and points of view brings richness and clarifications that explain the results.

Conclusions

The enhanced TTT program in a PCC, including patient trainers, is an innovative intervention centered on the very perspective of the Patient’s Medical Home and comprehensive care [ 2 ] promoted by many healthcare organizations. Our evaluation process supported our enhanced TTT program’s success as the initial goal was to help clinical trainees become more familiar with the four founding principles of primary care. The process used to create the enhanced TTT program, structure, and evaluation method can be used in other contexts. The crucial aspect is not the training content but how the TTT enables multiple professionals to receive training on different topics [ 24 ]. However, further work is needed to understand the long-term effects of enhanced TTT programs on primary care trainees and how these effects concretely translate into PCCs’ performance and patient care.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Primary care clinic

• Train-the-trainer

Fondation canadienne pour l’amélioration des services de santé. Vers une stratégie des soins primaires pour le Canada. Ottawa, Canada: Fondation canadienne pour l’amélioration des services de santé; 2012.

Google Scholar  

Patient’s Medical Home. The patient’s medical home is the future of family practice in Canada 2022. https://patientsmedicalhome.ca/ . [Accessed November 11 2021].

Barrett C, Mathews M, Poitras M-E, Norful AA, Martin-Misener R, Tranmer J, et al. Job titles and education requirements of registered nurses in primary care: an international document analysis. Int J Nurs Stud Adv. 2021;3:100044.

Article   PubMed   PubMed Central   Google Scholar  

Lukewich J, Poitras M-E, Mathews M. Unseen, unheard, undervalued: advancing research on registered nurses in primary care. Pract Nurs. 2021;32(4):158–62.

Article   Google Scholar  

Tadic V, Ashcroft R, Brown JB, Dahrouge S. The role of social workers in interprofessional primary healthcare teams. Healthc Policy = Politiques de sante. 2020;16(1):27–42.

Article   PubMed   Google Scholar  

Couturier Y, Lanoue S, Karam M, Guillette M, Hudon C. Social workers coordination in primary healthcare for patients with complex needs: a scoping review. Int J Care Coord. 2023;26(1):5–25.

Canadian Nurses Association. Interprofessional collaboration. Canada. p. 5.

College of Registered Nurses of Alberta. Interprofessional collaboration guidelines. Edmonton, Canada: College of Registered Nurses of Alberta 2023. p. 15.

Orchard C, Bainbridge L, Bassendowski S, Stevenson K, Wagner SJ, Weinberg L et al. A national interprofessional competency framework. 2010.

Ministère de la Santé et des Services Sociaux. Guide pratique à l’intention des infirmières cliniciennes qui travaillent dans un groupe de médecine de famille ou un groupe de médecine de famille universitaire. 2019.

Ministère de la Santé et des Services sociaux. Guide pratique à l’intention des travailleurs sociaux issus d’un établissement du réseau de la santé et des services sociaux et qui travaillent dans un groupe de médecine de famille ou un groupe de médecine de famille universitaire. 2019.

Seckler E, Regauer V, Rotter T, Bauer P, Müller M. Barriers to and facilitators of the implementation of multi-disciplinary care pathways in primary care: a systematic review. BMC Fam Pract. 2020;21(1):113.

McNeely J, Kumar PC, Rieckmann T, Sedlander E, Farkas S, Chollak C, et al. Barriers and facilitators affecting the implementation of substance use screening in primary care clinics: a qualitative study of patients, providers, and staff. Addict Sci Clin Pract. 2018;13(1):1–15.

Racic M, Pekez-Pavlisko T, Jokovic S. Barriers and facilitators for implementation of family medicine–oriented model of primary care in Bosnia and Herzegovina: a qualitative study. Int J Health Plann Manag. 2018;33(1):e378–90.

Benzer JK, Beehler S, Miller C, Burgess JF, Sullivan JL, Mohr DC, et al. Grounded theory of barriers and facilitators to mandated implementation of mental health care in the primary care setting. Depress Res Treat. 2012;2012:597157.

PubMed   PubMed Central   Google Scholar  

Bloch G, Rozmovits L. Implementing social interventions in primary care. Can Med Assoc J. 2021;193(44):E1696.

Rust C, Prior RM, Stec M. Implementation of a clinical practice guideline in a primary care setting for the prevention and management of obesity in adults. Nurs Forum. 2020;55(3):485–90.

Ohr SO, Maguire D, Lord N, Talluri C, Solman A. A unique model for developing continuing education resources for health professionals in geographically dispersed health organizations. J Continuing Educ Health Professions. 2021;41(3).

Pomey MP, Flora L, Karazivan P, Dumez V, Lebel P, Vanier MC, et al. [The Montreal model: the challenges of a partnership relationship between patients and healthcare professionals]. Sante publique (vandoeuvre-les-Nancy. France). 2015;27(1 Suppl):S41–50.

Bird M, Ouellette C, Whitmore C, Li L, Nair K, McGillion MH, et al. Preparing for patient partnership: a scoping review of patient partner engagement and evaluation in research. Health Expect. 2020;23(3):523–39.

Pomey M-P, Hihat H, Khalifa M, Lebel P, Néron A, Dumez V. Patient partnership in quality improvement of healthcare services: patients’ inputs and challenges faced. Patient Experience J. 2015;2(1):29–42.

Pearce J, Mann MK, Jones C, van Buschbach S, Olff M, Bisson JI. The most effective way of delivering a train-the‐trainers program: a systematic review. J Continuing Educ Health Professions. 2012;32(3):215–26.

Poitras ME, Bélanger E, Vaillancourt VT, Kienlin S, Körner M, Godbout I, et al. Interventions to improve trainers’ learning and behaviors for educating health care professionals using train-the-trainer method: a systematic review and meta-analysis. J Contin Educ Health Prof. 2021;41(3):202–9.

Pearce J, Mann MK, Jones C, van Buschbach S, Olff M, Bisson JI. The most effective way of delivering a train-the-trainers program: a systematic review. J Contin Educ Health Prof. 2012;32(3):215–26.

Cross V, Larkins S. Management of chronic hepatitis B infection in the remote primary health care setting: the search for a suitable guideline. Aust J Rural Health. 2011;19(2):95–100.

Krist AH, Tong ST, Aycock RA, Longo DR. Engaging patients in decision-making and behavior change to promote prevention. Studies in health technology and informatics. 2017;240:284–302.

Gillam S, Newbould J. Patient participation groups in general practice: what are they for, where are they going? BMJ. 2016;352.

Kiran T, Tepper J, Gavin F. Working with patients to improve care. Can Med Assoc J. 2020;192(6):E125–7.

Lau R, Stevenson F, Ong BN, Dziedzic K, Treweek S, Eldridge S, et al. Achieving change in primary care—effectiveness of strategies for improving implementation of complex interventions: systematic review of reviews. BMJ open. 2015;5(12):e009993.

Brousselle A, Champagne F, Contandriopoulos A-P, Hartz Z. L’évaluation: concepts et méthodes: Deuxième édition. Les Presses de l’Université de Montréal; 2011.

Poitras ME, Couturier Y, Massé S, Doucet E, T Vaillancourt V, Poirier M-D. Implementation and evaluation of a train-the-trainer strategy to support the sustainability of evidence-based practice guides for family practice nurses and social workers in family medicine groups: a study protocol. BMC Fam Pract. 2022.

Graham ID, Logan J, Fau - Harrison MB, Harrison Mb Fau - Straus SE, Straus Se Fau -, Tetroe J, Tetroe J, Fau - Caswell W, Caswell W, Fau - Robinson N et al. Lost in knowledge translation: time for a map? The Journal of Continuing Education in the Health. 2006;26(1):13–24.

Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation. Association for Talent Development; 2016.

Kirkpatrick J. An introduction to the new world Kirkpatrick ® model. 2015. Newnan, GA: Kirkpatrick Partners, LLC. 2017.

Kienlin S, Poitras ME, Stacey D, Nytrøen K, Kasper J. Ready for SDM: evaluating a train-the-trainer program to facilitate implementation of SDM training in Norway. BMC Med Inf Decis Mak. 2021;21(1):140.

Armitage CJ, Conner M. Efficacy of the theory of planned behaviour: a meta-analytic review. Br J Soc Psychol. 2001;40:471–99.

Article   CAS   PubMed   Google Scholar  

Godin G. Les comportements dans le domaine de la santé. Les presses de l’Université de Montréal; 2013.

Warshaw PR, Davis FD. Disentangling behavioral intention and behavioral expectation. J Exp Soc Psychol. 1985;21(3):213–28.

Kirkpatrick D, Kirkpatrick J. Evaluating training programs: the four levels. Berrett-Koehler; 2006.

Kirkpatrick DL. Implementing the four levels: A practical guide for effective evaluation of training programs: Easyread super large 24pt edition: ReadHowYouWant. com; 2009.

Bandura A. Guide for constructing self-efficacy scales. Self-efficacy Beliefs Adolescents. 2006;5(1):307–37.

Praslova L. Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assess Evaluation Account. 2010;22(3):215–25.

Huberman AM, Miles M, Saldana J. Qualitative data analysis: a methods sourcebook. The United States of America: SAGE; 2014.

Sator M, Holler P, Rosenbaum M. National train-the-trainer certificate programme for improving healthcare communication in Austria. Patient Educ Couns. 2021;104(12):2857–66.

Faija CL, Gellatly J, Barkham M, Lovell K, Rushton K, Welsh C, et al. Enhancing the behaviour change wheel with synthesis, stakeholder involvement and decision-making: a case example using the ‘Enhancing the quality of psychological interventions delivered by telephone’ (EQUITy) research programme. Implement Sci. 2021;16(1):53.

Maciver D, Hunter C, Johnston L, Forsyth K. Using stakeholder involvement, expert knowledge and naturalistic implementation to co-design a complex intervention to support children’s inclusion and participation in schools: the CIRCLE framework. Children (Basel, Switzerland). 2021;8(3).

Cranley LA, Cummings GG, Profetto-McGrath J, Toth F, Estabrooks CA. Facilitation roles and characteristics associated with research use by healthcare professionals: a scoping review. BMJ Open. 2017;7(8):e014384.

D’Amour D, Oandasan I. Interprofessionality as the field of interprofessional practice and interprofessional education: an emerging concept. J Interprof Care. 2005;19(Suppl 1):8–20.

Beaulieu MD, Denis JL, D’Amour D, Goudreau J, Haggerty J, Hudon E, et al. Montréal: Fondation canadienne de la recherche sur les services de santé. Fonds de la recherche en santé du Québec. Ministère de la Santé et des Services sociaux.; 2006. L’implantation des Groupes de médecine de famille: le défi de la réogranisation de la pratique et de la collaboration interprofessionnelle : Étude de cas de cinq GMF de la première vague au Québec.

Poitras ME, Chouinard MC, Fortin M, Girard A, Crossman S, Gallagher F. Nursing activities for patients with chronic disease in family medicine groups: a multiple-case study. Nurs Inq. 2018:e12250.

Poitras ME, Chouinard M-C, Gallagher F, Fortin M. The description of nursing activities for patients with chronic disease in primary care settings: a practice analysis. Nurs Res. 2018;67(1):35–42.

Bernier ACM, Lacroix M, Poitras M-E. The expected, enacted and desired role of family practice nurses in family medicine groups as perceived by nurses and family physicians. Sci Nurs Health Practices / Sci infirmière et pratiques en santé. 2020;3(2):1–15.

Fraser MW, Lombardi BM, Wu S, Zerden LS, Richman EL, Fraher EP. Integrated primary care and social work: a systematic review. J Soc Social Work Res. 2018;9(2):175–215.

Mormina M, Pinder S. A conceptual framework for training of trainers (ToT) interventions in global health. Globalization Health. 2018;14(1):1–11.

Bonnie LHA, Visser MRM, Kramer AWM, van Dijk N. Insight in the development of the mutual trust relationship between trainers and trainees in a workplace-based postgraduate medical training programme: a focus group study among trainers and trainees of the Dutch general practice training programme. BMJ open. 2020;10(4):e036593–e.

Morin A, Couturier Y, Poirier MD, Vaillancourt T, Massé V, Tardif SD. The impact of patients as trainers on registered nurses’ patient engagement in primary care clinics: a qualitative study. BMC Prim care. 2023;24(1):265.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Fraser C, Grundy A, Meade O, Callaghan P, Lovell K. EQUIP training the trainers: an evaluation of a training programme for service users and carers involved in training mental health professionals in user-involved care planning. J Psychiatr Ment Health Nurs. 2017;24(6):367–76.

Couturier Y, Belzile L. La collaboration interprofessionnelle en santé et services sociaux. Les Presses de l’Université de Montréal; 2018.

Reeves S, Pelone F, Harrison R, Goldman J, Zwarenstein M. Interprofessional collaboration to improve professional practice and healthcare outcomes. Cochrane Database Syst Reviews. 2017;(6).

Consortium pancanadien pour l’interprofessionalisme en santé. Référentiel national de compétences en matière d’interprofessionnalisme. 2010.

O’Donnell CA, Jabareen H, Watt GCM. Practice nurses’ workload, career intentions and the impact of professional isolation: a cross-sectional survey. BMC Nurs. 2010;9(1):2.

Download references

Acknowledgements

Not applicable.

This work was supported by the Fonds de Recherche du Québec en Santé under Grant number PCIBL-2.

Author information

Authors and affiliations.

Department of Family Medicine and Emergency Medicine, Université de Sherbrooke, 2500, boul. de l’Université, Sherbrooke, QC, J1K 2R1, Canada

Marie-Eve Poitras, Emmauelle Doucet, Vanessa T. Vaillancourt, Gilles Gauthier, Marie-Dominique Poirier, Catherine Hudon, Isabelle Gaboury, Priscilla Beaupré & Anaëlle Morin

School of Social Work, Université de Sherbrooke, 2500, boul. de l’Université, Sherbrooke, QC, J1K 2R1, Canada

Yves Couturier & Nathalie Delli-Colli

Integrated University Health and Social Services Centres of Saguenay–Lac-St-Jean, 305 Rue Saint Vallier, Chicoutimi, QC, G7H 5H6, Canada

Sylvie Massé

Human and Social Sciences Department, Université du Québec en Abitibi-Témiscamingue, 663 1re Avenue, Val-d’Or, QC, J9P 1Y3, Canada

Dominique Gagnon

Department of Rehabilitation, Université Laval, 1050 Av. de la Médecine, G1V 0A6, Québec, QC, Canada

Emmanuelle Careau

Faculty of Nursing, University of Montreal, 2375, Chemin de la Côte-Sainte-Catherine, Montréal, QC, H3T 1A8, Canada

Arnaud Duhoux

Charles-LeMoyne Research Center on Innovations in Health, Department of Community Health Sciences, Université de Sherbrooke, 2500, boul. de l’Université, Sherbrooke, QC, J1K 2R1, Canada

Djamal Berbiche

CubecXpert, Quebec, QC, Canada

Ali Ben Charif

Faculty of Social Work, University of Toronto, 246 Bloor Street W, Toronto, ON, M5S 1V4, Canada

Rachelle Ashcroft

Faculty of Nursing, Memorial University of Newfoundland, 100 Signal Hill, St. John’s, NL, A1C 5S7, Canada

Julia Lukewich

Department of General Practice and Laboratory of Ergonomics and Epidemiology in Occupational Health, University of Angers, 40 Rue de Rennes, Angers, 49100, France

Aline Ramond-Roquin

You can also search for this author in PubMed   Google Scholar

Contributions

MEP, YC, SM, MDP and GG are the principal investigators of this research. They were lead of the conception and design of the work. ED was the project manager and made substantial contribution to the study (acquisition, analysis, interpretation). VTV was involved in the design of the study, the acquisition and analysis and drafted the manuscript. PB and AM are research assistants and contributed to the acquisition and interpretation. CH, NDC, DG, EC, AD, IG, ABC, RA, JL, ARR, are co-investigators and were involved in the conception and design of the work. They substantially revised the manuscript. DB performed the quantitative analysis and wrote the corresponding section. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Marie-Eve Poitras .

Ethics declarations

Ethics approval and consent to participate.

We obtained ethical approval for the study from the Comité d’éthique et de la recherche du Saguenay-Lac-Saint-Jean , project number 2019-037 and informed consent was obtained from all participants.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1: Additional file 1

. Self-reported questionnaire

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Poitras, ME., Couturier, Y., Doucet, E. et al. Enhanced train-the-trainer program for registered nurses and social workers to apply the founding principles of primary care in their practice: a pre-post study. BMC Prim. Care 25 , 322 (2024). https://doi.org/10.1186/s12875-024-02574-3

Download citation

Received : 15 March 2024

Accepted : 16 August 2024

Published : 03 September 2024

DOI : https://doi.org/10.1186/s12875-024-02574-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Primary care
• Social worker
• Patient-oriented research

BMC Primary Care

ISSN: 2731-4553

• General enquiries: [email protected]

• Open access
• Published: 31 August 2024

Effects of pecha kucha presentation pedagogy on nursing students’ presentation skills: a quasi-experimental study in Tanzania

• Setberth Jonas Haramba 1 ,
• Walter C. Millanzi 1 &
• Saada A. Seif 2  

BMC Medical Education volume  24 , Article number:  952 ( 2024 ) Cite this article

1 Altmetric

Metrics details

Introduction

Ineffective and non-interactive learning among nursing students limits opportunities for students’ classroom presentation skills, creativity, and innovation upon completion of their classroom learning activities. Pecha Kucha presentation is the new promising pedagogy that engages students in learning and improves students’ speaking skills and other survival skills. It involves the use of 20 slides, each covering 20 seconds of its presentation. The current study examined the effect of Pecha Kucha’s presentation pedagogy on presentation skills among nursing students in Tanzania.

The aim of this study was to establish comparative nursing student’s presentation skills between exposure to the traditional PowerPoint presentations and Pecha Kucha presentations.

The study employed an uncontrolled quasi-experimental design (pre-post) using a quantitative research approach among 230 randomly selected nursing students at the respective training institution. An interviewer-administered structured questionnaire adopted from previous studies to measure presentation skills between June and July 2023 was used. The study involved the training of research assistants, pre-assessment of presentation skills, training of participants, assigning topics to participants, classroom presentations, and post-intervention assessment. A linear regression analysis model was used to determine the effect of the intervention on nursing students’ presentation skills using Statistical Package for Social Solution (SPSS) version 26, set at a 95% confidence interval and 5% significance level.

Findings revealed that 63 (70.87%) participants were aged ≤ 23 years, of which 151 (65.65%) and 189 (82.17%) of them were males and undergraduate students, respectively. Post-test findings showed a significant mean score change in participants’ presentation skills between baseline (M = 4.07 ± SD = 0.56) and end-line (M = 4.54 ± SD = 0.59) that accounted for 0.4717 ± 0.7793; p  < .0001(95%CI) presentation skills mean score change with a medium effect size of 0.78. An increase in participants’ knowledge of Pecha Kucha presentation was associated with a 0.0239 ( p  < .0001) increase in presentation skills.

Pecha Kucha presentations have a significant effect on nursing students’ presentation skills as they enhance inquiry and mastery of their learning content before classroom presentations. The pedagogical approach appeared to enhance nursing students’ confidence during the classroom presentation. Therefore, there is a need to incorporate Pecha Kucha presentation pedagogy into nursing curricula and nursing education at large to promote student-centered teaching and learning activities and the development of survival skills.

Trial registration

It was not applicable as it was a quasi-experimental study.

Peer Review reports

The nursing students need to have different skills acquired during the learning process in order to enable them to provide quality nursing care and management in the society [ 1 ]. The referred nursing care and management practices include identifying, analyzing, synthesizing, and effective communication within and between healthcare professionals [ 1 ]. Given an increasing global economy and international competition for jobs and opportunities, the current traditional classroom learning methods are insufficient to meet such 21st - century challenges and demands [ 2 ]. The integration of presentation skills, creativity, innovation, collaboration, information, and media literacy skills helps to overcome the noted challenges among students [ 2 , 3 , 4 ]. The skills in question constitute the survival skills that help the students not only for career development and success but also for their personal, social and public quality of life as they enable students to overcome 21st challenges upon graduation [ 2 ].

To enhance the nursing students’ participation in learning, stimulating their presentation skills, critical thinking, creativity, and innovation, a combination of teaching and learning pedagogy should be employed [ 5 , 6 , 7 , 8 ]. Among others, classroom presentations, group discussions, problem-based learning, demonstrations, reflection, and role-play are commonly used for those purposes [ 5 ]. However, ineffective and non-interactive learning which contribute to limited presentation skills, creativity, and innovation, have been reported by several scholars [ 9 , 10 , 11 ]. For example, poor use and design of student PowerPoint presentations led to confusing graphics due to the many texts in the slides and the reading of about 80 slides [ 12 , 13 , 14 ]. Indeed, such non-interactive learning becomes boring and tiresome among the learners, and it is usually evidenced by glazing eyes, long yawning, occasional snoring, the use of a phone and frequent trips to the bathroom [ 12 , 14 ].

With an increasing number of nursing students in higher education institutions in Tanzania, the students’ traditional presentation pedagogy is insufficient to stimulate their presentation skills. They limit nursing student innovation, creativity, critical thinking, and meaningful learning in an attempt to solve health challenges [ 15 , 16 ].These hinder nursing students ability to communicate effectively by being able to demonstrate their knowledge and mastery of learning content [ 17 , 18 ]. Furthermore, it affects their future careers by not being able to demonstrate and express their expertise clearly in a variety of workplace settings, such as being able to present at scientific conferences, participating in job interviews, giving clinic case reports, handover reports, and giving feedback to clients [ 17 , 18 , 19 ].

Pecha Kucha presentation is a new promising approach for students’ learning in the classroom context as it motivates learners’ self-directed and collaborative learning, learner creativity, and presentation skills [ 20 , 21 , 22 ]. It encourages students to read more materials, enhances cooperative learning among learners, and is interesting and enjoyable among students [ 23 ].

Pecha Kucha presentation originated from the Japanese word “ chit chat , ” which represents the fast-paced presentation used in different fields, including teaching, marketing, advertising, and designing [ 24 , 25 , 26 ]. It involves 20 slides, where each slide covers 20 s, thus making a total of 6 min and 40 s for the whole presentation [ 22 ]. For effective learning through Pecha Kucha presentations, the design and format of the presentation should be meaningfully limited to 20 slides and targeted at 20 s for each slide, rich in content of the presented topic using high-quality images or pictures attuned to the content knowledge and message to be delivered to the target audiences [ 14 , 16 ]. Each slide should contain a primordial message with well-balanced information. In other words, the message should be simple in the sense that each slide should contain only one concept or idea with neither too much nor too little information, thus making it easy to be grasped by the audience [ 14 , 17 , 19 ].

The “true spirit” of Pecha Kucha is that it mostly consists of powerful images and meaningful specific text rather than the text that is being read by the presenter from the slides, an image, and short phrases that should communicate the core idea while the speaker offers well-rehearsed and elaborated comments [ 22 , 28 ]. The presenter should master the subject matter and incorporate the necessary information from classwork [ 14 , 20 ]. The audience’s engagement in learning by paying attention and actively listening to the Pecha Kucha presentation was higher compared with that in traditional PowerPoint presentations [ 29 ]. The creativity and collaboration during designing and selecting the appropriate images and contents, rehearsal before the presentation, and discussion after each presentation made students satisfied by enjoying Pecha Kucha presentations compared with traditional presentations [ 21 , 22 ]. Time management and students’ self-regulation were found to be significant through the Pecha Kucha presentation among the students and teachers or instructors who could appropriately plan the time for classroom instruction [ 22 , 23 ].

However, little is known about Pecha Kucha presentation in nursing education in Sub-Saharan African countries, including Tanzania, since there is insufficient evidence for the research(s) that have been published on the description of its effects on enhancing students’ presentation skills. Thus, this study assessed the effect of Pecha Kucha’s presentation pedagogy on enhancing presentation skills among nursing students. In particular, the study largely focused on nursing students’ presentation skills during the preparation and presentation of the students’ assignments, project works, case reports, or field reports.

The study answered the null hypothesis H 0  = H 1, which hypothesized that there is no significant difference in nursing students’ classroom presentation skills scores between the baseline and end-line assessments. The association between nursing students’ presentation skills and participants’ sociodemographic characteristics was formulated and analyzed before and after the intervention. This study forms the basis for developing new presentation pedagogy among nursing students in order to stimulate effective learning and the development of presentation skills during the teaching and learning process and the acquisition of 21st - century skills, which are characterized by an increased competitive knowledge-based society due to changing nature and technological eruptions.

The current study also forms the basis for re-defining classroom practices in an attempt to enhance and transform nursing students’ learning experiences. This will cultivate the production of graduates nurses who will share their expertise and practical skills in the health care team by attending scientific conferences, clinical case presentations, and job interviews in the global health market. To achieve this, the study determined the baseline and end-line nursing students’ presentation skills during the preparation and presentation of classroom assignments using the traditional PowerPoint presentation and Pecha Kucha presentation format.

Methods and materials

This study was conducted in health training institutions in Tanzania. Tanzania has a total of 47 registered public and private universities and university colleges that offer health programs ranging from certificate to doctorate degrees [ 24 , 25 ]. A total of seven [ 7 ] out of 47 universities offer a bachelor of science in nursing, and four [ 4 ] universities offer master’s to doctorate degree programs in nursing and midwifery sciences [ 24 , 26 ]. To enhance the representation of nursing students in Tanzania, this study was conducted in Dodoma Municipal Council, which is one of Tanzania’s 30 administrative regions [ 33 ]. Dodoma Region has two [ 2 ] universities that offer nursing programs at diploma and degree levels [ 34 ]. The referred universities host a large number of nursing students compared to the other five [ 5 ] universities in Tanzania, with traditional students’ presentation approaches predominating nursing students’ teaching and learning processes [ 7 , 32 , 35 ].

The two universities under study include the University of Dodoma and St. John’s University of Tanzania, which are located in Dodoma Urban District. The University of Dodoma is a public university that provides 142 training programs at the diploma, bachelor degree, and master’s degree levels with about 28,225 undergraduate students and 724 postgraduate students [ 26 , 27 ]. The University of Dodoma also has 1,031 nursing students pursuing a Bachelor of Science in Nursing and 335 nursing students pursuing a Diploma in Nursing in the academic year 2022–2023 [ 33 ]. The St. John’s University of Tanzania is a non-profit private university that is legally connected with the Christian-Anglican Church [ 36 ]. It has student enrollment ranging from 5000 to 5999 and it provides training programs leading to higher education degrees in a variety of fields, including diplomas, bachelor degrees, and master’s degrees [ 37 ]. It hosts 766 nursing students pursuing a Bachelor of Science in Nursing and 113 nursing students pursuing a Diploma in Nursing in the academic year 2022–2023 [ 30 , 31 ].

Study design and approach

An uncontrolled quasi-experimental design with a quantitative research approach was used to establish quantifiable data on the participants’ socio-demographic profiles and outcome variables under study. The design involved pre- and post-tests to determine the effects of the intervention on the aforementioned outcome variable. The design involved three phases, namely the baseline data collection process (pre-test via a cross-sectional survey), implementation of the intervention (process), and end-line assessment (post-test), as shown in Fig.  1 [ 7 ].

A flow pattern of study design and approach

Target population

The study involved nursing students pursuing a Diploma in nursing and a bachelor of science in nursing in Tanzania. The population was highly expected to demonstrate competences and mastery of different survival and life skills in order to enable them to work independent at various levels of health facilities within and outside Tanzania. This cohort of undergraduate nursing students also involved adult learners who can set goals, develop strategies to achieve their goals, and hence achieve positive professional behavioral outcomes [ 7 ]. Moreover, as per annual data, the average number of graduate nursing students ranges from 3,500 to 4,000 from all colleges and universities in the country [ 38 ].

Study population

The study involved first- and third-year nursing students pursuing a Diploma in Nursing and first-, second-, and third-year nursing students pursuing a Bachelor of Science in Nursing at the University of Dodoma. The population had a large number of enrolled undergraduate nursing students, thus making it an ideal population for intervention, and it approximately served as a good representation of the universities offering nursing programs [ 11 , 29 ].

Inclusion criteria

The study included male and female nursing students pursuing a Diploma in nursing and a bachelor of science in nursing at the University of Dodoma. The referred students included those who were registered at the University of Dodoma during the time of study. Such students live on or off campus, and they were not exposed to PK training despite having regular classroom attendance. This enhanced enrollment of adequate study samples from each study program, monitoring of study intervention, and easy control of con-founders.

Exclusion criteria

All students recruited in the study were assessed at baseline, exposed to a training package and obtained their post-intervention learning experience. None of the study participants, who either dropped out of the study or failed to meet the recruitment criteria.

Sample size determination

A quasi-experimental study on Pecha Kucha as an alternative to traditional PowerPoint presentations at Worcester University, United States of America, reported significant student engagement during Pecha Kucha presentations compared with traditional PowerPoint presentations [ 29 ]. The mean score for the classroom with the traditional PowerPoint presentation was 2.63, while the mean score for the Pecha Kucha presentation was 4.08. This study adopted the formula that was used to calculate the required sample size for an uncontrolled quasi-experimental study among pre-scholars [ 39 ]. The formula is stated as:

Where: Zα was set at 1.96 from the normal distribution table.

Zβ was set at 0.80 power of the study.

Mean zero (π0) was the mean score of audiences’ engagement in using PowerPoint presentation = 2.63.

Mean one (π1) was the mean score of audience’s engagement in using Pecha Kucha presentation = 4.08.

Sampling technique

Given the availability of higher-training institutions in the study area that offer undergraduate nursing programs, a simple random sampling technique was used, whereby two cards, one labelled “University of Dodoma” and the other being labelled “St. Johns University of Tanzania,” were prepared and put in the first pot. The other two cards, one labelled “yes” to represent the study setting and the other being labelled “No” to represent the absence of study setting, were put in the second pot. Two research assistants were asked to select a card from each pot, and consequently, the University of Dodoma was selected as the study setting.

To obtain the target population, the study employed purposive sampling techniques to select the school of nursing and public health at the University of Dodoma. Upon arriving at the School of Nursing and Public Health of the University of Dodoma, the convenience sampling technique was employed to obtain the number of classes for undergraduate nursing students pursuing a Diploma in Nursing and a Bachelor of Science in Nursing. The study sample comprised the students who were available at the time of study. A total of five [ 5 ] classes of Diploma in Nursing first-, second-, and third-years and Bachelor of Science in Nursing first-, second-, and third-years were obtained.

To establish the representation for a minimum sample from each class, the number of students by sex was obtained from each classroom list using the proportionate stratified sampling technique (sample size/population size× stratum size) as recommended by scholars [ 40 ]. To recruit the required sample size from each class by gender, a simple random sampling technique through the lottery method was employed to obtain the required sample size from each stratum. During this phase, the student lists by gender from each class were obtained, and cards with code numbers, which were mixed with empty cards depending on the strata size, were allocated for each class and strata. Both labeled and empty cards were put into different pots, which were labeled appropriately by their class and strata names. Upon arriving at the specific classroom and after the introduction, the research assistant asked each nursing student to pick one card from the respective strata pot. Those who selected cards with code numbers were recruited in the study with their code numbers as their participation identity numbers. The process continued for each class until the required sample size was obtained.

To ensure the effective participation of nursing students in the study, the research assistant worked hand in hand with the facilitators and lecturers of the respective classrooms, the head of the department, and class representatives. The importance, advantages, and disadvantages of participating in the study were given to study participants during the recruitment process in order to create awareness and remove possible fears. During the intervention, study participants were also given pens and notebooks in an attempt to enable them to take notes. Moreover, the bites were provided during the training sessions. The number of participants from each classroom and the sampling process are shown in Fig.  2 [ 7 ].

Flow pattern of participants sampling procedures

Data collection tools

The study adapted and modified the students’ questionnaire on presentation skills from scholars [ 20 , 23 , 26 , 27 , 28 , 29 ]. The modification involved rephrasing the question statement, breaking down items into specific questions, deleting repeated items that were found to measure the same variables, and improving language to meet the literacy level and cultural norms of study participants.

The data collection tool consisted of 68 question items that assessed the socio-demographic characteristics of the study participants and 33 question items rated on a five-point Likert scale, which ranges from 5 = strongly agree, 4 = agree, 3 = not sure, 2 = disagree, and 1 = strongly disagree. The referred tool was used to assess the students’ skills during the preparation and presentation of the assignments using the traditional PowerPoint presentation and Pecha Kucha presentation formats.

The students’ assessment specifically focused on the students’ ability to prepare the presentation content, master the learning content, share presentation materials, and communicate their understanding to audiences in the classroom context.

Validity and reliability of research instruments

Validity of the research instrument refers to whether the instrument measures the behaviors or qualities that are intended to be measured, and it is a measure of how well the measuring instrument performs its function [ 41 ]. The structured questionnaire, which intends to assess the participants’ presentation skills was validated for face and content validity. The principal investigator initially adapted the question items for different domains of students’ learning when preparing and presenting their assignment in the classroom.

The items were shared and discussed by two [ 2 ] educationists, two [ 2 ] research experts, one [ 1 ] statistician, and supervisors in order to ensure clarity, appropriateness, adequacy, and coverage of the presentation skills using Pecha Kucha presentation format. The content validity test was used until the saturation of experts’ opinions and inputs was achieved. The inter-observer rating scale on a five-point Likert scale ranging from 5-points = very relevant to 1-point = not relevant was also used.

The process involved addition, input deletion, correction, and editing for relevance, appropriateness, and scope of the content for the study participants. Some of the question items were broken down into more specific questions, and new domains evolved. Other question items that were found to measure the same variables were also deleted to ease the data collection and analysis. Moreover, the grammar and language issues were improved for clarity based on the literacy level of the study participants.

Reliability of the research instruments refers to the ability of the research instruments or tools to provide similar and consistent results when applied at different times and circumstances [ 41 ]. This study adapted the tools and question items used by different scholars to assess the impact of PKP on student learning [ 12 , 15 , 18 ].

To ensure the reliability of the tools, a pilot study was conducted in one of the nursing training institutions in order to assess the complexity, readability, clarity, completeness, length, and duration of the tool. Ambiguous and difficult (left unanswered) items were modified or deleted based on the consensus that was reached with the consulted experts and supervisor before subjecting the questionnaires to a pre-test.

The study involved 10% of undergraduate nursing students from an independent geographical location for a pilot study. The findings from the pilot study were subjected to explanatory factor analysis (Set a ≥ 0.3) and scale analysis in order to determine the internal consistency of the tools using the Cronbach alpha of ≥ 0.7, which was considered reliable [ 42 , 43 , 44 ]. Furthermore, after the data collection, the scale analysis was computed in an attempt to assess their internal consistency using SPPSS version 26, whereby the Cronbach alpha for question items that assessed the participants’ presentation skills was 0.965.

Data collection method

The study used the researcher-administered questionnaire to collect the participants’ socio-demographic information, co-related factors, and presentation skills as nursing students prepare and present their assignments in the classroom. This enhanced the clarity and participants’ understanding of all question items before providing the appropriate responses. The data were collected by the research assistants in the classroom with the study participants sitting distantly to ensure privacy, confidentiality, and the quality of the information that was provided by the research participants. The research assistant guided and led the study participants to answer the questions and fill in information in the questionnaire for each section, domain, and question item. The research assistant also collected the baseline information (pre-test) before the intervention, which was then compared with the post-intervention information. This was done in the first week of June 2023, after training and orientation of the research assistant on the data collection tools and recruitment of the study participants.

Using the researcher-administered questionnaire, the research assistant also collected the participants’ information related to presentation skills as they prepared and presented their given assignments after the intervention during the second week of July 2023. The participants submitted their presentations to the principle investigator and research assistant to assess the organization, visual appeal and creativity, content knowledge, and adherence to Pecha Kucha presentation requirements. Furthermore, the evaluation of the participants’ ability to share and communicate the given assignment was observed in the classroom presentation using the Pecha Kucha presentation format.

Definitions of variables

Pecha kucha presentation.

It refers to a specific style of presentation whereby the presenter delivers the content using 20 slides that are dominated by images, pictures, tables, or figures. Each slide is displayed for 20 s, thus making a total of 400 s (6 min and 40 s) for the whole presentation.

Presentation skills in this study

This involved students’ ability to plan, prepare, master learning content, create presentation materials, and share them with peers or the audience in the classroom. They constitute the learning activities that stimulate creativity, innovation, critical thinking, and problem-solving skills.

Measurement of pecha kucha preparation and presentation skills

The students’ presentation skills were measured using the four [ 4 ] learning domains. The first domain constituted the students’ ability to plan and prepare the presentation content. It consisted of 17 question items that assessed the students’ ability to gather and select information, search for specific content to be presented in the classroom, find out the learning content from different resources, and search for literature materials for the preparation of the assignment using traditional PowerPoint presentations and Pecha Kucha formats. It also aimed to ascertain a deeper understanding of the contents or topic, learning ownership and motivation to learn the topics with clear understanding and the ability to identify the relevant audience, segregate, and remove unnecessary contents using the Pecha Kucha format.

The second domain constituted the students’ mastery of learning during the preparation and presentation of their assignment before the audience in the classroom. It consisted of six [ 6 ] question items that measured the students’ ability to read several times, rehearse before the classroom presentation, and practice the assignment and presentation harder. It also measures the students’ ability to evaluate the selected information and content before their actual presentation and make revisions to the selected information and content before the presentation using the Pecha Kucha format.

The third domain constituted the students’ ability to prepare the presentation materials. It consisted of six [ 6 ] question items that measured the students’ ability to organize the information and contents, prepare the classroom presentation, revise and edit presentation resources, materials, and contents, and think about the audience and classroom design. The fourth domain constituted the students’ ability to share their learning. It consisted of four [ 4 ] question items that measured the students’ ability to communicate their learning with the audience, present a new understanding to the audience, transfer the learning to the audience, and answer the questions about the topic or assignment given. The variable was measured using a 5-point Likert scale. The average scores were computed for each domain, and an overall mean score was calculated across all domains. Additionally, an encompassing skills score was derived from the cumulative scores of all four domains, thus providing a comprehensive evaluation of the overall skills level.

Implementation of intervention

The implementation of the study involved the training of research assistants, sampling of the study participants, setting of the venue, pre-assessment of the students’ presentation skills using traditional PowerPoint presentations, training and demonstration of Pecha Kucha presentations to study participants, and assigning the topics to study participants. The implementation of the study also involved the participants’ submission of their assignments to the Principal Investigator for evaluation, the participants’ presentation of their assigned topic using the Pecha Kucha format, post-intervention assessment of the students’ presentation skills, data analysis, and reporting [ 7 ]. The intervention involved Principal Investigator and two [ 2 ] trained research assistants. The intervention in question was based on the concept of multimedia theory of cognitive learning (MTCL) for enhancing effective leaning in 21st century.

Training of research assistants

Two research assistants were trained with regard to the principles, characteristics, and format of Pecha Kucha presentations using the curriculum from the official Pecha Kucha website. Also, research assistants were oriented to the data collection tools and methods in an attempt to guarantee the relevancy and appropriate collection of the participants’ information.

Schedule and duration of training among research assistants

The PI prepared the training schedule and venue after negotiation and consensus with the research assistants. Moreover, the Principle Investigator trained the research assistants to assess the learning, learn how to collect the data using the questionnaire, and maintain the privacy and confidentiality of the study participants.

Descriptions of interventions

The intervention was conducted among the nursing students at the University of Dodoma, which is located in Dodoma Region, Tanzania Mainland, after obtaining their consent. The participants were trained regarding the concepts, principles, and characteristics of Pecha Kucha presentations and how to prepare and present their assignments using the Pecha Kucha presentation format. The study participants were also trained regarding the advantages and disadvantages of Pecha Kucha presentations. The training was accompanied by one example of an ideal Pecha Kucha presentation on the concepts of pressure ulcers. The teaching methods included lecturing, brainstorming, and small group discussion. After the training session, the evaluation was conducted to assess the participants’ understanding of the Pecha Kucha conceptualization, its characteristics, and its principles.

Each participant was given a topic as an assignment from the fundamentals of nursing, medical nursing, surgical nursing, community health nursing, mental health nursing, emergency critical care, pediatric, reproductive, and child health, midwifery, communicable diseases, non-communicable diseases, orthopedics and cross-cutting issues in nursing as recommended by scholars [ 21 , 38 ]. The study participants were given 14 days for preparation, rehearsal of their presentation using the Pecha Kucha presentation format, and submission of the prepared slides to the research assistant and principle investigator for evaluation and arrangement before the actual classroom presentation. The evaluation of the participants’ assignments involved the number of slides, quality of images used, number of words, organization of content and messages to be delivered, slide transition, duration of presentation, flow, and organization of slides.

Afterwards, each participant was given 6 min and 40 s for the presentation and 5 min to 10 min for answering the questions on the topic presented as raised by other participants. An average of 4 participants obtained the opportunity to present their assignments in the classroom every hour. After the completion of all presentations, the research assistants assessed the participant’s presentation skills using the researcher-administered questionnaire. The collected data were entered in SPSS version 26 and analyzed in an attempt to compare the mean score of participants’ presentation skills with the baseline mean score. The intervention sessions were conducted in the selected classrooms, which were able to accommodate all participants at the time that was arranged by the participant’s coordinators, institution administrators, and subject facilitators of the University of Dodoma, as described in Table  1 [ 7 ].

Evaluation of intervention

During the classroom presentation, there were 5 to 10 min for classroom discussion and reflection on the content presented, which was guided by the research assistant. During this time, the participants were given the opportunity to ask the questions, get clarification from the presenter, and provide their opinion on how the instructional messages were presented, content coverage, areas of strength and weakness for improvement, and academic growth. After the completion of the presentation sessions, the research assistant provided the questionnaire to participants in order to determine their presentation skills during the preparation of their assignments and classroom presentations using the Pecha Kucha presentation format.

Data analysis

The findings from this study were analyzed using the Statistical Package for Social Science (SPSS) computer software program version 26. The percentages, frequencies, frequency distributions, means, standard deviations, skewness, and kurtosis were calculated, and the results were presented using the figures, tables, and graphs. The mean score analysis was computed, and descriptive statistical analysis was used to analyze the demographic information of the participants in an attempt to determine the frequencies, percentages, and mean scores of their distributions. A paired sample t-test was used to compare the mean score differences of the presentation skills within the groups before and after the intervention. The mean score differences were determined based on the baseline scores against the post-intervention scores in order to establish any change in terms of presentation skills among the study participants.

The association between the Pecha Kucha presentation and the development of participants’ presentation skills was established using linear regression analysis set at a 95% confidence interval and 5% (≤ 0.05) significance level in an attempt to accept or reject the null hypothesis.

However, N-1 dummy variables were formed for the categorical independent variables so as to run the linear regression for the factors associated with the presentation skills. The linear regression equation with dummy variables is presented as follows:

Β 0 is the intercept.

Β 1 , Β 2 , …. Β k-1 are the coefficients which correspond to the dummy variables representing the levels of X 1 .

Β k is the coefficient which corresponds to the dummy variable representing the levels of X 2 .

Β k+1 is the coefficient which corresponds to the continuous predictor X 3 .

X 1,1 , X 1,2 ,……. X 1,k-1 are the dummy variables corresponding to the different levels of X 1 .

ε represents the error term.

The coefficients B1, B2… Bk indicate the change in the expected value of Y for each category relative to the reference category. If the Beta estimate is positive for the categorical or dummy variables, it means that the corresponding covariate has a positive impact on the outcome variable compared to reference category. However, if the beta estimate is positive for the case of continuous covariates, it means that the corresponding covariate has direct proportion effect on the outcome variables.

The distribution of the outcome variables was approximately normally distributed since the normality of the data is one of the requirements for parametric analysis. A paired t test was performed to compare the presentation skills of nursing students before and after the intervention.

Social-demographic characteristics of the study participants

The study involved a total of 230 nursing students, of whom 151 (65.65%) were male and the rest were female. The mean age of study participants was 23.03 ± 2.69, with the minimum age being 19 and the maximum age being 37. The total of 163 (70.87%) students, which comprised a large proportion of respondents, were aged less than or equal to 23, 215 (93.48%) participants were living on campus, and 216 (93.91) participants were exposed to social media.

A large number of study participants (82.17%) were pursuing a bachelor of Science in Nursing, with the majority being first-year students (30.87%). The total of 213 (92.61%) study participants had Form Six education as their entry qualification, with 176 (76.52%) participants being the product of public secondary schools and interested in the nursing profession. Lastly, the total of 121 (52.61%) study participants had never been exposed to any presentation training; 215 (93.48%) students had access to individual classroom presentations; and 227 (98.70%) study participants had access to group presentations during their learning process. The detailed findings for the participants’ social demographic information are indicated in Table  2 [ 46 ].

Baseline nursing students’ presentation skills using traditional powerPoint presentations

The current study assessed the participant’s presentation skills when preparing and presenting the materials before the audience using traditional PowerPoint presentations. The study revealed that the overall mean score of the participants’ presentation skills was 4.07 ± 0.56, including a mean score of 3.98 ± 0.62 for the participants’ presentation skills during the preparation of presentation content before the classroom presentation and a mean score of 4.18 ± 0.78 for the participants’ mastery of learning content before the classroom presentation. Moreover, the study revealed a mean score of 4.07 ± 0.71 for participants’ ability to prepare presentation materials for classroom presentations and a mean score of 4.04 ± 0.76 for participants’ ability to share the presentation materials in the classroom, as indicated in Table  3 [ 46 ].

Factors Associated with participants’ presentation skills through traditional powerPoint presentation

The current study revealed that the participants’ study program has a significant effect on their presentation skills, whereby being the bachelor of science in nursing was associated with a 0.37561 (P value < 0.027) increase in the participants’ presentation skills.The year of study also had significant effects on the participants’ presentation skills, whereby being a second-year bachelor student was associated with a 0.34771 (P value < 0.0022) increase in the participants’ presentation skills compared to first-year bachelor students and diploma students. Depending on loans as a source of student income retards presentation skills by 0.24663 (P value < 0.0272) compared to those who do not depend on loans as the source of income. Furthermore, exposure to individual presentations has significant effects on the participants’ presentation skills, whereby obtaining an opportunity for individual presentations was associated with a 0.33732 (P value 0.0272) increase in presentation skills through traditional PowerPoint presentations as shown in Table  4 [ 46 ].

Nursing student presentation skills through pecha kucha presentations

The current study assessed the participant’s presentation skills when preparing and presenting the materials before the audience using Pecha Kucha presentations. The study revealed that the overall mean score and standard deviation of participants’ presentation skills using the Pecha Kucha presentation format were 4.54 ± 0.59, including a mean score of 4.49 ± 0.66 for participant’s presentation skills during preparation of the content before classroom presentation and a mean score of 4.58 ± 0.65 for participants’ mastery of learning content before classroom presentation. Moreover, the study revealed a mean score of 4.58 ± 0.67 for participants ability to prepare the presentation materials for classroom presentation and a mean score of 4.51 ± 0.72 for participants ability to share the presentation materials in the classroom using Pecha Kucha presentation format as indicated in Table  5 [ 46 ].

Comparing Mean scores of participants’ presentation skills between traditional PowerPoint presentation and pecha kucha Presentation

The current study computed a paired t-test to compare and determine the mean change, effect size, and significance associated with the participants’ presentation skills when using the traditional PowerPoint presentation and Pecha Kucha presentation formats. The study revealed that the mean score of the participants’ presentation skills through the Pecha Kucha presentation was 4.54 ± 0.59 (p value < 0.0001) compared to the mean score of 4.07 ± 0.56 for the participants’ presentation skills using the traditional power point presentation with an effect change of 0.78. With regard to the presentation skills during the preparation of presentation content before the classroom presentation, the mean score was 4.49 ± 0.66 using the Pecha Kucha presentation compared to the mean score of 3.98 ± 0.62 for the traditional PowerPoint presentation. Its mean change was 0.51 ± 0.84 ( p  < .0001) with an effect size of 0.61.

Regarding the participants’ mastery of learning content before the classroom presentation, the mean score was 4.58 ± 0.65 when using the Pecha Kucha presentation format, compared to the mean score of 4.18 ± 0.78 when using the traditional power point presentation. Its mean change was 0.40 ± 0.27 ( p  < .0001) with an effect size of 1.48. Regarding the ability of the participants to prepare the presentation materials for classroom presentations, the mean score was 4.58 ± 0.67 when using the Pecha Kucha presentation format, compared to 4.07 ± 0.71 when using the traditional PowerPoint presentation. Its mean change was 0.51 ± 0.96 ( p  < .0001) with an effect size of 0.53.

Regarding the participants’ presentation skills when sharing the presentation material in the classroom, the mean score was 4.51 ± 0.72 when using the Pecha Kucha presentation format, compared to 4.04 ± 0.76 when using the traditional PowerPoint presentations. Its mean change was 0.47 ± 0.10, with a large effect size of 4.7. Therefore, Pecha Kucha presentation pedagogy has a significant effect on the participants’ presentation skills than the traditional PowerPoint presentation as shown in Table  6 [ 46 ].

Factors associated with presentation skills among nursing students through pecha kucha presentation

The current study revealed that the participant’s presentation skills using the Pecha Kucha presentation format were significantly associated with knowledge of the Pecha Kucha presentation format, whereby increase in knowledge was associated with a 0.0239 ( p  < .0001) increase in presentation skills. Moreover, the current study revealed that the presentation through the Pecha Kucha presentation format was not influenced by the year of study, whereby being a second-year student could retard the presentation skills by 0.23093 (p 0.039) compared to a traditional PowerPoint presentation. Other factors are shown in Table  7 [ 46 ].

Social-demographic characteristics profiles of participants

The proportion of male participants was larger than the proportion of female participants in the current study. This was attributable to the distribution of sex across the nursing students at the university understudy, whose number of male nursing students enrolled was higher than female students. This demonstrates the high rate of male nursing students’ enrolment in higher training institutions to pursue nursing and midwifery education programs. Different from the previous years, the nursing training institutions were predominantly comprised of female students and female nurses in different settings. This significant increase in male nursing students’ enrollment in nursing training institutions predicts a significant increase in the male nursing workforce in the future in different settings.

These findings on Pecha Kucha as an alternative to PowerPoint presentations in Massachusetts, where the proportion of female participants was large as compared to male participants, are different from the experimental study among English language students [ 29 ]. The referred findings are different from the results of the randomized control study among the nursing students in Anakara, Turkey, where a large proportion of participants were female nursing students [ 47 ]. This difference in participants’ sex may be associated with the difference in socio-cultural beliefs of the study settings, country’s socio-economic status, which influence the participants to join the nursing profession on the basis of securing employment easily, an opportunity abroad, or pressure from peers and parents. Nevertheless, such differences account for the decreased stereotypes towards male nurses in the community and the better performance of male students in science subjects compared to female students in the country.

The mean age of the study participants was predominantly young adults with advanced secondary education. Their ages reflect adherence to national education policy by considering the appropriate age of enrollment of the pupils in primary and secondary schools, which comprise the industries for students at higher training institutions. This age range of the participants in the current study suits the cognitive capability expected from the participants in order to demonstrate different survival and life skills by being able to set learning goals and develop strategies to achieve their goals according to Jean Piaget’s theory of cognitive learning [ 41 , 42 ].

Similar age groups were noted in the study among nursing students in a randomized control study in Anakara Turkey where the average age was 19.05 ± 0.2 [ 47 ]. A similar age group was also found in a randomized control study among liberal arts students in Anakara, Turkey, on differences in instructor, presenter, and audience ratings of Pecha Kucha presentations and traditional student presentations where the ages of the participants ranged between 19 and 22 years [ 49 ].

Lastly, a large proportion of the study participants had the opportunity for individual and group presentations in the classroom despite having not been exposed to any presentation training before. This implies that the teaching and learning process in a nursing education program is participatory and student-centered, thus giving the students the opportunity to interact with learning contents, peers, experts, webpages, and other learning resources to become knowledgeable. These findings fit with the principle that guides and facilitates the student’s learning from peers and teachers according to the constructivism theory of learning by Lev Vygotsky [ 48 ].

Effects of pecha kucha presentation pedagogy on participants’ presentation skills

The participants’ presentation skills were higher for Pecha Kucha presentations compared with traditional PowerPoint presentations. This display of the Pecha Kucha presentation style enables the nursing students to prepare the learning content, master their learning content before classroom presentations, create good presentation materials and present the materials, before the audience in the classroom. This finding was similar to that at Padang State University, Indonesia, among first-year English and literature students whereby the Pecha Kucha Presentation format helped the students improve their skills in presentation [ 20 ]. Pecha Kucha was also found to facilitate careful selection of the topic, organization and outlining of the students’ ideas, selection of appropriate images, preparation of presentations, rehearsing, and delivery of the presentations before the audience in a qualitative study among English language students at the Private University of Manila, Philippines [ 23 ].

The current study found that Pecha Kucha presentations enable the students to perform literature searches from different webpages, journals, and books in an attempt to identify specific contents during the preparation of the classroom presentations more than traditional PowerPoint presentations. This is triggered by the ability of the presentation format to force the students to filter relevant and specific information to be included in the presentation and search for appropriate images, pictures, or figures to be presented before the audience. Pecha Kucha presentations were found to increase the ability to perform literature searches before classroom presentations compared to traditional PowerPoint presentations in an experimental study among English language students at Worcester State University [ 29 ].

The current study revealed that Pecha Kucha presentations enable the students to create a well-structured classroom presentation effectively by designing 20 meaningful and content-rich slides containing 20 images, pictures, or figures and a transitional flow of 20 s for each slide, more than the traditional PowerPoint presentation with an unlimited number of slides containing bullets with many texts or words. Similarly, in a cross-sectional study of medical students in India, Pecha Kucha presentations were found to help undergraduate first-year medical students learn how to organize knowledge in a sequential fashion [ 26 ].

The current study revealed that Pecha Kucha presentations enhance sound mastery of the learning contents and presentation materials before the classroom presentation compared with traditional PowerPoint presentations. This is hastened by the fact that there is no slide reading during the classroom Pecha Kucha presentation, thus forcing students to read several times, rehearse, and practice harder the presentation contents and materials before the classroom presentation. Pecha Kucha presentation needed first year English and literature students to practice a lot before their classroom presentation in a descriptive qualitative study at Padang State University-Indonesia [ 20 ].

The current study revealed that the participants became more confident in answering the questions about the topic during the classroom presentation using the Pecha Kucha presentation style than during the classroom presentation using the tradition PowerPoint presentation. This is precipitated by the mastery level of the presentation contents and materials through rehearsal, re-reading, and material synthesis before the classroom presentations. Moreover, Pecha Kucha was found to significantly increase the students’ confidence during classroom presentation and preparation in a qualitative study among English language students at the Private University of Manila, Philippines [ 23 ].

Hence, there was enough evidence to reject the null hypothesis in that there was no significant difference in nursing students’ presentation skills between the baseline and end line. The Pecha Kucha presentation format has a significant effect on nursing student’s classroom presentation skills as it enables them to prepare the learning content, have good mastery of the learning contents, create presentation materials, and confidently share their learning with the audience in the classroom.

The current study’s findings complement the available pieces of evidence on the effects of Pecha Kucha presentations on the students’ learning and development of survival life skills in the 21st century. Pecha kucha presentations have more significant effects on the students’ presentation skills compared with traditional PowerPoint presentations. It enables the students to select the topic carefully, organize and outline the presentation ideas, select appropriate images, create presentations, rehearse the presentations, and deliver them confidently before an audience. It also enables the students to select and organize the learning contents for classroom presentations more than traditional PowerPoint presentations.

Pecha Kucha presentations enhance the mastery of learning content by encouraging the students to read the content several times, rehearse, and practice hard before the actual classroom presentation. It increases the students’ ability to perform literature searches before the classroom presentation compared to a traditional PowerPoint presentation. Pecha Kucha presentations enable the students to create well-structured classroom presentations more effectively compared to traditional PowerPoint presentations. Furthermore, Pecha Kucha presentations make the students confident during the presentation of their assignments and project works before the audience and during answering the questions.

Lastly, Pecha Kucha presentations enhance creativity among the students by providing the opportunity for them to decide on the learning content to be presented. Specifically, they are able to select the learning content, appropriate images, pictures, or figures, organize and structure the presentation slides into a meaningful and transitional flow of ideas, rehearse and practice individually before the actual classroom presentation.

Strength of the study

This study has addressed the pedagogical gap in nursing training and education by providing new insights on the innovative students’ presentation format that engages students actively in their learning to bring about meaningful and effective students’ learning. It has also managed to recruit, asses, and provide intended intervention to 230 nursing students without dropout.

Study limitation

The current study has pointed out some of the strengths of the PechaKucha presentations on the students’ presentation skills over the traditional students’ presentations. However, the study had the following limitations: It involved one group of nursing students from one of the public training institutions in Tanzania. The use of one university may obscure the interpretation of the effects of the size of the intervention on the outcome variables of interest, thus limiting the generalization of the study findings to all training institutions in Tanzania. Therefore, the findings from this study need to be interpreted by considering this limitation. The use of one group of nursing students from one university to explore their learning experience through different presentation formats may also limit the generalization of the study findings to all nursing students in the country. The limited generalization may be attributed to differences in socio-demographic characteristics, learning environments, and teaching and learning approaches. Therefore, the findings from this study need to be interpreted by considering this limitation.

Suggestions for future research

The future research should try to overcome the current study limitations and shortcomings and extend the areas assessed by the study to different study settings and different characteristics of nursing students in Tanzania as follows: To test rigorously the effects of Pecha Kucha presentations in enhancing the nursing students’ learning, the future studies should involve nursing students’ different health training institutions rather than one training institution. Future studies should better use the control students by randomly allocating the nursing students or training institutions in the intervention group or control group in order to assess the students’ learning experiences through the use of Pecha Kucha presentations and PowerPoint presentations consecutively. Lastly, future studies should focus on nursing students’ mastery of content knowledge and students’ classroom performance through the use of the Pecha Kucha presentation format in the teaching and learning process.

Data availability

The datasets generated and analyzed by this study can be obtained from the corresponding author on reasonable request through [email protected] & [email protected].

Abbreviations

Doctor (PhD)

Multimedia Theory of Cognitive Learning

National Council for Technical and Vocational Education and Training

Principle Investigator

Pecha Kucha presentation

Statistical Package for Social Sciences

Tanzania Commission for Universities

World Health Organization

International Council of Nurses. Nursing Care Continuum Framework and Competencies. 2008.

Partnership for 21st Century Skills. 21st Century Skills, Education & Competitiveness. a Resour Policy Guid [Internet]. 2008;20. https://files.eric.ed.gov/fulltext/ED519337.pdf

Partnership for 21st Century Skills. 21St Century Knowledge and Skills in Educator Preparation. Education [Internet]. 2010;(September):40. https://files.eric.ed.gov/fulltext/ED519336.pdf

Partnership for 21st Century Skills. A State Leaders Action Guide to 21st Century Skills: A New Vision for Education. 2006; http://apcrsi.pt/website/wp-content/uploads/20170317_Partnership_for_21st_Century_Learning.pdf

World Health Organization. Four-Year Integrated Nursing And Midwifery Competency-Based Prototype Curriculum for the African Region [Internet]. Republic of South Africa: WHO Regional Office for Africa. 2016; 2016. 13 p. https://apps.who.int/iris/bitstream/handle/10665/331471/9789290232612-eng.pdf?sequence=1&isAllowed=y

World Health Organization, THREE-YEAR REGIONAL PROTOTYPE PRE-SERVICE COMPETENCY-BASED NURSING, CURRICULUM [Internet]. 2016. https://apps.who.int/iris/bitstream/handle/10665/331657/9789290232629-eng.pdf?sequence=1&isAllowed=y

Haramba SJ, Millanzi WC, Seif SA. Enhancing nursing student presentation competences using Facilitatory Pecha kucha presentation pedagogy: a quasi-experimental study protocol in Tanzania. BMC Med Educ [Internet]. 2023;23(1):628. https://bmcmededuc.biomedcentral.com/articles/ https://doi.org/10.1186/s12909-023-04628-z

Millanzi WC, Osaki KM, Kibusi SM. Non-cognitive skills for safe sexual behavior: an exploration of baseline abstinence skills, condom use negotiation, Self-esteem, and assertiveness skills from a controlled problem-based Learning Intervention among adolescents in Tanzania. Glob J Med Res. 2020;20(10):1–18.

Google Scholar  

Millanzi WC, Herman PZ, Hussein MR. The impact of facilitation in a problem- based pedagogy on self-directed learning readiness among nursing students : a quasi- experimental study in Tanzania. BMC Nurs. 2021;20(242):1–11.

Millanzi WC, Kibusi SM. Exploring the effect of problem-based facilitatory teaching approach on metacognition in nursing education: a quasi-experimental study of nurse students in Tanzania. Nurs Open. 2020;7(April):1431–45.

Article   Google Scholar  

Millanzi WC, Kibusi SM. Exploring the effect of problem based facilitatory teaching approach on motivation to learn: a quasi-experimental study of nursing students in Tanzania. BMC Nurs [Internet]. 2021;20(1):3. https://bmcnurs.biomedcentral.com/articles/ https://doi.org/10.1186/s12912-020-00509-8

Hadiyanti KMW, Widya W. Analyzing the values and effects of Powerpoint presentations. LLT J J Lang Lang Teach. 2018;21(Suppl):87–95.

Nichani A. Life after death by power point: PechaKucha to the rescue? J Indian Soc Periodontol [Internet]. 2014;18(2):127. http://www.jisponline.com/text.asp?2014/18/2/127/131292

Uzun AM, Kilis S. Impressions of Pre-service teachers about Use of PowerPoint slides by their instructors and its effects on their learning. Int J Contemp Educ Res. 2019.

Unesco National Commission TM. UNESCO National Commission Country ReportTemplate Higher Education Report. [ UNITED REPUBLIC OF TANZANIA ]; 2022.

TCU. VitalStats on University Education in Tanzania. 2020. 2021;1–4. https://www.tcu.go.tz/sites/default/files/VitalStats 2020.pdf.

Kwame A, Petrucka PM. A literature-based study of patient-centered care and communication in nurse-patient interactions: barriers, facilitators, and the way forward. BMC Nurs [Internet]. 2021;20(1):158. https://bmcnurs.biomedcentral.com/articles/ https://doi.org/10.1186/s12912-021-00684-2

Kourkouta L, Papathanasiou I. Communication in Nursing Practice. Mater Socio Medica [Internet]. 2014;26(1):65. http://www.scopemed.org/fulltextpdf.php?mno=153817

Foulkes M. Presentation skills for nurses. Nurs Stand [Internet]. 2015;29(25):52–8. http://rcnpublishing.com/doi/ https://doi.org/10.7748/ns.29.25.52.e9488

Solusia C, Kher DF, Rani YA. The Use of Pecha Kucha Presentation Method in the speaking for Informal Interaction Class. 2020;411(Icoelt 2019):190–4.

Sen G. What is PechaKucha in Teaching and How Does It Work? Clear Facts About PechaKucha in Classroom [Internet]. Asian College of Teachers. 2016 [cited 2022 Jun 15]. https://www.asiancollegeofteachers.com/blogs/452-What-is-PechaKucha-in-Teaching-and-How-Does-It-Work-Clear-Facts-About-PechaKucha-in-Classroom-blog.php

Pecha Kucha Website. Pecha Kucha School [Internet]. 2022. https://www.pechakucha.com/schools

Mabuan RA. Developing Esl/Efl Learners Public Speaking Skills through Pecha Kucha Presentations. Engl Rev J Engl Educ. 2017;6(1):1.

Laieb M, Cherbal A. Improving speaking performance through Pecha Kucha Presentations among Algerian EFL Learners. The case of secondary School students. Jijel: University of Mohammed Seddik Ben Yahia; 2021.

Angelina P, IMPROVING INDONESIAN EFL STUDENTS SPEAKING, SKILL THROUGH PECHA KUCHA. LLT J A. J Lang Lang Teach [Internet]. 2019;22(1):86–97. https://e-journal.usd.ac.id/index.php/LLT/article/view/1789

Abraham RR, Torke S, Gonsalves J, Narayanan SN, Kamath MG, Prakash J, et al. Modified directed self-learning sessions in physiology with prereading assignments and Pecha Kucha talks: perceptions of students. Adv Physiol Educ. 2018;42(1):26–31.

Coskun A. The Effect of Pecha Kucha Presentations on Students’ English Public Speaking Anxiety. Profile Issues Teach Prof Dev [Internet]. 2017;19(_sup1):11–22. https://revistas.unal.edu.co/index.php/profile/article/view/68495

González Ruiz C, STUDENT PERCEPTIONS OF THE USE OF PECHAKUCHA, In PRESENTATIONS IN SPANISH AS A FOREIGN LANGUAGE. 2016. pp. 7504–12. http://library.iated.org/view/GONZALEZRUIZ2016STU

Warmuth KA. PechaKucha as an Alternative to Traditional Student Presentations. Curr Teach Learn Acad J [Internet]. 2021;(January). https://www.researchgate.net/publication/350189239

Hayashi PMJ, Holland SJ. Pecha Kucha: Transforming Student Presentations. Transform Lang Educ [Internet]. 2017; https://jalt-publications.org/files/pdf-article/jalt2016-pcp-039.pdf

Solmaz O. Developing EFL Learners ’ speaking and oral presentation skills through Pecha Kucha presentation technique. 2019;10(4):542–65.

Tanzania Commission for Universities. University Institutions operating in Tanzania. THE UNITED REPUBLIC OF TANZANIA; 2021.

The University of Dodoma. About Us [Internet]. 2022 [cited 2022 Aug 22]. https://www.udom.ac.tz/about

NACTVET. Registered Institutions [Internet]. The United Republic of Tanzania. 2022. https://www.nacte.go.tz/?s=HEALTH

TCU. University education in tanzania 2021. VitalStats, [Internet]. 2022;(May):63. https://www.tcu.go.tz/sites/default/files/VitalStats 2021.pdf.

St. John University of Tanzania. About St. John University [Internet]. 2022 [cited 2022 Aug 22]. https://sjut.ac.tz/our-university/

TopUniversitieslist. St John’s University of Tanzania Ranking [Internet]. World University Rankings & Reviews. 2023 [cited 2023 Jul 1]. https://topuniversitieslist.com/st-johns-university-of-tanzania/

Tanzania Nursing and Midwifery Council. TANZANIA NURSING AND MIDWIFERY COUNCIL THE REGISTRATION AND LICENSURE EXAMINATION GUIDELINE FOR NURSESAND MIDWIVES IN TANZANIA REVISED VERSION. : 2020; https://www.tnmc.go.tz/downloads/

Salim MA, Gabrieli P, Millanzi WC. Enhancing pre-school teachers’ competence in managing pediatric injuries in Pemba Island, Zanzibar. BMC Pediatr. 2022;22(1):1–13.

Iliyasu R, Etikan I. Comparison of quota sampling and stratified random sampling. Biometrics Biostat Int J [Internet]. 2021;10(1):24–7. https://medcraveonline.com/BBIJ/comparison-of-quota-sampling-and-stratified-random-sampling.html

Surucu L, Ahmet M, VALIDITY, AND RELIABILITY IN QUANTITATIVE RESEARCH. Bus Manag Stud An Int J [Internet]. 2020;8(3):2694–726. https://bmij.org/index.php/1/article/view/1540

Lima E, de Barreto P, Assunção SM. Factor structure, internal consistency and reliability of the posttraumatic stress disorder checklist (PCL): an exploratory study. Trends Psychiatry Psychother. 2012;34(4):215–22.

Taber KS. The Use of Cronbach’s alpha when developing and Reporting Research Instruments in Science Education. Res Sci Educ. 2018;48(6):1273–96.

Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011;2(2011):53–5.

Madar P, London W, ASSESSING THE STUDENT :. PECHAKUCHA. 2013;3(2):4–10.

Haramba, S. J., Millanzi, W. C., & Seif, S. A. (2023). Enhancing nursing student presentation competencies using Facilitatory Pecha Kucha presentation pedagogy: a quasi-experimental study protocol in Tanzania. BMC Medical Education, 23(1), 628. https://doi.org/10.1186/s12909-023-04628-z

Bakcek O, Tastan S, Iyigun E, Kurtoglu P, Tastan B. Comparison of PechaKucha and traditional PowerPoint presentations in nursing education: A randomized controlled study. Nurse Educ Pract [Internet]. 2020;42:102695. https://linkinghub.elsevier.com/retrieve/pii/S1471595317305097

Mcleod G. Learning theory and Instructional Design. Learn Matters. 2001;2(2003):35–43.

Warmuth KA, Caple AH. Differences in Instructor, Presenter, and Audience Ratings of PechaKucha and Traditional Student Presentations. Teach Psychol [Internet]. 2022;49(3):224–35. http://journals.sagepub.com/doi/10.1177/00986283211006389

Download references

Acknowledgements

The supervisors at the University of Dodoma, statisticians, my employer, family members, research assistants and postgraduate colleagues are acknowledged for their support in an attempt to facilitate the development and completion of this manuscript.

The source of funds to conduct this study was the registrar, Tanzania Nursing and Midwifery Council (TNMC) who is the employer of the corresponding author. The funds helped the author in developing the protocol, printing the questionnaires, and facilitating communication during the data collection and data analysis and manuscript preparation.

Author information

Authors and affiliations.

Department of Nursing Management and Education, The University of Dodoma, Dodoma, United Republic of Tanzania

Setberth Jonas Haramba & Walter C. Millanzi

Department of Public and Community Health Nursing, The University of Dodoma, Dodoma, United Republic of Tanzania

Saada A. Seif

You can also search for this author in PubMed   Google Scholar

Contributions

S.J.H: conceptualization, proposal development, data collection, data entry, data cleaning and analysis, writing the original draft of the manuscript W.C.M: Conceptualization, supervision, review, and editing of the proposal, and the final manuscript S.S.A: Conceptualization, supervision, review, and editing of the proposal and the final manuscript.

Corresponding author

Correspondence to Setberth Jonas Haramba .

Ethics declarations

Ethics approval and consent to participate.

All methods were carried out under the relevant guidelines and regulations. Since the study involved the manipulation of human behaviors and practices and the exploration of human internal learning experiences, there was a pressing need to obtain ethical clearance and permission from the University of Dodoma (UDOM) Institution of Research Review Ethics Committee (IRREC) in order to conduct this study. The written informed consents were obtained from all the participants, after explaining to them the purpose, the importance of participating in the study, the significance of the study findings to students’ learning, and confidentiality and privacy of the information that will be provided. The nursing students who participated in this study benefited from the knowledge of the Pecha Kucha presentation format and how to prepare and present their assignments using the Pecha Kucha presentation format.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Haramba, S.J., Millanzi, W.C. & Seif, S.A. Effects of pecha kucha presentation pedagogy on nursing students’ presentation skills: a quasi-experimental study in Tanzania. BMC Med Educ 24 , 952 (2024). https://doi.org/10.1186/s12909-024-05920-2

Download citation

Received : 16 October 2023

Accepted : 16 August 2024

Published : 31 August 2024

DOI : https://doi.org/10.1186/s12909-024-05920-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Nursing students
• Pecha Kucha presentation pedagogy and presentation skills

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

Formaldehyde contamination in seafood industry: an update on detection methods and legislations

• Review Article
• Published: 03 September 2024

Cite this article

• Naresh Kumar Mehta   ORCID: orcid.org/0000-0002-0688-886X 1   na1 ,
• Anand Vaishnav 1   na1 ,
• Mocherla Bhargavi Priyadarshini 1 ,
• Payel Debbarma 1 ,
• Mohammad Sazedul Hoque 2 ,
• Pronoy Mondal 2 ,
• Mahmud Ab Rashid Nor-Khaizura 3 , 4 ,
• Gioacchino Bono 5 , 6 ,
• Pankaj Koirala 7 ,
• Aikkarach Kettawan 7 &
• Nilesh Prakash Nirmal 7   na1  

1 Altmetric

Seafood is abundant in high-quality protein, healthy fats (n-3 and n-6 PUFAs), minerals (calcium, magnesium, copper, selenium, and so on), and vitamin D. Functional compounds in seafood can protect against lifestyle-related diseases. Having had all the merits mentioned, it is also a highly putrefiable food item. To maintain quality and prolong seafood’s shelf life, various chemicals have been added, including nitrite, sulfur dioxide, and formaldehyde. In this review, we summarize the uses, the incidence of added formalin contamination, and the approved limit for seafood products. Additionally, worldwide regulations or standards for the use of formalin in seafood products, as well as recent changes relevant to new methods, are highlighted. Although strict limits and regulations have been placed on the utilization of formaldehyde for seafood preservation, there are few incidences reported of formalin/formaldehyde detection in seafood products around Asian countries. In this context, various qualitative and quantitative detection methods for formaldehyde have been developed to ensure the presence of formaldehyde within acceptable limits. Besides this, different rules and regulations have been forced by each country to control formaldehyde incidence. Although it is not an issue of formaldehyde incidence in European countries, strict regulations are implemented and followed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Biogenic amines in seafood: a review, a method for identification of exogenous formaldehyde in bombay duck ( harpodon nehereus ).

Organoleptic quality and formaldehyde content of mackerel marketed in Dar es Salaam, Tanzania

Explore related subjects.

• Environmental Chemistry

Data availability

The data sets generated are available from the corresponding author upon reasonable request.

AFSSA (2004) Evaluation des risques lies a la presence de formaldéhyde dans les aliments. Agence Francaise de Securite Sanitire des Aliments. https://www.anses.fr/en/system/files/NUT2004sa0363ErEN.pdf . Accessed Jan 2024 to June 2024

Agrinnovateindia.co.in (n.d.) CIFLIN®-Kit for detection of formaldehyde in fish. https://www.agrinnovateindia.co.in/subTechnologyDescription?id=470&Subtitle=CIFLIN%C2%AE-Kit%20for%20Detection%20of%20Formaldehyde%20in%20Fish&Techtitle=ICAR-Central%20Inland%20Fisheries%20Research%20Institute%20(CIFRI) . Accessed Jan 2024 to June 2024

Aksornneam L, Kanatharana P, Thavarungkul P, Thammakhet C (2016) 5-Aminofluorescein doped polyvinyl alcohol film for the detection of formaldehyde in vegetables and seafood. Anal Methods 8(6):1249–1256. https://doi.org/10.1039/C5AY02719E

Article   CAS   Google Scholar  

Aminah AS, Zailina H, Fatimah AB (2013) Health risk assessment of adults consuming commercial fish contaminated with formaldehyde. Food Public Health 3(1):52–58. https://doi.org/10.5923/j.fph.20130301.06

Article   Google Scholar  

Anissah U, Putri AK, Barokah GR (2019) An estimation of endogenous formaldehyde exposure due to consumption of Indonesian opah fish ( Lampris guttatus ) in three major export destination countries. Squalen Bull Mar Fish Postharvest Biotechnol 14(1):1–8. https://doi.org/10.15578/squalen.v14i1.369

Anissah U, Ariyani F, Barokah G, Januar HI (2021) NMR metabolomics of Saurida tumbil fish treated with formaldehyde solution as misconduct food preservation method. J Aquat Food Prod Technol 30(3):263–270. https://doi.org/10.1080/10498850.2021.1880510

Arumsari GP, Krianto T, Wispriyono B (2016) Behaviour of traders and manufacturers in the use of formalin in wet noodle and tofu in Jakarta. J Kesehat Masy Andalas 11(1):39–48. https://doi.org/10.24893/jkma.v11i1.261

Asare-Donkor NK, Adaagoam RA, Voegborlo RB, Adimado AA (2018) Risk assessment of Kumasi metropolis population in Ghana through consumption of fish contaminated with formaldehyde. J Toxicol 2018:4785031. https://doi.org/10.1155/2018/4785031

ASEAN (2013) Guidelines for the use of chemicals in aquaculture and measures to eliminate the use of harmful chemicals. Jakarta: ASEAN Secretariat. https://asean.org/wp-content/uploads/2021/09/ASEAN-Guideliness-forChemicals.pdf . Accessed Jan 2024 to June 2024

Asmawati A, Ahmadi A, Saputrayadi A (2021) Analysis of the use of formalin as a seafood preservative by traders at Mataram city traditional market. J Agrikan (agribisnis Perikanan) 14(2):367–375. https://doi.org/10.52046/agrikan.v14i2.367-375

Barokah GR, Anissah U, Januar HI, Ariyani F (2023) Mathematical formulation to differentiate between naturally occurred and artificially added formaldehyde in the ice stored of lizardfish ( Saurida tumbil ). Omni-Akuatika 19(1):27–33. https://doi.org/10.20884/1.oa.2023.19.1.952

BCSIR (2023) Formalin detection kit for fish and milk. Bangladesh Council of Scientific and Industrial Research. https://ifst.bcsir.gov.bd/site/notices/caee1e34-b839-42f0-87ac-2d5da620a33b/Formalin-detection-kit-for-fish-and-Milk . Accessed Jan 2024 to June 2024

BfR (2020) Fillable articles made from melamine formaldehyde resin, such as coffee-to-go cups sold as ‘Bambooware’, may leak harmful substances into hot foods. BfR opinion No 046/2019 issued 25 November 2020. https://doi.org/10.17590/20200123-134155

Bhowmik S, Begum M, Alam AN (2020) Formaldehyde-associated risk assessment of fish sold in local markets of Bangladesh. Agric Res 9(1):102–108. https://doi.org/10.1007/s40003-019-00414-w

Bhowmik S, Begum M, Hossain MA, Rahman M, Alam AKMN (2017) Determination of formaldehyde in wet marketed fish by HPLC analysis: a negligible concern for fish and food safety in Bangladesh. Egypt J Aquat Res 43(3):245–248. https://doi.org/10.1016/j.ejar.2017.08.001

Bianchi F, Careri M, Musci M, Mangia A (2007) Fish and food safety: determination of formaldehyde in 12 fish species by SPME extraction and GC-MS analysis. Food Chem 100(3):1049–1053. https://doi.org/10.1016/j.foodchem.2005.09.089

BuRO (2021) Advice From BuRO on the health risks of bamboo cups. The Netherlands Food and Consumer Product Safety Authority. https://english.nvwa.nl/documents/consumers/food/safety/documents/advice-from-buro-on-the-health-risks-of-bamboo-cups . Accessed Jan 2024 to June 2024

Chandralekha A, Baranage C, Samarajeewa U (2017) Formaldehyde levels in fish from the Kandy market. J Natl Sci Found Sri Lanka. 20(1):115–121. https://dl.nsf.gov.lk/bitstream/handle/1/6096/JNSF%2020_1_115.pdf?sequence=1 . Accessed Jan 2024 to June 2024

Chutia BC, Borah MP, Bordoloi U, Goswami LM, Bharadwaj S, Borthakur M, ... Borkataki S (2024) Qualitative detection of formaldehyde in challani fish obtained from selected fish markets of Nagaon, Assam, India. Discov Environ 2(1):26. https://doi.org/10.1007/s44274-024-00056-y

CIFT (2019) Guidance note on formaldehyde in fish. Central Institute of Fisheries Technology. https://www.google.com/search?q=9102f051224d%2FGuidance_Note_FormalineFish_20_07_2018.pdf&oq=9102f051224d%2FGuidance_Note_Formaline_Fish_20_07_2018.pdf&aqs=chrome..69i57j69i58.1213j0j7&sourceid=chrome&ie=UTF-8 . Accessed Jan 2024 to June 2024

Commission European (2018) Commission Implementing Regulation (EU) 2018/183 of 7 February 2018 concerning the denial of authorisation of formaldehyde as a feed additive belonging to the functional groups of preservatives and hygiene condition enhancers. http://data.europa.eu/eli/reg_impl/2018/183/oj . Accessed Jan 2024 to June 2024

Cruz JCD, Garcia RG, Collado ANM, Jovero RJS, Macalangcom RV, Tud RC (2019) Formalin on fresh tilapia via electronic nose and assessment of toxicity levels with reference to average adult Filipino weight. In: 2019 IEEE 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM). IEEE, p 1–6. https://doi.org/10.1109/HNICEM48295.2019.9072914

De Mesquita Cunha M, Rodrigues CR, Ferreira VS, Rodrigues AR, Alves SR, Chaves DA et al (2015) EFICIÃ? NCIA DO MÃ? TODO DE ESPECTROMETRIA DE MASSAS EM DROGAS DE ABUSO. Rev EVS-Rev Cienc Ambient Saude 42(4):409–423. https://doi.org/10.18224/est.v42i4.4356

De Souza A, Cunha ICM, Chagas JO, Pontes Carlos EB, Santos LL, Campos e Figueredo TR et al (2017) Use of mass spectrometry for the determination of formaldehyde in samples potentially toxic to humans: a brief review. Mass Spectrom. https://doi.org/10.5772/intechopen.68922

Delbono V, Larch CP, Newlands KC, Rhydderch S, Baddeley TC, Storey JMD (2022) Novel method of analysis for the determination of residual formaldehyde by high-performance liquid chromatography. Int J Anal Chem 2022. https://doi.org/10.1155/2022/9171836

Devaraj P, Babu V, Cengiz EI (2021) Qualitative detection of formaldehyde and ammonia in fish and other seafoods obtained from Chennai’s (India) fish markets. Environ Monit Assess 193:1–8. https://doi.org/10.1007/s10661-021-08871-z

Dhruve D, Shalini R, Shakila RJ, Lowanshi A, Arisekar U, Jeyasekaran G, Neethiselvan N (2022) Detection of free and bound formaldehyde concentration in commercially important fishes and their effect on cooking: a preliminary study. Pharma Innov 11(9S):3048–3053. https://doi.org/10.22271/tpi.2022.v11.i9Saj.15939

Duarte AM, Silva F, Pinto FR, Barroso S, Gil MM (2020) Quality assessment of chilled and frozen fish—mini review. Foods 9(12):1739. https://doi.org/10.3390/foods9121739

EFSA (2014a) Endogenous formaldehyde turnover in humans compared with exogenous contribution from food sources. EFSA J 12(2):1–11. https://doi.org/10.2903/j.efsa.2014.3550

EFSA (2014b) Scientific opinion on the safety and efficacy of formaldehyde for all animal species based on a dossier submitted by Adiveter S.L. EFSA J 12(2). https://doi.org/10.2903/j.efsa.2014.3562

Eills J, Budker D, Cavagnero S, Chekmenev EY, Elliott SJ, Jannin S et al (2023) Spin hyperpolarization in modern magnetic resonance. Chem Rev 123(4):1417–1551. https://doi.org/10.1021/acs.chemrev.2c00534

Ekinci Dogan C, Sancı R (2015) Formaldehyde migration in aqueous extracts from paper and cardboard food packaging materials in Turkey. Food Addit Contam Part B 8(3):221–226. https://doi.org/10.1080/19393210.2015.1057872

Encyclopedia of Cancer (2019) https://www.sciencedirect.com/referencework/9780128124857/encyclopedia-of-cancer . Accessed Jan 2024 to June 2024

European Union (2008) Regulation (EC) No 1272/2008 of the European Parliament and of the Council of 16 December 2008 on classification, labelling and packaging of substances and mixtures, amending and repealing directives 67/548/EEC and 1999/45/EC, and amending regulation (EC) No 1907/2006. http://data.europa.eu/eli/reg/2008/1272/oj . Accessed Jan 2024 to June 2024

Fappiano L, Carriera F, Iannone A, Notardonato I, Avino P (2022) A review on recent sensing methods for determining formaldehyde in agri-food chain: a comparison with the conventional analytical approaches. Foods 11(9):1351. https://doi.org/10.3390/foods11091351

FSSAI (2019) Direction under Section 16 (5) of Food Safety and Standards Act, 2006 regarding operationalisation of the food safety and standards (contaminants, toxins and residues) amendment regulation, 2018 relating to limit of formaldehyde in fresh water and marine fish. Food Safety and Standard Authority of India. https://fssai.gov.in/upload/Formalin.contamination.in.fish.advisories/2019/06/5cfe3cfa54f82Direction_Limit_Fisheries_Product_10_06_2019.pdf . Accessed Jan 2024 to June 2024

FSSAI (2023) Manual of methods of analysis of foods fish and fish products. Food Safety and Standard Authority of India. https://www.fssai.gov.in/upload/uploadfiles/files/Manual_Fish_05_01_2023_compressed.pdf . Accessed Jan 2024 to June 2024

Garcia A, Barbas C (2011) Gas chromatography-mass spectrometry (GC-MS)-based metabolomics. In: Metabolic profiling: methods and protocols. p 191–204. https://doi.org/10.1007/978-1-61737-985-7_11

Goon S, Bipasha M, Islam MS, Hossain MB (2014) Fish marketing status with formalin treatment in Bangladesh. Int J Public Health Sci 3(2):95. https://doi.org/10.11591/.v3i2.4680

Gu D-C, Zou M-J, Guo X-X, Yu P, Lin Z-W, Hu T et al (2017) A rapid analytical and quantitative evaluation of formaldehyde in squid based on Tri-step IR and partial least squares (PLS). Food Chem 229:458–463. https://doi.org/10.1016/j.foodchem.2017.02.082

Han XX, Rodriguez RS, Haynes CL, Ozaki Y, Zhao B (2021) Surface-enhanced Raman spectroscopy. Nat Rev Methods Prim 1(1):87. https://doi.org/10.1038/s43586-021-00083-6

He Y, Xu W, Qu M, Zhang C, Wang W, Cheng F (2022) Recent advances in the application of Raman spectroscopy for fish quality and safety analysis. Comp Rev Food Sci Food Saf 21(4):3647–3672. https://doi.org/10.1111/1541-4337.12968

Hladová M, Martinka J, Rantuch P, Nečas A (2019) Review of spectrophotometric methods for determination of formaldehyde. Res Pap Fac Mater Sci Technol Slovak Univ Technol 27(44):105–120. https://doi.org/10.2478/rput-2019-0012

Hoque MS, Jacxsens L, De Meulenaer B, Alam AN (2016) Quantitative risk assessment for formalin treatment in fish preservation: food safety concern in local market of Bangladesh. Procedia Food Sci 6:151–158. https://doi.org/10.1016/j.profoo.2016.02.037

Hoque MS, Jacxsens L, Rahman MB, Nowsad AA, Azad SO, De Meulenaer B et al (2018) Evaluation of artificially contaminated fish with formaldehyde under laboratory conditions and exposure assessment in freshwater fish in Southern Bangladesh. Food Control 84:1–9. https://doi.org/10.1016/j.chemosphere.2017.12.111

IARC (2004) Monographs on the evaluation of carcinogenic risk to humans – yearbook 88. International Agency for Research on Cancer. https://monographs.iarc.who.int/wp-content/uploads/2018/06/mono88.pdf . Accessed Jan 2024 to June 2024 

IARC (2012) A review of human carcinogens: chemical agents and related occupations. IARC Monogr Eval Carcinog Risks to Humans 100:225–248.  https://www.ncbi.nlm.nih.gov/books/NBK304416/

IFAS Extension (2024) Formalin Use in Aquaculture. https://edis.ifas.ufl.edu/publication/VM061 . Accessed 09 January 2024

Immaculate J, Jamila P (2018) Quality characteristics including formaldehyde content in selected sea foods of Tuticorin, southeast coast of India. Int Food Res J 25(1):293–302. http://www.ifrj.upm.edu.my/25%20(01)%202018/(39).pdf . Accessed Jan 2024 to June 2024

Islam R, Mahmud S, Aziz A, Sarker A, Nasreen M (2015) A comparative study of present status of marketing of formalin treated fishes in six districts of Bangladesh. Food Nutr Sci 06(01):124–134. https://doi.org/10.4236/fns.2015.61013

Jaman N, Hoque MS, Chakraborty SC, Hoq ME, Seal HP (2015) Determination of formaldehyde content by spectrophotometric method in some fresh water and marine fishes of Bangladesh. Int J Fish Aquat Stud 2(6):94–98. https://www.fisheriesjournal.com/archives/2015/vol2issue6/PartB/2-5-32.pdf . Accessed Jan 2024 to June 2024

Jiang L, Hu Q, Chen T, Min D, Yuan HQ, Bao GM (2020) Highly sensitive and rapid responsive fluorescence probe for determination of formaldehyde in seafood and in vivo imaging application. Spectrochim Acta Part A Mol Biomol Spectrosc 228:117789. https://doi.org/10.1016/j.saa.2019.117789

Jinadasa BKKK, Elliott C, Jayasinghe GDTM (2022) A review of the presence of formaldehyde in fish and seafood. Food Control 136:108882. https://doi.org/10.1016/j.foodcont.2022.108882

Joshi R, Bhatta R, Paudel PN, Kafle BK (2015) Formaldehyde content of selected fish from the wet markets of Kathmandu valley. Int Food Res J 22(4):1434. http://www.ifrj.upm.edu.my/22%20(04)%202015/(18).pdf . Accessed Jan 2024 to June 2024

Kaneko G, Ushio H, Ji H (2019) Application of magnetic resonance technologies in aquatic biology and seafood science. Fish Sci 85(1):1–17. https://doi.org/10.1007/s12562-018-1266-6

Karu K, Hornshaw M, Woffendin G, Bodin K, Hamberg M, Alvelius G, Griffiths WJ (2007) Liquid chromatography-mass spectrometry utilizing multi-stage fragmentation for the identification of oxysterols. J Lipid Res 48(4):976–987. https://doi.org/10.1194/jlr.M600497-JLR200

Kashyap A, Chakraborty B, Siddiqui MS, Tyagi H, Kalita H (2023) Selective and sensitive detection of formaldehyde at room temperature by tin oxide nanoparticles/reduced graphene oxide composite. ACS Appl Nano Mater 6(9):7948–7959. https://doi.org/10.1021/acsanm.0c02033

Khan MK (2013) Food adulteration and its effect on health. Community Based Med J 2(2):1–3. https://www.cbmcb.org/download/journal/Editorial.pdf . Accessed Jan 2024 to June 2024

Kontominas MG, Badeka AV, Kosma IS, Nathanailides CI (2021) Innovative seafood preservation technologies: recent developments. Animals 11(1):92. https://doi.org/10.3390/ani11010092

Kundu A, Dey P, Bera R, Sarkar R, Kim B, Kacew S et al (2020) Adverse health risk from prolonged consumption of formaldehyde-preserved carps in eastern region of Indian population. Environ Sci Pollut Res 27(14):16415–16425. https://doi.org/10.1007/s11356-020-08305-0

Laly SJ, Priya ER, Panda SK, Zynudheen AA (2018) Formaldehyde in seafood: a review. https://krishi.icar.gov.in/jspui/bitstream/123456789/20617/1/Formaldehyde%20in%20Seafood%20A%20review.pdf . Accessed Jan 2024 to June 2024

Letertre MP, Giraudeau P, De Tullio P (2021) Nuclear magnetic resonance spectroscopy in clinical metabolomics and personalized medicine: current challenges and perspectives. Front Mol Biosci 8:698337. https://doi.org/10.3389/fmolb.2021.698337

Lin C, Li Y, Peng Y, Zhao S, Xu M, Zhang L, Yang Y (2023) Recent development of surface-enhanced Raman scattering for biosensing. J Nanobiotechnology 21(1):149. https://doi.org/10.1186/s12951-023-01890-7

Lovestead T, Urness K (2019) Gas chromatography-mass spectrometry (GC−MS).  https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=926655

Ma H, Haque M, Afmiu Z, Aziz M, Sharmin K (2020) Association of natural formaldehyde level with quality attributes of selected sea fish, collected from southeast coast of Bangladesh. Sci Res J 8(4):1–7. https://doi.org/10.31364/SCIRJ/v8.i4.2020.P0420757

Malaysian Food Regulations 1985 (2006) In Food Act 1983 & regulations. International Law Book Services. https://faolex.fao.org/docs/pdf/mal27309.pdf . Accessed Jan 2024 to June 2024

Manna SK, Achar TK, Mondal S (2021) Recent advances in selective formaldehyde detection in biological and environmental samples by fluorometric and colorimetric chemodosimeters. Anal Methods 13(9):1084–1105. https://doi.org/10.1039/D0AY02201A

Manyanga VP, Uledi MA, Makoye PM, Daniel IJ, Shedafa R, Masota NE ... Kaale E (2024) Organoleptic quality and formaldehyde content of mackerel marketed in Dar es Salaam, Tanzania. Food Saf Risk 11(1):1. https://doi.org/10.1186/s40550-024-00107-5

Mastracci L, Gambella A, Bragoni A, Pigozzi S, Pastorino L, Vanni I et al (2019) Coping with formalin banning in pathology: under vacuum long-term tissue storage with no added formalin. Histochem Cell Biol 151(6):501–511. https://doi.org/10.1007/s00418-019-01765-0

Mathaweesansurn A, Detsri E (2022) A new colorimetric method for determination of formaldehyde in sea food based on anti-aggregation of gold nanoparticles. J Food Compos Anal 114:104802. https://doi.org/10.1016/j.jfca.2022.104802

Mehta NK, Chouksey MK, Balange AK, Tripathi G, Nayak BB (2017) Physicochemical and gel properties of myofibrillar protein from sin croaker (Johnius dussumieri) fish during ice storage. J Aquat Food Prod Technol 26(1):71–85. https://doi.org/10.1080/10498850.2015.1092485

Mehta NK, Majumdar RK, Pal D, Priyadarshini MB, Dhar B (2021) Screening of fish from retail markets for the presence of formalin and its effect on the quality of fresh catla. Indian J Anim Health 60(2):244–252. https://doi.org/10.36062/ijah.2021.spl.01121

Mehta NK, Rout B, Balange AK, Nayak BB (2023a) Dynamic viscoelastic behaviour, gelling properties of myofibrillar proteins and histological changes in shrimp ( L . vannamei ) muscles during ice storage. Aquacult Fish 8(2):180–189. https://doi.org/10.1016/j.aaf.2021.08.011

Mehta NK, Pal D, Majumdar RK, Priyadarshini MB, Das R, Debbarma G, Acharya PC (2023b) Effect of artificial formaldehyde treatment on textural quality of fish muscles and methods employed for formaldehyde reduction from fish muscles. Food Chem Adv 3:100328. https://doi.org/10.1016/j.focha.2023.100328

MINSAN-telegram (1985) MINSAN-telegram no. 703/3266/6/1377 08.11.1985. World J Eng Technol 3(3C):256–262. https://doi.org/10.4236/wjet.2015.33C040

Morisasa M, Kimura K, Sumida M, Fukumoto S, Tamura T, Takeuchi R, ... Goto-Inoue N (2020) Application of matrix-assisted laser desorption/ionization massspectrometry imaging for evaluating the quality of fish fillets. Foods 9(4):402. https://doi.org/10.3390/foods9040402

Nagana Gowda GA, Raftery D (2022) Quantitative NMR methods in metabolomics. In: Ghini V, Stringer KA, Luchinat C (eds) Metabolomics and its impact on health and diseases. Springer International Publishing, Cham, pp 143–164. https://doi.org/10.1007/164_2022_612

Chapter   Google Scholar  

Nash T (1953) The colorimetric estimation of formaldehyde by means of the Hantzsch reaction. Biochem J 55(3):416. https://doi.org/10.1042/bj0550416

Natsir M, Nurdin M, Syah ZR, Astuti SD, Azis T, Mulkiyan LOMZ, ... Maulidiyah M (2024) Design and fabrication of a high-performance sensor for formaldehyde detection based on graphene-TiO2/Ag electrode. Surf Eng Appl Electrochem 60(2):247–255. https://doi.org/10.3103/S1068375524020078

Nayana A and Sabu (2018) Incidence of formaldehyde residues in fresh seer fish (Scomberomorus guttatus) collected from Ernakulam fish market. Indian J Sci Res. 19(2):56–61. https://www.researchgate.net/publication/326200187_INCIDENCE_OF_FORMALDEHYDE_RESIDUES_IN_FRESH_SEER_FISH_Scomberomorus_guttatus_COLLECTED_FROM_ERNAKULAM_FISH_MARKET . Accessed Jan 2024 to June 2024

Nielsen SS, Ismail BP (2017) Mass spectrometry with high-performance liquid chromatography. In: Suzanne Nielsen S (ed) Food analysis laboratory manual. Springer International Publishing, Cham, pp 97–104. https://doi.org/10.1007/978-3-319-44127-6_9

Nirmal NP, Santivarangkna C, Rajput MS, Benjakul S, Maqsood S (2022) Valorization of fish byproducts: sources to end-product applications of bioactive protein hydrolysate. Compr Rev Food Sci Food Saf 21(2):1803–1842. https://doi.org/10.1111/1541-4337.12917

Nirmal KV, Pillay VV, Ramakrishnan, UK, Arathy SL, Bhaskaran R (2021) Estimation of formaldehyde contamination in selected sea fish species sold in Ernakulam District of Kerala State. J Punjab Acad Forensic Med Toxicol 2021; 21(1):83–90. https://doi.org/10.5958/0974-083X.2021.00014.5

Noor Aini B, Siddiquee S, Ampon K (2016) Development of formaldehyde biosensor for determination of formalin in fish samples; malabar red snapper (Lutjanus malabaricus) and longtail tuna (Thunnus tonggol). Biosensors 6(3):32. https://doi.org/10.3390/bios6030032

Noordiana N, Fatimah AB, Farhana YCB (2011) Formaldehyde content and quality characteristics of selected fish and seafood from wet markets. Int Food Res J 18(1). http://www.ifrj.upm.edu.my/18%20(01)%202011/(13)%20IFRJ-2010-073%20Noordiana[1].pdf . Accessed Jan 2024 to June 2024

Nowshad F, Islam MN, Khan MS (2018) Concentration and formation behavior of naturally occurring formaldehyde in foods. Agric Food Secur 7(1):17. https://doi.org/10.1186/s40066-018-0166-4

Nurlely AM, Yook Heng L, Ling TL (2021) Potentiometric enzyme biosensor for rapid determination of formaldehyde based on succinimide-functionalized polyacrylate ion-selective membrane. Measurement 175:109112. https://doi.org/10.1016/j.measurement.2021.109112

Paul L, Mondal DK, Paul M, Riar MGS, Ali A (2014) Intensity of formalin misuse for fish preservation in five markets of Jessore district, Bangladesh. Int J Nat Sci 77–81. http://ijnss.org/wp-content/uploads/2014/12/IJNSS-V1I2-11-pp-77-81.pdf . Accessed Jan 2024 to June 2024

Peng Y, Zhang Z, He L, Li C, Liu M (2024) NMR spectroscopy for metabolomics in the living system: recent progress and future challenges. Anal Bioanal Chem 1–16. https://doi.org/10.1007/s00216-024-05137-8

Rahman S, Majumder MAA, Ahasan R, Ahmed SM, Das P, Rahman N (2016) The extent and magnitude of formalin adulteration in fish sold in domestic markets of Bangladesh: a literature review. Int J Consum Stud 40(2):152–159. https://doi.org/10.1111/ijcs.12238

Rahman MM, Ahmed S, Hosen MM, Talukder AK (2012) Detection of formalin and quality characteristics of selected fish from wet markets at Sylhet city in Bangladesh. Bangladesh Res Publ J 7(2):161–169. https://www.researchgate.net/profile/Md-Rahman-297/publication/299820574_DETECTION_OF_FORMALIN_AND_QUALITY_CHARACTERISTICS_OF_SELECTED_FISH_FROM_WET_MARKETS_AT_SYLHET_CITY_IN_BANGLADESH/links/5705b5f508ae13eb88b97bd3/DETECTION-OF-FORMALIN-AND-QUALITY-CHARACTERISTICS-OF-SELECTED-FISH-FROM-WET-MARKETS-AT-SYLHET-CITY-IN-BANGLADESH.pdf . Accessed Jan 2024 to June 2024

Rathod NB, Nirmal NP, Pagarkar A, Özogul F, Rocha JM (2022) Antimicrobial impacts of microbial metabolites on the preservation of fish and fishery products: a review with current knowledge. Microorganisms 10(4):773. https://doi.org/10.3390/microorganisms10040773

Rovina K, Vonnie JM, Shaeera SN, Yi SX, Halid NFA (2020) Development of biodegradable hybrid polymer film for detection of formaldehyde in seafood products. Sens Bio-Sens Res 27:100310. https://doi.org/10.1016/j.sbsr.2020.100310

Saha S, Sahoo P (2021) Luminescence turn-on response of naphthalene diimide based chemosensor with formaldehyde: a novel stratagem for estimation of formaldehyde in storage fish samples. Bioorg Med Chem Lett 49:128287. https://doi.org/10.1016/j.bmcl.2021.128287

Sahu SK, Mehta NK, Samantaray S, Majumdar RK (2018) Incidence of formalin in fishes in Asian countries: some solutions. World Aquacult 43(9):66–69. https://www.was.org/Magazine/ArticleContent/1440 . Accessed Jan 2024 to June 2024

Sanyal S, Sinha K, Saha S, Banerjee S (2017) Formalin in fish trading: an inefficient practice for sustaining fish quality. Arch Pol Fish 25(1):43–50. https://doi.org/10.1515/aopf-2017-0005

Senthong P, Sirikitputtisak T, Wittayasilp S (2021) Determination of formaldehyde in fresh seafood under different washing and cooking conditions. Int J Food Saf Nutr Public Health 6(2):158–166. https://doi.org/10.1504/IJFSNPH.2021.113405

Shen Y, Wang WF, Wu YY (2015) A method for identification of exogenous formaldehyde in Bombay duck ( Harpodon nehereus ). Food Anal Methods 8:2474–2481. https://doi.org/10.1007/s12161-015-0138-2

Suwanaruang T (2018) Formalin contaminated in seafood and frozen meat at Somdet Market, Kalasin Province. J Environ Prot 9(12):1286–1293. https://doi.org/10.4236/jep.2018.912080

Thazin Y, Eamsa-Ard T, Pobkrut T, Kerdcharoen T (2019) Formalin adulteration detection in food using E-nose based on nanocomposite gas sensors. In: 2019 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia). IEEE, p 64–67.  https://doi.org/10.1109/ICCE-Asia46551.2019.8941601

TNJFU (2020) Technology Release. https://www.tnjfu.ac.in/directorates/othersdir/dor/downloads/Technology-Release-English.pdf . Accessed Jan 2024 to June 2024

USEPA (1988) Health and environmental effects profile for formaldehyde. https://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=49092 . Accessed Jan 2024 to June 2024

Valadares J, Majik MS, Tilve SG (2021) Comparison of the formaldehyde content in Indian mackerel ( Rastrelliger kanagurta ) fish using high performance liquid chromatography and UV-Vis spectrophotometry. J Aquat Food Prod Technol 30(8):980–987. https://doi.org/10.1080/10498850.2021.1961961

Vongkhampheang C, Lorbiayao B, Sulyya S, Keovongsuk V, Oudone P (2023) Detection of formalin in seafoods at markets and wholesaler-retailer stores (Case study: Xaythany District and Xaysetha District, Vientiane capital). Sujournal 9(1):286–294. http://www.su-journal.com/index.php/su/article/view/364 . Accessed Jan 2024 to June 2024

Wahed P, Razzaq MdA, Dharmapuri S, Corrales M (2016) Determination of formaldehyde in food and feed by an in-house validated HPLC method. Food Chem 202:476–483. https://doi.org/10.1016/j.foodchem.2016.01.136

Wang C, Gao Y, Qiu X, Nie L, Liu Y, Zhou R, Xiong S (2024) A rapid and interference-resistant formaldehyde detection method based on surface-enhanced Raman spectroscopy with a reaction-induced self-amplification strategy. Chemosensors 12(7):132. https://doi.org/10.3390/chemosensors12070132

Wang S, Cui X, Fang G (2007) Rapid determination of formaldehyde and sulphur dioxide in food products and Chinese herbals. Food Chem 103:1487–1493. https://doi.org/10.1016/j.foodchem.2006.09.023

WHO (World Health Organization) (2002) Formaldehyde. Concise International Chemical Assessment, Document 40, Geneva, Switzerland. https://apps.who.int/iris/handle/10665/42430 . Accessed Jan 2024 to June 2024

Wijaya DR, Syarwan NF, Nugraha MA, Ananda D, Fahrudin T, Handayani R (2023) Seafood quality detection using electronic nose and machine learning algorithms with hyperparameter optimization. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3286980

Wongniramaikul W, Limsakul W, Choodum A (2018) A biodegradable colorimetric film for rapid low-cost field determination of formaldehyde contamination by digital image colorimetry. Food Chem 249:154–161. https://doi.org/10.1016/j.foodchem.2018.01.021

World Health Organization (2001) Organic pollutants: formaldehyde. In: Air quality guidelines for Europe. Copenhagen: WHO Regional Office for Europe. https://apps.who.int/iris/handle/10665/107335 . Accessed Jan 2024 to June 2024

World Health Organization (2020) Code of practice for fish and fishery product. https://iris.who.int/bitstream/handle/10665/336524/9789240013179-eng.pdf . Accessed Jan 2024 to June 2024

WTO (World Trade Organization) (2022) Technical barriers to trade. https://docs.wto.org/dol2fe/Pages/SS/directdoc.aspx?filename=q:/G/TBTN22/EU888.pdf&Open=True . Accessed 09 January 2024

Yasin M, Irawati N, Zaidan AH, Mukti AT, Soegianto A, Rosalia DKP, Perego AM (2019) Fiber bundle sensor for detection of formaldehyde concentration in fish. Opt Fiber Technol 52:101984. https://doi.org/10.1016/j.yofte.2019.101984

Yasuhara A, Shibamoto T (1995) Quantitative analysis of volatile aldehydes formed from various kinds of fish flesh during heat treatment. J Agric Food Chem 43:94–97. https://doi.org/10.1021/jf00049a017

Yeasmin T, Reza MS, Khan MNA, Shikha FH, Kamal M (2010a) Present status of marketing of formalin treated fishes in domestic markets at Mymensingh district in Bangladesh. Int J Biol Res 1(4):21–24. https://www.researchgate.net/profile/Dr_Md_Reza/publication/268304955_Present_status_of_marketing_of_formalin_treated_fishes_in_domestic_markets_at_Mymensingh_district_in_Bangladesh/links/546ae74a0cf2f5eb18078352/Present-status-of-marketing-of-formalin-treated-fishes-in-domestic-markets-at-Mymensingh-district-in-Bangladesh.pdf . Accessed Jan 2024 to June 2024

Yeasmin T, Reza MS, Shikha FH, Khan MNA, Kamal M (2010b) Quality changes in formalin treated rohu fish ( Labeo rohita , Hamilton) during ice storage condition. Asian J Agric Sci 2(4):158–163. https://maxwellsci.com/print/ajas/v2-158-163.pdf . Accessed Jan 2024 to June 2024

Yeerum C, Issarangkura Na Ayutthaya P, Kesonkan K, Kiwfo K, Boochathum P, Grudpan K, Vongboot M (2022) Modified natural rubber as a simple chemical sensor with smartphone detection for formaldehyde content in a seafood sample. Molecules 27(7):2159. https://doi.org/10.3390/molecules27072159

Yeh TS, Lin TC, Chen CC, Wen HM (2013) Analysis of free and bound formaldehyde in squid and squid products by gas chromatography–mass spectrometry. J Food Drug Anal 21(2):190–197. https://doi.org/10.1016/j.jfda.2013.05.010

Zhang L (2018) Formaldehyde: exposure, toxicity and health effects. Vol. 37. Royal Society of Chemistry. https://books.google.co.in/books?hl=en&lr=&id=cSFdDwAAQBAJ&oi=fnd&pg=PA1&dq=103.%09Zhang+L.+2018.+Formaldehyde:+exposure,+toxicity+and+health+effects.+Vol.+37.+Royal+Society+of+Chemistry.&ots=t889wNmhwk&sig=1UsXWdvGqq6U-HV5BA93vCg2P3o&redir_esc=y#v=onepage&q&f=false . Accessed Jan 2024 to June 2024 

Zhang Z, Zhao C, Ma Y, Li G (2014) Rapid analysis of trace volatile formaldehyde in aquatic products by derivatization reaction-based surface enhanced Raman spectroscopy. Analyst 139(14):3614–3621. https://doi.org/10.1039/C4AN00200H

Zhang X, Yang G, Kong C, Shen X, Cai Y, Huang D (2019) A rapid determination of free formaldehyde content in marine products. https://doi.org/10.5220/0008185200470050

Zheng H, Ding Q, Li C, Chen W, Chen X, Lin Q, Lin D (2022) Recent progress in surface-enhanced Raman spectroscopy-based biosensors for the detection of extracellular vesicles. Anal Methods 14(42):4161–4173

Download references

Acknowledgements

The first author acknowledges the support from the Vice Chancellor, Central Agricultural University (CAU), Imphal, and the Dean, College of Fisheries, Central Agricultural University, Tripura. The first author also acknowledges the financial help received from the Institutional Development Plan-NAHEP, CAU, Imphal, for undergoing foreign training at Prince of Songkla University, Hat Yai, Thailand. This research work was supported by Mahidol University.

No funding was obtained for this study.

Author information

Naresh Kumar Mehta, Anand Vaishnav, and Nilesh Prakash Nirmal contributed equally and are first authors.

Authors and Affiliations

Department of Fish Processing Technology and Engineering, College of Fisheries, Central Agricultural University, Lembucherra, Tripura, 799210, India

Naresh Kumar Mehta, Anand Vaishnav, Mocherla Bhargavi Priyadarshini & Payel Debbarma

Department of Fisheries Technology, Faculty of Fisheries, Patuakhali Science and Technology University, Dumki, Patuakhali, 8602, Bangladesh

Mohammad Sazedul Hoque & Pronoy Mondal

Department of Food Science, Faculty of Food Science and Technology, Universiti Putra Malaysia, UPM, 43400, Serdang, Selangor, Malaysia

Mahmud Ab Rashid Nor-Khaizura

Laboratory of Food Safety and Food Integrity, Institute of Tropical Agricultural and Food Security, Universiti Putra Malaysia, UPM, 43400, Serdang, Selangor, Malaysia

Institute for Biological Resources and Marine Biotechnologies, National Research Council (IRBIM-CNR), Via L. Vaccara 61, Mazara del Vallo, 91026, Trapani, Italy

Gioacchino Bono

Dipartimento Di Scienze E Tecnologie Biologiche, Chimiche E Farmaceutiche (STEBICEF), Università Di Palermo, Palermo, Italy

Institute of Nutrition, Mahidol University, 999 Phutthamonthon 4 Road, Salaya, Nakhon Pathom, 73170, Thailand

Pankaj Koirala, Aikkarach Kettawan & Nilesh Prakash Nirmal

You can also search for this author in PubMed   Google Scholar

Contributions

Naresh Kumar Mehta: conceptualization, data curation, writing original draft and visualization.

Anand Vaishnav: data curation, writing original draft and visualization.

Mocherla Bhargavi Priyadarshini: original draft preparation, tables and figures.

Payal Debbarma: data curation and writing.

Mohammad Sazedul Hoque: reviewing and visualization.

Pronoy Mondal: original draft preparation, data curation

Mahmud Ab Rashid Nor-Khaizura: original draft preparation, reviewing and editing

Gioacchino Bono: original draft preparation, reviewing and editing

Pankaj Koirala: data curation, original draft preparation

Aikkarach Kettawan: reviewing and editing

Nilesh Prakash Nirmal: conceptualization, visualization, reviewing, editing, supervision and project management.

Corresponding author

Correspondence to Naresh Kumar Mehta .

Ethics declarations

Ethics approval.

This article does not need any ethical approval.

Competing interests

The authors declare no competing interests.

Additional information

Responsible Editor: Ester Heath

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Mehta, N.K., Vaishnav, A., Priyadarshini, M.B. et al. Formaldehyde contamination in seafood industry: an update on detection methods and legislations. Environ Sci Pollut Res (2024). https://doi.org/10.1007/s11356-024-34792-8

Download citation

Received : 04 June 2024

Accepted : 20 August 2024

Published : 03 September 2024

DOI : https://doi.org/10.1007/s11356-024-34792-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Contaminants
• Detection methods
• Legislation
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Korean Med Sci
• v.37(16); 2022 Apr 25

A Practical Guide to Writing Quantitative and Qualitative Research Questions and Hypotheses in Scholarly Articles

Edward barroga.

1 Department of General Education, Graduate School of Nursing Science, St. Luke’s International University, Tokyo, Japan.

Glafera Janet Matanguihan

2 Department of Biological Sciences, Messiah University, Mechanicsburg, PA, USA.

The development of research questions and the subsequent hypotheses are prerequisites to defining the main research purpose and specific objectives of a study. Consequently, these objectives determine the study design and research outcome. The development of research questions is a process based on knowledge of current trends, cutting-edge studies, and technological advances in the research field. Excellent research questions are focused and require a comprehensive literature search and in-depth understanding of the problem being investigated. Initially, research questions may be written as descriptive questions which could be developed into inferential questions. These questions must be specific and concise to provide a clear foundation for developing hypotheses. Hypotheses are more formal predictions about the research outcomes. These specify the possible results that may or may not be expected regarding the relationship between groups. Thus, research questions and hypotheses clarify the main purpose and specific objectives of the study, which in turn dictate the design of the study, its direction, and outcome. Studies developed from good research questions and hypotheses will have trustworthy outcomes with wide-ranging social and health implications.

INTRODUCTION

Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses. 1 , 2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results. 3 , 4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the inception of novel studies and the ethical testing of ideas. 5 , 6

It is crucial to have knowledge of both quantitative and qualitative research 2 as both types of research involve writing research questions and hypotheses. 7 However, these crucial elements of research are sometimes overlooked; if not overlooked, then framed without the forethought and meticulous attention it needs. Planning and careful consideration are needed when developing quantitative or qualitative research, particularly when conceptualizing research questions and hypotheses. 4

There is a continuing need to support researchers in the creation of innovative research questions and hypotheses, as well as for journal articles that carefully review these elements. 1 When research questions and hypotheses are not carefully thought of, unethical studies and poor outcomes usually ensue. Carefully formulated research questions and hypotheses define well-founded objectives, which in turn determine the appropriate design, course, and outcome of the study. This article then aims to discuss in detail the various aspects of crafting research questions and hypotheses, with the goal of guiding researchers as they develop their own. Examples from the authors and peer-reviewed scientific articles in the healthcare field are provided to illustrate key points.

DEFINITIONS AND RELATIONSHIP OF RESEARCH QUESTIONS AND HYPOTHESES

A research question is what a study aims to answer after data analysis and interpretation. The answer is written in length in the discussion section of the paper. Thus, the research question gives a preview of the different parts and variables of the study meant to address the problem posed in the research question. 1 An excellent research question clarifies the research writing while facilitating understanding of the research topic, objective, scope, and limitations of the study. 5

On the other hand, a research hypothesis is an educated statement of an expected outcome. This statement is based on background research and current knowledge. 8 , 9 The research hypothesis makes a specific prediction about a new phenomenon 10 or a formal statement on the expected relationship between an independent variable and a dependent variable. 3 , 11 It provides a tentative answer to the research question to be tested or explored. 4

Hypotheses employ reasoning to predict a theory-based outcome. 10 These can also be developed from theories by focusing on components of theories that have not yet been observed. 10 The validity of hypotheses is often based on the testability of the prediction made in a reproducible experiment. 8

Conversely, hypotheses can also be rephrased as research questions. Several hypotheses based on existing theories and knowledge may be needed to answer a research question. Developing ethical research questions and hypotheses creates a research design that has logical relationships among variables. These relationships serve as a solid foundation for the conduct of the study. 4 , 11 Haphazardly constructed research questions can result in poorly formulated hypotheses and improper study designs, leading to unreliable results. Thus, the formulations of relevant research questions and verifiable hypotheses are crucial when beginning research. 12

CHARACTERISTICS OF GOOD RESEARCH QUESTIONS AND HYPOTHESES

Excellent research questions are specific and focused. These integrate collective data and observations to confirm or refute the subsequent hypotheses. Well-constructed hypotheses are based on previous reports and verify the research context. These are realistic, in-depth, sufficiently complex, and reproducible. More importantly, these hypotheses can be addressed and tested. 13

There are several characteristics of well-developed hypotheses. Good hypotheses are 1) empirically testable 7 , 10 , 11 , 13 ; 2) backed by preliminary evidence 9 ; 3) testable by ethical research 7 , 9 ; 4) based on original ideas 9 ; 5) have evidenced-based logical reasoning 10 ; and 6) can be predicted. 11 Good hypotheses can infer ethical and positive implications, indicating the presence of a relationship or effect relevant to the research theme. 7 , 11 These are initially developed from a general theory and branch into specific hypotheses by deductive reasoning. In the absence of a theory to base the hypotheses, inductive reasoning based on specific observations or findings form more general hypotheses. 10

TYPES OF RESEARCH QUESTIONS AND HYPOTHESES

Research questions and hypotheses are developed according to the type of research, which can be broadly classified into quantitative and qualitative research. We provide a summary of the types of research questions and hypotheses under quantitative and qualitative research categories in Table 1 .

Research questions in quantitative research

In quantitative research, research questions inquire about the relationships among variables being investigated and are usually framed at the start of the study. These are precise and typically linked to the subject population, dependent and independent variables, and research design. 1 Research questions may also attempt to describe the behavior of a population in relation to one or more variables, or describe the characteristics of variables to be measured ( descriptive research questions ). 1 , 5 , 14 These questions may also aim to discover differences between groups within the context of an outcome variable ( comparative research questions ), 1 , 5 , 14 or elucidate trends and interactions among variables ( relationship research questions ). 1 , 5 We provide examples of descriptive, comparative, and relationship research questions in quantitative research in Table 2 .

Hypotheses in quantitative research

In quantitative research, hypotheses predict the expected relationships among variables. 15 Relationships among variables that can be predicted include 1) between a single dependent variable and a single independent variable ( simple hypothesis ) or 2) between two or more independent and dependent variables ( complex hypothesis ). 4 , 11 Hypotheses may also specify the expected direction to be followed and imply an intellectual commitment to a particular outcome ( directional hypothesis ) 4 . On the other hand, hypotheses may not predict the exact direction and are used in the absence of a theory, or when findings contradict previous studies ( non-directional hypothesis ). 4 In addition, hypotheses can 1) define interdependency between variables ( associative hypothesis ), 4 2) propose an effect on the dependent variable from manipulation of the independent variable ( causal hypothesis ), 4 3) state a negative relationship between two variables ( null hypothesis ), 4 , 11 , 15 4) replace the working hypothesis if rejected ( alternative hypothesis ), 15 explain the relationship of phenomena to possibly generate a theory ( working hypothesis ), 11 5) involve quantifiable variables that can be tested statistically ( statistical hypothesis ), 11 6) or express a relationship whose interlinks can be verified logically ( logical hypothesis ). 11 We provide examples of simple, complex, directional, non-directional, associative, causal, null, alternative, working, statistical, and logical hypotheses in quantitative research, as well as the definition of quantitative hypothesis-testing research in Table 3 .

Research questions in qualitative research

Unlike research questions in quantitative research, research questions in qualitative research are usually continuously reviewed and reformulated. The central question and associated subquestions are stated more than the hypotheses. 15 The central question broadly explores a complex set of factors surrounding the central phenomenon, aiming to present the varied perspectives of participants. 15

There are varied goals for which qualitative research questions are developed. These questions can function in several ways, such as to 1) identify and describe existing conditions ( contextual research question s); 2) describe a phenomenon ( descriptive research questions ); 3) assess the effectiveness of existing methods, protocols, theories, or procedures ( evaluation research questions ); 4) examine a phenomenon or analyze the reasons or relationships between subjects or phenomena ( explanatory research questions ); or 5) focus on unknown aspects of a particular topic ( exploratory research questions ). 5 In addition, some qualitative research questions provide new ideas for the development of theories and actions ( generative research questions ) or advance specific ideologies of a position ( ideological research questions ). 1 Other qualitative research questions may build on a body of existing literature and become working guidelines ( ethnographic research questions ). Research questions may also be broadly stated without specific reference to the existing literature or a typology of questions ( phenomenological research questions ), may be directed towards generating a theory of some process ( grounded theory questions ), or may address a description of the case and the emerging themes ( qualitative case study questions ). 15 We provide examples of contextual, descriptive, evaluation, explanatory, exploratory, generative, ideological, ethnographic, phenomenological, grounded theory, and qualitative case study research questions in qualitative research in Table 4 , and the definition of qualitative hypothesis-generating research in Table 5 .

Qualitative studies usually pose at least one central research question and several subquestions starting with How or What . These research questions use exploratory verbs such as explore or describe . These also focus on one central phenomenon of interest, and may mention the participants and research site. 15

Hypotheses in qualitative research

Hypotheses in qualitative research are stated in the form of a clear statement concerning the problem to be investigated. Unlike in quantitative research where hypotheses are usually developed to be tested, qualitative research can lead to both hypothesis-testing and hypothesis-generating outcomes. 2 When studies require both quantitative and qualitative research questions, this suggests an integrative process between both research methods wherein a single mixed-methods research question can be developed. 1

FRAMEWORKS FOR DEVELOPING RESEARCH QUESTIONS AND HYPOTHESES

Research questions followed by hypotheses should be developed before the start of the study. 1 , 12 , 14 It is crucial to develop feasible research questions on a topic that is interesting to both the researcher and the scientific community. This can be achieved by a meticulous review of previous and current studies to establish a novel topic. Specific areas are subsequently focused on to generate ethical research questions. The relevance of the research questions is evaluated in terms of clarity of the resulting data, specificity of the methodology, objectivity of the outcome, depth of the research, and impact of the study. 1 , 5 These aspects constitute the FINER criteria (i.e., Feasible, Interesting, Novel, Ethical, and Relevant). 1 Clarity and effectiveness are achieved if research questions meet the FINER criteria. In addition to the FINER criteria, Ratan et al. described focus, complexity, novelty, feasibility, and measurability for evaluating the effectiveness of research questions. 14

The PICOT and PEO frameworks are also used when developing research questions. 1 The following elements are addressed in these frameworks, PICOT: P-population/patients/problem, I-intervention or indicator being studied, C-comparison group, O-outcome of interest, and T-timeframe of the study; PEO: P-population being studied, E-exposure to preexisting conditions, and O-outcome of interest. 1 Research questions are also considered good if these meet the “FINERMAPS” framework: Feasible, Interesting, Novel, Ethical, Relevant, Manageable, Appropriate, Potential value/publishable, and Systematic. 14

As we indicated earlier, research questions and hypotheses that are not carefully formulated result in unethical studies or poor outcomes. To illustrate this, we provide some examples of ambiguous research question and hypotheses that result in unclear and weak research objectives in quantitative research ( Table 6 ) 16 and qualitative research ( Table 7 ) 17 , and how to transform these ambiguous research question(s) and hypothesis(es) into clear and good statements.

a These statements were composed for comparison and illustrative purposes only.

b These statements are direct quotes from Higashihara and Horiuchi. 16

a This statement is a direct quote from Shimoda et al. 17

The other statements were composed for comparison and illustrative purposes only.

CONSTRUCTING RESEARCH QUESTIONS AND HYPOTHESES

To construct effective research questions and hypotheses, it is very important to 1) clarify the background and 2) identify the research problem at the outset of the research, within a specific timeframe. 9 Then, 3) review or conduct preliminary research to collect all available knowledge about the possible research questions by studying theories and previous studies. 18 Afterwards, 4) construct research questions to investigate the research problem. Identify variables to be accessed from the research questions 4 and make operational definitions of constructs from the research problem and questions. Thereafter, 5) construct specific deductive or inductive predictions in the form of hypotheses. 4 Finally, 6) state the study aims . This general flow for constructing effective research questions and hypotheses prior to conducting research is shown in Fig. 1 .

Research questions are used more frequently in qualitative research than objectives or hypotheses. 3 These questions seek to discover, understand, explore or describe experiences by asking “What” or “How.” The questions are open-ended to elicit a description rather than to relate variables or compare groups. The questions are continually reviewed, reformulated, and changed during the qualitative study. 3 Research questions are also used more frequently in survey projects than hypotheses in experiments in quantitative research to compare variables and their relationships.

Hypotheses are constructed based on the variables identified and as an if-then statement, following the template, ‘If a specific action is taken, then a certain outcome is expected.’ At this stage, some ideas regarding expectations from the research to be conducted must be drawn. 18 Then, the variables to be manipulated (independent) and influenced (dependent) are defined. 4 Thereafter, the hypothesis is stated and refined, and reproducible data tailored to the hypothesis are identified, collected, and analyzed. 4 The hypotheses must be testable and specific, 18 and should describe the variables and their relationships, the specific group being studied, and the predicted research outcome. 18 Hypotheses construction involves a testable proposition to be deduced from theory, and independent and dependent variables to be separated and measured separately. 3 Therefore, good hypotheses must be based on good research questions constructed at the start of a study or trial. 12

In summary, research questions are constructed after establishing the background of the study. Hypotheses are then developed based on the research questions. Thus, it is crucial to have excellent research questions to generate superior hypotheses. In turn, these would determine the research objectives and the design of the study, and ultimately, the outcome of the research. 12 Algorithms for building research questions and hypotheses are shown in Fig. 2 for quantitative research and in Fig. 3 for qualitative research.

EXAMPLES OF RESEARCH QUESTIONS FROM PUBLISHED ARTICLES

• EXAMPLE 1. Descriptive research question (quantitative research)
• - Presents research variables to be assessed (distinct phenotypes and subphenotypes)
• “BACKGROUND: Since COVID-19 was identified, its clinical and biological heterogeneity has been recognized. Identifying COVID-19 phenotypes might help guide basic, clinical, and translational research efforts.
• RESEARCH QUESTION: Does the clinical spectrum of patients with COVID-19 contain distinct phenotypes and subphenotypes? ” 19
• EXAMPLE 2. Relationship research question (quantitative research)
• - Shows interactions between dependent variable (static postural control) and independent variable (peripheral visual field loss)
• “Background: Integration of visual, vestibular, and proprioceptive sensations contributes to postural control. People with peripheral visual field loss have serious postural instability. However, the directional specificity of postural stability and sensory reweighting caused by gradual peripheral visual field loss remain unclear.
• Research question: What are the effects of peripheral visual field loss on static postural control ?” 20
• EXAMPLE 3. Comparative research question (quantitative research)
• - Clarifies the difference among groups with an outcome variable (patients enrolled in COMPERA with moderate PH or severe PH in COPD) and another group without the outcome variable (patients with idiopathic pulmonary arterial hypertension (IPAH))
• “BACKGROUND: Pulmonary hypertension (PH) in COPD is a poorly investigated clinical condition.
• RESEARCH QUESTION: Which factors determine the outcome of PH in COPD?
• STUDY DESIGN AND METHODS: We analyzed the characteristics and outcome of patients enrolled in the Comparative, Prospective Registry of Newly Initiated Therapies for Pulmonary Hypertension (COMPERA) with moderate or severe PH in COPD as defined during the 6th PH World Symposium who received medical therapy for PH and compared them with patients with idiopathic pulmonary arterial hypertension (IPAH) .” 21
• EXAMPLE 4. Exploratory research question (qualitative research)
• - Explores areas that have not been fully investigated (perspectives of families and children who receive care in clinic-based child obesity treatment) to have a deeper understanding of the research problem
• “Problem: Interventions for children with obesity lead to only modest improvements in BMI and long-term outcomes, and data are limited on the perspectives of families of children with obesity in clinic-based treatment. This scoping review seeks to answer the question: What is known about the perspectives of families and children who receive care in clinic-based child obesity treatment? This review aims to explore the scope of perspectives reported by families of children with obesity who have received individualized outpatient clinic-based obesity treatment.” 22
• EXAMPLE 5. Relationship research question (quantitative research)
• - Defines interactions between dependent variable (use of ankle strategies) and independent variable (changes in muscle tone)
• “Background: To maintain an upright standing posture against external disturbances, the human body mainly employs two types of postural control strategies: “ankle strategy” and “hip strategy.” While it has been reported that the magnitude of the disturbance alters the use of postural control strategies, it has not been elucidated how the level of muscle tone, one of the crucial parameters of bodily function, determines the use of each strategy. We have previously confirmed using forward dynamics simulations of human musculoskeletal models that an increased muscle tone promotes the use of ankle strategies. The objective of the present study was to experimentally evaluate a hypothesis: an increased muscle tone promotes the use of ankle strategies. Research question: Do changes in the muscle tone affect the use of ankle strategies ?” 23

EXAMPLES OF HYPOTHESES IN PUBLISHED ARTICLES

• EXAMPLE 1. Working hypothesis (quantitative research)
• - A hypothesis that is initially accepted for further research to produce a feasible theory
• “As fever may have benefit in shortening the duration of viral illness, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response when taken during the early stages of COVID-19 illness .” 24
• “In conclusion, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response . The difference in perceived safety of these agents in COVID-19 illness could be related to the more potent efficacy to reduce fever with ibuprofen compared to acetaminophen. Compelling data on the benefit of fever warrant further research and review to determine when to treat or withhold ibuprofen for early stage fever for COVID-19 and other related viral illnesses .” 24
• EXAMPLE 2. Exploratory hypothesis (qualitative research)
• - Explores particular areas deeper to clarify subjective experience and develop a formal hypothesis potentially testable in a future quantitative approach
• “We hypothesized that when thinking about a past experience of help-seeking, a self distancing prompt would cause increased help-seeking intentions and more favorable help-seeking outcome expectations .” 25
• “Conclusion
• Although a priori hypotheses were not supported, further research is warranted as results indicate the potential for using self-distancing approaches to increasing help-seeking among some people with depressive symptomatology.” 25
• EXAMPLE 3. Hypothesis-generating research to establish a framework for hypothesis testing (qualitative research)
• “We hypothesize that compassionate care is beneficial for patients (better outcomes), healthcare systems and payers (lower costs), and healthcare providers (lower burnout). ” 26
• Compassionomics is the branch of knowledge and scientific study of the effects of compassionate healthcare. Our main hypotheses are that compassionate healthcare is beneficial for (1) patients, by improving clinical outcomes, (2) healthcare systems and payers, by supporting financial sustainability, and (3) HCPs, by lowering burnout and promoting resilience and well-being. The purpose of this paper is to establish a scientific framework for testing the hypotheses above . If these hypotheses are confirmed through rigorous research, compassionomics will belong in the science of evidence-based medicine, with major implications for all healthcare domains.” 26
• EXAMPLE 4. Statistical hypothesis (quantitative research)
• - An assumption is made about the relationship among several population characteristics ( gender differences in sociodemographic and clinical characteristics of adults with ADHD ). Validity is tested by statistical experiment or analysis ( chi-square test, Students t-test, and logistic regression analysis)
• “Our research investigated gender differences in sociodemographic and clinical characteristics of adults with ADHD in a Japanese clinical sample. Due to unique Japanese cultural ideals and expectations of women's behavior that are in opposition to ADHD symptoms, we hypothesized that women with ADHD experience more difficulties and present more dysfunctions than men . We tested the following hypotheses: first, women with ADHD have more comorbidities than men with ADHD; second, women with ADHD experience more social hardships than men, such as having less full-time employment and being more likely to be divorced.” 27
• “Statistical Analysis
• ( text omitted ) Between-gender comparisons were made using the chi-squared test for categorical variables and Students t-test for continuous variables…( text omitted ). A logistic regression analysis was performed for employment status, marital status, and comorbidity to evaluate the independent effects of gender on these dependent variables.” 27

EXAMPLES OF HYPOTHESIS AS WRITTEN IN PUBLISHED ARTICLES IN RELATION TO OTHER PARTS

• EXAMPLE 1. Background, hypotheses, and aims are provided
• “Pregnant women need skilled care during pregnancy and childbirth, but that skilled care is often delayed in some countries …( text omitted ). The focused antenatal care (FANC) model of WHO recommends that nurses provide information or counseling to all pregnant women …( text omitted ). Job aids are visual support materials that provide the right kind of information using graphics and words in a simple and yet effective manner. When nurses are not highly trained or have many work details to attend to, these job aids can serve as a content reminder for the nurses and can be used for educating their patients (Jennings, Yebadokpo, Affo, & Agbogbe, 2010) ( text omitted ). Importantly, additional evidence is needed to confirm how job aids can further improve the quality of ANC counseling by health workers in maternal care …( text omitted )” 28
• “ This has led us to hypothesize that the quality of ANC counseling would be better if supported by job aids. Consequently, a better quality of ANC counseling is expected to produce higher levels of awareness concerning the danger signs of pregnancy and a more favorable impression of the caring behavior of nurses .” 28
• “This study aimed to examine the differences in the responses of pregnant women to a job aid-supported intervention during ANC visit in terms of 1) their understanding of the danger signs of pregnancy and 2) their impression of the caring behaviors of nurses to pregnant women in rural Tanzania.” 28
• EXAMPLE 2. Background, hypotheses, and aims are provided
• “We conducted a two-arm randomized controlled trial (RCT) to evaluate and compare changes in salivary cortisol and oxytocin levels of first-time pregnant women between experimental and control groups. The women in the experimental group touched and held an infant for 30 min (experimental intervention protocol), whereas those in the control group watched a DVD movie of an infant (control intervention protocol). The primary outcome was salivary cortisol level and the secondary outcome was salivary oxytocin level.” 29
• “ We hypothesize that at 30 min after touching and holding an infant, the salivary cortisol level will significantly decrease and the salivary oxytocin level will increase in the experimental group compared with the control group .” 29
• EXAMPLE 3. Background, aim, and hypothesis are provided
• “In countries where the maternal mortality ratio remains high, antenatal education to increase Birth Preparedness and Complication Readiness (BPCR) is considered one of the top priorities [1]. BPCR includes birth plans during the antenatal period, such as the birthplace, birth attendant, transportation, health facility for complications, expenses, and birth materials, as well as family coordination to achieve such birth plans. In Tanzania, although increasing, only about half of all pregnant women attend an antenatal clinic more than four times [4]. Moreover, the information provided during antenatal care (ANC) is insufficient. In the resource-poor settings, antenatal group education is a potential approach because of the limited time for individual counseling at antenatal clinics.” 30
• “This study aimed to evaluate an antenatal group education program among pregnant women and their families with respect to birth-preparedness and maternal and infant outcomes in rural villages of Tanzania.” 30
• “ The study hypothesis was if Tanzanian pregnant women and their families received a family-oriented antenatal group education, they would (1) have a higher level of BPCR, (2) attend antenatal clinic four or more times, (3) give birth in a health facility, (4) have less complications of women at birth, and (5) have less complications and deaths of infants than those who did not receive the education .” 30

Research questions and hypotheses are crucial components to any type of research, whether quantitative or qualitative. These questions should be developed at the very beginning of the study. Excellent research questions lead to superior hypotheses, which, like a compass, set the direction of research, and can often determine the successful conduct of the study. Many research studies have floundered because the development of research questions and subsequent hypotheses was not given the thought and meticulous attention needed. The development of research questions and hypotheses is an iterative process based on extensive knowledge of the literature and insightful grasp of the knowledge gap. Focused, concise, and specific research questions provide a strong foundation for constructing hypotheses which serve as formal predictions about the research outcomes. Research questions and hypotheses are crucial elements of research that should not be overlooked. They should be carefully thought of and constructed when planning research. This avoids unethical studies and poor outcomes by defining well-founded objectives that determine the design, course, and outcome of the study.

Disclosure: The authors have no potential conflicts of interest to disclose.

Author Contributions:

• Conceptualization: Barroga E, Matanguihan GJ.
• Methodology: Barroga E, Matanguihan GJ.
• Writing - original draft: Barroga E, Matanguihan GJ.
• Writing - review & editing: Barroga E, Matanguihan GJ.