• Digital Manufacturing
• Smart Factory
Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).
Research questions examples.
Research Questions examples |
---|
• : What are the current challenges of context-aware systems that support the decision-making of business processes in smart manufacturing? • : Which technique is most appropriate to support decision-making for business process management in smart factories? • : In which scenarios are semantic web and machine learning used to provide context-awareness in business process management for smart manufacturing? |
The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.
Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.
Database | Description | URL | Area | Advanced Search Y/N |
---|---|---|---|---|
Scopus | From Elsevier. sOne of the largest databases. Very user-friendly interface | Interdisciplinary | Y | |
Web of Science | From Clarivate. Multidisciplinary database with wide ranging content. | Interdisciplinary | Y | |
EI Compendex | From Elsevier. Focused on engineering literature. | Engineering | Y (Query view not available) | |
IEEE Digital Library | Contains scientific and technical articles published by IEEE and its publishing partners. | Engineering and Technology | Y | |
ACM Digital Library | Complete collection of ACM publications. | Computing and information technology | Y |
Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.
Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.
Criteria Type | Description | Example |
---|---|---|
Period | Articles can be selected based on the time period to review, e.g., reviewing the technology under study from the year it emerged, or reviewing progress in the field since the publication of a prior literature review. | : From 2015 to 2021 Articles prior 2015 |
Language | Articles can be excluded based on language. | : Articles not in English |
Type of Literature | Articles can be excluded if they are fall into the category of grey literature. | Reports, policy literature, working papers, newsletters, government documents, speeches |
Type of source | Articles can be included or excluded by the type of origin, i.e., conference or journal articles or books. | : Articles from Conferences or Journals Articles from books |
Impact Source | Articles can be excluded if the author limits the impact factor or quartile of the source. | Articles from Q1, and Q2 sources : Articles with a Journal Impact Score (JIS) lower than |
Accessibility | Not accessible in specific databases. | : Not accessible |
Relevance to research questions | Articles can be excluded if they are not relevant to a particular question or to “ ” number of research questions. | Not relevant to at least 2 research questions |
Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.
Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.
Do the researchers discuss any problems (limitations, threats) with the validity of their results (reliability)? | 1 – No, and not considered (Score: 0) 2 – Partially (Score: 0.5) 3 – Yes (Score: 1) |
Is there a clear definition/ description/ statement of the aims/ goals/ purposes/ motivations/ objectives/ questions of the research? | 1 – Disagree (Score: 1) 2 – Somewhat disagree (Score: 2) 3 – Neither agree nor disagree (Score: 3) 4 – Somewhat agree (Score: 4) 5 – Agree (Score: 5) |
The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .
Planning Step 6 “Define data extraction form”. Examples of fields.
Classification and fields to consider for data extraction | Description and examples |
---|---|
Research type | • focuses on abstract ideas, concepts, and theories built on literature reviews . • uses scientific data or case studies for explorative, descriptive, explanatory, or measurable findings . an SLR on context-awareness for S-PSS and categorized the articles in theoretical and empirical research. |
By process phases, stages | When analyzing a process or series of processes, an effective way to structure the data is to find a well-established framework of reference or architecture. : • an SLR on self-adaptive systems uses the MAPE-K model to understand how the authors tackle each module stage. • presented a context-awareness survey using the stages of context-aware lifecycle to review different methods. |
By technology, framework, or platform | When analyzing a computer science topic, it is important to know the technology currently employed to understand trends, benefits, or limitations. : • an SLR on the big data ecosystem in the manufacturing field that includes frameworks, tools, and platforms for each stage of the big data ecosystem. |
By application field and/or industry domain | If the review is not limited to a specific “Context” or “Population" (industry domain), it can be useful to identify the field of application : • an SLR on adaptive training using virtual reality (VR). The review presents an extensive description of multiple application domains and examines related work. |
Gaps and challenges | Identifying gaps and challenges is important in reviews to determine the research needs and further establish research directions that can help scholars act on the topic. |
Findings in research | Research in computer science can deliver multiple types of findings, e.g.: |
Evaluation method | Case studies, experiments, surveys, mathematical demonstrations, and performance indicators. |
The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.
After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.
Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:
(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”
Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.
Example of Advanced search on Scopus.
In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.
The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.
In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.
Performing quality assessment (QA) in Parsif.al.
Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.
Example of data extraction form using Parsif.al.
The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.
The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.
The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.
Techniques for bibliometric analysis and examples.
Publication-related analysis | Description | Example |
---|---|---|
Years of publications | Determine interest in the research topic by years or the period established by the SLR, by quantifying the number of papers published. Using this information, it is also possible to forecast the growth rate of research interest. | [ ] identified the growth rate of research interest and the yearly publication trend. |
Top contribution journals/conferences | Identify the leading journals and conferences in which authors can share their current and future work. | , |
Top countries' or affiliation contributions | Examine the impacts of countries or affiliations leading the research topic. | [ , ] identified the most influential countries. |
Leading authors | Identify the most significant authors in a research field. | - |
Keyword correlation analysis | Explore existing relationships between topics in a research field based on the written content of the publication or related keywords established in the articles. | using keyword clustering analysis ( ). using frequency analysis. |
Total and average citation | Identify the most relevant publications in a research field. | Scatter plot citation scores and journal factor impact |
Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.
[1] Keyword co-relationship analysis using clusterization in vos viewer.
This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.
Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.
The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].
SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.
Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.
Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.
1 https://parsif.al/
Reference management. Clean and simple.
Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.
A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.
Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:
➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.
Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.
Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.
A classic systematic literature review can take different approaches:
Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.
When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.
You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.
Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.
You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:
Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.
The protocol for your systematic literature review should include:
For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.
When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .
This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.
➡️ Read more on on how to efficiently search research databases .
The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.
➡️ Take a look at our list of the top academic research databases .
Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.
Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:
Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:
➡️ Read more about gray literature .
This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:
Make sure reviewers keep a log of which studies they exclude, with reasons why.
➡️ Visit our guide on what is an abstract?
Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.
The kinds of questions you'll come up with are:
Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.
Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.
Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.
Measures of outcome for studies with:
Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.
Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.
Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.
Apply appropriate grading of your evidence and consider the strength of your recommendations.
It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.
Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .
Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.
The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.
Your systematic review should include the following details:
A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.
The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .
Part of the book series: Familienforschung ((FAFO))
1741 Accesses
Die dieser Arbeit zu Grunde liegenden Forschungsfragen sind konzeptioneller Art. Als Methode zur (Weiter-) Entwicklung konzeptioneller und theoretischer Modelle wird ein systematisches Review verwendet. Nachfolgend werden die einzelnen Arbeitsschritte des systematischen Reviews vorgestellt und deren Umsetzung für die vorliegende Forschungsarbeit beschrieben.
This is a preview of subscription content, log in via an institution to check access.
Subscribe and save.
Tax calculation will be finalised at checkout
Purchases are for personal use only
Institutional subscriptions
Der British Education Index konnte wegen fehlender Lizenz nicht als Datenquelle herangezogen werden.
Seit Ende 2017 steht die Metadatenbank nicht mehr zur Verfügung.
Bedingt zum einen durch die hohe Trefferanzahl und zum anderen durch die getroffenen Maßnahmen zur Qualitätssicherung, vor allem durch die Wiederholung des Selektionsprozesses, hat die Durchführung des systematischen Reviews hohe zeitliche Ressourcen benötigt. Um dennoch die Aktualität der Ergebnisse zu bestätigen, wurde ein weiteres Review für das Jahr 2016 durchgeführt. Ausführliche Erläuterungen dazu inklusive der Ergebnisse des zweiten Reviews sind im Abschnitt 6.4 enthalten.
Literatur, die den Familienalltag bis zur Zeit des zweiten Weltkrieges thematisiert, wurde ausgeschlossen.
Ausnahme bildeten Artikel, die sich zwar auf die gleiche Studie bzw. theoretischen Ansatz bezogen, jedoch eine erweiterte Darstellung von Familienalltag anboten.
Erst bei der Volltextrecherche wird die Qualitätsbeurteilung als Ausschlusskriterium einbezogen. Nur Studien mit unzureichender Qualität werden ausgeschlossen.
Mit Blick auf die Ergebnisse des Selektionsprozesses der elektronischen Datenbanken zeigte sich, dass mittels dieses Ausschlusskriteriums nur 234 Artikel (1,0 % der Ausgangstreffer) ausgeschlossen wurden (vgl. Abschnitt 5.3.2.1 ). Somit ist eine negative Auswirkung auf das Ergebnis der vorliegenden Arbeit im Sinne fehlender Dimensionen bzw. Konkretisierungen der Dimensionen nicht zu erwarten.
In einer ersten groben Literatursichtung zur Erstellung der Suchmatrix für die Literatursuche in den elektronischen Datenbanken (vgl. Abschnitt 5.2.1 ) war auffällig, dass ein großer Anteil an Literatur vorhanden war, welcher sich mit medizinischen Themen bzw. Themen des abweichenden Verhaltens beschäftigt. Diese Erkenntnis ermöglichte vorab, dies als Ausschlusskriterium aufzunehmen. Im Selektionsprozess der Treffer der elektronischen Datenbanken konnten so 824 Artikel (3,5 % der Ausgangstreffer) ausgeschlossen werden (vgl. Abschnitt 5.3.2.1 ).
Falls mehrere Ausschlussgründe gleichzeitig zutrafen, wurde der Grund ausgewählt, der am meisten im Fokus des jeweiligen Titels stand.
Die Interpretation von Yule’s Y erfolgt wie folgt: 0 = kein Zusammenhang, -1 = sehr starker, perfekter negativer Zusammenhang, + 1 = sehr starker, perfekter positiver Zusammenhang.
Die meisten der eingeschlossenen Studien verwenden qualitative Verfahren, was zudem eine objektive Qualitätsbewertung etwas erschweren würde, da keine festen allgemeingültigen Gütekriterien wie in der quantitativen Forschung vorliegen. Seit Mitte der 1980er Jahre wird verstärkt eine Diskussion über die Qualität der qualitativen Forschung geführt (Flick, 2016, S. 485–567), welche bis heute noch anhält (zur Qualitätsbeurteilung qualitativer Forschung siehe vertiefend Reid und Gough (2000); Seale (1999); Steinke (1999), (2005); Tong, Sainsbury und Craig (2007)).
Zur Erhöhung des kreativen Prozesses wird daher empfohlen, diesen Arbeitsschritt in einer Gruppe umzusetzen. Auf Grund des Charakters einer Qualifizierungsarbeit konnte das für die vorliegende Arbeit nicht umgesetzt werden.
Authors and affiliations.
Otto-Friedrich-Universität Bamberg, Bamberg, Deutschland
Doreen Müller
You can also search for this author in PubMed Google Scholar
Correspondence to Doreen Müller .
Zusatzmaterial 1 (pdf 1208 kb), rights and permissions.
Reprints and permissions
© 2023 Der/die Autor(en), exklusiv lizenziert durch Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature
Müller, D. (2023). Methode: Durchführung eines systematischen Reviews. In: Familienalltag: Ein systematisches Review zur Entwicklung eines konzeptionellen Modells. Familienforschung. Springer VS, Wiesbaden. https://doi.org/10.1007/978-3-658-35599-9_5
DOI : https://doi.org/10.1007/978-3-658-35599-9_5
Published : 12 September 2022
Publisher Name : Springer VS, Wiesbaden
Print ISBN : 978-3-658-35598-2
Online ISBN : 978-3-658-35599-9
eBook Packages : Social Science and Law (German Language)
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
Policies and ethics