ACADEMIC WRITING GUIDE
How to Assess Risk of Bias in a Systematic Review for a Dissertation (With Worked Examples)
Two studies may investigate the same intervention, use similar sample sizes and even report similar effects. Yet one may have concealed random allocation and measured outcomes consistently, while the other may have lost many participants and reported only its most favourable outcome. Treating their findings as equally reliable would weaken a dissertation systematic review.
A risk-of-bias assessment asks a focused question: Could systematic problems in the way a study was designed, conducted, analysed or reported have distorted the result I want to use? It is not an exercise in finding faults with authors. It helps readers judge how much confidence to place in particular findings and why.
This guide explains how to choose appropriate appraisal tools, work through the evidence, document reasoned judgments, and report the implications in your methods, results and discussion chapters. It includes a detailed hypothetical randomised-trial example, a non-randomised intervention example and a model results table. Every study scenario, number and judgment below is invented for teaching purposes, not evidence from an actual systematic review.
What is risk of bias in a systematic review?
Risk of bias is the possibility that systematic errors have shifted a study's estimate or interpretation away from the answer it would otherwise provide. These errors are different from random sampling variation. A result can be extremely precise yet biased; a small, carefully conducted study can have little identifiable systematic bias but substantial statistical uncertainty.
For example, imagine a study of an academic-support intervention that compares students who voluntarily attend weekly sessions with students who do not. If those who volunteer are already more motivated, the groups may differ before the sessions begin. An apparent improvement in grades may reflect pre-existing motivation, the intervention, or a combination. The issue is confounding, not merely that the study is observational.
Crucially, risk of bias is often outcome- or result-specific. A trial might report attendance from administrative records with relatively little measurement concern, while reporting self-rated confidence from participants who know which intervention they received. A single blanket verdict for the entire article can conceal this difference. Cochrane's RoB 2 and ROBINS-I frameworks focus on the result being assessed, rather than awarding one universal score to a paper.
Risk-of-bias appraisal usually follows study selection and data extraction. If you are still identifying the evidence base, begin with our systematic literature-search guide. For a transparent account of how records became included studies, see our PRISMA 2020 flow-diagram guide. The PRISMA flow diagram records selection; it does not replace appraisal of included evidence.
Risk of bias, critical appraisal and certainty of evidence: what is the difference?
These terms overlap but are not interchangeable.
| Question | Main concern | Illustration |
|---|---|---|
| Risk of bias | Could a systematic flaw distort a particular study result? | Knowledge of group assignment may influence a subjective outcome. |
| Critical appraisal / methodological quality | How trustworthy, relevant and methodologically appropriate is this research more broadly? | Does a qualitative study adequately explain its analytic approach and reflexivity? |
| Reporting quality | Has the study described enough detail for readers to understand and evaluate it? | The paper does not explain how participants were allocated. |
| Certainty in a body of evidence | How confident should we be in the collective evidence for an outcome? | Across several studies, bias, inconsistency, imprecision and other considerations may lower confidence. |
For broader methods of evaluating research articles, see our critical-appraisal guide.
Poor reporting may prevent a reviewer from judging risk of bias, but lack of detail is not automatically proof that a flawed procedure occurred. Conversely, beautifully written methods can describe a genuinely biased design. Likewise, an individual study with a favourable appraisal does not automatically make a whole body of evidence certain.
PRISMA 2020 treats study risk of bias, risk of bias due to missing results, and certainty of evidence as distinguishable reporting topics. This matters when planning your dissertation: do not assume one colour-coded table answers all three questions.
Step 1: Define the specific result you are evaluating
Before choosing a tool, write down four items: the review question, the design of the included study, the outcome or finding of interest, and the comparison or effect being interpreted.
Suppose your review asks: Among university students, do structured feedback workshops improve assignment performance compared with usual teaching? One included randomised trial reports an adjusted mean difference in assessment scores at eight weeks. Your assessment should focus on that result and its intended interpretation. If the same trial also reports self-efficacy scores, that second outcome may require its own assessment.
Record the result's source: article page/table, supplementary material, trial registration, protocol or other verified study documentation. This makes it possible to explain exactly what informed each judgment rather than relying on an impression of the paper.
Do not decide a study is low risk because its results support your hypothesis. The assessment must be independent of whether the effect is large, statistically significant or personally persuasive.
Step 2: Choose a tool that fits the study design and review question
There is no single best checklist for every systematic review. Select a recognised instrument based on the studies and outcomes actually included, then use its current manual and judgment rules.
| Included evidence | Suitable starting point | Important qualification |
|---|---|---|
| Randomised trials estimating intervention effects | Cochrane RoB 2 | Result-specific assessment across five bias domains, using its official signalling questions and algorithms. |
| Non-randomised studies estimating effects of interventions | ROBINS-I | Requires a carefully defined comparison and attention to confounding, selection and other intervention-related biases; specialist judgment may be needed. |
| Analytical cross-sectional, cohort and other non-trial designs | Design-specific JBI critical appraisal / risk-of-bias tools, when suitable | Confirm the exact tool version, research purpose and whether it evaluates the kind of bias relevant to your review. |
| Diagnostic test accuracy studies | An appropriate diagnostic-accuracy tool, such as QUADAS-2 where applicable | Test accuracy questions require different domains from intervention-effect trials; follow current discipline guidance. |
| Qualitative evidence syntheses | A suitable qualitative critical-appraisal framework, such as JBI's qualitative tool | Credibility, methodology and interpretation are not well represented by forcing qualitative studies into an intervention-trial tool. |
The Cochrane RoB 2 guidance describes a structured trial-result assessment; Cochrane's ROBINS-I resource explains its intended use for non-randomised intervention studies; and JBI's critical appraisal tools provide design-specific resources. These tools are not interchangeable labels for a generic “quality checklist.”
If your review includes several study designs, document which tool applies to which design. Do not force every study into a single instrument merely to produce a tidier table. For a scoping review, clarify whether critical appraisal is appropriate to its objectives rather than assuming it is always required. For a conventional narrative literature review, structured critical appraisal may still be helpful, but do not claim to have conducted a formal systematic review unless the overall methods justify that label.
Step 3: Prespecify how appraisals will be conducted
A transparent protocol explains who will assess each result, which materials reviewers will inspect, how disagreements will be handled and how judgments will influence synthesis. Ideally these decisions are made before the reviewers know which results would favour the emerging conclusion.
For a formal review, two independent assessors can reduce idiosyncratic judgments; whether this is mandatory depends on the review method, supervisor and protocol. If you are the sole dissertation researcher, do not claim that a second independent assessor participated when they did not. Instead, describe the actual arrangement and its limitations, and obtain a permitted second assessment or adjudication where feasible.
A practical extraction sheet includes:
Study identifier, citation and study design.
Outcome, time point, contrast and effect/result being assessed.
Chosen appraisal tool and version.
Each domain's signalling-question responses and supporting source locations.
Domain judgments, overall judgment under the tool's rules, and rationale.
Assessor identity, date, disagreements and resolutions if applicable.
A pilot on a small number of different study designs can expose inconsistent interpretations. Agree on what counts as evidence for a decision before assessing the rest. If the appraisal plan changes, document the amendment instead of silently rewriting the protocol. Our preregistration guide explains why recorded decisions and transparent deviations matter.
Step 4: Apply the domains rather than relying on an overall impression
For randomised intervention trials, RoB 2 addresses five broad areas. The descriptions below are an accessible overview, not replacements for the official tool's signalling questions.
Bias arising from the randomisation process
Was the allocation sequence random, was it concealed until assignment, and do baseline differences suggest a problem? Describing groups as “randomly selected” is not enough: selecting people randomly from a population is different from randomly allocating them to comparison groups.
Bias due to deviations from intended interventions
Did participants or personnel depart from assigned interventions in ways relevant to the target effect? The correct assessment depends on whether the review is estimating the effect of assignment to the intervention or the effect of adhering to it. Do not assume that lack of blinding automatically makes every outcome high risk.
Bias due to missing outcome data
How much outcome information is missing, why might it be missing, and could missingness depend on the unobserved outcome? A percentage alone is not decisive. Five percent missing for reasons closely linked to poor outcomes may be more concerning than a larger amount missing for demonstrably unrelated reasons.
Bias in measurement of the outcome
Were outcomes measured appropriately and consistently across groups? Could an assessor's knowledge of the intervention have affected judgment? Administrative attendance data and unblinded subjective performance ratings can have different vulnerabilities.
Bias in selection of the reported result
Were the outcome measures, analysis methods and reporting decisions chosen after seeing results? Compare available protocols and registrations with the published analysis where possible. Multiple measures and flexible analytical choices may create opportunities to emphasise favourable findings.
The official Cochrane Handbook chapter on randomised-trial risk of bias explains how these domains, signalling questions and overall judgments work. Use the official algorithm; do not add domain scores or calculate an arbitrary percentage of quality.
Worked example 1: Assessing a hypothetical randomised trial
Imagine Study A, an entirely fictional randomised trial of weekly feedback workshops involving 120 undergraduate students. Sixty students were assigned to workshops and sixty to usual teaching. The outcome for the worked example is an eight-week assignment score, measured on a 0–100 scale. The trial reports an illustrative mean difference of +5 points favouring workshops, but the following information is all we know about its conduct:
Allocation was generated by a computer, but the publication does not establish whether the next assignment was concealed from recruiters.
The workshops were visible to participants; the study reports no clear intervention deviations relevant to the assignment-effect question.
Outcome scores are available for 104 of 120 participants. More students with missing scores were in the usual-teaching group, and reasons are incompletely documented.
Scores were based on standardised assignments, but markers may have known students' group assignments.
A protocol lists the eight-week assignment score as an outcome, but a complete pre-specified analysis plan is unavailable.
Example evidence-to-judgment worksheet
| RoB 2 domain | Evidence to check | Illustrative concern; not a formal tool output |
|---|---|---|
| Randomisation process | Generation and allocation concealment | Computer generation is reassuring; concealment remains unclear. |
| Deviations from intended interventions | Group awareness, departures, relevant analysis | Awareness alone does not establish bias; examine actual deviations and target effect. |
| Missing outcome data | 16/120 outcomes missing, distribution, reasons | Unequal and unexplained missingness could affect the estimated effect. |
| Measurement of outcome | Marking rules, assessor knowledge, objectivity | If markers knew allocation and exercise judgment, measurement bias may be possible. |
| Selection of reported result | Protocol, analysis plan, multiple outcome definitions | Outcome is listed, but analysis-selection flexibility remains uncertain. |
What can we conclude? It would be inappropriate to call Study A “low risk” simply because it used a computer-generated allocation sequence. There are unresolved questions about concealment, missing outcomes, assessor blinding and analysis selection. Equally, we cannot assign a defensible overall RoB 2 category from this short fictional description alone. A real assessor would answer the relevant signalling questions, consult the full reports and protocol, run the tool's specified judgment process, and retain supporting quotations or page references.
A defensible dissertation write-up would say: “In the illustrative trial, the available description raised concerns about allocation concealment, missing outcome data and possible assessor awareness. A final tool-based risk-of-bias judgment would require full source documentation and completed signalling-question assessments.” That is more transparent than inventing a “7/10 quality score.”
Notice also that the hypothetical +5-point estimate is not proof that workshops improve performance in a real population. The invented result is included solely to show why outcome-level appraisal matters.
Worked example 2: A non-randomised intervention study and confounding
Now imagine Study B, a hypothetical university cohort study comparing students who choose optional academic coaching with those who do not. The study reports that coaching participants achieved assignment scores 4 points higher on average, after statistical adjustment for year of study and prior GPA.
What could still bias that estimate? Students opting into coaching might differ in motivation, study time, family resources or access to other assistance. Some differences may be unmeasured; others may have been measured too imprecisely for adjustment to remove the bias. The intervention may also begin at different times or be defined inconsistently across participants.
With an appropriate version of ROBINS-I, reviewers would first specify the causal effect and hypothetical target trial, identify important confounding factors, and assess bias across the applicable domains. The tool requires more than writing “observational study = high risk.” It asks whether the study's particular methods plausibly distorted the particular comparison being interpreted.
A useful dissertation statement is: “Because coaching uptake was self-selected, the study's adjusted association may remain vulnerable to confounding by characteristics related both to enrolment and assignment performance. The assessment should consider measured and unmeasured confounders, intervention classification, participant selection, missing information, outcome measurement and selection of the reported analysis under the specified ROBINS-I version.”
Again, this is a reasoning demonstration, not a completed ROBINS-I judgment. Reviewers should use the actual instrument, including version-specific guidance, rather than assigning labels from a condensed example. For complex non-randomised intervention syntheses, specialist methodological supervision is valuable.
Step 5: Record decisions so another reader can understand them
A useful risk-of-bias table does not merely display coloured cells. It tells readers which result was evaluated, which instrument was applied and what evidence supports the conclusion. Where reporting space is limited, place the complete domain-level assessments in an appendix and summarise the relevant patterns in the main chapter.
Model dissertation results table — fictional illustration
| Fictional study | Result under assessment | Tool | Information prompting review | Assessment status |
|---|---|---|---|---|
| Study A | Eight-week assignment score difference | RoB 2 | Missing outcomes and assessor awareness need clarification | Final judgment not assigned: complete signalling questions first |
| Study B | Coaching versus no coaching, adjusted score association | ROBINS-I | Self-selection and residual confounding may distort the comparison | Final judgment not assigned: define target trial and complete tool |
| Study C | Student experiences of academic feedback | Design-appropriate qualitative appraisal | Researcher role, sampling and interpretive transparency need evaluation | Use a qualitative framework; not RoB 2 |
Table note: Studies A, B and C do not exist. The “assessment status” column intentionally avoids fabricated low/high labels. It illustrates responsible reporting when the full information required by the instruments has not been supplied.
For an actual dissertation, replace these rows with verified included studies and completed appraisals. Use the tool's approved categories exactly. For RoB 2, these include low risk, some concerns and high risk. Do not translate those categories into made-up numerical thresholds. If a judgement could not be made because important source details were unavailable, record that limitation honestly and follow the instrument's guidance.
Step 6: Explain how risk of bias changes the synthesis
Appraisal adds value only when it affects interpretation. A review should explain whether the main conclusion depends on studies with important bias concerns, not leave the reader to inspect a disconnected table.
Imagine that four eligible studies, also entirely hypothetical, all favour an intervention. Two have relatively robust methods, whereas two rely on unblinded subjective assessments and have substantial missing outcome data. The correct interpretation is not simply “four studies support the intervention.” Instead, compare the direction and magnitude of findings across the studies and ask whether the apparent consistency survives consideration of their vulnerabilities.
Where your pre-specified synthesis plan and sufficient data permit, you might conduct a sensitivity analysis that removes studies meeting a clearly defined high-risk criterion and compares the result with the primary synthesis. Report what happened, including when estimates are too imprecise to support a strong conclusion. Our sensitivity-analysis guide explains how to frame such comparisons without treating every analytical change as evidence of robustness.
Do not automatically exclude all studies with a high-risk judgment after seeing their results. Exclusion, down-weighting, subgrouping or narrative qualification should follow the review question, the protocol and a defensible synthesis strategy. Standard meta-analytic inverse-variance weighting reflects statistical precision, not a built-in discount for bias. Avoid arbitrary numerical “quality weights.”
Appraisal must also inform the discussion chapter: explain how bias could affect the direction of the observed estimate, whether evidence remains consistent across designs or outcomes, and which claims are still defensible. An appropriate conclusion may be that evidence is promising but uncertain—not that an intervention is proven effective.
How to report risk of bias under PRISMA 2020
PRISMA 2020 is a reporting guideline, not a substitute for conducting the appraisal. Several items are directly relevant:
| PRISMA 2020 item | What the dissertation should report |
|---|---|
| 11 — Study risk-of-bias assessment | The method, selected tool(s), number of reviewers, whether they worked independently and relevant process details. |
| 18 — Risk of bias in studies | The actual assessments for each included study/result, as appropriate. |
| 20a — Results of syntheses | A summary of risk of bias among studies contributing to each synthesis. |
| 13f / 20d — Sensitivity analyses | Planned robustness methods and the results of analyses actually performed. |
| 14 / 21 — Reporting biases | Methods and findings concerning results that may be missing from a synthesis. |
| 15 / 22 — Certainty of evidence | Methods and findings for outcome-level certainty when assessed. |
The PRISMA 2020 statement and checklist clarify these distinctions. Report only procedures actually undertaken. If one assessor performed all judgments, say so; if a tool was planned but could not be completed, explain why. Neither a PRISMA diagram nor a completed checklist alone establishes methodological quality.
Model methods paragraph — edit to reflect what you actually did
Risk of bias in the included studies was assessed using design-appropriate instruments selected before appraisal. For randomised intervention-effect results, the review used the RoB 2 framework; for non-randomised intervention-effect results, it used ROBINS-I. Each assessment was linked to a defined outcome and comparison, with domain-level reasoning documented against information from reports and, when available, protocols or registrations. The number of assessors, independence of judgments, disagreement-resolution process and handling of unclear information are reported in the completed review records. Findings were considered when interpreting the synthesis rather than converted into a numerical study-quality score.
Important: This is a model of structure, not a claim that any particular dissertation actually used those methods. Replace the conditional/general wording with truthful details of your tools, versions, assessors, judgments and synthesis procedures.
Model results paragraph — use only after real appraisal
Risk-of-bias concerns were evaluated separately for each result included in the synthesis. The main limitations identified were [insert verified domains and specific study examples]. These issues were considered when interpreting [insert named outcome]. [Describe any pre-specified sensitivity analysis and its actual findings, or explicitly state that none was performed.] Full domain-level judgments and supporting reasons appear in [insert genuine table or appendix reference].
Never fill the placeholders with imagined studies, invented agreement percentages or generic statements such as “all articles were high quality.”
Common mistakes to avoid
Using one checklist for every study design. Match the instrument to the question and evidence.
Treating small samples as proof of bias. Small samples often affect precision; bias requires separate analysis.
Treating missing reporting details as automatically proving misconduct or flawed methods. Locate additional documentation and apply the chosen tool's guidance.
Scoring methodological quality out of ten without a validated rationale. Domain-based tools are not interchangeable with arbitrary totals.
Appraising an entire paper when the relevant instrument is result-specific. Name the precise result, outcome and comparison.
Ignoring protocols or registrations. They may clarify whether outcomes and analyses were specified in advance.
Using appraisal as a way to remove inconvenient results. Predefine how findings affect synthesis.
Confusing study risk of bias with publication or selective non-publication bias across a synthesis. Both matter, but they answer different questions.
Reporting a two-reviewer process when only one person assessed the evidence. Accuracy about review methods is part of research integrity.
Displaying a traffic-light chart without interpreting it. Explain what the concerns mean for the review's actual conclusion.
Frequently asked questions
Is risk-of-bias assessment mandatory in every dissertation literature review?
No. A formal systematic review commonly requires a fit-for-purpose assessment of included evidence, but requirements depend on review type, research question, protocol and institution. A narrative review or scoping review may have different appraisal expectations. Describe the approach you actually used.
Can I use CASP or JBI instead of RoB 2?
Only when the chosen instrument appropriately addresses your study design and review purpose, and your method or supervisor permits it. CASP and JBI include varied appraisal checklists; they are not automatic substitutes for RoB 2 when the review aims to judge intervention-effect bias in randomised trials.
Should I assess every outcome separately?
For result-specific tools, assess the results relevant to your synthesis and follow the instrument's guidance. You may not need to assess every outcome a trial has ever reported, but you should not assume one outcome's judgment automatically covers another.
What if the paper does not say whether allocation was concealed?
Check the full report, supplements, protocol, registry and other legitimate study information. Record uncertainty, answer the tool's signalling questions based on available evidence, and apply the specified judgment process. Do not invent concealment or classify authors as careless without evidence.
Can a study with high risk of bias still be included?
Potentially, yes. Inclusion and interpretation are different decisions. Follow pre-specified eligibility and synthesis rules, explain the concern and avoid overstating conclusions. Some reviews may prespecify exclusions or sensitivity analyses; these choices should not be improvised to produce a preferred result.
Does a risk-of-bias assessment prove whether a study's conclusion is true?
No. It identifies plausible threats to a particular estimate or finding. Precision, relevance, inconsistency, indirectness, missing evidence and the wider body of research still affect the strength of a conclusion.
Final dissertation risk-of-bias checklist
Before submitting your review, check that you can answer the following questions accurately:
Is the review type and appraisal purpose stated?
Are study designs and results of interest defined?
Are the named tools appropriate for those designs and results?
Are tool versions, manuals and any adaptations documented?
Have you identified assessors, independence and disagreement resolution truthfully?
Do completed judgments refer to full source evidence rather than impressions?
Are domain-level reasons and source locations retained?
Have you avoided invented scores and unsupported overall labels?
Does the results chapter show assessments for included evidence?
Does the synthesis explain how bias affects conclusions?
Are missing results and outcome-level certainty distinguished from individual-study bias?
Do your claims meet the relevant PRISMA reporting items and institutional requirements?
Final takeaway
A credible systematic review does not simply count studies that agree. It examines how each important result was produced, whether design or reporting problems could have shifted that result, and how those concerns affect the combined interpretation. Select the right tool, use its actual rules, document each judgment, and write conclusions that reflect the strength and limitations of the available evidence.
If you would value careful, human-led help with the structure of a dissertation literature review, appraisal-table presentation, source organisation, academic editing or clearer methodological explanations, explore Acadelyra's dissertation and research support services or request academic help. Support is tailored to the requirements of your brief and should be used in accordance with your institution's academic-integrity policies. The assessment and academic decisions in your dissertation remain your responsibility.
References and further guidance
Cochrane Methods Bias. RoB 2: A revised Cochrane risk-of-bias tool for randomized trials.
Cochrane Handbook for Systematic Reviews of Interventions. Chapter 8: Assessing risk of bias in a randomized trial.
Cochrane Methods Bias. ROBINS-I: Risk of Bias In Non-randomized Studies – of Interventions.
Cochrane Handbook for Systematic Reviews of Interventions. Chapter 25: Assessing risk of bias in a non-randomized study.
JBI. Critical Appraisal Tools.
Page, M. J., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71.