The Tyranny of Average and Guideline Development for PPE

‘All models are wrong, but some are useful’ George Box

Link to The Dental Elf

3D_medical_animation_corona_virus

The Bottom Line

It is important when developing national and local guidelines to appreciate the extremely high variability  in clinical risk that Covid-19 poses to the dental team, and the patients they serve. Regarding the use of PPE and aerosol generating procedures there should be sufficient flexibility in the national guidelines to allow clinicians to safely apply their local knowledge so that the PPE selected matches the degree of risk for the individuals. Applying a rigid, unsustainable PPE policy will waste valuable resources while increasing potential harm to staff and patients.

Background

During the Q&A section of a webinar I was recently presenting about evidence-based dentistry one of the participants made a very interesting point. As a surgeon working in the front line of the Covid-19 pandemic not all areas have been effected the same, in some areas of the country we have hardly seen any deaths in the community, and in a North London hospital the staff there are seeing many severe cases a day. This point that the infection is not evenly distributed either geographically nor demographically is important, as it highlights the ‘tyranny of the average’(Merlo et al., 2017) in terms of healthcare policy and guideline development. From the data this virus is not an ‘equal opportunities killer’ as the highest mortality rates  are effecting the frail, disadvantaged, and highlighting strong racial/ethnic inequalities (Webb et al., 2020, Everson et al., 2002). A clear example of this is that out of 23,804 Covid-19 related deaths 31.4% (7466) occurred in people with Type II diabetes (prevalence in pop. approx. 6%). 15% of  Covid-19 admissions  to hospital had no reported medical conditions, but 50% had three or more significant comorbidities (ISARIC, 2020) (See Figure 1.).

Figure 1.The distribution of combinations of the four most common chronic diseases (ISARIC 2020)

Comorbidities

An example of this variation can be seen in regional mortality figures for England. Mortality with a diagnosis of Covid-19 is the only reliable endpoint measurement  until we have large scale reliable track and trace/antibody tests. From the background information we can deduce that there appears to be significant variation between geographic locations with lower risk in the more rural counties and smaller cities by as much as a factor of 2.5.(Table 1).

Table 1. Differences in regional mortality up to 8th May

Location Population Population density/km2 Covid-19 deaths % of population die of Covid-19 Risk Ratio
Plymouth 262100 8500 57 0.021 1
Lincolnshire 761224 127 207 0.027 1.29
Manchester 552858 4716 289 0.052 2.48
Tower Hamlets 324745 16000 172 0.053 2.52

Methods

The following calculations are a Fermi problem after Enrico Fermi, where an estimate is made with little or no data and used to identify orders of magnitude rather than point estimates. The methodology undertaken uses the mathematical principles laid out by Pólya in his famous book ‘How to Solve it’ (Polya, 1945). These are

  • Understand the problem.
  • Find the connection between the data and the unknown and devise a plan.
  • Run the model.
  • Examine the solution.

In a previous post I looked at base rates as a point estimate by using the mean without a confidence interval (Howe, 2020). Since then two reviews have been published  investigating the infection fatality rate (IFR) (Ioannidis, 2020, Meyerowitz-Katz and Merone, 2020). The IFR estimates the overall mortality rather than the number of deaths as a proportion of confirmed cases (CFR). Due to variations in analysis the two reviews produce significantly different results, Ioannidis’s review produces as IFR range of 0.02% to 0.40%, and the Meyerowitz-Katz review a mean of 0.75% (95% CI: 0.49 to 1.01). The results from the  Office of National Statistics (ONS) data for infection in England was  0.27% (95% CI: 0.17% to 0.41%). To calculate the risk range of dying following a dental aerosol generating procedure (AGP) is based on the assumption that there is a 100% chance of becoming infected in a single exposure to the virus, the second assumption is that we work with both the minima and the maxima as presented. The summary estimate for asymptomatic individuals in the population was previously calculated as 0.27% (95% CI: 0.12% to 0.45%) (Howe, 2020).

Figure 2. Frequency tree for maximum  and minimum AGP risk

Minimax

With this new range of asymptomatic risk as 1:540 to1:5000 we can calculate the risk that this infected patient passes on the virus to another patient or member of staff and they unfortunately die. We do not treat symptomatic Covid-19 patients in this model. The final range of AGP related death ranges from 1: 54000 to 1:25,000,000 (See Table 3.).

Table 3. Minimum and maximum infection fatality risk

IFR Review paper Minimum and maximum risk of death(natural frequency)
Ioannidis, 2020 1: 135,000 to 1:25,000,000
Meyerowitz-Katz and Merone, 2020 1: 54,000   to 1:1,020,000

Discussion

From the data presented we can make broad assumptions about the population most at risk form corona virus, namely the urban elderly with multiple chronic health conditions. We can also make a similar broad assumption that rural and less densely populated areas have had lower mortality rates. From the current IFR data the variability in trying to make an estimate of risk become incredibly wide from 1:54,000 to 1:25,000,000 which is a 462 times increase in risk. From this result we can see that even in the worst-case scenario the risk of a dental AGP being directly linked to an individual patients death is very low if good cross infection policy is employed which includes PPE as a component of the process. High level PPE (Fluid resistant disposable gowns with plastic aprons, FFP3 and visors) can soon lose its protective benefits with prolonged use, lack of comfort, complicated workflow and difficulties in removing (Phan et al., 2019a, Phan et al., 2019b). There is a huge variance in the volume of aerosol generated in a dental procedure from polishing a composite to removing a broken bridge. It is important in that case that any guidance developed around the use of PPE should consider this variability and set an absolute minimum standard, thus allowing the clinician to tailor the PPE requirement to local conditions/risk. Low risk AGPs could require a fluid resistant face mask and face shield/visor, and as the risk increases higher levels of PPE are added rather than going to maximum PPE in all circumstances (high volume suction is common to all procedures). Wasted resources will harm health care provision to the most vulnerable social groups disproportionally, so high quality studies will need to be rapidly developed to assess these risks.

 

EVERSON, S. A., MATY, S. C., LYNCH, J. W. & KAPLAN, G. A. 2002. Epidemiologic evidence for the relation between socioeconomic status and depression, obesity, and diabetes. Journal of psychosomatic research, 53, 891-895.

HOWE, M. S. 2020. Are we sleepwalking into PPE paralysis? [Online]. Available: https://www.nationalelfservice.net/dentistry/dental-workforce/sleepwalking-ppe-paralysis/ [Accessed].

IOANNIDIS, J. P. 2020. The infection fatality rate of COVID-19 inferred from seroprevalence data.

ISARIC. 2020. Data PlatformCOVID-19 Report 6 MAY 20 [Online]. Available: https://media.tghn.org/medialibrary/2020/05/ISARIC_Data_Platform_COVID-19_Report_6MAY20.pdf [Accessed].

MERLO, J., MULINARI, S., WEMRELL, M., SUBRAMANIAN, S. V. & HEDBLAD, B. 2017. The tyranny of the averages and the indiscriminate use of risk factors in public health: The case of coronary heart disease. SSM Popul Health, 3, 684-698.

MEYEROWITZ-KATZ, G. & MERONE, L. 2020. A systematic review and meta-analysis of published research data on COVID-19 infection-fatality rates. medRxiv.

PHAN, L. T., MAITA, D., MORTIZ, D. C., WEBER, R., FRITZEN-PEDICINI, C., BLEASDALE, S. C., JONES, R. M. & PROGRAM, C. P. E. 2019a. Personal protective equipment doffing practices of healthcare workers. Journal of occupational and environmental hygiene, 16, 575-581.

PHAN, L. T., SWEENEY, D., MAITA, D., MORITZ, D. C., BLEASDALE, S. C., JONES, R. M. & PROGRAM, C. D. C. P. E. 2019b. Respiratory viruses on personal protective equipment and bodies of healthcare workers. Infect Control Hosp Epidemiol, 40, 1356-1360.

POLYA, G. 1945. How to solve it: A new aspect of mathematical method, Princeton university press.

WEBB, H. M., NÁPOLES, A. & PÉREZ-STABLE, E. 2020. COVID-19 and Racial/Ethnic Disparities. JAMA.

Mouthwash; can it reduce levels of Covid-19 in the mouth?

mouthwaskhttps://www.nationalelfservice.net/dentistry/oral-medicine-and-pathology/mouthwash-reduce-levels-covid-19-mouth/

Question:

In dental practice what is the most effective mouthwash for a patient to use to reduce the potential concentration of Covid-19 in the oral cavity ?

 

Bottom line answer: 

From this review  four mouthwashes were identified: 0.2% chlorhexidine mouthwash (CHX), 1% povidone iodine (PI) , 1.5% hydrogen peroxide (H2O2), or 0.05% hypochlorous acid (HOCl). CHX had poor virucidal properties and the other three (PI, H2O2,HOCl) all have good virucidal properties but poor microbial substantivity, with the benefits being lost within a few minutes as saliva flow may potentially replace the virus.  Of the three the most clinically acceptable in terms of virucidal activity, commercial availability, and taste is the 1.5% hydrogen peroxide.

Background

Two recent papers from China mention the potential use of chlorhexidine in prevention and control of aerosol transmission risk (An et al., 2020; Su, 2020) based on CDC guidelines (Kohn et al., 2003). There are also a number of papers identifying the presence of viral particles in the saliva of patients diagnosed with Covid-19 (Khurshid et al., 2020; Sabino-Silva et al., 2020; Xu et al., 2020).  Even though there is good evidence that chlorhexidine is effective at reducing  the bacterial count in bioaerosols derived from dental treatment (Shetty et al., 2013; Gupta et al., 2014; Santos et al., 2014; Swaminathan et al., 2014; Saini, 2015; Mohan and Jagannathan, 2016; Retamal-Valdes et al., 2017) it has poor virucidal activity (Farzan and Firoozi, 2019; Kampf et al., 2020). Two common oral mouthwashes have been suggested for the reduction of Covid-19, these being providone-iodine, and hydrogen peroxide, one further mouthwash mentioned was hypochlorous acid.

Method

In this rapid review, the authors searched via the electronic database Medline (Ovid) and Google Scholar for studies comparing the pretreatment use PI, H2O2, HOCl mouthwashes in the reduction in viral microorganisms in the oral cavity. Reviews, and papers investigating non-respiratory viruses were excluded. There was no restriction by language or publication or year (see Appendix A).

Results

Only one paper could be found relating to the antiviral capacity of common oral mouthwashes and that was Eggers and co-workers (Eggers et al., 2018) regarding povidone-iodine. This pharmaceutical industry funded study was carried out in-vitro using a povidone-iodine solution diluted to 0.23% and after 15 seconds it had produced a Log10 reduction factor for SARS-CoV, and MERS-Cov of  (A reduction of 39,811 copies/ml).

Discussion

To date there are no high-quality studies that have been peer reviewed or otherwise relating to the virucidal efficacy of commonly used oral mouthwashes. Having said that it is possible to hypothesize that both povidone-iodine and hydrogen peroxide do exhibit substantially more virucidal activity than chlorhexidine ( Log10 reduction factor of 0.6)  against  respiratory viruses  by a factor of 8000 times (Kampf et al., 2020). It must also be borne in mind that apart from chlorhexidine the other mouthwashes have poor substantivity allowing the oral microflora to reestablish within several minutes (Addy and Wright, 1978; Lafaurie et al., 2018). If the patient were to have Covid-19 the virus could potentially be replaced quite rapidly via the saliva (Khurshid et al., 2020; Sabino-Silva et al., 2020; Xu et al., 2020).

Therefore, of the three mouthwashes the most clinically acceptable in terms of virucidal activity, commercial availability, and taste is hydrogen peroxide. As a clinical workflow regarding restorative aerosol generating procedures the patient could be asked to rinse with 10 ml of 1.5% hydrogen peroxide for 1 minute prior to placement of the rubber dam and then the exposed surface of the teeth, and rubber dam swabbed down with hydrogen peroxide prior to treatment. These results are hypothetical and due to the lack of specific studies of the virucidal activity of mouthwashes in dentistry are based on surrogate, and composite outcomes. There is an urgent need for specific studies to address mouthwash use in the dental surgery environment.

Disclaimer:  The article has not been peer-reviewed; it should not replace individual clinical judgement, and the sources cited should be checked. The views expressed in this commentary represent the views of the author and not necessarily those of the host institution. The views are not a substitute for professional advice.

References

ADDY, M. & WRIGHT, R. 1978. Comparison of the in vivo and in vitro antibacterial properties of povidone iodine and chlorhexidine gluconate mouthrinses. Journal of clinical periodontology, 5, 198-205.

AN, N., YUE, L. & ZHAO, B. 2020. [Droplets and aerosols in dental clinics and prevention and control measures of infection]. Zhonghua kou qiang yi xue za zhi = Zhonghua kouqiang yixue zazhi = Chinese journal of stomatology, 55, E004.

EGGERS, M., KOBURGER-JANSSEN, T., WARD, L. S., NEWBY, C. & MÜLLER, S. 2018. Bactericidal and virucidal activity of Povidone-Iodine and chlorhexidine gluconate cleansers in an in vivo hand hygiene clinical simulation study. Infectious diseases and therapy, 7, 235-247.

FARZAN, A. & FIROOZI, P. 2019. Which Mouthwash is Appropriate for Eliminating Coronaviruses? A. Regeneration, Reconstruction & Restoratiion, 5.

GUPTA, G., MITRA, D., ASHOK, K. P., GUPTA, A., SONI, S., AHMED, S. & ARYA, A. 2014. Efficacy of preprocedural mouth rinsing in reducing aerosol contamination produced by ultrasonic scaler: a pilot study. Journal of periodontology, 85, 562-8.

KAMPF, G., TODT, D., PFAENDER, S. & STEINMANN, E. 2020. Persistence of coronaviruses on inanimate surfaces and its inactivation with biocidal agents. Journal of Hospital Infection.

KHURSHID, Z., ASIRI, F. Y. I. & AL WADAANI, H. 2020. Human Saliva: Non-Invasive Fluid for Detecting Novel Coronavirus (2019-nCoV). International journal of environmental research and public health, 17.

KOHN, W. G., COLLINS, A. S., CLEVELAND, J. L., HARTE, J. A., EKLUND, K. J. & MALVITZ, D. M. 2003. Guidelines for infection control in dental health-care settings-2003.

LAFAURIE, G., ZAROR, C., DÍAZ‐BÁEZ, D., CASTILLO, D., DE ÁVILA, J., TRUJILLO, T. & CALDERÓN‐MENDOZA, J. 2018. Evaluation of substantivity of hypochlorous acid as an antiplaque agent: A randomized controlled trial. International journal of dental hygiene, 16, 527-534.

MOHAN, M. & JAGANNATHAN, N. 2016. The efficacy of pre-procedural mouth rinse on bacterial count in dental aerosol following oral prophylaxis. Dental and Medical Problems, 53, 78-82.

RETAMAL-VALDES, B., SOARES, G. M., STEWART, B., FIGUEIREDO, L. C., FAVERI, M., MILLER, S., ZHANG, Y. P. & FERES, M. 2017. Effectiveness of a pre-procedural mouthwash in reducing bacteria in dental aerosols: randomized clinical trial. Brazilian oral research, 31, e21.

SABINO-SILVA, R., JARDIM, A. C. G. & SIQUEIRA, W. L. 2020. Coronavirus COVID-19 impacts to dentistry and potential salivary diagnosis. Clinical oral investigations, 24, 1619-1621.

SAINI, R. 2015. Efficacy of preprocedural mouth rinse containing chlorine dioxide in reduction of viable bacterial count in dental aerosols during ultrasonic scaling: A double-blind, placebo-controlled clinical trial. Dental Hypotheses, 6, 65-71.

SANTOS, I. R. M. D., MOREIRA, A. C. A., COSTA, M. G. C. & CASTELLUCCI E BARBOSA, M. D. 2014. Effect of 0.12% chlorhexidine in reducing microorganisms found in aerosol used for dental prophylaxis of patients submitted to fixed orthodontic treatment. Dental press journal of orthodontics, 19, 95-101.

SHETTY, S. K., SHARATH, K., SHENOY, S., SREEKUMAR, C., SHETTY, R. N. & BIJU, T. 2013. Compare the efficacy of two commercially available mouthrinses in reducing viable bacterial count in dental aerosol produced during ultrasonic scaling when used as a preprocedural rinse. Journal of Contemporary Dental Practice, 14, 848-851.

SU, J. 2020. [Aerosol transmission risk and comprehensive prevention and control strategy in dental treatment]. Zhonghua kou qiang yi xue za zhi = Zhonghua kouqiang yixue zazhi = Chinese journal of stomatology, 55, E006.

SWAMINATHAN, Y., THOMAS, J. T. & MURALIDHARAN, N. P. 2014. The efficacy of preprocedural mouth rinse of 0.2% chlorhexidine and commercially available herbal mouth containing salvadora persica in reducing the bacterial load in saliva and aerosol produced during scaling. Asian Journal of Pharmaceutical and Clinical Research, 7, 71-74.

XU, R., CUI, B., DUAN, X., ZHANG, P., ZHOU, X. & YUAN, Q. 2020. Saliva: potential diagnostic value and transmission of 2019-nCoV. International journal of oral science, 12, 11.

 

Appendix A. Medline (Ovid) Search strategy

Keyword Search result
1 exp *dentistry/ or exp *dental care/ 291185
2 (dental or oral or tooth or teeth).mp. 1138218
3 1 or 2 1179282
4 exp Mouthwashes/ 14314
5 ((oral or dental or mouth) adj3 (wash$ or rins$)).mp. 2899
6 (mouthrins$ or “mouth rins$”).mp. 2821
7 (rinse$ or rinsing$).mp. 14275
8 (mouthwash$ or “mouth wash$”).mp. 6676
9 5 or 6 or 7 or 8 19294
10 chlorhexidine/ 8265
11 chlorhexidine.mp. 12146
12 chx.mp. 3444
13 10 or 11 or 12 13721
14 Antiviral Agents/ 77993
15 exp Coronavirus Infections/ or exp Coronavirus/ 16185
16 3 and 9 10412
17 15 and 16 3
18 3 and 15 294
19 3 and 9 and 15 3
20 Saliva/ 41183
21 15 and 20 14
22 from 21 keep 1-3, 10-12 6
23 Hypochlorous Acid/ 2330
24 Povidone-Iodine/ 2817
25 Hydrogen Peroxide/ 57630
26 from 17 keep 1-3 3
27 virus.mp. or exp Viruses/ 1023534
28 16 and 23 and 27 1
29 16 and 24 and 27 0
30 16 and 25 and 27 0
31 chlorhexidine.mp. or exp Chlorhexidine/ 12146
32 16 and 27 and 31 11
33 from 32 keep 9, 11 2
34 16 and 23 9
35 from 34 keep 2, 4, 7 3

Editors Note:

A Cochrane rapid review related to this question is in development.

Antimicrobial mouthwashes (gargling) and nasal sprays to protect healthcare workers when undertaking aerosol generating procedures (AGPs) on patients without suspected or confirmed Covid-19 infection

Spin the Odds

odds

I recently attended the Evidence-Based Medicine Live19 conference at Oxford University where Professor Isabella Boutron from the Paris Descartes University presented a lecture entitled ‘Spin or Distortion of Research Results’. Simply put, research spin is ‘reporting to convince readers that the beneficial effect of the experimental treatment is greater than shown by the results’(Boutron et al., 2014). In a study of oncology trials spin was prevalent  in 59% of the 92 trials where the primary outcome was negative (Vera-badillo et al., 2013). I would argue that spin also affects a large proportion of dental research papers.

To illustrate how subtle this problem can be  I have selected a recent systematic review (SR) that was posted on the Dental Elf website  regarding pulpotomy (Li et al., 2019). Pulpotomy  is the removal of a portion of the diseased pulp, in this case from a decayed permanent tooth, with the intent of maintaining the vitality of the remaining nerve tissue by means of a therapeutic dressing. Li’s SR was comparing the effectiveness of calcium hydroxide with the newer therapeutic dressing material mineral trioxide aggregate (MTA).

In the abstract Li states that the meta-analysis favours mineral trioxide aggregate (MTA), and  in the results sections of the SR that ‘MTA had higher success rates in all trial at 12 months (odds ratio, 2.23,  p= 0.02, I2=0%), finally concluding that ‘mineral trioxide aggregate appears to be the best pulpotomy medicament in carious permanent teeth with pulp exposures’. I do not agree with this assumption, and would argue that the results show substantial spin. Close appraisal of Li’s paper reveals several methodological problems that have magnified the beneficial effect of MTA.

The first problem is regarding the use of reporting guidelines, which in this case was the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher et al., 2009). The author states this was adhered to but there is no information regarding registration of a review protocol to establish predefined primary and secondary outcomes or methods of analysis. To quote Shamseer:

‘Without review protocols, how can we be assured that decisions made during the research process aren’t arbitrary, or that the decision to include/exclude studies/data in a review aren’t made in light of knowledge about individual study findings?’(Shamseer & Moher, 2015)

In the ‘Data synthesis and statistical analysis’ section the author states that the primary and secondary outcomes for this SR were only formulated after data collection. This post hoc selection makes the data vulnerable to selection bias. Additionally, there is no predefined rationale relating to the choice of an appropriate summary measure or method of synthesising the data.

The second problem relates to the post hoc choice of summary measure, in this case ‘odds ratio’ and the use of a fixed effects model in the meta-analysis (Figure.1).

Capture

Figure 1. Forest plot of 12-month clinical success (original).

Of all the options available to analyse the 5 randomised control trails odds ratio and a fixed effects model produced the largest significant effect size (OR 2.23 0.02). There was no explanation as to why odds ratio was selected over relative risk (RR), risk difference (RD), or arcsine difference (ASD) if the values were close to 0 or 1. Since the data for the SR is dichotomous the three most common effect measurements are:

  • the risk difference which is the actual size of the difference . This is probably the most  straightforward and useful analysis.
  • the relative risk highlights the relative difference between two probabilities .
  • the odds ratio .Odds ratios are approximately equal to the RR when outcomes are rare, however they are easy to misinterpret (in Li’s results an OR of 2.23 represents a 2.23 fold increase in odds). OR  and are best used for case-control studies.

The authors specifically chose a fixed-effects model for meta-analysis based on the small number of studies. There are two problems with this, firstly there is too much variability between the 5 studies in terms of methodology and patient factors, such as age (in 4  studies the average age is approximately 8 years and in  one study its 30 years). Secondly we don’t need to used a fixed effect model since there are 5 studies, therefore we can use a random effects model using a Hartung-Knapp adjustment specifically for handling the small number of studies (Röver, Knapp & Friede, 2015; Guolo & Varin, 2017).

Below I have reanalysed the original data using a more plausible random effects model (Hartung-Knapp) and RR to show the relative difference in treatments plus RD to highlight the actual difference (Figure 2. and 3.) using the ‘metabin’ package in R (Schwarzer, 2007).

Hakn_Z

Figure 2. 12-month clinical success using  Hartung-Knapp adjustment for random effects model and relative risk

HaknRDFigure 3. 12-month clinical success using  Hartung-Knapp adjustment for random effects model and risk difference

Both analyses now show a small effect size ( 8% to 9%) that slightly favours the MTA but is non-significant as opposed to a 2.23-fold increase in odds. In the pulpotomy review the OR magnifies the effect size by 51% using the formula   . In a paper by Holcomb reviewing 151 studies using odds ratios 26% had interpreted the odds ratio as a risk ratio (Holcomb et al., 2001).

There are a couple of further observations to note. Regarding the 5 studies, even combined one would need 199 individuals in each arm of the study for it to be sufficiently powered (  error prob = 0.05,   error prob = 0.8) putting the authors results into question about significance.

I have included a prediction interval in both my forest plots to signify the range of possible true values one could expect in a future RCT’s, which is more useful to know in clinical practice than the confidence interval (IntHout et al., 2016). Using the RD meta-analysis, a future RCT could produce a result that favours calcium hydroxide by 20% or MTA by 35% which is quite a wide range of uncertainty.

One of  Li’s primary outcomes was cost effectiveness and the paper concluded there was insufficient data to determine a result, it also mentions the high cost and technique sensitivity of MTA compared to the calcium hydroxide. I would argue that since there appears to be no significant difference between outcomes, we could conclude that on the evidence available calcium hydroxide must be more cost effective.

In conclusion researchers, reviewers and editors need to be aware of the harm spin can do. Many clinicians are not able to interrogate the main body of a research paper for detail as it is hidden behind a paywall and they rely heavily on the abstract for information(Boutron et al., 2014). Registration of a research protocol prespecifying appropriate outcome and methodology will help prevent post-hoc changes to the outcomes and analysis. I would urge researches to limit the use of odds ratios to case-control studies and use relative risk or risk difference as they are easier to interpret. For the meta-analysis avoid using a fixed effects model if the studies don’t share a common true effect and include a prediction interval to explore possible future outcomes.

References

Boutron, I., Altman, D.G., Hopewell, S., Vera-Badillo, F., et al. (2014) Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: The SPIIN randomized controlled trial. Journal of Clinical Oncology. [Online] 32 (36), 4120–4126. Available from: doi:10.1200/JCO.2014.56.7503.

Guolo, A. & Varin, C. (2017) Random-effects meta-analysis: The number of studies matters. Statistical Methods in Medical Research. [Online] 26 (3), 1500–1518. Available from: doi:10.1177/0962280215583568.

Holcomb, W.L., Chaiworapongsa, T., Luke, D.A. & Burgdorf, K.D. (2001) An Odd Measure of Risk. Obstetrics & Gynecology. [Online] 98 (4), 685–688. Available from: doi:10.1097/00006250-200110000-00028.

IntHout, J., Ioannidis, J.P.A., Rovers, M.M. & Goeman, J.J. (2016) Plea for routinely presenting prediction intervals in meta-analysis. British Medical Journal Open. [Online] 6 (7), e010247. Available from: doi:10.1136/bmjopen-2015-010247.

Li, Y., Sui, B., Dahl, C., Bergeron, B., et al. (2019) Pulpotomy for carious pulp exposures in permanent teeth: A systematic review and meta-analysis. Journal of Dentistry. [Online] 84 (January), 1–8. Available from: doi:10.1016/j.jdent.2019.03.010.

Moher, D., Liberati, A., Tetzlaff, J. & Altman, D.G. (2009) Systematic Reviews and Meta-Analyses: The PRISMA Statement. Annulas of Internal Medicine. [Online] 151 (4), 264–269. Available from: doi:10.1371/journal.pmed1000097.

Röver, C., Knapp, G. & Friede, T. (2015) Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Medical Research Methodology. [Online] 15 (1), 1–8. Available from: doi:10.1186/s12874-015-0091-1.

Schwarzer, G. (2007) meta: An R package for meta-analysis. [Online]. 2007. R News. Available from: https://cran.r-project.org/doc/Rnews/Rnews_2007-3.pdf.

Shamseer, L. & Moher, D. (2015) Planning a systematic review? Think protocols. [Online]. 2015. Research in progress blog. Available from: http://blogs.biomedcentral.com/bmcblog/2015/01/05/planning-a-systematic-review-think-protocols/.

Vera-badillo, F.E., Shapiro, R., Ocana, A., Amir, E., et al. (2013) Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer. Annals of Oncology. [Online] 24 (5), 1238–1244. Available from: doi:10.1093/annonc/mds636.