Schemitsch, J Bone Joint Surg Am 2009 - Orthopaedic Surgeons: Artists or Scientists? - Article Summary

Orthopaedic Surgeons: Artists or Scientists?

Schemitsch, Hawkins, et al.·J Bone Joint Surg Am·2009·14 citations·Review article

This 2009 JBJS symposium paper by Schemitsch and colleagues audits the quality of orthopedic evidence. It asks whether RCTs, observational studies, and expert opinion each adequately support clinical decision-making. The answer: orthopedic literature is dominated by low-level evidence, and even its RCTs are frequently underpowered and poorly reported.

Key Findings

Only 14% of JBJS articles represented Level-I evidence, while 57% were Level-IV case series — the dominant form of orthopedic literature is the single-center retrospective series, the design most prone to bias and overestimation of benefit.
91% of fracture care RCTs were underpowered, with spine and hand literature exceeding 80%. When a trial reports "no significant difference," that conclusion is statistically unproven in the vast majority of orthopedic studies. A negative result is not proof of equivalence.
Mean RCT quality in JBJS (1988–2000) scored only 68% on the Detsky index, below the 75% high-quality threshold. 60% of trials fell below that cutoff. Drug trials scored significantly higher than surgical trials (72.8% vs 63.9%, p < 0.05), reflecting the inherent difficulty of standardizing surgical interventions.
Published orthopedic RCTs met an average of only 36% of CONSORT reporting criteria, meaning the information needed to critically appraise a trial (allocation concealment, blinding methods, reasons for exclusion) is routinely missing from the published report.
Alpha error contaminated 37% of positive findings across 60 orthopedic studies. Testing 16 outcomes in a single study gives a 55% chance of a spurious p < 0.05. Statistically significant findings in multi-outcome trials should be interpreted with this in mind.
30% of continuous and 47% of dichotomous statistically significant trauma outcomes failed to meet thresholds for clinical importance. A p-value below 0.05 does not mean the difference matters to the patient. The two concepts are distinct.
Unreamed femoral nailing had a nonunion rate 4.5 times higher than reamed nailing (p = 0.049) in a powered multicenter RCT, reversing a practice that had become standard at many institutions based solely on retrospective single-center data. This is the paper's clearest illustration of why retrospective enthusiasm must be tested prospectively.
Fracture classification systems have poor interobserver reliability across the literature, with a mean kappa of only 0.23. When classification drives enrollment criteria or outcome stratification in a trial, unreliable classification undermines the entire study.

Board Pearl91% of orthopedic fracture RCTs are underpowered — a 'no difference' result rarely means the treatments are truly equivalent.

Supplemental

Orthopaedic Surgeons: Artists or Scientists?

Schemitsch, Hawkins, et al.J Bone Joint Surg Am200914 citationsReview article