Fundamental error in "Trends in serious head injuries..." Cook andSheikh 2003Fundamental error in "Trends in serious head injuries..." Cookand Sheikh 2003

I sent the following to "Injury prevention on-line" over a week ago, but
it shows no sign of being published and my follow-up email has not been
answered. I guess someone might as well see it, typo and all. Isn't the
internet great for vanity publishing?

Neither Cook nor Sheikh replied to my email, either.

Their original article, to which this refers, can be found on

http://ip.bmjjournals.com/cgi/content/full/9/3/266

-----------------------------------------------------------------
"Fundamental error in "Trends in serious head injuries..." Cook and
Sheikh 2003"
-----------------------------------------------------------------

!-- article ID: 9/3/266 --

P The main conclusion of Cook and Sheikh (2003), that a bicycle helmet
prevents 60% of head injuries, is incorrect due to a fundamental error in
the way they have treated their percentages. A correct analysis
demonstrates unequivocally that there must be major confounding factors in
their data set that they have failed to take into account, and therefore
any estimate of helmet effectiveness is purely speculative.

P Assuming that their basic analysis of the data is correct (although
the numbers they quote in the text do not actually appear to match the
figure plotted), they arrive at a figure of a 3.6% for the reduction in
the head injury (HI) rate for cyclists, over and above the "background"
reduction that pedestrians have also seen. They assume that this drop in
HI is due to increased helmet-wearing. However, this reduction is
presented in terms of the number of percentage points, and relative to the
baseline value of 27.9% HI for cyclists in 1995-6 it actually represents a
3.6/27.9 = 13% drop in the HI rate.

P The decrease in the number of helmetless cyclists over the same
interval is 5.8 percentage points from a baseline of 84% unhelmeted,
giving the percentage drop as 5.8/84 = 7%. Cook and Sheikh calculate
helmet effectiveness to be given by the ratio 3.6/5.8 = 60%. However the
correct expression to use is 13/7 = 186%. In other words, "helmet
effectiveness" is so high that each helmet does not just save its wearer,
but a non-wearer too. At this rate, head injuries would be eliminated
completely if just a little over half of all cyclists wore them! This is
clearly ludicrous.

P A more reasonable conclusion to draw from this would be that there
are some other factors that are responsible for the large drop in HI rate,
and therefore any attempt to attribute some part of the total 30%
(8.49/27.9) change to the provably marginal impact of a very small number
of extra helmet wearers is at best highly speculative and fraught with
inaccuracy.

P What makes this all the more poignant is the fact that the authors
have recently produced a book entitled "Basic skills in statistics"!

P James Annan

James Annan wrote in message ...
I sent the following to "Injury prevention on-line" over a week ago, but
it shows no sign of being published and my follow-up email has not been
answered. I guess someone might as well see it, typo and all. Isn't the
internet great for vanity publishing?

Sometimes these things take a little longer. Have you checked whether
you are in excess of the formal word limit?

The editor may also think that the nature of your criticism means he
should seek advice before publishing it. What you say is, of course,
correct. I was very suspicious when I first saw that claim and
thought I'd better write it out formally to check, but (shame on me!)
I never got round to it.

Below is a more formal way of expressing the same thing. I didn't
bother sending it to Injury Prevention. But if James or anyone else
thinks they can make use of it, feel free!

An alternative, more mathematical/formal way of making the same points
as James
----------------------------------------------------------------------------

There is a miscalculation in Cook and Sheikh's paper.[1]

From 1995/6 to 2000/1 the percentage of hospital admissions with head
injury (%HI) fell from 27.9% to 20.4% for cyclists and 26.9% to 22.8%
for pedestrians, declines of 7.5 and 4.1 percentage points.

Cook and Sheikh claimed that, because percent helmet wearing (%HW)
increased by 5.8% (from 16.0% to 21.8%) and %HI of cyclists fell by
3.6% more than pedestrians, helmets must prevent 3.6/5.8= 60% of head
injuries.

The above argument confuses percentages with percentage points. A
decline from 27.9 to 20.4 represents a fall of 27%. The predicted
fall in %HI of 3.6 percentage points more than pedestrians (25.9% to
22.3%, after subtracting pedestrian trends) represents a 13.9% drop in
numbers of head injuries. Could such a relatively large fall be
caused by an increase of just 5.8 percentage points in %HW?

A mathematical approach is needed to provide the answer.[2] If h and
n are the probabilities of head injury for helmet wearers and
non-wearers respectively, then:
%HI = %HW*h + (100-%HW)*n (1)
The predicted value of %HI at the midpoint of Cook and Sheikh's data
is 24.1%; with %HW of 18.9%. If helmets prevent 60% of head injuries,
h = 0.4n, so, from equation (1), n = 0.272.

Also from equation (1) with n = 0.272, increased %HW from 16.0% to
21.8% should decrease %HI from 24.59 to 23.64, 0.95 percentage points.
This is nowhere near the 3.6 percentage points reported by Cook and
Sheikh. Even if helmets prevented 100% of head injuries, %HI would
fall by only 1.7 percentage points.

Cook and Sheikh's estimate of 60% is therefore invalid. Increased
helmet wearing cannot explain the larger drop in %HI of cyclists
compared to pedestrians. Other factors must also have been involved,
such as gradual changes in the age composition of cyclists, or the
relative amounts of on vs off-road cycling. Cook and Sheikh's data
therefore provide no real evidence of reduced HI from increased helmet
wearing.

In Australia and New Zealand, helmet laws increased %HW dramatically,
in many cases from less than 30% to more than 80% of all cyclists in
less than a year, yet there were no large or obvious corresponding
changes in %HI over and above prevailing trends. In contrast to Cook
and Sheikh's analysis, this tells us a great deal about the benefits
of helmet laws.[3]

References
1. Cook A, Sheikh A. Trends in serious head injuries among English
cyclists and pedestrians. Inj Prev 2003; 9: 266–267.
2. Robinson DL. Head injuries and bicycle helmet laws. Accid Anal Prev
1996; 28: 463-475.
3. Robinson DL. Reasons for trends in cyclist injury data. Injury
Prevention 2004; 10: 126–127.

(Dorre) wrote in message om...
James Annan wrote in message ...
I sent the following to "Injury prevention on-line" over a week ago, but
it shows no sign of being published and my follow-up email has not been
answered. I guess someone might as well see it, typo and all. Isn't the
internet great for vanity publishing?

Sometimes these things take a little longer. Have you checked whether
you are in excess of the formal word limit?

There's not a hard limit, it took some searching to find out that I
might be slightly in excess of the guideline figure but I can't
imagine them agonising over that detail without telling me.

The editor may also think that the nature of your criticism means he
should seek advice before publishing it.

Well, it is possible, but a bit rude to not only not tell me of this
off his own bat (once he had exceeded his own time limit) but to
ignore my email asking what was going on. Also, IP have a deliberate
policy of wading in to controversial areas, and my letter is a lot
more polite and less controversial than several others they have
recently published ("Injury Prevention injured itself by publishing
such unprofessional work. A retraction is warranted, with support for
this methodology and its spurious conclusions disavowed.")

OTOH, a couple of years ago Science refused to publish a
straightforward correction to one of the most famous papers measuring
global warming over recent decades (it overestimated the trend by
about 10% due to a numerical error). It continues to be cited
regularly. So nothing really surprises me any more...

James

in message , Dorre
') wrote:

James Annan wrote in message
...
I sent the following to "Injury prevention on-line" over a week ago,
but it shows no sign of being published and my follow-up email has
not been answered. I guess someone might as well see it, typo and
all. Isn't the internet great for vanity publishing?

Sometimes these things take a little longer. Have you checked whether
you are in excess of the formal word limit?

[snip]

Below is a more formal way of expressing the same thing. I didn't
bother sending it to Injury Prevention. But if James or anyone else
thinks they can make use of it, feel free!

An alternative, more mathematical/formal way of making the same points
as James

----------------------------------------------------------------------------

There is a miscalculation in Cook and Sheikh's paper.[1]

[snip: excellent presentation]

Perhaps it would be a good thing to send this one in also, so that each
could reinforce the other?

--
(Simon Brooke) http://www.jasmine.org.uk/~simon/

;; no eternal reward will forgive us now for wasting the dawn.
;; Jim Morrison

Simon Brooke wrote:


Perhaps it would be a good thing to send this one in also, so that each
could reinforce the other?

It appeared at last:

http://ip.bmjjournals.com/cgi/eletters/9/3/266#59

James

On 11/6/04 10:24 pm, in article
, "James Annan"
wrote:

Simon Brooke wrote:


Perhaps it would be a good thing to send this one in also, so that each
could reinforce the other?

It appeared at last:

http://ip.bmjjournals.com/cgi/eletters/9/3/266#59


Well done!

And as it is linked from the original paper, everyone will see it (a great
improvement over dead tree journals).

If I were the authors I would be very embarrassed.

...d

On Sat, 12 Jun 2004 20:04:04 +0100, David Martin
wrote:

And as it is linked from the original paper, everyone will see it (a great
improvement over dead tree journals).

If I were the authors I would be very embarrassed.

Unfortunately it won't undo the damage already done by the stories in
the general press.

--
Dave...

Get a bicycle. You will not regret it. If you live. - Mark Twain

On Sat, 12 Jun 2004 20:04:04 +0100, David Martin wrote:

On 11/6/04 10:24 pm, in article
, "James Annan"
wrote:

Simon Brooke wrote:


Perhaps it would be a good thing to send this one in also, so that each
could reinforce the other?

It appeared at last:

http://ip.bmjjournals.com/cgi/eletters/9/3/266#59


Well done!

And as it is linked from the original paper, everyone will see it (a great
improvement over dead tree journals).

If I were the authors I would be very embarrassed.

I'm embarrassed for posting the link - nice one!

AC

Dave Kahn wrote:
On Sat, 12 Jun 2004 20:04:04 +0100, David Martin
wrote:

And as it is linked from the original paper, everyone will see it (a great
improvement over dead tree journals).

If I were the authors I would be very embarrassed.

Unfortunately it won't undo the damage already done by the stories in
the general press.

The recent press has been on their more recent paper anyway. Need to get some
responses up to that.

Tony

Tony Raven wrote:


The recent press has been on their more recent paper anyway. Need to get some
responses up to that.

AIUI, at least one has been sent (not by me), pointing out the shaky
foundations of their research (including, but not limited to, this
error). Not that the press will be interested, of course.

James