From Misrepresentation to Misrepresentation

The Church Times of 24 April includes a letter from Bev Botting (Head of Research and Statistics for the Archbishops' Council) and Kevin Norris (Senior Strategy Officer for the Church Commissioners), responding to my paper From Delusion to Reality, in which I critique From Anecdote to Evidence, the summary report of the Church Growth Research Programme.

The letter includes the following paragraph:

‘Dr Hart notes that the factors associated with growth account for only a small proportion of the difference between growing and declining churches (up to 25 per cent). From Anecdote to Evidence explicitly recognised this. Indeed, a quotation from one of the researchers making this point was on the back cover of the report.’

The image above is from the back cover of the report and shows the quotation to which they refer. It is flatly untrue that this quotation makes the point that the factors associated with growth account for only a small proportion of the difference between growing and declining churches. It beggars belief that Bev Botting & Kevin Norris are prepared to claim otherwise.

I challenge anyone to attempt to explain how this quotation makes the point they say it makes. And I challenge anyone to find any other reference to this point in From Anecdote to Evidence.

The truth is that this highly significant point was not included in From Anecdote to Evidence, consistent with the systematic misrepresentation of the underlying research which runs through the report.

Early in the letter, caveats included in From Anecdote to Evidence are quoted, followed by the statement that ‘It is simply not correct, therefore, to suggest that the summary report failed to include any caveats’. It would make sense to make this point if I had claimed that the report fails to include any caveat, but I never did made this charge; rather, I quote their caveats in my paper. This particular logical fallacy, called a straw man, is where you misrepresent someone’s argument to make it easier to attack.

There’s a similar straw man towards the end of the letter, where it is said that ‘Dr Hart highlights factors where findings were drawn from the self-reported data, or where the strength of correlation for self-reported data is higher than that found in the annual-returns data - though the direction of correlation is the same - and dismisses them’. It is just not true that I ‘dismiss’ these findings.

The letter opens by saying that ‘Dr Hart's critique was a selective critique of a document designed to provide a non-technical introduction to the research findings for a wide audience.’ I don’t have a problem with this, unless by ‘selective’ they are suggesting that I failed to consider parts of the report which would have changed my conclusions (there are no such parts) or by ‘non-technical’ they are suggesting that, for example, ‘small’ becoming ‘large’ and ‘hypothesis’ becoming ‘finding' are not misrepresentations after all.

More positively, the letter does not try to contradict 7 of my 8 concluding points, nor my overall conclusion that ‘according to the research, the increase in growth to be expected from the use of these factors will be nowhere near sufficient to halt the relentless generational decline, even if the resources could be found to move every lever as far as possible’.

However, they end by trying to defend the claim to have an evidence base for the Reform & Renewal programme by saying that the Church Growth Research Programme is just one part of the evidence. Yet it has repeatedly been cited as the basis, it is claimed as ‘hard information’ compared with the anecdotal, and it is undoubtedly the most comprehensive and detailed research available.