Monday 27 April 2015

From Misrepresentation to Misrepresentation

The Church Times of 24 April includes a letter from Bev Botting (Head of Research and Statistics for the Archbishops' Council) and Kevin Norris (Senior Strategy Officer for the Church Commissioners), responding to my paper From Delusion to Reality, in which I critique From Anecdote to Evidence, the summary report of the Church Growth Research Programme.

The letter includes the following paragraph:
‘Dr Hart notes that the factors associated with growth account for only a small proportion of the difference between growing and declining churches (up to 25 per cent). From Anecdote to Evidence explicitly recognised this. Indeed, a quotation from one of the researchers making this point was on the back cover of the report.’
The image above is from the back cover of the report and shows the quotation to which they refer. It is flatly untrue that this quotation makes the point that the factors associated with growth account for only a small proportion of the difference between growing and declining churches. It beggars belief that Bev Botting & Kevin Norris are prepared to claim otherwise.

I challenge anyone to attempt to explain how this quotation makes the point they say it makes. And I challenge anyone to find any other reference to this point in From Anecdote to Evidence.

The truth is that this highly significant point was not included in From Anecdote to Evidence, consistent with the systematic misrepresentation of the underlying research which runs through the report.

Early in the letter, caveats included in From Anecdote to Evidence are quoted, followed by the statement that ‘It is simply not correct, therefore, to suggest that the summary report failed to include any caveats’. It would make sense to make this point if I had claimed that the report fails to include any caveat, but I never did made this charge; rather, I quote their caveats in my paper. This particular logical fallacy, called a straw man, is where you misrepresent someone’s argument to make it easier to attack.

There’s a similar straw man towards the end of the letter, where it is said that ‘Dr Hart highlights factors where findings were drawn from the self-reported data, or where the strength of correlation for self-reported data is higher than that found in the annual-returns data - though the direction of correlation is the same - and dismisses them’. It is just not true that I ‘dismiss’ these findings.

The letter opens by saying that ‘Dr Hart's critique was a selective critique of a document designed to provide a non-technical introduction to the research findings for a wide audience.’ I don’t have a problem with this, unless by ‘selective’ they are suggesting that I failed to consider parts of the report which would have changed my conclusions (there are no such parts) or by ‘non-technical’ they are suggesting that, for example, ‘small’ becoming ‘large’ and ‘hypothesis’ becoming ‘finding' are not misrepresentations after all.

More positively, the letter does not try to contradict 7 of my 8 concluding points, nor my overall conclusion that ‘according to the research, the increase in growth to be expected from the use of these factors will be nowhere near sufficient to halt the relentless generational decline, even if the resources could be found to move every lever as far as possible’.

However, they end by trying to defend the claim to have an evidence base for the Reform & Renewal programme by saying that the Church Growth Research Programme is just one part of the evidence. Yet it has repeatedly been cited as the basis, it is claimed as ‘hard information’ compared with the anecdotal, and it is undoubtedly the most comprehensive and detailed research available.

3 comments:

  1. Justin Lewis-Anthony27 April 2015 at 21:40

    One of the pleasantly amusing parts of the letter was where they said self-reported data has little correlation with the annual returns, but at least it's more up to date! Never mind the quality, feel the speed!

    ReplyDelete
  2. Dr Hart - Your analysis of Anecdote to Evidence has provoked a lot of interest and comment. My experience of a large number of churches is that the situation is much more irretrievably far gone than many in authority are willing to admit, and that the most important thing to do in 90-95% of places is for dying congregations to create contingency plans to ensure that ancient church buildings are retained as public spaces with at least a residuum of Christian worship (perhaps on the festival churches model). I believe this ought to be a mandatory requirement akin to a quinquennial. The process for creating contingency plans might take several years - and several years is what a great many congregations have not got. The CCT budget also needs to be increased greatly. Sorry to be so gloomy! At the very least, the Church should have changed it service patterns to fit radically changed weekend timetables at least 20 years ago.

    ReplyDelete
  3. Including summaries in your essays isn't necessarily a bad thing. However, it is important to write one only when a summary is actually warranted. Excessive summaries aren't that uncommon in student writing and it often leads to very poor essays. See more paragraph summarizer generator

    ReplyDelete