environmentalresearchweb blog

« Too much cheap wind?! | Main | The cost of nuclear »

In praise of grey literature

In the aftermath of the fuss about Himalayan glaciers, I have noticed a tendency among my colleagues to hesitate about citing so-called “grey” literature – loosely, stuff that has not been reviewed by scientific peers and accepted for publication by an editor acting on recommendations from such reviewers.

Some have argued that we should stop citing any publication that has not appeared in a peer-reviewed journal. The snag about this idea is that it would make scientific studies of the climate in general, and glaciers in particular, almost impossible. Much of the raw data appears in documents, and nowadays files on the internet, published by governments or quangos. Sometimes there is a reviewed paper to document the work underlying the measurements, sometimes not.

In glaciology, some of our mass-balance measurements are superbly documented in high-profile journals. I don’t know of any wrong numbers in this kind of source, but by definition the documentation is not superb if it doesn’t include a thorough analysis of uncertainties. It is a pity, but hardly the fault of the authors, that readers in a hurry tend to read the name of the journal but to skip the thorough analysis.

Some measurements are mentioned only briefly in very obscure documents, with few or no accompanying details. You can only decide whether to accept this kind of measurement by reading critically and judging whether the measurers knew what they were doing. (I will come back to this idea of reading and judging.)

Most of the measurements lie between these extremes. You can find them in black and white, with some background information, but in a grey source. Your choices, in a context in which you are desperately short of hard facts, are to reject the measurements because they were not peer-reviewed, reject them because they do not stand up to judgement, or accept them with appropriate reservations.

It is not as if publication in the peer-reviewed literature is a guarantee of correctness. There are some appalling wrong results in the literature. Among the most famous examples is the 1989 claim by Fleischmann and Pons (in the Journal of Electroanalytical Chemistry, volume 261(2A), pp301–308) that they had observed cold fusion, that is, the fusion of atoms at room temperature. It would have altered our world forever, but it was a report better suited to the Journal of Irreproducible Results.

During the recent furore, a recurrent criticism of the Intergovernmental Panel on Climate Change was that it had cited grey literature, as if that were a mortal sin. Indeed, at the centre of the Himalayan maelstrom was a report by the World Wildlife Fund, usually berated by the ill-disposed or mischievous as a “green advocacy group”. Somebody then counted other IPCC references to WWF reports and found 16. I haven’t read the other 15, but the Himalayan one was in fact pretty good – thorough, reliable except for one howler and, ironically, reviewed. The WWF made the same regrettable error as the IPCC’s Himalayan-glacier authors, namely swallowing nonsense uncritically from a popular science magazine. It is further ironic that the WWF and the IPCC can be shown to have made this error independently, but that the IPCC erred additionally by splicing in a reference to the WWF instead of to the popular magazine. So the WWF was a victim of friendly fire, but not an innocent victim.

The culminating irony, however, is that the IPCC’s guidelines for the treatment of grey literature, in Appendix A of the Principles Governing IPCC Work, are a model of reasonableness. (See Annex 2 in particular.) Had they been followed with respect to the Himalayan glaciers the fiasco would never have happened. The IPCC’s guidelines for handling work that has not been reviewed by peers boil down to “read the darn thing for yourself and do your own review”.

It strikes me as good advice. If more people – preferably everybody – were to heed it, we would all be better off. And whether the source were grey or not wouldn’t make any difference. The most basic error is accepting authority as a substitute for reasonableness.

Posted by Graham Cogley on May 17, 2010 3:00 PM | Permalink