The American scientist Roger Revelle famously wrote in the 1950s that "human beings are now carrying out a large-scale geophysical experiment of a kind that could not have happened in the past nor be reproduced in the future". More than 50 years on we are well advanced with this experiment. Greenhouse-gas concentrations continue to rise, atmospheric aerosol loadings reflect human activities, and climates around the world are beginning to take on less familiar characters. We do not yet know the final outcome of the experiment.

Many would say that we are now engaged in another large-scale experiment, the like of which we also haven't seen before. We are in the early stages of a worldwide socio-political trial to see whether the whole panoply of human behaviours, preferences and practices can be directed towards achieving one over-arching goal: to neutralise the effects of Revelle's experiment by reducing global greenhouse-gas emissions and the production of other climate-changing agents. Unlike the inadvertent geophysical experiment, this socio-political one is both deliberate and purposeful. But as with the geophysical experiment underway, we do not know where it will lead, and we are even less sure about what its side effects will be.

The IPCC: a third experiment

But I believe there is a third experiment that it's important to identify and reflect on. This is the attempt to synthesise, globally, the body of scientific knowledge about the changing climate system and its present and future impact on matters of human concern. This experiment started, I suggest, in 1988 with the formation of the Intergovernmental Panel on Climate Change and is continuing in its most recent phase through the preparation of the Fifth IPCC Assessment Report due to be published in 2013/14. It is an undertaking that in some way connects the two other experiments: how do we organise and mobilise our limited human understanding of Revelle's geophysical experiment in such a way as to be useful for designing the emerging socio-political experiment?

Why do I suggest that the IPCC knowledge about climate change is experimental? The IPCC has three key characteristics: it is an international knowledge assessment; it is multidisciplinary; and it is governed by an intergovernmental process. These three characteristics distinguish the assessed knowledge produced by the IPCC from what might be called primary disciplinary knowledge generated by scientific research and scholarship. For the latter there are well-established and well-tested rules, protocols and practices that extend back to the emergence of modern science in the late 17th century. But the conventions and procedures that govern the international, multidisciplinary and intergovernmental process of IPCC knowledge-making have to be made up on the hoof. The IPCC is experimenting with methods of knowledge production and so brings into being a new hybrid form of scientific knowledge.

Blank canvas

When the IPCC was created in 1988 under a formal mandate from the United Nations, there was no existing model to follow, no analogous institution to copy. Decisions had to be taken from scratch on how expert authors from around the world were to be recruited, what forms of scientific knowledge were to be assessed, how successive drafts were to be checked and reviewed, and what roles governmental and non-governmental interests and expertise should play. Such an ambitious global form of public knowledge-making had not been attempted before. It was very different from making knowledge in the lab, the field or the library.

Because this has been an experimental process, the IPCC's rules and procedures have changed or been adapted over time. For example, the new role of review editor was introduced after the Second Assessment Report of 1995, and more specific guidelines have been developed on the use of non peer-reviewed, or grey, literature. All reviewer comments and the authors' responses to these comments are now made public. And formal attempts have been made to adopt a particular linguistic vocabulary to convey different levels of certainty and confidence when making specific knowledge claims, for example the phrase "very likely" denotes a subjective probability of more than 90%. The obligations of lead authors for the Fifth Assessment Report are very different – unrecognisable almost – from the quite informal duties of authors involved with the First Assessment Report in 1988.

Experimental error?

Given this context it is important to see the IPCC as producing experimental knowledge rather than yielding infallible texts (not that science ever does infallibility – only religious texts and authorities can do this). Not only does primary scientific understanding of the physical and social world evolve, but so too do the ways in which scientific knowledge is validated and made authoritative for public use. The IPCC needs to be responsive to such evolution.

So does it matter that errors, such as the incorrect date for the projected melting of Himalayan glaciers, get made? All mistakes matter, but more important is whether mistakes are honestly admitted and whether procedural lessons are learned. For an institution like the IPCC seeking to make authoritative public knowledge this learning needs to be conducted in the open, so that public confidence and trust in its judgements is retained. There are similarities here between the significance of the leaked CRU e-mails and the mistakes in the IPCC report. The former raised questions about how "primary disciplinary knowledge", as I have named it here, is made; the latter episode questions how this primary knowledge is synthesised into trustworthy global hybrid knowledge. Both cases appear to have contributed to the public mood of suspicion, and in some instances the admitting of mistakes appeared to be reluctant or tardy.

As to whether lessons have been learned, a verdict is premature. The recent report from the Dutch Environmental Assessment Agency (PBL) into the chapters of the IPCC Fourth Assessment on the regional impacts of climate change (PBL, 2010) provides a good opportunity for learning – both about the complex nature of this hybrid IPCC knowledge and also how to improve IPCC processes and protocols for the Fifth Assessment.

The PBL report identifies seven levels of IPCC assessment activity, starting with the creation of primary literature and ending with the production of the Summary for Policy-Makers (SPM) of the Synthesis Report. What becomes apparent is how heavily crafted are the core messages that emerge at the top of the IPCC pyramid. At each level, significant linguistic filtering of knowledge statements occurs before they emerge as a headline key message in the SPM of the Synthesis Report (such as "By 2080, an increase of 5 to 8% of arid and semi-arid land in Africa is projected under a range of climate scenarios"). This is a deliberative process of collective judgement, shaped by the primary knowledge base as interpreted by individual expertise, and further modified through intergovernmental negotiation. What emerges well deserves the label of "hybrid knowledge".

But this is what the IPCC does and we need to understand better the experimental process by which this knowledge comes into being.

A matter of trust

What the turbulence of recent months shows is that in the making of authoritative public knowledge about climate change – knowledge that is trustworthy, and hence trusted for public use – it is the social and institutional practices of knowledge-making that matter as much as the substance of the knowledge itself. It is unrealistic to expect infallibility from scientists; but it is necessary to demand they be subject to some form of public accountability. Equally, it is dangerous for scientists to be insensitive to the social contexts in which their knowledge assessments are made and used; but it is necessary for qualified experts to be able to do what society expects experts to do – make fair and considered judgements on the basis of their expertise.

The first formal independent review of the policies and procedures of the IPCC is now underway, conducted under UN mandate by the Inter-Academy Council. A report is due after the summer, for consideration by the IPCC governing body in South Korea in October. It is important that this review be fearless and thorough in its evaluation and its recommendations. In turn, the governing body of the IPCC must respond thoroughly to these recommendations. And scientists involved with the IPCC must abide by the outcomes, as part of this on-going experiment in knowledge-making.