How to audit the AR4 references

09 Mar

The Citizen’s Audit of the IPCC’s AR4 has begun! Read this post for details. Here are some general guidelines that may help you in your auditing, and also some tips on how to further pursue questionable citations if you find any.

This assumes you have already e-mailed Donna ( AT and then received a Microsoft Word document containing a numbered reference list from a particular chapter of the AR4. If you haven’t yet, e-mail her!

You’ve got your particular, numbered reference list. Now begins the simple, albeit tedious, task of looking at each reference in turn. What you are looking for is peer-reviewed material. This is usually easy to determine.

What’s peer-reviewed?

It’s as simple as this: if it is in a journal, consider it peer-reviewed. Don’t worry about how credible the journal is, we are going to give the IPCC the benefit of the doubt. If we ensure our results are not biased, they cannot be easily dismissed. Give the IPCC the benefit of the doubt.

How can I identify a journal?

The name is usually the easiest way. The name of a journal in the references should come near the end of the reference, before the journal and page numbers. Often they will have words such as these associated with them: research, policy, journal, studies, weekly, today, review, update, or other names. Often they reflect a specific field, such as ‘Forestry’ or ‘Biomass and Bioenergy’.

Sometimes these are abbreviated, for example, the ‘Journal of Geophysical Research’ becomes ‘J. Geophys. Res’. Annoyingly enough, different working groups actually have different formatting for their reference lists. Working group III doesn’t abbreviate Journal names, WGs I & II do. A list of abbreviations is here that may help you. Here is a list of common Journals in the AR4:

Journal of Geophysical Research
Geophysical Research Letters
Global Biogeochemical Cycles
Atmospheric Research
Journal of Climate
Monthly Weather Review
Journal of Hydrometeorology
Journal of Glaciology

This is far from complete, but if you see these they are peer-reviewed. Remember, these are often abbreviated. Journal of Climate –> J. Clim

Another hint is the inclusion of a journal number, then page numbers. Usually the page numbers are in the hundreds, sometimes thousands.

There are also signs that show a reference is not in a journal. The reference will often include such phrases as: Consultancy report, Technical Report, Proceedings of (group), Symposium on (subject), Working papers, Findings of (group), and others. Often they will be books, easily determined if they list the publisher (e.g. Island Press). Also, if the reference includes a website, it is usually not peer-reviewed.


Examples are helpful. One chapter has already been done and posted by Donna Lafamboise, chapter 5 of WG III. Here are some references from that chapter (bold mine):

6. Akerman, J. and M. Hojer, 2006: How Much Transport Can the Climate Stand? – Sweden on a Sustainable Path in 2050. Energy Policy, 34, pp. 1944-1957.

This is peer-reviewed. The journal name, Energy Policy, is right before two sets of numbers, 34, pp. 1944-1957. The journal number is first, followed by the page numbers. This reference should be highlighted yellow. Another example:

37. COWI, 2002: Fiscal measures to reduce CO2 emissions from new passenger cars. Main report, EC, Brussel, 191 pp.

This isn’t peer-reviewed. There is no clear journal name, and while there is a page number given there is no journal number associated. This is actually a report from the European Commission I believe. This shouldn’t be highlighted, unless you were unsure, then highlight it blue.

I’m done, now what?

Count up your total number of peer-reviewed references, then divide it by the total number of all references. Or, especially if you are auditing a chapter in WG I, it’s probably easier to count the non-peer reviewed. Include that percentage in your Word document. Save your work and send it back to Donna. Make sure you’ve communicated how you wish to be identified and included on the auditor’s list.

Other tips

When you stop reviewing your reference (lunch or bathroom break), make sure you clearly identify where you left off and save your work. There is nothing more frustrating than getting through 400 references and losing your place, or even worse, losing your work!

While this particular project is to determine the percentage of peer-reviewed references in the AR4, don’t be afraid to do a little digging on any one reference. If you find a suspicious reference, first make sure you save your work and keep your place. Then take the title of the reference, and google it. See what you find. Also, look what the IPCC used the reference for. Search the reference along with ‘IPCC AR4’ and you will find it easily. Let me give you an example. Let’s say you see this reference:

177. Nippon Steel, 2002: Advanced technology of Nippon Steel contributes to ULSAB-AVC Program. Nippon Steel News, 295, September 2002.

Obviously not peer-reviewed, it’s a news article. If you google “Advanced technology of Nippon Steel contributes to ULSAB-AVC Program. Nippon Steel News” you will find the newsletter. Here it is. The article they are quoting is on one single page, three paragraphs. Seems like a pretty poor source. What did the IPCC use the source for? Now, Google “Nippon Steel, 2002 IPCC AR4” and you will find that it is in WG III chapter 5. Open up that page (or pdf) and use the find function to find “nippon”. There you go, you’ll find that they quote some technical numbers from the article, nothing fishy there.

I need help!

If you need help, or find a fishy citation and want to bring attention to it, e-mail me at our about us page. Or, e-mail Donna ( AT Also, Donna has her own FAQ’s right here. Alright, get working.



Posted by on March 9, 2010 in IPCC audit project



5 responses to “How to audit the AR4 references

  1. Cloneof

    March 10, 2010 at 2:47 pm

    If I find refrences to a company that publishes journals (like Earthscan Publications or Oxford Press) these aren’t journals we are looking for, right?

  2. Cloneof

    March 10, 2010 at 2:53 pm

    Correction, the if I would find refrence to the company, not to a journal it publishes, it doesn’t count right?

  3. ClimateQuoter

    March 10, 2010 at 10:30 pm

    No, a reference to a publishing company does not mean it is peer-reviewed. In fact, often a reference to a publishing company means it is a book, which generally aren’t peer-reviewed.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: