Study claiming '97% of climate scientists agree' is flawed

10 Feb

Perhaps the most common argument used when urging action on climate change is the appeal to scientific authority. Previously this was accomplished by pointing at the IPCC, but since they have lost a significant portion of their credibility recently it has become more frequent to point out the scientists themselves. The most common claim that I encounter is a variation on this claim:

97% of climate experts agree humans are causing global warming.

I recently heard this claim on my own threads. I looked at the source (the study Doran and Zimmerman 2009), found some problems, and then wrote back on my threads. However, I have seen this claim so many times that I believe it would be good to make a post about it. I also e-mailed several prominent climate scientists who would be considered ‘skeptics’ to get their opinions on the study. Their responses are displayed at the end of the post. In this post I briefly comment on past responses to the study, then break my post into three sections. The first will focus on the flaw in the study (the second question), the second will look at the motives of the researcher, and the third will be posting responses from prominent ‘skeptical’ climate scientists.

First I’m going to address a common response to this study. In this post at The Hockey Schtick, it is pointed out that the 97% statistic is based on only 79 climatologists, and that those participating were self-selected. There are two concerns here. The first is sample size. While climate science isn’t a massive field, 79 participants is fairly small. To claim definitely that 97% believe this or that you would need to poll significantly more people. The second concern is the fact that the scientists were self-selected by an online survey. This may not have led to a representative sample.

Other concerns with the study deal with numbers behind it, or other reasons to consider it a poor study. However, these aren’t my primary concern. My concern is the actual questions asked in the study, which I will show in a moment.

The study on which these claims are based is available here. It is an paper by Peter Doran and Maggie Kendall Zimmerman written in 2009, entitled “Examining the Scientific Consensus on Climate Change”. Here is the citation:

Doran, P. T., and M. Kendall Zimmerman (2009), Examining the Scientific Consensus on Climate Change, Eos Trans. AGU, 90(3)

The questions

The study is fairly simple. It has a large database of earth scientists, and sends them an invitation to participate in their study. If they accept, then they take an online survey. The survey asks two primary questions:

1. When compared with pre-1800s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?

2. Do you think human activity is a significant contributing factor in changing mean global temperatures?

The first question is largely irrelevant. I’m unaware of any scientists who don’t believe the planet has gotten warmer when compared with pre-1800s levels. Not surprisingly, 76 of 79 climate scientists answered ‘risen’ to this question. I’m guessing that the other three didn’t consider the increase significant enough to warrant ‘risen’ and picked ‘constant’.

The major problem with this study is the second question. It is not phrased properly. In fact, the phrasing is so poor that I consider the entire study flawed because of it. There are multiple problems with the phrasing, so let me break them down.

1. The phrase “human activity”

Human activity comprises numerous actions which can affect the climate other than greenhouse gases. Agricultural changes and deforestation are two influences that come to mind. Now, any respondent who believes that ANY human activity can change the climate must answer yes to this question.

A better phrasing would be:

Do you think anthropogenic greenhouse gas emissions are a significant contributing factor in changing mean global temperatures?

2. The phrase “significant contributing factor”

The problem with this is obvious. What makes something significant? If 5% of recent temperature change is caused by mankind, is that significant? How about 10%? There is no context for answering the question. There is no way of knowing whether or not the respondents consider human activity the primary factor in temperature change.

A better phrasing would be:

Do you think that anthropogenic greenhouse gas emissions are the primary factor (50% or more) in changing mean global temperatures?

3. The phrase “changing mean global temperature”

This is the most problematic part of the question, because there is no indication of how much temperature change is considered worth answering ‘yes’. For example, if a respondent believed that human activities had increased the temperature of the planet by 1/10th of a degree, the answer would still be yes. Even so for 1/100th. There is no useful context here. Many climate skeptics believe that human activities have increased the temperature of the planet, but not by any significant amount. The survey should specifically ask if the warming is a statistically significant amount. Also, the word “changing” should be changed to “increasing”, because otherwise a respondent could consider human activities as cooling the planet and still answer yes.

Actually, I’d change the ending altogether. In my mind, the following is a much better question:

Do you believe that anthropogenic greenhouse gas emissions were the primary factor (50% or more) in the observed mean global temperature increase since the mid-20’th century?

I believe this question is more appropriate then theirs to determine if a consensus exists among climate scientists regarding AGW. My guess is less than 97% would agree with this statement. But I didn’t create the survey.

The second question they ask isn’t able to determine if climate scientists believe that GHGs have caused significant global warming. The poor phrasing and the ambiguity of certain essential terms renders the entire study useless. All it proves is that 97% climatologists agree that the planet has warmed since 1800, and that human activity has caused an unspecified amount of that warming. Let me be very clear and break out the bold:

The survey does not ask climate scientists if they believe global warming is primarily driven by human activity. Because of this, the survey responses cannot answer this question.

Instead of asking if human activity is the primary driver, they asked if it was a “significant contributing factor”, which is completely subjective. A respondent could believe that 10% of current warming is due to human activities, and might consider this significant enough to answer yes to the question.

Another problem with citing this study is that is only asks the opinion of scientists about whether or not global warming is occurring and humans are having an impact, but it does not address whether or not scientists are concerned about climate change. The study is being used to urge action, yet it makes no claims about whether or not scientists are urging action. This isn’t a problem with the study itself, only with how AGW proponents are using it.

Motives of the researcher

There is another aspect to this study which hasn’t been mentioned; the motives of the researcher. I think this is a valid question, especially considering that climate skeptics are constantly having their motives questioned. Anytime a skeptical paper is published the accusations of ‘big-oil funding’ start to fly, implying that the researchers have impure motives. What was the motivation behind this study? Let’s listen to the researcher himself. In combination with this press release, Peter Doran gave an interview on January 19th, 2009 to a University of Illinois at Chicago news program called Research News. It is available here. There is no transcript so I’ve written a section myself. It starts about 3/4’s of the way through the interview:

Some people have asked me since this paper came out, “Barack Obama is in office now, the Democrats are in control, do we have to worry about this anymore?”, and the answer is yes, because the general public is still about 50% convinced that global warming is an issue that’s real, let alone do we have to do something about it. And so the public needs convincing, and also, there are still people in government that need convincing. As recently as December there was a senate minority report put forward that said exactly the opposite of what our paper said, and was trying to convince people in the senate that scientists don’t agree on global warming. So there is a still a battle, if you will, to be fought here, and I hope our paper pushes the numbers towards more people believing that global warming is a reality. I think if people don’t believe that scientists agree then they can use that as an excuse for inaction, and that’s a dangerous thing.

Clearly this researcher wants his paper to change the public’s mind and politicians’ minds about the scientific consensus. I am not claiming that his paper is invalid solely because of this reason, but I do want to make the motivations of this particular study clear: the study is intended to convince the public, and politicians, that global warming is real and we have no excuse for inaction.

Response from climate scientists

In looking at the phrasing of the survey questions, I kept thinking to myself “I’m fairly certain that most climate scientists would agree with these even if they aren’t concerned about future climate change.” In order to verify if this were actually true, I e-mailed multiple ‘skeptical’ climate scientists. I included the two survey questions, and then asked the two following questions to them:

1. What are your answers to these two questions?

2. Do you believe the second question is phrased correctly to determine if climatologists consider AGW a concern?

I asked the first because I was curious to see if they did agree with the survey questions even though they are not considered part of the consensus opinion. I asked the second question to see if they agreed with my assessment that the phrasing of the second question was poor. Here are their responses.

Patrick Michaels

Yes and yes. The second question is phrased precisely to NOT determine whether or not the respondent feels it is a pressing concern.

Anyone with experience in survey development (and I know people who do this) would recognize the hidden motive here. It is telling that such a paper would be accepted with such poor design and such a foregone conclusion.

Pat Michaels

Richard Lindzen

As you know, polling is a dicey business. With respect to your first question, my answer to (1) is probably, but the amount is surprisingly small — suggesting that global mean temperature anomaly is not a particularly good index. My answer to (2) would be yes, but dependent on what is meant by significant. As to your second question, I agree that one can answer yes without any implication of alarm. Remember, according to the IPCC, we have already reached a level of radiative forcing that is almost as large as one would expect from a doubling of CO2. Even if climate sensitivity were 0.5C (which is generally considered to be of no concern) we would still be making a significant contribution to the small observed ‘warming.’

John Christy

1. What are your answers to these two questions?

Generally temperatures have risen from the little-ice age minimum in the 19th century to the present.

No one knows how much of this warming is due to human effects. In my opinion, most of the warming since the 19th century is due to natural variations.

2. Do you believe the second question is phrased correctly to determine if climatologists consider AGW a concern?

It was not phrased properly. For example someone might think that 10 percent of any warming constitutes a “significant” contribution, and so would answer yes to that question, even though the proportion of warming due to any human effect might in fact be small.

Bob Carter

Both the questions that you report from Doran’s study are (scientifically) meaningless because they ask what people “think”. Science is not about opinion but about factual or experimental testing of hypotheses – in this case the hypothesis that dangerous global warming is caused by human carbon dioxide emissions.

When tested against empirical data, this global warming hypothesis fails. For example, there has been no increase in global temperature for more than 10 years despite an increase in atmospheric carbon dioxide of more than 5%.

Rephrasing them appropriately, the scientific answers to the two questions are therefore

“1. When compared with pre-1800s levels, have mean global temperatures generally risen, fallen, or remained relatively constant?”

The answer depends (a) on what dataset you use (MSU satellites, ground thermometers, radiosondes), (b) the ways in which you plot and/or average the data points, and (c) the precise choice of endpoints.

For all datasets, however, a true statement is that there has been no significant (i.e. within the bounds of error) global warming since 1998, and some of the datasets demonstrate cooling.

“2. Is human activity a significant contributing factor in changing mean global temperatures?”

It is unchallenged that human activity has an effect (in different places either cooling or warming) on local and regional temperatures, not least as a result of land-use changes. When averaged across the globe, however, the net human effect on the global average temperature statistic is indeterminable, presumably because it is so small that it is lost in the noise of natural variation.

In addition, because carbon dioxide is a greenhouse gas and because the gas mixes globally in the atmosphere, human emissions must exert a prima facie global warming effect. In actuality, both positive and negative feedbacks then occur, which are poorly known, so there is an ongoing debate as to the magnitude of the net human greenhouse effect (the climate sensitivity issue). In any case, and again, the empirical data fail to demonstrate an unequivocal warming trend of human origin against the background of natural climate variation.

Therefore, the null hypothesis that the temperature changes that have been measured since the advent of thermometers are natural remains unchallenged. The onus of providing substantive evidence for a dangerous human-caused greenhouse effect therefore rests with the proponents of that hypothesis, and to date they have failed utterly to provide it, basing their arguments instead on speculative deterministic computer models that are known to be inaccurate.

[Dr. Carter also mentioned that he addresses these issues in more detail in his recent book on pp. 38-70, available here].

I’d like to thank these scientists for responding to my questions. I appreciate it.

On the whole, I would say that these four climatologists agree with my assessment of this study. Patrick Michaels believes the study’s results were a foregone conclusion, Richard Lindzen points out that even at a very low climate sensitivity the second answer is yes, John Christy agrees that the second question was phrased incorrectly, and Bob Carter admits that human activity has changed climate but asserts that it is too small to even ascertain. If scientists such as Patrick Michaels or Richard Lindzen, often called ‘deniers’, can agree with the survey questions asked, how can the study claim to prove any consensus? This study is seriously flawed.


This survey should not be cited as evidence that a consensus exists among climate scientists regarding AGW. This is due to the fact that it does not ask the scientists if human activities are the primary cause of increasing temperatures. The questions asked only pertained to ascertaining whether or not climate scientists agree that the earth has warmed and humans have played any role, and it did a poor job at ascertaining these facts as well. Anyone using this study to claim that 97% of climate scientists agree that humans are the primary cause of global warming is ignoring the ambiguous and poor phrasing of this survey questions. The survey does not ask if global warming is primarily driven by human activity, so the survey responses cannot answer this question.



Posted by on February 10, 2011 in Uncategorized



27 responses to “Study claiming '97% of climate scientists agree' is flawed

  1. Donna Laframboise

    February 11, 2011 at 4:29 pm

    Great piece. Don’t know if you saw this similar analysis here:

    97% cooked stats

    And for a complete change of pace, here’s a headline I stumbled across recently:

    Thirty-two years of consensus (as if that were a good thing)

  2. Sam

    February 11, 2011 at 5:08 pm

    Thanks for the link, that piece seems to raise similar issues with the study as mine. Interesting that in the comments it was dismissed because it was in a financial publication.

    I’m hoping that in the climate blogosphere, anytime someone makes the 97% claim someone else will send them to this article, because I really do believe that anyone with an open mind can see this is a flawed study.

  3. GM

    February 11, 2011 at 9:57 pm

    Good dissection. Enjoyed the contributions from the Four Horsemen of the Non-pocalypse™

  4. DocBud

    February 12, 2011 at 11:22 pm

    “Do you think human activity is a significant contributing factor in changing mean global temperatures?”

    Well of course it is, but I’m referring to the activity of James Hansen and all the other adjusters of raw data:

    And there are also those people who park adjacent to weather stations or put aircon outlets near them:

  5. Jevaughn Brown

    February 15, 2011 at 1:25 pm

    “There are three kinds of lies: lies, damned lies, and statistics.” – Benjamin Disraeli

    Case in point: “97% of climate experts agree humans are causing global warming. ”

    Nicely rational and methodical article. It does my heart good to read something of intelligence online regarding climate science.

  6. Pablo

    March 3, 2011 at 8:43 am

    It’s actually 98%. “Some 98% of climate scientists that publish research on the subject support the view that human activities are warming the planet, a study suggests.

    It added there was little disagreement among the most experienced scientists.

    But climate sceptics questioned the findings, saying that publication in scientific journals was not a fair test of expertise.

    The findings have been published in the journal Proceedings of the National Academy of Sciences.”

    Too many, mostly very right wing folk, searching the web for straws to clutch, so they can shout
    down the majority and most respected scientists. Sceptic views generally don’t get published in science journals much because they know they’ll fail or their research doesn’t pass muster.

  7. Otter

    March 9, 2011 at 4:58 am

    I see we got some Pablo(m) from the true deniers on this one.

  8. Sam

    March 9, 2011 at 10:45 am

    I’m afraid you’re right Otter.


    I write a fairly long post which involved contacting several prominent scientists for their opinion. In my opinion, it contains strong evidence that the focus of my post, Doran and Zimmerman 09, is flawed.

    Your response?

    Quoting something which you don’t identify, then claiming we are grasping at straws. If you think I’m wrong, why not explain how my article is wrong.

    I suspect you are (accidentally) quoting an article about Anderegg et al 2010. That’s an entirely different story.

  9. CalvinX

    July 21, 2011 at 4:26 pm

    Thank you for posting the article. It amazes me how the GW crowd is hijacking science for their ends.

  10. Darren

    July 24, 2011 at 8:46 pm

    If you want to dispute something for being biased, perhaps you shouldn’t only ask well-known skeptics for their responses to back up your claim. Did you ask yourself whether Bob Carter was just angry that he wasn’t included in this figure considering he doesn’t have the knowledge or expertise to get published in any reputable climate science journal?

  11. Sam

    July 27, 2011 at 8:00 pm

    Darren, thanks for joining the discussion.

    The questions I asked would only be appropriate for skeptics, because the point is to demonstrate that the survey questions were flawed. These skeptics would have answered in such a way so as to have been considered in the “consensus” camp, which invalidates the point of the survey.

    Your contention that a skeptic scientist might be angry doesn’t make sense. This was a self-selected survey done of climatologists. Perhaps you are confusing this with Anderegg 2010?

    Or maybe you didn’t actually read the post at all?

  12. Jack

    August 30, 2011 at 4:13 pm

    The source of the 97% statistic claimed by politicians, etc. is in fact the Anderegg study, so refuting the Doran and Zimmerman study is pointless. There are a ton of flawed studies out there.

  13. Ellen

    August 30, 2011 at 9:19 pm

    It is unfortunate that those who wrote the “review” and responding comments above clearly do not understand what “significance” means to scientists, that it is the standard by which all research (whether medical, nutritional, environmental, or…) is evaluated to determine if the research results should be considered important and of value. The whole premise of the critique above is that the question of “significance” isn’t a good way to ask researchers the question, and a more commonly understood (but meaningless from a scientific research standpoint) question should have been asked.

    In stating that the researchers should have asked a different question, the author is simply reflecting his ignorance of science and the scientific method.

    The truth is that Mother Nature simply doesn’t give a damn whether people believe that climate change is occurring, or whether they believe that humans are playing an important role in accelerating it (and I have not seen any scientific reports that state that ALL the change is human caused). The only important thing is that we have a chance to make changes now to perhaps reduce the rapidity of the change to reduce its ultimate harm. It is much easier to believe the comforting words of the scientifically ignorant than to respond with appreciation to the scientists who have offered us a chance to reduce the effects of climate change by highlighting the importance of taking action now.

    It is my understanding that recently sunspot activity was actually reduced, not increased, which raises the question of what will happen to earth’s temperatures when it increases once again. We are already on the roller coaster of the magnified (in size, not number) weather events that had been predicted — witness the 500 mile wide hurricane that just hit the US coast (the largest on record?); and the multiplied tornados that hit the South a while back. So hang on for the ride!

  14. Sam

    August 30, 2011 at 10:20 pm

    Jack, Doran and Zimmerman is very frequently cited, nearly every time the 97% number is used it is Doran.

    For example, look at Cook’s site:

    Scientific Consensus

    Anderegg 2010 said 97-98%. I wrote about Anderegg here:

    Is it 97% or 66% of climate scientists who believe in AGW?

    Ellen, I understand significance, but the question confuses the term. It does not ask for statistical significance. You can argue it is implied, but if that is true it may still be statistically significant yet relatively small. The poll doesn’t define how large an impact humans have.

    Perhaps you are arguing that they are not asking for statistical significance, but a measure of “scientific importance”. Fine, but this leads to my claim of subjectivity. If humans impact the climate by 0.1c is that scientifically important? 0.4c? 0.0005c?

    The poll is really only asking if humans have any impact. That would be fine (I accept humans impact the climate) except this study is constantly being used to urge for action since “97% of climate scientists believe in climate change”. 97% of them believe in what exactly? Humans impact the climate? Of course they do! That doesn’t mean it is a concern, which is what those scientists above are all saying.

  15. Marty Koessel

    September 12, 2011 at 2:25 am

    Just 79 climatologists? Try 1,372:

    Anderegg, Prall, Harold, and Schneider, 2010A 2010 paper in the Proceedings of the National Academy of Sciences of the United States (PNAS) reviewed publication and citation data for 1,372 climate researchers and drew the following two conclusions:

    (i) 97–98% of the climate researchers most actively publishing in the field support the tenets of ACC (Anthropogenic Climate Change) outlined by the Intergovernmental Panel on Climate Change, and (ii) the relative climate expertise and scientific prominence of the researchers unconvinced of ACC are substantially below that of the convinced researchers.[107]

    The methodology of the Anderegg et al. study was challenged in PNAS by Lawrence Bodenstein for “treat[ing] publication metrics as a surrogate for expertise”. He would expect the much larger side of the climate change controversy to excel in certain publication metrics as they “continue to cite each other’s work in an upward spiral of self-affirmation”.[108] Anderegg et al. replied that Bodenstein “raises many speculative points without offering data” and that his comment “misunderstands our study’s framing and stands in direct contrast to two prominent conclusions in the paper.[109] The Anderegg et al. study was also criticized by Roger A. Pielke,[110] Pat Michaels, Roger Pielke, Jr., and John Christy.[111] Pielke Jr. commented that “this paper simply reinforces the pathological politicization of climate science in policy debate.” [111

  16. Sam

    September 12, 2011 at 7:34 am

    I’ve mentioned Anderegg multiple times already, if you bothered to read the comments:

    Your number is wrong, 1372 is TOTAL number, of which 903 accept human caused climate change. Go read it and inform yourself.

  17. john

    November 26, 2011 at 2:23 pm

    There’s your problem you don’t know squat about the scientific method! “I also e-mailed several prominent climate scientists who would be considered ‘skeptics’ to get their opinions on the study. ” Yes there are 3% who question weather or not mans activities are contributing. Some of them say they don’t know how much… so it might be man is not.

  18. Sam

    November 26, 2011 at 7:53 pm

    I try to respond to every comment on here but this last one doesn’t even make sense. Can you clarify what I have got wrong?

  19. Mike Everet

    January 11, 2012 at 8:05 am

    I know this point has already been made, but it is worth reinforcing for the sake of accuracy: when people talk about 97-98% of expert climate change scientists agreeing on the tenets of anthropogenic climate change (as outlined by the IPCC), they are referring to the research paper by Anderegg, Prall, Harold and Schneider:

    This research started with a dataset of 1372 climate researchers and from this group selected the 200 most published – and hence supposedly the most ‘expert’ – scientists. This is where there was the 97-98% consensus: among the most published scientists.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: