Wednesday, April 02, 2008

Weitzman's Dismal Theorem again

Via John Fleck, I see that Marty Weitzman's manuscript is getting more air-time. It has changed a lot since I previously wrote about it, but my fundamental criticism remains unchanged.

You have to read through about 15 pages of background and overview to get to it, but the crux of the argument is as follows. Marty describes the situation as if climate sensitivity S "has a pdf" of unknown width, and our process of learning about the width of its pdf is akin to drawing samples from "the pdf of S". In this case, our estimate for the next sample from the pdf will be a long-tailed pdf.

However, I think that S is best considered as an unknown constant, and we learn about S by making imprecise observations of it. In this case, our pdf for S (and our next observation) may or may not have long tails, depending on the nature of the observations and their errors.

This distinction is not a purely semantic one, but is fundamental to the whole process of estimation - ie, are we estimating the (finite but unknown) width of a pdf, or the location of a parameter? I think that basically all of the climate science literature follows the latter point of view (there is some oddball stuff that is sufficiently ambiguous that it's not clear what the authors intend). To be perfectly honest, I don't even know what it might mean to say that "S has a pdf" from which we draw samples. Drawing samples from a pdf is an inherently frequentist construct. In my worldview, we can treat observational error (including natural variability) as a pdf from which we sample, because we can in principle make more observations. But I don't see at all how we can sample from "the pdf of S". The fact that we will never know S precisely, and thus any estimate of S will always take the form of a pdf, is not the same thing at all. This latter pdf is "my pdf of S" and is fundamentally attached to me, not S. Any future observations that are taken will not be influenced by my beliefs about S - the thermometer can't tell who is reading it and change its output accordingly!

Some pedants may argue (correctly) that Bayesians should only really care about observations, and that the concept of a probabilistic estimate of a non-observable parameter doesn't really make much sense anyway. While that may be technically true, it's a dodge that does not materially alter things. Bernardo and Smith give this attitude short shrift in their famous tome:
However, as we noted on many occasions in Chapter 4, if we proceed purely formally, from an operationalist standpoint it is not at all clear, at first sight, how we should interpret "beliefs about parameters" as represented by p(theta) and p(theta|x), or even whether such "beliefs" have any intrinsic interest. We also answered these questions on many occasions in Chapter 4, by noting that, in all the forms of predictive model representations we considered, the parameters had interpretations as strong law limits of (appropriate functions of) observables.[...]
Inference about parameters is thus seen to be a limiting form of predictive inference about observables. [their emphasis]
As far as I can tell, nothing in Marty's basic argument is specific to climate science and climate sensitivity. If he is correct that S naturally has a long tail, then his argument appears to apply equally to all Bayesian estimates of all unknown parameters in all fields of science. I find this a priori unlikely.

I've had a frustrating time trying to debate this with Marty, and am still unclear at which point he parts company with my argument. He certainly seems to agree that the two viewpoints about the nature of S are fundamentally incompatible and that his argument rests on taking his particular approach. I've tried to get Andrew Gelman to read and comment on this manuscript, so far to no avail (mind you he seems to have gone off Bayesian statistics recently). I'd also be interested in hearing the views of any other people who really do know about Bayesian statistics.

No comments: