The Dog Ate my Data

Good grief. Is any of the adjusted data worth a damn?

The million dollar question. Without reasons for the adjustments who knows. I see more andmore people coming out with these sorts of plots asking precisely that question.

Shouldn't the raw data at least hint at warming? Yet, it is only after the chicen bones have been cast and the incense burned : ) that we get this magical transformation.

Seriously though, the graph I produced and yours, both point to the data being manipulated pre-1980, yet left alone thereafter.

So in otherwords, temperature gauges were defective pre-1980 and then perfectly calibrated post 1980?

That is what we are to believe?

Sent the link to my brother in Cairns.

Thanks.

I have a new post up that links to both of your work.

I did a temperature comparison of raw vs homogenized data for my hometown, Cincinnati, Ohio.

In this case, they deflated the warm spike between 1930 and 1950 to make the post 1980 period look like it had a run of "unparalleled" warming.

http://thevirtuousrepublic.com/?p=4828

This is getting good! I can smell a danRather!

I am very new at looking at the actual data on "Climate Change". As I examine the storyline and learn a bit about the techniques being used by the eggspurts some things become pretty obvious. There is a lot less land mass in the southern hemisphere and a lot more ocean. Looking at the land in the southern hemisphere I am struck by some interesting facts. Comparing the continents, Antarctica is 5th and Australia is 7th in size. The two of them combined are larger than S. America and a bit smaller than N. America. Their combined poulation is 32 million. Why all these factoids?

It is painfully obvious that the best way to skew global warming results with the least amount of obvious chicanery is to misrepresent these two continents. Add it all up and you should not be too surprised at all at what you are finding in Australia. New Zealand has similar data problems IIRC. When this story is finally completed, it may be amazingly obvious how easy it was to pull off the AGW scam misusing the southern hemisphere. I am not ignoring Africa or South America. They have there own stories to tell in the data I am sure.

I admire the work and effort you have put into your blog and love to read it every day! I apologize if I am stating the obvious to people here with a lot more knowledge than a newbie.

The Machiavellian said...

"Shouldn't the raw data at least hint at warming? Yet, it is only after the chicen bones have been cast and the incense burned : ) that we get this magical transformation.

Seriously though, the graph I produced and yours, both point to the data being manipulated pre-1980, yet left alone thereafter.

So in otherwords, temperature gauges were defective pre-1980 and then perfectly calibrated post 1980?

That is what we are to believe?"

Hi Machiavellian and thanks for the comments.

With my chart of the Birsbane airport temperatures for 1950 - 2008 (that station was opened part way through 1949) the GHCN adjusted figures have adjusted the early temperatures (pre 1978) down and the later temperatures up (note the see-saw effect of the flicking image I did) which totally changes the cooling raw trend to a warming adjusted trend. As I said in the article we need to know why and we need to know why all the adjustments were made to every station.

The Machiavellian said...

"I did a temperature comparison of raw vs homogenized data for my hometown, Cincinnati, Ohio.

In this case, they deflated the warm spike between 1930 and 1950 to make the post 1980 period look like it had a run of "unparalleled" warming.

http://thevirtuousrepublic.com/?p=4828"

Certainly following the Climategate scandal I have no faith in the adjustments without documentation. When one looks at the satellite data for the lower troposhpere it shows a very minor warming trend. I have much more faith in that data but it only rund from the late seventies. Interesting that adjusted data by humans shows warming and satellite data does not show any significant warming. I also think that to use adjusted temperature data and then say there has been warming of say 0.4 deg C is rediculous. How can you be sure of tenths of a degree warming when you have made SUBJECTIVE adjustments to data. I supposed if you could say the adjustments are unbiased then on average you would expect any errors to cancel out - but can we say the adjustments are unbiased? I don't know and the problem is the CHCN/GISS/CRU/IPCC cannot or will not prove it.

The debate continues..

Sir Henry, thank you kindly Sir.

Borepatch, thanks for the link. As I said elsewhere that little animated graphic has gone semi-viral thanks I think to your link. I usually get about 15 visits a day and it looks like it will crack over 500 today.

DirtcrashR, Well certainly the more people made aware of the issues and the uncertainties the better. The debate is NOT over. The debate is only just beginning thanks to the furore over the Climategate scandal. Truth will out in the end.

Thanks again for the comments everyone (and anyone I missed) - I have enjoyed it very much.

Hi David.

Thank you for your kind words! Keep reading. Check out some of the links in the sidebar. Dr Roy Spencer is a good place to start. He is a real expert, easy to understand (former NASA climatologist) and has no discernable political agenda. Climate Audit and Watts Up With That are great but heavy going until you get into it. As to the hemishperes/continents, I will be posting on some of the issues in due course. Note that that Antartica and the north polar regions show warming in the terrestrial temperature data. The tropics, southern hemisphere much less. Antartica shows cooling but I did note a discussion on WUWT questioning some sort of cluster of sensors in the western pen. which is the warmest end of antartica - I didn't read it deeply as I had other things on but will get back to it.

C.

Adding in the "correction factor" in black as in the Darwin plots would be a nice touch. The difference between the raw and adjusted values is, afterall, the "algorithm" used to make the "correction". The rationale for using that "algorithm" is, of course, quite another matter - one which I don't expect to read about very soon.

How do you know that the GHCN "raw"
data has not already been adjusted (by the local national weather service or other source(s) of the data to GHCN)?

How do you know that the GHCN "raw" data have not been adjusted by the data source (National weather Service?) prior to being included in the GHCN database?

How convienient that after "calibrating" the thermometers post 1978, not only did all temperature to either side of that year move either up or down, the amount they were adjusted up or down looks exactly linear as referenced to to 1978. I guess it really depends on what the meaning of "calibrated" is. If we define "calibration" as the act of fixing data to generate the maximum quantity of AGW research grants, I can then see how the AGW proponents can still believe this crap and keep a straight face.

What is the accuracy of the RAW temperature measurement system (I would be more interested in how this is calculated than even seeing the raw data)? Neither adjusted nor unadjusted may tell us anything if it is simply noise.

The Grand Canyon has been played with like crazy too.

http://i45.tinypic.com/bguywn.jpg

Raw data is quite interesting for the longest running records around as well. Even without urban heat adjustment, few show any AGW signal:

http://i45.tinypic.com/125rs3m.jpg

If some data during the reference period is adjusted, it changes the whole dataset since the average temperatures change.

I am pretty sure there is a good reason for adjustments. You can see that the Australian Bureau for Meteorology clearly demonstrates that pretty much the whole Australia has been warming.

Google Australian climate variablity and change

...and you'll see that the warming is real, or the Australian Bureau for Meteorology is in this huge conspiracy.

Tuukka

You misinterpret the data. My plots investigate trends in the temperature anomaly using simple straight line least squares regression. Change the normal to whatever you like; it will not change the trend as we are not looking at absolute values. You should be able to eyeball that easily from the animated gif I prepared. You can easily check this for yourself by downloading the data for Brisbane Aero from the GISS website and perform your own tests to satisfy yourself.

With respect to you being “pretty sure there is a good reason for adjustments” one of the main points to the exercise was to see what adjustments were made to data from my local weather station and look at the reasons for those adjustments in light of what I know from living here most of my life. Although there are general statements as to adjustments it is not possible to determine precisely what adjustments were made and that is the question. Major adjustments were made prior to the mid 1970’s in a downward direction and post that time in an upward direction. Given that Brisbane has gone from a sleepy country town to a fairly major city in that time and that UHI effect would dictate a downward adjustment to the later temperatures rather than upwards, all other things being equal, I am particularly interested in the precise reasons for these counter intuitive adjustments. The only other station I can find within 100 km with a long temperature record is Cape Moreton. I though perhaps it had a huge upward trend and that Brisbane had been adjusted upwards based on that but no – Cape Moreton lighthouse shows a downward trend too on the raw data.

As for getting the data from the Australian BOM – I was interested in the GHCN data as it bears a very close resemblance to the data used for the IPCC reports. Therefore BOM data is irrelevant. Out of interest I did look at BOM and found they only make available a little over a decade of data for Brisbane Aero however even that shows a slight downward trend. So we are left with the reasonable question – why is the data adjusted from a negative trend based on observational records to a positive trend based on homogenised data. Perhaps you could advise me of the answer.

With respect to BOM findings based on their "value added" or "adjusted" data for Australia as a whole I would need to know the data set and the resonas for all adjustments to the original raw data before I comment. When you get that information to me I would be pleased to look at it and comment in detail. My post looks at specifically one station, namely Brisbane Aero. If you wish to widen the debate I will let you get the data together before we go further.

Your attempt to label me as a conspiracy theorist is a standard attack by catastrophic AGW proponents and I shall dispense with it using my standard response. Please see my next post for that part of my response.

Tuukka (continued),

I am cut and pasting with minor modification a response to your consipiracy theory allegation. I do this with non-arguements such as that to save time.

The emails released by the Climategate whistleblower contain clear evidence, now being investigated by the prosecuting authorities in the United Kingdom, that climate scientists prominent in the UN’s climate panel and in leading meteorological institutions round the world conspired to block, withhold, or destroy scientific data that had been validly requested under the UK’s Freedom of Information Act. The emails also show how The Team, as these bad scientists called themselves, tampered with the peer-review process, leaned upon editors to delay publication of papers by those with whom they disagreed and to provide them with advance copies of such papers, and – in essence – made up the global temperature record for the past 150 years. Their scope for fabrication was limited over the past 30 years by independent satellite temperature data: however, it cannot now be said with any certainty whether or to what extent the warming of the previous 120 years was real. There is indeed a legitimate consensus to the effect that CO2 is a greenhouse gas and that enhancing its atmospheric concentration will cause some warming, since this consensus rests upon hypotheses that have long been proven. However, there is no consensus to support the UN’s hypothesis that the warming to be expected from CO2 will be very large and potentially catastrophic.

Privately, one of the CRU Team (see recent blog post on this) wrote to his colleagues that there had indeed been nine years’ global cooling; that The Team could not explain the cooling, and that it was “a travesty” that they could not explain it. We can explain it, of course: natural factors drown out the tiny warming signal from CO2.

The father of the scientific method, Abu Ali Ibn al-Hassan Ibn al-Hussain Ibn al-Haytham, in 11th-century Iraq, wrote, “The seeker after truth does not place his trust in any mere consensus, however broad and however venerable. Instead, he subjects what he has learned of it to his hard-won scientific knowledge, and to measurement, scrutiny, and verification. … The road to the truth is long and hard, but that is the road we must follow.” If the CRU scientists had followed Abu Ali Ibn al-Hassan Ibn al-Hussain Ibn al-Haytham’s advice they would not now be faced with investigations from the prosecuting authorities. It does not take a conspiracy theory to believe this – it is fact and you know it as does the entire world.

(continued next post)

Tuukka - continued from previous post

Sometimes Catastrophic Anthropogenic Global Warming believers try to maintain – without offering any evidence – that climate scientists who “do not agree about climate change” are “very much a minority”. This is the tired, old argumentum ad populum again, and it is the very antithesis of the scientific method, in which counting heads plays no part whatsoever. However, the only survey of active climatologists ever conducted found that 97% of them agreed that CO2 causes some warming, and may have contributed to warming in recent decades. No skeptical scientist would disagree with that. (Note: A peer-reviewed analysis of 539 papers (following Oreskes) showed not a single paper provided any evidence that “global warming” might prove catastrophic (Schulte, 2008).)

Dr. Art Robinson decided to conduct a wider survey. He approached tens of thousands of scientists and researchers, half of whom were in climate-relevant fields. Some 31,000 of them individually signed statements that they did not agree with the UN’s principal conclusion that “global warming” in recent decades is chiefly manmade. That is the largest head-count ever conducted on this subject. So, if one uses the head-count argument, far more scientists have expressed clear, written opposition to the UN’s official position than those who have expressed the contrary opinion; but, in any event, the head-count argument is unsound, as is the reputation argument, and should not be relied upon.

Just on the topic of whether there was a medieval warm period there are studies by 770 scientists worldwide that refute what the UN’s climate panel tells us is the “consensus”. On each of the UN’s major conclusions, there are very large numbers of scientific papers that question what we are told is the “consensus”, and the question of the medieval warm period is by no means the only one on which the true consensus in the peer-reviewed literature is entirely the opposite of what the UN wishes us to believe.

So in summary, no there is no conspiracy, there are thousands of scientists who question catastrophic AGW, leading IPCC authors from the CRU are under investigation from prosecuting authorities, Mann is under investigation by Penn State University – don’t you think it wise under the circumstances based on this alone to be a little skeptical. I certainly do not and will not take your word for it that, “I am pretty sure there is a good reason for adjustments” when we are being asked to fund such tenuous conclusions to the tune world wide of trillions of dollars over the next decade alone. I wouldn’t “trust” data such as that to base a business decision to open a lemonade stand and quite frankly I don’t believe you would either if it were your investment decision.

Wasting your time on Tuuka, his blog smells very warmist. Had to go by smell, I don't read Finnish.

GHCN has a nice offering of raw data ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily

If only CRU would cough up theirs... we could have some real fun.

Thanks Gene for that link. I am redoing this in R as we speak and including other Australian stations and Antartica and see what results so the link is particularly timely.

Hi Charles, Nice article, cool animation. I too live in Brisbane so was very interested in this.

Some of my thoughts are:
o - Monckton is doing a lecture tour of Oz in Feb 2010, I think he should use this graph, at least for the Brisbane gig. Have you contacted him.
o - Do you have the links to where to get the data from GISS so I can reproduce your analysis. I looked on GISS but could not find it.
o - Is there a way to contact you if I have more questions.
o - Have you sent this to the Courier Mail?

John in Brisbane

Charles, being a resident of Brisbane, the RAW data points post 2000 amaze me greatly as we had some very hot (and dry) periods with the post 2000 summers. You must remember them as they coincided with Brisbane's dam levels declining to 18%.

But the BOM data does not reflect those hotter (than normal?) summers. That seems very curious?

-Steve

John,

Re your queries,

1. Monckton – no not about this. I understood from a recent update on his tour that it was going to be in the form of a debate with a scientist and, I think a journalist, in Brisbane rather than a presentation.

2. GHCN data can be obtained from an ftp server (see post by Gene above) but if this is your first time looking at it the easiest format is to go to the GISS website and explore it there. Note that the unadjusted GHCN may have been adjusted somewhat (I have not looked into this but I think it was EM Smith who posted that) but I am prepared at the moment to take it on face value for the moment. The GISS website is here:

http://data.giss.nasa.gov/gistemp/station_data/

Take a look at that and feel free to ask questions if you run into difficulty. Not that when you manage to produce your graph on line there is a link below it to download the data in text format so you can then feed it into R or excel or whatever you are using and reproduce the above.

3. You can contact me by posting a message to me with your email. As I have to approve posts before they appear I will email you and delete the post so it is not public. You will then have my email to contact me.

4. the CM – no. I will discuss by email but I have no interest in that and it is not worth a media piece. I am merely showing what happens with a single station and that is of little relevance to a general discussion except to show the importance of total transparency (w.r.t. what the adjustments are and why they were made to each station – GISS handle it with an algorithm which is of little comfort to me) in situations like this where adjusted data is being used as the basis for national and global policy making involving huge economic changes and costs as well as changes in the ability to enjoy the fruits of one’s labour and life. Also note I have only shown GHCN data here no BOM data. BOM actually modifies the data significantly more than GHCN homogenised data for Brisbane Aero. There are papers published by current and former BOM climatologists addressing the adjustments in a general sense and it will be the subject of a later post but I can give you a quick heads up in email if you like and link you to the relevant papers you should read.

Hi Steve,

Actually I think I was in the UK at that time but I certainly recall hearing about it.
First, please note none of the data in the graphs above is taken directly from BOM; it is GHCN data. No doubt it had its origins with BOM. As it happens I am preparing a post which will include more detail and the BOM data but it will take another week or two of work to finish so you might like to check back for that. I’ll be including links to the data so that may interest you as it will save a lot of tiresome digging around to find it.

As you would have noted the above graphs are of yearly anomalies. The yearly anomalies are calculated by first calculating a yearly mean temperature from monthly means. Briefly GISS does this by dividing the year into seasons, calculating mean monthly temperatures by (Tmax+Tmin)/2 and then calculating a seasonal anomaly and finally averaging that and calculating the yearly anomaly. A complete explanation is available in the link I posted in my reply above and as it is a little involved I recommend you check that out if it is of relevance to you. The anomaly is based on variances from a baseline which varies according to what dataset you are looking at and who calculates it but that is not that important when looking at trends as it is rather arbitrary in the first place and won’t affect the outcome except in an absolute sense.

All this averaging, no matter how one goes about it, loses a lot of information in the form of variability. There is no way on earth you are going to “feel” the tiny few decimal points in average yearly temperatures that we are talking about in just about any data series. The adjusted GHCN graph above has a positive linear trend (based on simple linear regression) of about 0.6C per century. Below are the mean annual temperatures for Brisbane around the year 2000 from BOM data (these are adjusted by BOM for various reasons to produce what they technically refer to as their High Quality data set for temperature). The mean temperature is the first figure and the second figure is the anomaly from the standard BOM baseline period 1961-1990. The mean temperature and anomaly is in degrees C.

1999, 20.0, -0.1
2000, 20.1, 0.0
2001, 20.5, 0.4
2002, 20.5, 0.4
2003, 20.2, 0.1
2004, 20.6, 0.5
2005, 20.9, 0.8
2006, 20.5, 0.4
2007, 20.6, 0.5
2008, 20.1, 0.0

These mean temperatures and anomalies are not of any use in judging anyone’s local weather conditions. The daily temperature range might be say 10C and we are talking about an anomaly of say 0.6. The discussion is based on these tiny anomalies. Whether they are reliable, at least in the surface station record, after all the adjustments are made to the historical temperature record is something I am interested in and it is what prompted me to make this post in the first place. Of course I haven’t answered that question and nor could I without much more data which I do not have but it is still interesting to look at the trend evident in any adjustments. I’ll go into this in a little more detail in my upcoming post but you can find a lot of work done on this on WUWT and Climate Audit amongst others (see sidebar of links). Personally I place my reliance on the UAH (& RSS) satellite data as it far fewer problems and adjustments due to say changes in orbital parameters over time are fully documented and available for us to review.

I recommend that if you are interested to go to BOM and get the data yourself and do your own calculations and don’t rely on what I or anyone else tells you. Verify for yourself directly. I would welcome any corrections to any hard data I post if you find anything.

Steve - a quick PS - available on BOM is data for the period you refer to which you could analyse by season to see if the trend you supsect over summer is evident.

Hi,

To confirm what was done here, today I went to http://data.giss.nasa.gov/gistemp/station_data/, and did a search for Eagle Farm, I downloaded both the "raw GHCN data + USHCN corrections" and the "after homogeneity adjustment" data sets. The column labelled metANN in the downloaded data looks like the annual average, and the years 1950 - 2009 are available.

Using MS excel I plotted a linear trend line to both data series. As shown in the original post, the raw data trend line has a negative gradient (a decline) and the "homogenised" trend line shows a positive gradient (a rise).

Isn't it great being able to download raw data and verify other people's interpretations and calculations. I wonder if any of our IPCC friends would be open to such a radical method ;)

I suppose the next logical step is to ask GISS for an explanation of the method used to apply the homogeneity and do they have an opinion as to why this would change a declining trend into a rising trend.

John in Brisbane

Antarctica got cooler too.

http://en.wikipedia.org/wiki/Antarctica_cooling_controversy

Not only is there adjustment, there is selection bias.
http://www.kusi.com/weather/colemanscorner/82827162.html?video=YHI&t=a

This is the biggest bunch of scientifically-challenged, manipulated morons I have ever come across in my many years of observing right wing morons.

Incapable of an objective survey of all the data, they grasp to bits and pieces that fit an agenda they've been sold by an entity with everything to gain: fossil fuel energy.

They're susceptible to being "rallied", but resistant to objective debate.

Clearly we're overdo for a major population scalding meltdown to thin the herd.

So typical of the CAGW supporters such as the last post by anonymous, rather than discuss the topic the far left statist fool reverts to its true colours and states its aim, the killing of men, women and children in order to achieve its green is red aims or as you so eloquently call mankind, “the herd”.

Anonymous when your gonads have finally dropped I suggest you spend a few years actually working for a living before shooting your mouth off and issuing death threats to all and sundry. It might just make a useful citizen of you. If however you are intent on herd thinning why not lead the way and start with yourself. Nothing is more inspirational than a population reductionist having the moral courage to lead by example.

Of course that assumes you are not a coward. You aren’t a coward are you anon.

Watch out for those evil power stations too – they are out to get you.

So once again Wibjorn is correct, this looks nothing like the corresponding IPCC temperature record for Australia. But it’s too soon to tell. Professor Karlen is only showing 3 stations. Three is not a lot of stations, but that’s all of the century-long Australian records we have in the IPCC specified region. OK, we’ve seen the longest stations record, so lets throw more records into the mix. Here’s every station in the UN IPCC specified region which contains temperature records that extend up to the year 2000 no matter when they started, which is 30 stations.

Nice one.. I'm looking forward for more information.