David Cage prompted me to write a reply which on reflection is a nice summary of why academic science got into this mess with global warming so I’m posting it as an article.
He was replying to my comment that “academics … are looking increasingly sheepish and trying to talk about anything but their proven inability to predict the climate.” and said that this was not true as some academics working on signal processing had understood the real nature but had failed to get grants and so moved into engineering  which is how David got to know them.
As someone who started University in Physics but eventually chose to take electronics as well and then eventually worked in the very traditional area of a textile mill, I can remember my own transition from “science” to engineering. And sometimes I really do feel quite schizophrenic, because these are very different cultures and outlooks and both as a scientist in engineering and as a engineer interfacing with academia, I’m found myself on either side of this cultural and philosophical divide.
And to be frank, there just aren’t many (any?) academics who look themselves in the mirror and say “what is wrong with us”? So, the academic view of themselves and how they behave is perhaps one of the least studied areas of research.

My comment

Thanks David, I can remember where I was standing when I realised that if I had personally recorded the global temperature signal on one of my instruments as part of my work and I had been asked “is it correlated with this other rising signal” … I would have had to say “I could not possibly say – there’s far too little signal to draw any real conclusion”.
And then I realised that I had been applying a very different standard to the “big” temperature signal to any other “ordinary” signals. I tried to explain it in my submission to the Climategate inquiry. However, how do you explain something that you really just learn on the job by looking at lots and lots of signals?
But it was interesting seeing my own perception change as I started to view it as a signal and not as “science”. I think the difference is because I had been taught in science to look at signals in an entirely different way to that which I did as an experienced engineer.
After studying it, I think the big difference is that in the real world, real world signals are full of 1/f noise. However, in most areas where science works, the signal is a relationship that is “static”. So, the “signal” is a long term relationships and so almost all noise is higher in frequency like white noise. So, in physics the philosophy was “if you average it enough, the noise will disappear”, whereas in the real world of engineering, if you average a signal often, the signal disappears leaving only the noise.
When you play around with short duration real world signals with multiple frequencies, you learn that averaging can only go so far and eventually it’s a question of judgement. But in science, the experiments will keep going until they have enough data, and/or they only work in areas where averaging is a useful technique.
The other huge cultural difference, is that in “science” the aim is to “understand” something and anyone who doesn’t understand a system is considered to be … morally corrupt would be a good way to put it.
In contrast, in engineering (and it also applies to GP doctors), the aim is to “keep something working”. So those real world scientists called engineers, focus on identifying and fixing problems — and if understanding is not necessary — then they aren’t fixated on trying to understand what doesn’t help fix the problem.
So, engineers tend to look at the world and ask “is it broken” … so they look for symptoms of problems try to assess the validity of available data and look if there are any real and worrying trends.
In contrast, those from a “science” background are fixated on “understanding” the climate. Then once they have even a minuscule understanding of the climate (akin to the Met Office understanding of the weather on a day one year ahead). They use their minuscule knowledge to make massive predictions of WHAT WILL HAPPEN, [… if … their models are correct].
And they justify this along the lines of “there is no one who understands the climate better …. therefore we must have the best predictions of what will happen … therefore the politicians must listen to us and only us”.
So the basic methodology is
In “Science”

PROBLEM
-> (attempt) UNDERSTANDING
-> MODEL
-> PREDICTION
-> APPLY MORAL STANDARD OF “WHAT SHOULD BE DONE”
-> IF MORALLY “REQUIRED”: DEMAND POLITICAL ACTION

In Engineering

PROBLEM
-> LOOK AT SYMPTOMS
-> ASSESS TRENDS AND DANGER THRESHOLD
-> GIVE POLITICIANS THE TOOLS TO MAKE THE DECISION ABOUT WHETHER ACTION IS NEEDED AND WHETHER IT IS COST EFFECTIVE

And those last bit of why action is needed is highly illuminating. In engineering, the engineer will tell the client the technical reasons why action is needed, perhaps give a best prediction of what might happen if action is taken or not, the cost of action … but ultimately, it’s up to the client to decide whether they take action.
In “science”, their works produces a model. That should be the end of it … because then it should be up to others [climate consultants?] to use those models to advise government.
However, when the political “eco” wing of academia saw that governments were not acting as their model suggested they should, they invented organisations to lobby on behalf of “science” to try to force government to act. So, perversely, the supposed “dispassionate” scientists, found themselves working as MORALLY driven political lobbyists. (Which is why I call them “science”.)
In other words, I have some sympathy for the academics who thought they should act to ensure their work that appeared [to them] to be showing a problem was acted on by government. However, by doing so, they became morally biased about their own work, they then corrupted the system to prevent those who did not share their views getting money and so the whole of academic science became corrupted by confirmation bias and society lost the benefit of having a dispassionate group of real SCIENTISTS who could advise us.
Fortunately, a few altruistic engineers chose to give their own time and resources to stop the worst excesses until the climate itself started proving the academic scientists had been wrong.
And to be frank, this is how science really develops. Academics go into new areas. They try to understand them. They are then bold and assertive in stating what they know. And then it is tested against what actually happens. And eventually, the subject matures enough to know how bold and assertive they can be within their given subject area without crossing the boundary into just looking plain stupid.
It’s not that academia is any better than engineering. It’s just that they would like to think they are better – or at least have a better understanding! And there’s nothing wrong with having a group dedicated to understanding, but they’ve got to know their limits.

Commercial realism.

Lastly, I must mention the driving force behind the culture of engineering. An engineer is a scientist working in the real world where you get sued if they are wrong. So, for obvious reasons, engineers tend to be very cautious about their advice.
Academics, on the other hand, USED TO stick to areas of academic interest and ACADEMIC was almost the same as saying “not being commercially liable”.
That ended in climate when they decided to give specific advice insisting on action costing huge amounts of public money.
When academic scientists decided that they would start advising government on how it MUST act on climate, they stopped being “academic”. They stopped being “science” and they started being commercially liable like engineers when their advice is wrong without the culture of caution of engineers.
And now it looks like the advice was very clearly wrong and very clearly they failed to act with due diligence and care and they insisted their advice was acted on, all those “academics” are now personally liable along with their institutions. THAT IS THE LAW – THE LAW DOES NOT GIVE IMMUNITY TO ACADEMICS AND UNIVERSITIES FROM A DUTY OF CARE TO THOSE THEY ADVISE IF THEY TELL OTHERS THEIR ADVICE CAN BE (NOT SHOULD) BE RELIED ON AND THOSE ACTING USE THAT ADVICE AND SUFFER FINANCIALLY.
This is the culture from which sceptics come. We use the same science and data – we just know that in the real world giving bad advice, even if well intentioned, can be catastrophic to both those being advised and those giving the advice.