I’ve just published in the peer-reviewed literature explaining:
“Much of the present research effort by government institutions in Australia, which is focused on monthly and seasonal rainfall forecasts, is limited to the application of general circulation models, in particular, POAMA. However, results so far have been disappointing with medium-term monthly forecasts consistently about equivalent to, or worse than, climatology, Table 4. Nevertheless in June 2013, POAMA was adopted as the system for generating the BOM’s official seasonal forecasts.”
http://www.sciencedirect.com/science/article/pii/S0169809513003141
[Open access until March 26, 2014]
In short, the Bureau could better serve the Australian public by just updating and publishing the long-term average rainfall each month for localities for which it has statistics, rather than pretending it can generate a reliable forecast. And I offer the same advice to the UK Met Office which yet again botched its seasonal weather forecast. The following quotes via Benny Peiser at The Global Warming Policy Foundation.
BUNGLING weather bosses predicted a drier than usual winter, it has emerged. The Met Office’s staggeringly inaccurate forecast was made at the end of November last year – just a month before the record-breaking deluge began. And the agency gave just a one in seven chance the three following months would “fall into the wettest category”. On Nov 21, its experts predicted: “For the December-January-February period as a whole, there is a slight signal for below-average precipitation”. The calamitous estimate emerged as Downing Street dubbed the devastation caused by the floods as “Biblical”. –Tom Newton Dunn, Political Editor, The Sun, 11 February 2014
Contingency planners were advised by the Met Office to expect a dry winter less than four weeks before the heaviest rainfall in 250 years. The official guidance to expect “drier than normal” conditions was issued in mid November, just weeks before the onset of the wettest new year period on record. The Met Office report, which claims to use “cutting edge science” to “help contingency planners prepare for and respond to emergencies”, predicted a pattern of high pressure weather systems would lead to dry conditions. It added: “The weakening of the prevailing westerly flow means that the normally wetter western or northwestern parts of the country may see a significant reduction in precipitation compared to average, while the east or southeast may be closer to average.” –ClickGreen News, 11 February 2014
I don’t blame the climate scientists for not knowing. Climate and weather is quite extraordinarily complex and this is a very new form of science. All I blame them for is pretending they know when they don’t. What we ought to focus on is what we’re going to do. I think this is a wake-up call. We need to abandon this crazy and costly policy of spending untold millions on littering the countryside with useless wind turbines and solar panels, and moving from a sensible energy policy of having cheap and reliable forms of energy to a policy of having unreliable and costly energy. Give up that. What we want to focus on – it’s very important – is making sure this country is really resilient and robust to whatever nature throws at us, whether there’s a climate element or not. Flood defences, sea defences – that’s what we want to focus on. –Nigel Lawson, BBC Radio 4, 13 February 2014
Met Office Forecasts – A Reality Check
The Global Warming Policy Foundation, 10 February 2014
The Met Office’s temperature forecasts issued in 13 out of the last 14 years have been too warm. None of the forecasts issued ended up too cold. That makes the errors systemic and significant.
2014: Met Office global forecasts too warm in 13 of last 14 years – BBC Weather, 27 January 2014: So far this century, of 14 yearly headline predictions made by the Met Office Hadley Centre, 13 have been too warm.
2013: Met Office winter forecast: For the December-January-February period as a whole there is a slight signal for below-average precipitation.
Reality Check: Met Office confirms England is suffering wettest period for 250 years
2013: Met Office Spring Forecast: Feb-March Above-Average UK Temps More Likely – Met Office, 20 December 2012: For February and March the range of possible outcomes is also very broad, although above-average UK-mean temperatures become more likely.
Reality Check: Met Office confirms coldest March in more than 50 years – Press Association, 29 March 2013: This March is the coldest in the UK since 1962, forecasters have confirmed. After weeks of speculation about whether this miserable March would top the list, the Met Office has announced it is the coldest in 51 years according to provisional statistic.
2012: Met Office Spring Forecast: “The forecast for average UK rainfall slightly favours drier-than-average conditions for April-May-June as a whole, and also slightly favours April being the driest of the 3 months. With this forecast, the water resources situation in southern, eastern and central England is likely to deteriorate further during the April-May-June period… This forecast is based on information from observations, several numerical models and expert judgement.”
Reality Check: Wettest April for 100 years – April: 2012 had wettest April for 100 years, Met Office says “It has been the wettest April in the UK for over 100 years, with some areas seeing three times their usual average, figures from the Met Office show. Some 121.8mm of rain has fallen, beating the previous record of 120.3mm which was set in 2000.”
25 June: Spring is wettest in Britain for 250 years – England and Wales are on course for the wettest late spring and early summer for 250 years, experts said yesterday. June has just seen its fourth washout weekend and yet more downpours are forecast. Now it is feared combined rainfall for April, May and June will break the record of 13.2in (336mm) set in 1782 and be the highest since records began in 1766.
2010: Met Office Winter Forecast: Winter To Be Mild Predicts Met Office – Daily Express, 28 October 2010: IT’S a prediction that means this may be time to dig out the snow chains and thermal underwear. The Met Office, using data generated by a £33million supercomputer, claims Britain can stop worrying about a big freeze this year because we could be in for a milder winter than in past years… The new figures, which show a 60 per cent to 80 per cent chance of warmer-than-average temperatures this winter, were ridiculed last night by independent forecasters. The latest data comes in the form of a December to February temperature map on the Met Office’s website.
Reality Check: December 2010 “Almost Certain” To Be Coldest Since Records Began – The Independent, 18 December 2010: December 2010 is “almost certain” to be the coldest since records began in 1910, according to the Met Office.
2009: Met Office Winter Forecast: Trend To Milder Winters To Continue, Snow And Frost Becoming Less Of A Feature – Met Office, 25 February 2009: Peter Stott, Climate Scientist at the Met Office, said: “Despite the cold winter this year, the trend to milder and wetter winters is expected to continue, with snow and frost becoming less of a feature in the future. “The famously cold winter of 1962/63 is now expected to occur about once every 1,000 years or more, compared with approximately every 100 to 200 years before 1850.”
Reality Check: Winter Of 2009/10 Coldest Winter For Over 30 Years – Met Office, 1 March 2010: Provisional figures from the Met Office show that the UK winter has been the coldest since 1978/79. The mean UK temperature was 1.5 °C, the lowest since 1978/79 when it was 1.2 °C.
2008: Met Office Winter Forecast: Trend of Mild Winters Continues – Met Office, 25 September 2008: The Met Office forecast for the coming winter suggests it is, once again, likely to be milder than average. It is also likely that the coming winter will be drier than last year.
Reality Check: Winter of 2008/09 Coldest Winter For A Decade – Met Office, March 2009: Mean temperatures over the UK were 1.1 °C below the 1971-2000 average during December, 0.5 °C below average during January and 0.2 °C above average during February. The UK mean temperature for the winter was 3.2 °C, which is 0.5 °C below average, making it the coldest winter since 1996/97 (also 3.2 °C).
It has emerged that the Environment Agency rejected calls to dredge the flood-hit lower reaches of the Thames because of the presence of the endangered mollusc. In a 2010 report, seen by the Mail, they ruled out dredging between Datchet and Staines because the river bed was home to the vulnerable creatures. And even though a public consultation indicated support for de-silting work, the quango said it would be ‘environmentally unacceptable’ due to the ‘high impact on aquatic species’. –Daniel Martin, Daily Mail, 13 February 2014
******
The photograph of the duck pond was taken in May 2013 in Birmingham.
Jennifer Marohasy BSc PhD is a critical thinker with expertise in the scientific method.
Yes I did notice the reporting that the UK Met office’s Winter/Spring rainfall forecast was a similar sort of ‘not wrong’ as BoM’s.
Jonova summarised some of it here:
http://joannenova.com.au/2014/02/uk-met-office-predicts-15-chance-of-heavy-rain-britian-gets-biblical-floods/
Jennifer as an ex-farm manager I can tell you that rainfall averages mean squat,they are only statistics that show very little.
Gos,
That is your assessment.
But the BOM comes out with something less reliable, dresses it up as an official seasonal forecast from a “state of the art forecasting system” and… what do you think?
Plans are made on the basis of BOM official forecasts, but everything so far published would suggest they are rubbish, they are just official snake oil.
Gos,
I agree that yearly statistical averages mean very little on farm. It is entirely possible to have better than average production in lower than average rainfall years depending on WHEN or in WHICH SEASON the precipitation occurs (as I’m sure you must know if you are an ex-farm manager)
BoM reports that 2012 was 30% wetter than average but as far as agricultural production is concerned it is a rather meaningless statistic.
2012 saw massive rainfall and flooding through much of the Eastern Seaboard in early Autumn 2012. . .which heavily influenced that statistic.
What it doesn’t show is that the remainder of 2012 was much drier than average and dry land winter production did not finish well… and yields were down.
I would encourage Jen and Abbot ( and of course anyone else who is researching in this space) to keep working on improving skill in SEASONAL (as opposed to yearly) rainfall forecasting as improved skill would most definitely be a useful tool for those of us who work with the vagaries of weather and variable seasons in Australia 24/7/365.
BoM forecast an 80% probability of ‘wetter than average’ Winter/Spring for 2013 in Eastern Australia using POAMA.
I have been lectured here and other places several times that even though a DROUGHT eventuated for great swathes of inland QLD and NSW . . . (which you and I would most likely agree is not the same as ‘drier than average’). . . that still means that BoM and POAMA were right (although I think what they’re really saying is they were ‘not wrong’).
It does appear something similar has occurred in the UK with their GCM?. . .except the forecast and what eventuated are reversed to forecast ‘drier’: eventuated ‘massive flooding’.
Jen also points out that Govt resource management plans are made partly on the basis of these GCM forecasts. That is indeed a trifle concerning. . .especially re water storage and water resource management .
They predict Global Warming, but can’t predict the weather that comes with Global Warming.
Debbie
Thanks for the reply,yes it would be a much better idea to have seasonal forecasts than annual but when it comes to farming you really need to be able to plan it out yourself,to know how your own seasons happen ie frosts in autumn will mean a drier winter,prolonged frost in winter will point to drought,all these types of things only come from experience,they can’t be quantified.
Rainfall is measured by gauges that are spread widely apart,1k either side of a rain gauge and the falls can be drastically different.
What we have with the BoM is an organisation that is more concerned with the political aspect of its charter than what it was first designed for,they have concerned themselves with this thing called climate(the word climate like the word environment is abstract,it can mean anything) instead of concentrating on the weather,we need to know what the weather is going to do 3 months from now not be worrying about the faint possibility of what it will be doing 100 years from now.
The other BIG problem that is happening is that environmental ideology is the driving force for the making of water policy,which isn’t good for those humans who are reliant on it for their livelihoods.
Jennifer,
Congratulations on your publication.
As an American, who has never been to Australia, and who has a ‘purple thumb’ to boot, here’s my 2 cents worth. Putting too much faith in skilled seasonal forecasts is like Russian Roulette, applied to your farm. Even with the best of intentions, you can still get clobbered.
For farmers, seasonal forecasts based upon POAMA are a different ball of wax. Taken at face value, they’re more like Cuban Roulette, in which there are live rounds in 5 out of 6 chambers of the revolver! (Or should I say 13 out of 14 chambers?) Nevertheless there’s still a way to game the system in your favor.
Look at the DIRECTION of change, temperature-wise and rainfall-wise, in the seasonal forecasts. Then assume that the opposite is true! That information is better than no information at all. If your informed guess is lucky, then your bumper crops will fetch a higher price, because your competitors will naively accept whatever drought myths your goobermint tells them, and will ‘play it safe’, by leaving their fields fallow for that year.
After you gain more self-confidence in DIY seasonal ‘forecasting’, take a gander at the Marohasy-Abbot seasonal forecasts, as well as their track record.
Gos,
Yes. . .I agree. . .we still need to plan it out ourselves from the best available information and much of that best available information is indeed our own generational knowledge of pre-seasonal behaviour (eg frosts and wind direction). It is indeed true that rainfall data can be suspect because rainfall is not uniform and to add to your point, one of my properties, only about 4 km from the home property can record quite different figures from the same rain event. Our closest ‘official’ rainfall data is approximately 15 kilometres away (at Yanco), and very close to the ‘bidgee and their rainfall figures are often dramatically different to ours.
I also agree that it appears that BoM has lost its way and the organisation is not focusing on presenting information that is USEFUL for the public that must take risks with the weather/climate all the time.
I don’t think it’s because they’re bad people. . .I think it’s more likely that something has gone amiss with the organisational goals and purposes.
But definitely congratulations to Jen and Abbot who are choosing to focus on developing skill in seasonal forecasting rather than focusing on what the weather/climate might or might not be doing in 100 years time.
Congratulations Jennifer, on this new paper, farmers and fishermen are much better at forecasting it seems. On one occasion we forced BoM to declare a cyclone against their will when there wasn’t supposed to one prior to the New Year some years ago. We nearly lost 2 fishing vessels then operating out of Broome.
““Much of the present research effort by government institutions in Australia, which is focused on monthly and seasonal rainfall forecasts, is limited to the application of general circulation models, in particular, POAMA.” Just BoM ! But not others ….
“However, results so far have been disappointing with medium-term monthly forecasts consistently about equivalent to, or worse than, climatology,” – on 3 stations – I don’t think so !!! ROFL !!
“But definitely congratulations to Jen and Abbot who are choosing to focus on developing skill in seasonal forecasting rather than focusing on what the weather/climate might or might not be doing in 100 years time.” MISDIRECTION and irrelevant twaddle and Debs how do you know?? You haven’t even read the paper.
“Putting too much faith in skilled seasonal forecasts is like Russian Roulette, applied to your farm. Even with the best of intentions, you can still get clobbered.’ YES ! if you don’t get that – don’t play !
So as nobody has actually read the paper – this is what you’re saying yes to – it’s only Qld – it’s only 3 sites – rest of Australia who knows?
2 of the 3 sites have POAMA 2 the latest version – which only has one statistic. 2 of the 3 sites have POAMA 1.5 – out of date.
So this all encompassing take all claim, unchallenged, and not thought about for even 10 seconds by the silly persons here, is based on the r value for two sites in Qld.
And no web site to follow progress ….
And only one lead period. No other leads or periods.
Current drought – Qld graziers would like to have known it was coming. And what summer rainfall would have been before June. So do we have a track record on the current drought – nope !
CALL ME SCEPTICAL ! AND how about the rest of you get minimally serious. Debbie you’re prattling on,
Luke,
We get it. . .and we don’t play. . .read the comments.
HOWEVER! We would like to see some FOCUS and IMPROVED SKILL in regional seasonal forecasting.
That could be very useful don’t you think?
QLD is as good as anywhere else to start.
It’s an encouraging START Luke. . .your criticisms re number of sites and presentation are not relevant.
GCMs are not showing any real skill in this space either despite claims otherwise.
Here is Obama’s science?????? advisor Holdren yesterday explaining that the California drought is caused by AGW and just about all weather is now caused by CC. Work that out if you can?
http://wattsupwiththat.com/2014/02/14/quote-of-the-week-21/#more-103236
I used to think that Flannery was unique but he looks to be very run of the mill among all the other extremists.
You must remember this ..
http://climateaudit.org/2013/09/30/ipcc-disappears-the-discrepancy/
The IPCC graph that showed discrepancy between the models and observations.
The initial graph where the observations are outside the envelope of projections
and then the, er, reconstructed scribbly graph. Say, the classics teach us the gods
are capricious, and here, the climate models likewise.
Don’t waffle on Debbie. We’d all like to have better shopping experiences and free fairy floss too. You’re prattling on about material you have no idea about and are convinced Jen has cracked the golden egg with TWO stations.
WHAT A BUNCH OF NON-SCEPTICS ! Come one Debs – you’d believe a sample size of two would you – did you come down in the last shower? It’s a try-on.
Neville take your off-topic trash elsewhere.
Luke
You know every well that our first paper looked at 17 sites. This one at three in detail.
Regardless, the levels of skill from the POAMA forecasts would indicate that the Bureau is misleading the Australian public with its seasonal fore castings. Suggesting some level of expertise when none exists.
Luke,
Clearly you are the one not getting it.
WE don’t play but. . . as Jen points out above . . .WHO or WHAT is using the POAMA seasonal forecasting?
BTW. . .please stop with those straw man thingos. . .I am not convinced that Jen has cracked a golden egg. . .but I am more than happy to CONGRATULATE & ENCOURAGE anyone who is willing to:
a) point out that skill is poor in this space and
b) have a genuine go at improving seasonal forecasting
Luke is clearly suffering from short term memory loss…
We benchmarked against POAMA for 17 sites in the following paper…
see Abbot J., and J. Marohasy, 2012. Application of artificial neural networks to rainfall forecasting in Queensland, Australia. Advances in Atmospheric Sciences, Volume 29, Number 4, Pages 717-730. doi: 10.1007/s00376-012-1259-9 .
We had a lower RMSE for 16 of the 17 sites and that using a very basic ANN model developed in about 7 months run on a very ordinary computer versus the Bureau’s supercomputers and how many decades of sunk tax payer dollars.
Hi Jennifer,
Congratulations on your ANN papers. Any improvement to seasonal forecasting would be appreciated. I’m not a farmer, but I am subsisting on tank water and using dam water for the garden, and the ‘rainy’ season in Gympie has been very dry.
It isn’t just the UK and BoM that suck at seasonal forecasting, Environment Canada has an equally dismal record. Their one saving grace was that they published on-line analyses of their poor record, so you knew not to trust it – well, at least if you were willing to make the effort to find and try to understand the analyses (they didn’t trumpet them). Unfortunately, EnvCan weekly forecasts are also completely unreliable after day 2, and not very reliable for day 2. They also publish daily ‘normal’ temperatures – an excellent example of why means are useless in a highly variable system. ‘Normal’ temperatures almost never occur. BoM seems to do much better with their weekly temperature forecasts, but not so well with the rainfall (even though they keep it vague, e.g. ‘A shower or two’).
I sometimes suspect that weather bureaus are now dominated by game-players who never look up from their models and out the window. One would think that experienced meteorologists who had been studying weather patterns for a while would be able to do better, but if they are wedded to their models and the models are crap …
Cheers,
Dave
Fair suck of the sauce bottle. On an out of date POAMA version which BoM did not go into production with. Using a downscale method they don’t use. Call me sceptical. Anyway did your system predict this drought.
Luke,
Nothing wrong with maintaining healthy scepticism. . . but don’t forget it works both ways.
The current POAMA did not predict this drought either. . .just to help with that apparent short term memory loss.
Abbot J., and J. Marohasy, 2011 ??
Bazza, That was a typo should be 2012. I keep thinking we have been working on this stuff for longer than we have. We only started looking at the application of ANN to medium-term rainfall forecasting after the Brisbane floods (so after January 2011). We have other jobs, so our efforts so far have been part-time.
Luke, Everything published would suggest no measurable improvement in skill with each new version of POAMA. OUr most recent paper, table 4, includes reference to the latest POAMA.
We are keen to benchmark against the latest output from POAMA, but have been told we aren’t getting any more output as time series data for direct comparison with output from our ANN model. To get the first lot I had to fly to Melbourne and beg.
As regards the current drought, we are still refining our method of forecasting. We don’t presume to be able to give a particularly skilled forecast yet. But we do know that we can give a much better forecast than POAMA.
We believe we could make significant further improvements to our method of forecasting if we could employee an electrical engineer with expertise in signal processing.
We have applied for ARC and other sources of funding, but have been told to piss-off. There are many gate-keepers within the climate science community all committed to GCMs (General Circulation Models) and AGW (Anthropogenic Global Warming). And indeed their careers dependent on them, never mind the long suffering farmers, miners (who would like to be able to schedule) and the list goes on…
Jen, I have the answer to your wish. “We believe we could make significant further improvements to our method of forecasting if we could employee an electrical engineer with expertise in signal processing.” Bob FJ with his moving averages!
Really, snark from luke and bazza?! Bazza’s the betting man; what about a wager; Jennifer’s prediction for a month ahead vs the BOM?
Laughable Cohenite on a single throw of the dice. I have asked Jen for the track record for Queensland for the last few years. We don’t have it. Sounds simple enough doesn’t it? Show us….
If the system is working – let’s run it and see !
Too much bluff here Cohers and not enough serious work. A really big anti-GCM spruik based on frankly bugger all. It’s sceptic show pony stuff like without a much larger case. POAMA tries to many more things than forecast a few Queensland rainfall stations for one lag period for one period ahead. There’s this thing called the rest of the nation and the surrounding oceans and how the whole ocean/atmosphere behaves. Jen has compared one thing in a 100 things you could look at.
So very very very simple stuff Cohers –
(1) show us the last 3 years track record up to now for Qld.
(2) show us ANN vs POAMA since May 2012 when they released it for action
(3) AND can we be assured that Jen is comparing with what BoM forecast on with POAMA. Now let’s be precise here to avoid tricks – not whether the GCM produces fields as an off-shoot or not – but exactly how BoM produce the Qld rainfall forecast for POAMA. Do they really use rainfall downscaled out the GCM? Otherwise I simply don’t know if it’s ANN vs Jen’s home brew POAMA index. You see I don’t see a forecast product for a month ahead from them? Why is that?
Bazza and I love mavericks and outside horses. But you have to have some form. In my experience the best ones are usually derived from humble beginning whereby you’d don’t start out claiming your perceived competition is crap.
Now as a fair minded chappy Cohenite are the above requests sound too onerous? No traps here – I’m interested – but show me some real ANN in action?
If it doesn’t come – I think that tells me what I need to know.
Happy to benchmark against POAMA since May 2012. But we need the POAMA forecast as time series output, not as a coloured chart.
I’m stopping/ending this thread here. Thanks for your contributions.