THERE are at least 3 practical ways in which medium-term rainfall forecasts for Queensland can be improved:
1. Through the use of sophisticated statistical modelling techniques, in particular artificial neural networks to mine historical data for recurrent patterns,
2. The incorporation of relevant climate indices, including the Inter-decadal Pacific Oscillation, and also
3. The presentation of forecasts as charts showing total forecast rainfall, rather than as coloured maps with assigned probabilities.
And there is a fourth suggestion, also detailed in the recent paper by myself and John Abbot that has just been made available for free download by Elsevier, publishers of the journal Atmospheric Research.
The link is here: http://elsarticle.com/1ej97n3
Free access will only be available until March 26, 2014. Enjoy!
Jennifer Marohasy BSc PhD is a critical thinker with expertise in the scientific method.
Practical improvements?
🙂 🙂
Now there’s a novel idea.
We expect reruns this time of year. So:
Simple forecast of Cairns Aero monthly rainfall for the 12 months using only the monthly mean gives an R of 0.91 for 2008. Who needs a neural net .So this patch up rerun is a travesty.
I was still getting a laugh from the first paper which used Sydney as a proxy for Qld temperatures. As I recall Qld split from NSW in 1859. But you had to go to Sydney to find a site without a trend.
I’m working on a next paper to a deadline and want to insert shading into an excel chart… so that there is shading between an upper and lower limit, presented as two data series.
Can someone tell me how to do this or do it for me if I send them the chart?
I’m in a rush.
jennifermarohasy at gmail.com
Not easy, this is probably the best starting point
http://peltiertech.com/WordPress/excel-charts-with-horizontal-bands/
‘———————————————–
Shading the full sheet on the other hand is easy.
Select the cells you want to shade/right click inside selected cell/ choose format cells/ selct ‘Fill’ tab/
select fill effects/select shading style you want to use.
If you don’t like the style click undo (on top left next to ‘Save’ icon and try a different style.
Forget it Jen
Right click on data point in the series you want to format and select format data series
select shadow/presets inside top/colour/distance Bingo
Don’t ask Phil Jones about excel.
Say, good to develop a forecasting process with less error,
nevertheless, prediction is a …. well you know.
Sorry,
It’s probably a bit off topic:
http://www.independent.co.uk/news/science/cooler-pacific-is-causing-global-warming-pause-9117816.html
“An international team of researchers has calculated that the stronger trade winds blowing from South America to Australia have had the net effect of cooling surface temperatures by a global average of between 0.1C and 0.2C, which would be enough to account for the apparent hiatus in global average temperatures over the past 15 years.
The scientists warn however that the cooling capacity of the Pacific Ocean is not expected to continue much beyond 2020, when global surface temperatures are expected to start rising again rapidly as a result of increasing concentrations of man-made carbon dioxide.”
So they’ve given themselves at least another 6 years?
I do think that Jen & Abbot (and of course all who are working in this space) are on a more relevant, more practical and useful track by focusing on improving skill in regional seasonal forecasting.
Johnathan
It’s the stacked area series that looks closest to what I want to achieve (at the link your provided here http://peltiertech.com/WordPress/excel-charts-with-horizontal-bands/ )
I’ve sent MA the chart.
Debbie, thanks.
Beth, I’ve been reading ‘Einstein: His life and universe’ based on some of your comments last year about the history of science. He doesn’t seem to fit the Kuhnian model particularly well.
Thx fer comment, Jennifer. Of course I recognise that I am jest a serf and you
a scientist in the field, but to me, some of the issues argued here by Weinberg,
are how I regard ideas of religious conversion in science rather than attempts
at problem solving, and also issues of incommensurability, but I genuinly recognise
my mega-sised limitations. How could I not. )
Beth.
http://www.physics.utah.edu/~detar/phys4910/readings/fundamentals/weinberg.html
Downloaded the paper, looks good but have to read it more thoroughly. Do not know anything about the graphing in Excel.
The graphing seems to be too much aimed for accountants instead of scientific. Maybe someone could point to free or low cost graphing packages.
Sigmaplot and Coplot are often mentioned for scientific graphics but expensive.
There is R if you want to master it – sophisticated and free – http://www.sr.bham.ac.uk/~ajrs/R/r-gallery.html but not simple
Wiki-Pravda lists the following free software http://en.wikipedia.org/wiki/Category:Free_plotting_software
I don’t use this sort of thing, but there are some good free cross-platform products that are free, such as R, mentioned by Luke, and Veusz. Trouble is, a fair bit of learning.
Then there’s these guys, maybe fewer hoops, trial period, modest price, 50% academic discount, and a FREE student version which looks okay.
http://magicplot.com/magicplot.php
Cross platform, Linux included. The forum is quite active and current, so chance of free help.
Anyway, good luck, Jen! Won’t know if your plan works till it works, but take a swing.
Here’s the recipe – scroll down to “Fill Between Two XY Series”
http://peltiertech.com/WordPress/fill-under-between-series-in-excel-chart/
Excel whizz kids may wish to help Jen. I’ve struck a few issues and a dab hand may get there quicker.
Hmmm,M. England from UNSW has a paper out about variations on the strength of trade winds to explain away “the hiatus”.No matter what causes wind to blow- England reckons he can pick up anything and rebadge it- this case La Niña/El Niño to explain away the obvious:that CO2 is not the main driver of the worlds’climate.
http://www.abc.net.au/news/2014-02-10/scientists-find-explanation-for-global-warming-pause/5248456
Researcher Dr Matthew England says that when the trade winds return to, er, ‘normal,’
then ‘we’d expect global warming to kick back in and start to rise … We reject the
arguments from sceptics that the slowdown suggests global warming is not as bad
as we thought … We want the community to have confidence in the climate models’.
Hey, what do they say … ‘there’ll always be an Encland (or jaycee or ?’ )
What is it with these new age computer modelling scientists? Beholden to their political masters. Like the Monty Python sketch.”this new learning amazes me Sir Bedevere;tell me again how sheeps bladder may be employed to prevent earthquakes.
Fair enough Jen,
It was the best plan to delete those comments. . .(mine included).
You’re trying for practical improvements. . .best to stay in that space.
OK Everyone,
It’s be nice to Luke for a week, week!
I sent him my chart just now, and he sent it back fixed! Yes, he got the grey shading between the two lines defining the upper and lower limits of an envelop/area.
He can be useful.
I’ll just now have to add Luke Walker to the acknowledgements. 🙂
PS Beth, I edited Luke out of your most recent comment. Now go write him a poem.
“mine historical data for recurrent patterns”
In a highly variable climate such as Australia’s how many years of records do we need to have a good idea of what is ‘normal’?
100? 400? Any idea?
Yea so-called Beth – be nice – offers hand for kissing and looks away at the sky.
Halcyon
Yes. A major limitations is the length of the input data. In this paper in Atmospheric Science we go into some detail discussing this very issue see pages 169 and 170, that you can access from the above link.
There is generally about 120 years of rainfall record. Monthly values for indices like the IPO go back to 1871. We use 85% of the data to train the model, and then the remainder for testing.
Good for you Luke.
How now jennifer, censorship? 🙂
A pome fer Luke? (Very off the cuff.)
Lucky Luke Skywalker, lives
in a great southern land where
he’s free ter take a stand and
usually does most vocally. and
sometimes even acts heroically
like helpin’ a scientist in distress
when her excel sheets are
in a mess. Bravo, Luke Skywalker!
Awww shucks …
I was fearing a limerick
There once was an alarmist called Luke
Who used to bedwet and rebuke
He liked to opine when Neville did whine
And call him a tropical fruit
To be useful is the highest state we humans can aim for. Well done luke.
On England’s wind problem, a light hearted response here:
http://theclimatescepticsparty.blogspot.com.au/2014/02/england-passes-wind-and-discovers-heat.html
Just filing this here…
CQ Uni Media Release
Best rainfall forecasts come from computers that can learn
The best monthly rainfall forecasts come from computer programs able to represent complex relationships between climate data while acquiring knowledge from many examples over time for better pattern detection.
That’s according to CQUniversity researchers Dr John Abbot and Jennifer Marohasy, who have considered relationships between lagged values for temperature, atmospheric pressure and rainfall as well as climate data.
They have recently published their findings in the Atmospheric Research journal published by Elsevier.
The authors have compared results from their artificial neural networks (ANNs) analysis of the Inter-decadal Pacific Oscillation index against government-based seasonal rainfall-forecasting programs. This index has never before been used for official seasonal forecasts for Queensland.
“Forecasts using the ANN for sites in three geographically distinct regions within Queensland are shown to be superior … compared to forecasts from the Predictive Ocean Atmosphere Model for Australia (POAMA), which is the general circulation model used to produce the official season rainfall forecasts,” say Abbot and Marohasy.
They say a major limitation of government forecasts is that they provide no information about the magnitude of the expected deviation from the median rainfall value within the defined forecast period.
They discuss the Bureau of Meteorology’s acknowledged gap in rainfall prediction capability ‘beyond one week and shorter than a season’.
Abbot and Marohasy say that for purposes including management of water infrastructure or scheduling mine operations, the distribution of rainfall within the three-month period is more important than an averaged seasonal value.
They say ANNs have been investigated for rainfall forecasting in many parts of the world but they have rarely been applied in Australia and are not used to generate official forecasts.
The full article will be available via http://elsarticle.com/1ej97n3 until March 26.
[ Abbot and Marohasy say that for purposes including management of water infrastructure or scheduling mine operations, the distribution of rainfall within the three-month period is more important than an averaged seasonal value.]
THIS…is the “operative phrase”…..
praytell fellow researchers….If I wanted to open a mine in a “water sensitive” area…what would your “forcasts” show?……NO, NO!…don’t tell me!
Jaycee,
What is your definition of a “water sensitive” area?
High rainfall areas? Groundwater? Surface Water? Waterways? Townships? Critical Industry Clusters? S&D supplies? Low rainfall areas? Industrial zones? Irrigation areas? Dry land cropping areas? National parks? NSW? QLD? Vic? SA? WA? NT? TAS? ACT? Mining development? Coastline? Inland?
MIA. for a start?
C’mon, Debbie…tell the good folk about water-table problems in the MIA.
I have no doubt that the MIA is “water sensitive”.
“1838 and 1839 saw the champion drought of the century. Stock were all but exterminated. The Murrumbidgee is a great river, 150ft. wide, 60ft. deep, and overflows its banks, like the Nile, when the head snows melt, for five miles on each side to a depth of 3ft. This gives a volume of water equal to a river of 1450 ft. wide and 120 ft. deep, and besides this it fills a group of lakes each from seven to eighteen miles in diameter. Yet this great river dried up so thoroughly in 1839 that the fish died and putrefied at the bottom of it.”
Brisbane Courier 1839 (Also reported in Sydney Gazette in 1839)
Yes, they were actually able to stage horse races in the bed of Murrumbidgee. But this is just one episode in that main story called Drought in Oz. It’s the main story that stopped briefly in the mid 1970s…then took up where it left off. Through La Ninas and neutral years and El Ninos. Regardless of IOD, PDO etc. Not that there is no value in all these indicators – quite the contrary! – but there are mountains of knowledge we don’t have on the greatest of all subjects for Oz: drought. (I suppose the next big story is Flood in Australia. Oxley gave up on the interior in 1818 because he thought it was one huge marsh. Eight years late, Sturt found a huge dustbowl.)
To ignore the dominance of drought in Australia is to ignore Australia, rainfall, topography, geography, climate, history…plus whatever I just left out. To frame the latest drought as somehow new or different in order to fit it to a political purpose or manipulation would be obscene if it wasn’t so obviously lame.
Yes, Deb, you’re in might water sensitive country. It’s called Australia, and it’s been that way forever.
(My apologies for broaching the naughty topic of Stuff Which Actually Happened.)
A typo. The Brisbane Courier correspondence was dated 1889, and the guy had much post-1839 perspective. He records, for example, how that drought was broken by the 1841 flood, when the Bremer rose 70 feet. He remembers the world’s biggest known bushfire in 1851 before the floods of the 1850s, especially the amazing but awful Gundagai flood the year after Black Tuesday. He also recalls that flood in 1806 where the Hawkesbury somehow managed to rise 101 feet at Windsor (huh?).
And so it goes on. I think there is much we can do to make things better, and the first step will be to ignore those who choose to ignore what has happened in the past. Ignorance can only breed ignorance, however much you dress it up with numbers and stats. It’s worth remembering that Australia’s longest drought, starting around 1958, persisted through a period deemed to have been mostly, or more frequently, PDO neg. This does not mean there is no value in PDO or Mantua’s work, and the IOD explains a few things about the Long Drought. Now that we have some clues in hand, wouldn’t this be a good time to enquire more and project less? If it takes ten years for someone to come up with just a slightly more reliable seasonal outlook for rain, that would be ten years well spent. Gawd, look what we’ve got now.
Nay-chur is a great teacher, past weather events transmitteded down the ages via
the historical record…. Tony Brown on CET does a great service regardin’ the historical
*record.*To echo the words of a well known poster and wit at Climate Etc … ‘Ignore
the millenial at your perennial.’ H/t kim.
Jaycee,
No argument from me that the MIA is a ‘water sensitive’ area. . .like duh!
However. . that does not answer my question.
Yes Robert & Beth. . .ignore history at your peril.
I also second Robert that even if it takes ten years to come up with better seasonal outlook for rainfall. . .we would all be better off.
Go for it Jen (and whomever else is prepared to take this one on)
Pretty sure those in England would agree atm as well.
Just filing these regional newspaper article here…
Morning Bulletin, Rockhampton QLD by Ashleigh Truscott 14 Feb 2014
General News – page 8 – 338 words – ID 237076348 – Photo: No – Type: News Item – Size: 183.67cm2
DR John Abbot doesn’t claim to be a brainiac, but he knows of a computer that is one.
The CQUniversity researcher and Dr Jennifer Marohasy have been looking into t technology -. that can 5 accurately predict a specific amount of rainfall.
Dr Abbot (pictured) said neural network technology simulates what goes on inside our brain so it can learn characteristics of past data.
The research will be published in the journal Advances in Atmospheric Science.
It comes as Russian, German and US scientists forecast high chances of an El Nino weather event later this year.
The revolutionary technology would greatly help the mining and agriculture industries, as both can be badly affected by heavy rainfall.
“In comparison to the Bureau of Meteorology, this new technology is able to determine a specific amount of rainfall, whereas BOM just says that rainfall might be above or below average,” Dr Abbot.
“We aren’t saying we can predict what will happen tomorrow but based on complex patterns from past data we can predict what might happen over the next few months.
“We aren~t claiming to forecast the weather and we can’t compete with satellites, but we can perform pattern analysis to predict rainfall.
Dr Abbot started using the technology 10 years ago for share market trading, and was inspired to get involved with rainfall forecasting after experiencing flooding himself.
“Perhaps one of the things that contributed to the flooding was people didn’t get accurate forecasts,” he said.
WHAT IS EL NINO? It isaband of anomalously warm ocean water temperatures that occur periodically It causes extreme weather, such as floods and droughts It is Spanish for “the boy” Peruvian fishermen originally used the term to describe the appearance, around Christmas, of a warm ocean current off the South American coast. It is now the commonly accepted term to describe the warming of the central and eastern tropical Pacific Ocean. Every three to seven years, an El Nino event may last for many months During the past 40 years, 10 of these major El Nino events have been recorded
****
It’s in the patterns
Port Douglas & Mossman Gazette, Port Douglas QLD by None 13 Feb 2014
General News – page 10 – 187 words – ID 236890141 – Photo: No – Type: News Item – Size: 105.88cm2
THE best monthly rainfall forecasts come from computer programs able to represent complex relationships between climate data while acquiring knowledge from many examples over time for better pattern detection.
That’s according to CQUniversity researchers Dr John Abbot (pictured left) and Jennifer Marohasy (pictured right), who have considered relationships between lagged values for temperature, atmospheric pressure and rainfall as well as climate data.
They have published their findings in the Atmospheric Research journal.
The authors have compared results from their artificial neural networks (ANNs) analysis of the Inter-decadal Pacific Oscillation index against government-based seasonal rainfallforecasting programs. This index has never before been used for official seasonal forecasts for Queensland.
Forecasts using the ANN for sites in three geographically distinct regions within Queensland are shown to be superior … compared to forecasts from the Predictive Ocean Atmosphere Model for Australia (POAMA), which is the general circulation model used to produce the official season rainfall forecasts,’’ say Abbot and Marohasy.
They say a major limitation of government forecasts is they provide no information about the magnitude of the expected deviation from the median rainfall value within the defined forecast period.
****
Superior forecast
North West Star, Mount Isa QLD by Jasmine Barber 13 Feb 2014
General News – page 3 – 231 words – ID 236875221 – Photo: No – Type: News Item – Size: 147.30cm2
NEW weather prediction technology could result in more accurate rainfall estimates in remote areas of the country.
With television news reports often giving false hope to Mount Isa residents and predicting rain in the city, weather research experts have said new technology could give more accurate predictions.
John Abbot and Jennifer Marohasy, of CQ University in Rockhampton, said the best monthly rainfall forecasts come from computer programs able to represent complex relationships between climate data while acquiring knowledge from many examples over time for better pattern detection.
They have recently compared weather prediction results from their artificial neural networks with government-based seasonal rainfall-forecasting programs and found the results to be superior.
Mount Isa Mayor Tony McGrady said improved weather prediction technology had been ignored for too long, and was a vital source of information for rural and remote parts of the country.
“It is so important that we have accurate weather – reporting, and I don’t like drawing distinctions between the city and the bush, but unless we get accurate predictions in the bush it places us a difficult situation,” he said.
“Good on them for doing this, anything at all that will enhance and improve the weather prediction technology for regional Queensland is most welcome.” Cr McGrady said more accurate weather predictions would have enabled the council to better prepare for the current water management plans throughout the city.
As sceptics are wont to do I will go again and see if it lasts?
Censored from 11:53 am Feb 16th from the now closed lead thread in response to a challenge from Cohenite.
If cohenite is a serious fan of the A&M model and as a Bayesian presumably, he will offer me odds as deduced from the performance to date . As Jen proclaims “We had a lower RMSE for 16 of the 17 sites “.
So offering me any odds less than 16:1 would be a vote of no confidence in A&M. What odds Cohers?
filing this here…
ABC Southern Queensland, Toowoomba
hosted by Arlie Felton-Taylor
21 Feb 2014 6:20 AM
Rural Report – 4 mins 19 secs – ID: W00056298963
New research from the University of Central Qld shows medium-term rainfall forecasts may be more accurately predicted by a form of artificial intelligence, which aim to predict the weather through pattern detection, and predicts not what the weather will do day-to-day, but in a month or more. John Abbot, researcher, says he was motivated to do the research after the huge flood at the start of 2011.
Abbot talks about how his research works, and the problems with modern-day weather forecasting systems, and says farmers and mine-operators need accurate data as opposed to vagaries. Abbot says he would like the Bureau of Meteorology take more interest in what he is doing.
****
ABC Capricornia, Rockhampton hosted by Paul Robinson
21 Feb 2014 7:33 AM
07:30 News – 1 min 35 secs – ID: W00056301562
Dr John Abbot, CQUniversity, is researching the use of artificial intelligence to improve medium term rain forecasts.