Last week Melbourne’s rail network ground to a halt after the main control centre was evacuated due to a false alarm – the latest in a growing list of similar outages. So why is Melbourne’s rail control centre so vulnerable to these kind of issues, when other cities seem to keep on ticking?
In the beginning
Way back in the beginning of railways there was no such thing as a central rail control centre – staff at railway stations pulled mechanical levers to direct trains.
Some levers moved the tracks to direct trains where they needed to go, while others operated semaphore signals to tell train drivers whether to stop or go.
The coming of electricity removed the need for as many staff – trains could be automatically detected by the signalling system, with electric motors and coloured light signals taking over from human muscle.
Improvements to electronics made the lever frames themselves redundant, with push button controls allowing remotely located staff to control trains over a widely dispersed network.
PROV image, VPRS 12800/P1, item H 5445
Melbourne moved into this modern age with the opening of the ‘Metrol’ train control facility in the 1980s. Located in a specially built facility on Batman Avenue, the brown brick bunker was for more secure than the timber signal boxes that it replaced.
Flinders Street C signal box with Metrol in background, photo by Weston Langford
When commissioned in 1980, Metrol had three responsibilities:
- operation of the electronic passenger information displays at city stations,
- communications with station staff, signal boxes and train drivers, and
- directing trains through Flinders Street Station, Spencer Street Station and the City Loop.
In the years that followed the scope of control expanded to cover the area bounded by Clifton Hill, East Richmond, Richmond, and South Kensington stations.
Sowing the seeds of failure
In the late 1990s the decision was made to cover over the Jolimont rail yards, requiring the demolition of the original Metrol site to make way for Federation Square. The existing control systems from the 1980s were relocated to a ‘temporary’ site in the Melbourne CBD, pending their replacement by a modern system by June 2001. Originally costed at $11 million, the cost had grown to $18 million by the time the ‘Train Management Facility’ project was abandoned in 2003.
A second attempt at rejuvenating the now aging system was made in 2006, along with building a backup train control facility, when $88 million was made available in the State Government’s ‘Meeting Our Transport Challenges’ package. Completion was due in 2010 but again delays were encountered, with the modern ‘Train Control Management System’ not taking over until late 2014.
And the failures begin
The first failure of note occurred in June 2005, when a leaking air conditioner flooded the control room and forced trains to stop running:
A simple leak in an air-conditioning hose stopped Melbourne’s trains in their tracks yesterday. The evacuation of a flooded control centre halted city services for two hours.
Up to 30,000 metropolitan and country passengers were stranded and 66 trains cancelled during the afternoon crisis. Dozens of trains sat empty at City Loop platforms. Another 21 trains were cancelled after services restarted and delays lasted into the evening peak period.
The chaos began at 11.40am, when water gushed through the hose into the Metrol control centre on Collins Street, from which Connex runs the train network. As the centre was evacuated, trains were ordered to stop at the nearest station and were stranded for two hours.
Mr Hughes said an investigation was under way. “The management of the problem was fine, our emergency response and our shutdown process worked fine,” he said. “But we’ve got to examine what happened and what re-engineering we might do to make sure it doesn’t happen again.”
October 2014 saw a different issue shut down the control room.
A false fire alarm briefly shut down Melbourne’s entire train network this morning, resulting in flow-on delays.
Metro Trains said an alarm sounded in the building housing the train control centre in the CBD before 10:00am (AEDT).
Trains were halted for about 15 minutes and resumed when it was determined to be a false alarm.
But June 2015 saw the same issue reoccur.
Melbourne’s train operator Metro has apologised to commuters after the city’s rail service was brought to a standstill by a false alarm in the control centre during the morning peak period.
There were delays of up to 40 minutes across the system this morning after the Flinders Street control centre was evacuated due to an alarm at 8:30am.
About 100 trains were stopped and staff were evacuated.
It was quickly determined it was a false alarm and within 18 minutes trains started running again.
The cause of the issue – another water leak.
Metro’s operations director Ron Bria apologised to Metro commuters but said safety was paramount.
“There was a water leak that made its way into the fire panel which made the alarm go into a default position, evacuating the building,” he said.
Mr Bria said a backup did not kick in because that took 50 minutes but the situation was sorted within 15 minutes.
He said communications with building maintenance was usually made before to check if it was a false alarm but because it was a default system, the entire building was ordered out.
The water leak was not on a Metro floor.
But this time around the backup train control centre was an option.
Public Transport Victoria (PTV) chief executive Mark Wild said they did not switch to the back-up control centre because it was faster to get staff back into the main control centre. following the false alarm.
It takes about 50 minutes to get the back-up control centre up and operational.
“When you know it’s going to be a relatively minor delay due to a false alarm, which we were pretty sure this was, it’s best just to re-populate the existing centre and get the system going again, rather than starting up the back-up centre,” Mr Wild said.
“I think Metro are always in that difficult thing about whether they start up the permanent back-up centre or just wait to go back in.”
He said he thought Metro had made the right call by waiting to go back into the existing centre.
History never repeats? Another false alarm caused the November 2016 outage.
Metro Trains say they are “very sorry” after a false alarm brought down Melbourne’s entire train system last night.
The alarm caused the Metro Trains control centre to be evacuated, causing all trains to grind to a halt. The blame was laid on a power fault.
Metro Trains managing director Andrew Lezala said a power fault caused the alarm to activate.
Mr Lezala said that the building’s backup power supply also failed.
“We immediately dispatched a crew then to our other centre,” he told 3AW.
Notice a common theme?
Fixing the problem
When originally built in the 1980s, Melbourne’s train control centre was a bunker that could keep on ticking away despite what the rest of the city threw at it.
Today it is just a room full of computers sitting in an anonymous CBD office block. Sure – backup power supplies allow it to ride out a short power outage, but as soon as a tenant on another floor burns their toast and sets off the fire alarm, it’s everyone out until the all clear is given.
The only solution is to heed the lessons of the past and move Melbourne’s train control centre back into a purpose built secure facility, with redundant systems built in and a management team ready to fix any minor issue before it becomes a major one.
Footnote
While I was writing this post many others came to the same conclusion – Rail meltdown ignites calls to move network control centre to secure location from The Age.
“Anonymous CBD office block” untill the news published the address…
Oooops.
Railfans have known the address for years – it appears 2015 is when the news media let it slip.
I was on the job for 30 years.
That was an excellent summary of the problem.
Well done!
Glad I didn’t miss the mark. 🙂
The construction of Fed Square was a golden opportunity to replace the ageing computer systems at Metrol – still based on PDP11/70 technology (even by the mid 90s, Treasury had already knocked back several proposals). Unfortunately, the timing was a problem. The Metrol building had to be vacated by December 1999, and this was too short a time to commission a new system. A two step approach was taken. The first step was to build a replica Metrol – using PDP11/84s – in a ‘central city office building’. This was commissioned, very successfully, in October 1999. This was the ‘temporary’ Metrol.
This was to be followed by commissioning of a completely new system, the TMF, in April 2001, including an off-site backup system. AFAIK there was never any intention to build a replacement Metrol building – the new system was also to be installed in the same ‘central city office building’. The TMF was delayed, and was quietly cancelled in mid 2003. No reason for the cancellation was ever formally announced, but National Express, who were funding the replacement had collapsed in late 2002. About the time the TMF was cancelled, the government and National Express’ parent announced a partial payout of National Express’ unsecured creditors.
This left Metrol operating on very obsolete technology. To ‘stabilise’ the system a further interim system was developed using a PDP hardware emulator. Development appears to have commenced around June 2004 and was practically complete by June 2006.
On 27 June 2005 Metrol had to be evacuated after a burst water pipe in the air conditioning system. There was a lot of criticism of the government due to the lack of a backup Metrol system, which had been part of the cancelled TMF project. The government promptly announced that a back-up system would be built – this, of course, was only feasible because of the work done on the PDP emulator based system. Work commenced on the backup in July 2006 (after the main site had been commissioned), and was completed in either 2007 or 2008.
This emulation system was still considered an interim solution, with an expected life of 10 years. Immediately after the 2005 failure, the government announced a new project to replace Metrol (the TCMS). This was finally commissioned on the main site in 2015, but even then the backup was still operating on the PDP11 emulators.
Thanks for that additional information Andrew – the 1990s relocation of Metrol was before my time!
I’ve had other people point out that I missed the project that delivered the PDP hardware emulator and enabled the creation of a disaster recovery site.
[…] automated signalling system referred to is the Metrol signal control centre. The history is a saga in itself – it moved physical home in late-1990s, then moved again into an emulated computer system in […]
Your correspondents, are right on the money. In their analysis of this situation.
Without too much boring history, Marcus made a good point about the original Metrol site. Not only was it ‘bullet’ proof but would be still running today if the Kennett land grab did not occur. The building and the hardware for housing the train control function, would be still there today. As a former employee that looked after the “system” we all knew that the application software and associated hardware, was originally set up so as to be ‘cloned’ onto new lines or additional expansion.
What is not know by the public, was that the new Clifton Hill group to Epping workshops, was done exactly like this on the computing platform known as System ‘C’
This was totally cloned, set up, tested & commissioned in house. circa 1989.
Both political parties were responding to the groups like the “Lets cover the rail yards”
This was the impending demise of METROL.
Back to the security, redundancy & the system availability, for a start;
Triple back up power systems in place, GAS turbine generator on roof of building,
software redundancy on real ‘hot standby’ transfer.
The system was designed by real computer people both hardware & software.
System specifications included the use of fully running platforms, where a flick of switch would transfer any system ( A B or C ) to an up and running network.
It is very sad to see such ingenuity & design being traded in for ‘windows’ based pc’s and other similar virus infected & hacked “cheap off the shelf hardware”
Thanks for the extra info Regent!
[…] But by mid-1978 the intended opening date of December 1979 was looking unachievable – October 1980 set as the new date, following delays to the new ‘Metrol’ train control centre. […]