Airline CEOs and reciprocal first class travel

I’m currently reading the 1987 book Struggle For Take Off: The British Airways Story by Duncan Campbell-Smith, and the start of chapter 4 contained an interesting anecdote about Sir Freddie Laker, founder of Laker Airways.

Freddie was visiting his estranged wife and their son, Little Freddie, in Miami at the beginning of March 1982. As he had done countless times before, he asked the British Airways office manager in Miami for a first-class ticket home. Since his last trip, however, the world has changed for Freddie.

Top men in most of the world’s airlines have reciprocal arrangements to pick up free first-class air tickets as and where they want them. When Laker Airways has passed into receivership, on 5 February 1982, Freddie had lost this entitlement – in theory, anyway. But by March Freddie was already proclaiming plans for a new ‘People’s Airline’. And this, as he told BA’s Miami office, should have kept him on the free travel list.

There followed a frantic exchange between the office manager and BA’s brass at Heathrow. It seemed there were going to be quite a few economy-class passengers on the same flight who had each paid a normal fare and then been asked for an additional £130 by BA. The normal fare, unhappily, had been paid to Laker Airways: they were among the 6,000 tour and charter passengers stranded abroad by the collapse.

BA was caused some embarrassment by the whole incident. In the end, Freddie got his free first-class ticket; but it was quickly made clear that it would be his last from BA.

Free first-class tickets anywhere in the world sounds like quite a deal, and the above story is corroborated by this article from the March 6, 1982 edition of the New York Times.

For Sir Freddie Laker, Still Another Blow
By Albin Krebs and Robert McG. Thomas

First Sir Freddie Laker lost his Laker Airways to bankruptcy and receivership, and now comes what may be the unkindest cut of all: He’s also lost his right to ride free on other airlines.

It has been the policy of British Airways, and other airlines, to accord heads of competing airlines the privilege of riding free, but a spokesman for B.A. said in London yesterday that since the Laker line was no longer in operation, it was withdrawing the courtesy in Sir Freddie’s case.

Spokesmen for other airlines said they would follow suit. “There is no reason to suppose he will ask for, or will be granted, this facility again,” said the British Airways spokesman. On Tuesday, Sir Freddie rode British Airways on a free $1,935 first-class ticket from Florida to London. Twenty other Britons stranded in Miami when his airline collapsed last month were also on board the flight, in tourist class. They had paid $238 each for their tickets.

Since his 4-year-old son, Freddie Jr., lives in Key Biscayne with Sir Freddie’s estranged wife, Sir Freddie has flown frequently to Florida.

Fast forward to today, and the I wonder if the gentleman’s agreement still exists between airline CEOs – I find it hard to imagine Michael O’Leary, controversial CEO of super low cost airline Ryanair, being given a free ticket for the super plush ‘The Residence’ first-class suite with Etihad Airways!

Back in the real world

Reciprocal travel privileges for airline staff still exist today, and it isn’t just for the men in suits – officially known as interline travel, it is governed by the Zonal Employee Discount (ZED) multilateral agreement, which covers around 180 airlines around the world.

Further reading

Melbourne’s exploding Siemens trains

Last week Melbourne’s railway network almost came to a grinding halt when industrial action threatened to pull dozens of trains of service. So how did Metro Trains Melbourne find themselves in that situation?

Siemens 729M approaches South Kensington on a down Werribee service

The story starts in mid-December 2014, when a number of Siemens trains were pulled from service following explosions beneath the floor – capacitors located in the static inverter unit under the train were experiencing violent failures, resulting in the metal cabinet door being blown off.

The Age went into further detail about the problem:

An instruction notice written after Tuesday’s crisis meeting states: “As we have no capacitors on stock, we are allowed to cannibalise units where the capacitance check reveals a faulty capacitor, but we are not allowed to take any capacitors from a unit which had a reported explosion!”

Defective train units have been taken to the Newport workshop for repair.

Metro said just three of its fleet of 36 Siemens trains had been found to have an electrical fault forcing its removal, and the rest were running fine.

“We are confident the capacitors on the remainder of the Siemens fleet are operating without fault,” spokeswoman Larisa Tait said.

“We will continue to inspect and test all capacitors on Siemens trains during routine maintenance exams, which occur every six to eight weeks. We are confident we will pick up any capacitor faults before they occur and certainly before they fail.”

The article also featured CCTV footage showing a static inverter exploding beneath a parked Siemens train – presumably sent to the newspaper by a source wanting to force the hand of Metro Trains.

With each Siemens train having one static inverter located beneath the middle ‘T’ carriage, Metro installed tie-down straps to every one of the 72 trains in the Siemens fleet as a temporary fix.

Tie-down straps affixed to a static inverter beneath a Siemens train

Fast forward to close of business on 6 January 2015 and the Locomotive Division of the Rail, Tram and Bus Union (RTBU) got involved, issuing a circular to their members.

The circular read:

Siemens trains – exploding static inverters

As a result of Metro Trains failure to ensure that there is no imminent threat to drivers safety by providing a physical barrier to prevent serious injury or death by the force and flames of the exploding static inverters on Siemens trains the following will apply:

No driver is to attend a Siemens train with the pantographs raised.

This includes but is not limited to the following tasks – train preparation, stabling of trains, changing of ends at rail level, etc, etc.

Any driver that is threatened or coerced by management to place themselves in harm’s way should contact the Union immediately.

The union directive was a virtual black ban on the Siemens trains – before entering service each train has to be inspected at ground level by the train driver, and a ban on this practice effectively welded the entire fleet of 72 Siemens trains to the tracks for the foreseeable future.

Metro Trains considered the actions of the union as “unprotected industrial action” and took the RTBU to a late night Fair Work Commission hearing – chaos on the rail network as averted when the union agreed to withdraw their circular to drivers, and Metro agreed to start addressing the safety concerns.

So what was the agreed resolution to the issue of exploding capacitors?

The agreement stipulates Metro must “implement additional mitigation, such as a fire blanket or similar” to address exploding static inverters on its Siemens trains within a week from Wednesday.

Metro and the union must meet daily to monitor progress on the directives.

Said fire blankets have since started appearing in static inverter cabinets, but it does raise the question – what is the long-term fix for the issue?

'Access panel' side of a static inverter beneath a Siemens train, with fire blanket and tie down straps fitted

The bigger story

Metro Trains cutting corners on safety isn’t anything new – trains without working headlights, broken passenger intercoms and ‘temporary’ fixes to flawed rails are all ways that the company has tried to save money by taking shortcuts. Combine that with their station skipping to avoid fines tactics, and intimidation of injured staff, they are a company that is happy to milk the Victorian taxpayer for everything possible while screwing over both passengers and staff.

As for the relationship between Metro Trains and the RTBU Locomotive Division, I’m betting that it is going to become more explosive than the static inverters in the coming months – the current enterprise bargaining agreement comes to an end in the middle of 2015, and the union doesn’t seem to be afraid of beating Metro around the head in the court of public opinion in order to get what they want.

Footnote

The exact number of static inverter explosions is so far unknown, but on 3 December 2014 a small electrical fire broke out underneath a train during peak hour at Yarraville – a Siemens train was involved. Related?

Public Transport Victoria fixing the small things

You don’t need to wait for a billions of dollars to drop from the sky to improve Melbourne’s public transport – making small tweaks to existing services and infrastructure can often make things a little better, and for far less money.

Unfortunately for us, Public Transport Victoria seems to have misread the memo – small tweaks to branding is their modus operandi, such as what I found at the ‘PTV Hub’ at Southern Cross Station.

Main 'PTV Hub' on the concourse at Southern Cross Station

Originally opened as the “Myki Discovery Centre” to assist passengers in the transition to the new ticketing system, in November 2012 it was expanded to cover all public transport inquiries (minus V/Line ticket sales, but that is a story in itself).

Fast forward to December 2014, and I noticed a minor change outside the hub – they’ve updated the bus photo to one featuring the PTV livery.

PTV Hub at Southern Cross has replaced their bus photo with their current livery

Normally I only accuse dribbly trainspotters of caring about the “wrong” train in a photo, but presumably there is someone at PTV who is so anal retentive that they thought that the organisation’s highest priority should be ordering a replacement vinyl sign because the “wrong” bus featured in the existing one.

Bus spotting minutiae

The original photo was of a NationalBus vehicle in the Ventura-style blue and yellow livery, driving somewhere in the eastern suburbs of Melbourne.

Now the sign features Westrans Altona bus #129 (registration 9407AO) in PTV livery outside Williams Landing Station – along with bus #130 they were the first two PTV liveried buses in Melbourne, entering service in April 2013.

Tracing a performance issue on my web server

Managing my various web sites can be difficult at times, and my experience the other weekend was no different. My day started normally enough, as I logged onto my VPS and installed the latest security patches, then set to work on uploading new photos to my site. It was then I noticed my web site was taking minutes to load pages, not seconds, so I started to dig into the cause.

Server statistics logged by New Relic

My initial setup

After I moved from shared web hosting, my collection of websites had been running on a $5 / month VPS from Digital Ocean – for that I got 1 CPU, 512 MB of RAM, and 20 GB of disk space. On top of that I used an out-of-the-box Ubuntu image, and installed Apache for the web server and MySQL for the database server.

I then installed a number of separate WordPress instances for my blogs, a few copies of Zenphoto to drive my different photo galleries, and then a mishmash of custom code for a number of other side projects. All of that is exposed via four different domain names, all of which sit behind the CloudFlare CDN to reduce the load on my server.

With some many web sites running on just 512 MB of RAM, performance was an issue! My first fix was to setup a 1 GB swap file to give some breathing room, which did stabilise the situation, but MySQL would still crash every few days when the server ran out of memory.

Swapping out Apache for the much less memory intensive Nginx web server is one way to fix the issue, but I didn’t have time for that. My solution – cron jobs to check the status of my server and restart the services as required!

The first script I came up with checked if the MySQL service was running, and start it up if it wasn’t.

service mysql status| grep 'mysql start/running' > /dev/null 2>&1
if [ $? != 0 ]
then
SUBJECT="MySQL service restarted $(date)"
service mysql status|mail -s "$SUBJECT" [email protected]
sudo service mysql start
fi

My second attempt negated the need for the first script, as it checked to see how much memory was free on my server, and restarted Apache if it was less than a given threshold.

#Minimum available memory limit, MB
THRESHOLD=300

available=$(free -m|awk '/^Swap:/{print $4}')
if [ $available -lt $THRESHOLD ]
then
SUBJECT="Apache service restarted $(date)"
service apache2 status|mail -s "$SUBJECT" [email protected]
sudo service apache2 restart
fi

Under normal load my cron job would restart Apache every day or so, but it did keep the database server up for the rest of the time.

Something is not right

After realising my web site was taking minutes to load pages, not seconds, I started to dig into my server logs. CPU load was hitting 100%, as was memory consumption, and my cron job was restarting Apache every few minutes – something wasn’t quite right!

My first avenue of investigation was Google Analytics – I wanted to find out if the spike in load was due to a flood of new traffic. While the Slashdot effect is a nice problem to have, but in my case it wasn’t to be – incoming traffic was normal.

I then took a look at my Apache access logs – they are split up by virtual host, so I had a number of log files to check out. The first suspicious entries I found were brute force attacks on my WordPress login pages – blocking those was simple, but the server load was still high.

Spending my way out

When looking to upgrade a system to handle more traffic, there are two completely different ways to go about it:

  • Be smart and optimise what you already have, to do more with the same resources
  • Throw more resources at the problem, and just ignore the cause

My server was already nearing the 20 GB disk space limitation set by Digital Ocean on their $5 / month VPS, so I figured an upgrade to next size VPS might fix my problem. Upgrading a Digital Ocean ‘droplet’ is simple job with their ‘Fast-Resize’ functionality – it takes about a minute, but in my case the option wasn’t available – I had to do it the hard way:

  1. shut down my server,
  2. create a snapshot of the stopped virtual machine,
  3. spin up a new Digital Ocean server,
  4. restore my snapshot to the new server,
  5. point CloudFlare from my old server IP address to the new one.

All up it took around 30 minutes to migrate from my old server to my new one, but at least with CloudFlare being my public facing DNS host, I didn’t have to wait hours for my new IP address to propagate across the internet!

Unfortunately, the extra resources didn’t fix my problem – CPU load was still through the roof.

Digging for the root cause

I first installed the htop process viewer on my server, and was able to see that MySQL was using up far much more CPU than normal – presumably my caching wasn’t working right, and my web pages were having to be generated with fresh database queries each time.

Next I fired up a MySQL console, and had a look at the currently running queries. Here I noticed a curious looking query over and over again:

SELECT @serachfield ...

A check of the code deployed to my server indicated that the query was thanks to the search function in Zenphoto, and when I went back into my Apache access logs, I eventually found the problem – a flood of hits on my photo gallery.

Apache web server access logs

Each line in the logs looked like the following:

108.162.250.234 – – [21/Dec/2014:04:32:03 -0500] “GET /page/search/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/beacon-3.newrelic.com HTTP/1.1″ 404 2825 “http://railgallery.wongm.com/page/search/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/maintenance/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/nr-476.min.js” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)”

Each request was bound for “http://js-agent.newrelic.com/nr-476.min.js” or other files hosted at newrelic.com, and the user agent always appeared to be Internet Explorer 8.

New Relic is a software analytics tool I have installed on my server, and on seeing the multiple references to it in my access logs, I remembered that I had updated my version of the New Relic agent just before my performance issues had started. Had I found a bug in it?

The cause

A check of the HTML source of the page in question showed a link to js-agent.newrelic.com embedded in the page, so I came up with the following explanation for the load on my server:

  1. A user hits http://railgallery.wongm.com/page/search/SEARCH_TERM
  2. The New Relic Javascript file at http://js-agent.newrelic.com/nr-476.min.js somehow gets loaded as a relative path, and not an absolute one, which results in a request to:

    http://railgallery.wongm.com/page/search/SEARCH_TERM/js-agent.newrelic.com/nr-476.min.js

  3. My server would then treat the above URL as valid, delivering a page, which then includes a relative link to js-agent.newrelic.com/nr-476.min.js a second time, which then results in a page request to this URL:

    http://railgallery.wongm.com/page/search/SEARCH_TERM/js-agent.newrelic.com/js-agent.newrelic.com/nr-476.min.js

  4. And so on recursively:

    http://railgallery.wongm.com/page/search/SEARCH_TERM/js-agent.newrelic.com/js-agent.newrelic.com/js-agent.newrelic.com/nr-476.min.js

With the loop of recursive page calls for a new set of search results, each requiring a fresh database query, it was no wonder my database server was being hit so hard.

As an interim fix, I modified the Zenphoto code to ignore search terms that referenced New Relic, and then rolled back to the older version of the New Relic agent.

sudo apt-get remove newrelic-php5
sudo apt-get remove newrelic-php5-common
sudo apt-get remove newrelic-daemon
sudo apt-get autoremove newrelic-php5
sudo apt-get install newrelic-php5-common=4.15.0.74
sudo apt-get install newrelic-daemon=4.15.0.74
sudo apt-get install newrelic-php5=4.15.0.74

I then raised a support case for New Relic to look into my issue. In an attempt to reproduce the issue, I rolled forward with the current version of the New Relic agent to play ‘spot the difference’, but I couldn’t find any, and the errors also stayed away.

I’m writing this one off as a weird conflict between the updated New Relic agent running my server, and an old version of the browser monitor javascript file cached by a single remote user.

Conclusion

After working through my performance issues I now know more about what my web server is doing, and the extra RAM available following the upgrade means my horrible cron job hacks are no longer required to keep the lights on!

As for the steps I will follow next time, here are the places to check:

  • Google Analytics to check if I am getting a flood of legitimate traffic,
  • Apache access logs for any odd looking sources of traffic,
  • current process list to see where the CPU usage is coming from,
  • currently running MySQL queries for any reoccurring patterns.

My most viewed blog posts for 2014

I sat down the other evening and had a look at my top 20 most viewed blog posts for 2014 – entries with an asterisk (*) beside them were published this year.

My top post was one I published early in the year, digging into the story behind a viral image of a fire hose crossing railway tracks – the post seems to be getting a lot of traffic even today. Next up was my 15 minutes of fame when my story about confronting a racist guy on the tram ended up in the news.

Heading further down the list we find a number of railway themed entries, some older posts about abandoned buildings in Melbourne, and three old faithfuls – fixing the power jack of a Samsung laptop, the history of National Mutual, and a how-to for fixing digital camera timestamps after daylight savings time changes.

With only eight of my top 20 posts having been written this year, it goes to show the value that writing a “timeless” blog post can give.