An ever-increasing number of energy suppliers and manufacturers are being hit by the most spectacular and far-reaching software failures. This is reflected in a recently-conducted survey by SQS Software Quality Systems AG for 2013. Every year, the software quality specialist asks its consultants to name the top 10 software failures from the past 12 months. Here are its top ten from 2013.
1. Launch of ‘Obamacare’: software swallows a third of insurance applications
The politically controversial ‘Obamacare’ health insurance programme in the USA made fresh headlines when it was launched. In the autumn, the healthcare portal responsible for handling the insurance was exposed to fierce criticism of its software from many quarters. The most serious defect was that around a third of the insurance applications could not be processed at all to begin with. Because of software errors, the application forms filled in on the portal were stuck there and did not reach the insurers concerned.
One reason was a system routine which prevented the social security number from being entered. The people affected then risked receiving no insurance cover even though their applications were correct. Along with this major fault, many users also complained about the poor usability of the software and errors in the forms. Around 35,000 users also had difficulties logging in for more than a week, although the system was in fact designed to support around 800,000 users. Those responsible for the healthcare portal therefore advised applicants to contact the insurers directly to check whether they had received their applications as indicated.
2. Near-blackout of the national grid in Austria
A simple meter reading query brought the Austrian national grid to the brink of total collapse in May last year, also jeopardising electricity networks in other European countries. A command had been misdirected into the control system for the European electricity networks during the commissioning of a new natural gas control system in southern Germany, almost bringing the whole grid down. The automated function then acted like a ‘distributed denial of service’ (DDoS) attack, where hackers bring a server to its knees with a deluge of queries. When the system failure was publicised, a representative of the electricity regulatory authority told Austrian Radio (ÖRF) that security levels in the energy providers’ IT systems needed to be improved.
3. Private data on six million members of a social network made public
The software failure that produced the most headlines in 2013 was down to one of the world’s leading social networks. This caused the private contact details of around six million of its members to be made public. Their e-mail addresses and telephone numbers were visible to outsiders for about a year. The cause of this was a software error in a system on the network that generated recommendations for new ‘friending’ requests. On average, it seems the contact details were shared once or twice without permission. All the users concerned were informed of the failure by e-mail.
4. German university ‘dismisses’ 48,000 students and staff
"In twelve days your login will be blocked. This is because you have left the university, your employment contract has ended or your guest login has expired." This e-mail went out to 37,000 students and 11,000 employees of a major German university one Sunday morning last year. A software error in the university’s data centre was responsible for the misinformation. Problems in reconciling staff and student data then caused the e-mail to be circulated. A spokesperson for the university tried to see the funny side of the supposed mass dismissal: "I presume the Rector got the message as well," she said.
5. Preparation for SEPA: banks transfer sums of money twice
A regional banking group in Germany transferred sums of money for thousands of customers twice by error. The fault, which affected its standing orders, also failed to execute many transfers on the specified processing date. The cause was a software error in a new program to convert the accounts to the new SEPA (Single Euro Payments Area) standard. In the course of this conversion, the new SEPA-compliant standing orders were set up successfully in the new system, but the old ones were not deleted. This caused duplicate transactions. The customers concerned had the incorrectly transferred sums refunded immediately, with interest.
6. Trading on the US Nasdaq exchange crippled for three hours
For about three hours, trading in shares and options on New York’s Nasdaq exchange ground to a halt. Software errors brought about this total failure two and a half hours after the start of trading one day last August. The source of the problems was the ‘Securities Information Processor’, which swaps share prices with other exchanges. During the shutdown, the exchange operator temporarily froze all prices and listings. Almost 30 per cent of all shares on US exchanges are traded on the Nasdaq, including Apple, Facebook, Google and Microsoft. This meant that the blackout also affected the pricing behind other major US indices like the Dow Jones and the S&P 500. The failure was not the first of this kind to hit US exchanges. The year before, for example, a trading platform had to call off its own IPO because of a software error in the trading system.
7. New high-speed train hits the tracks two years late
A major European rail operator had to wait two years longer than originally agreed with the manufacturer for 16 new high-speed trains. One reason for the delay to the high-tech trains was that a software error held up the scheduled acceptance testing by the authorities. The error in the train control system caused the brakes to be applied with a second’s delay. Other problems with the brakes, and also with the air-conditioning system, had already blown the scheduled delivery date. The delayed delivery led to significant bottlenecks for the railway company concerned. Train delays and cancellations were the result. In compensation, the manufacturer of the new trains is offering to supply the operator with an extra train worth EUR 30 million.
8. Hundreds of flights cancelled or delayed by telephone crash
When a software error crippled the internal phone system of a major European air traffic control centre, both passengers and airlines were faced with huge delays. Hundreds of flights were cancelled altogether. Some 1,300 flights, or eight per cent of European air traffic, suffered delays of up to several hours. The telephone crash, which lasted for twelve hours, meant that the authority could only process the day’s flights after a delay, if at all. The fault arose when the systems were switched from night-time to day-time operation in the early morning. The affected system was not just a simple telephone exchange. It was used by air traffic controllers to communicate with their colleagues in other national and international control centres.
9. Millions of customers without phone and text services
Six to seven million customers of a national network provider had to get by without mobile calls and text messages for four or five hours one day in October last year. Because of a software error, the relays in the telephone network were temporarily flooded with network signals. This caused the relays themselves to shut down and restart, in turn causing the whole network for phone calls and text messages to collapse, especially as the relays took longer to reactivate. Data services were not affected by the system failure.
10. New logistics software stops spare parts for cars
Last year, the workshop customers of a major German carmaker had to wait for over a week to receive the spare parts that they needed. The reason was a software change in the firm’s central logistics system. In the days and weeks after the change, a backlog of up to 200,000 parts built up. Some 5,000 dealers and branches around the world were affected. Curiously enough, customers in the immediate vicinity of the central warehouse suffered especially long waiting times, as this supplies them directly. To limit the damage to customer satisfaction, the carmaker ordered its staff in the central stores to work extra shifts at the weekend and overnight.
"At financial service providers, telecommunications companies and public authorities, important services have been dependent on IT systems ever since the 1990s. That is why they have also dominated the annual top 10 for a long time," Jochen Brunnstein, Principal Consultant at SQS Software Quality Systems, says of this year’s list. "Manufacturers and utility companies, on the other hand, are relatively new to the list. Their core business is only now being really revolutionised by software. Examples of this are the great advances in automotive electronics, and smart metering from energy suppliers."
This year’s list also shows that serious software failures rarely cause just a loss of image. "Let’s take the example of Obamacare: the jerky start caused by poor software also left substantial political flood damage in its wake," says Brunnstein. For private companies, on the other hand, major software failures almost always bring significant financial losses. As Brunnstein says, "In these cases, the financial damage is always many times more than the necessary preventive and systematic software quality assurance would have cost."
