Mark your calendars! Sept. 9 is hereby declared Debugging Day. It's been associated with removing bugs for more than 50 years now but is rarely formally celebrated. So let's start the tradition this year.
It all began with a log entry from 1947 by Harvard University's Mark II technical team. The now-classic entry features a moth taped to the page, time-stamped 15:45, with the caption "Relay #70 Panel F (moth) in relay" and the proud boast, "First actual case of bug being found."
[ Also on InfoWorld: Imagine a world with no cyber secrets. | Master your security with InfoWorld's interactive Security iGuide. | Stay up to date on the latest security developments with InfoWorld's Security Central newsletter. ]
OK, the history of computer bugs didn't really begin on this date (see "Moth in the machine" for the real story), but nevertheless, its anniversary seems a perfect time to examine famous bugs and other ghosts in the machine.
Here is a highly selective -- and therefore incomplete -- collection of infamous software bugs. Unlike the relatively benign tale of the moth in the relay, some bugs have wreaked disaster, embarrassment and destruction on the world. Some have literally killed people.
Don't expect this collection to contain tales of the Ping of Death or other faults exploited by hackers and malware -- such as the Spanair crash of 2008 or the possibly apocryphal tale of the CIA sabotaging the Soviet gas pipeline. Nor will it include deliberate decisions by programmers that came back to haunt them later, as with Y2K.
Instead, this story is about outright programming errors that caused key failures in their own right.
Have I missed anything important? Consider this a call for nominations for the biggest bugs of all time. These are my suggestions; if you have any honorable mentions, bring 'em on. The worst anyone can do is swat them.
Space bugs
Programming errors that derail high-profile space-exploration missions -- especially bugs that cause spectacular explosions -- are frightening, expensive and career-killingly embarrassing for those who let them slip through. They provide extremely vivid reminders for all of us to check and recheck (and recheck and recheck and recheck) every line of code.
The Mars Climate Orbiter doesn't orbit
Back in physics class, our teachers leaped all over answers that consisted of a number. If the answer was 2.5, they'd take their red pens and write "2.5 what? Weeks? Puppies? Demerits?" And proceed to mark the answer wrong.
Back then, we thought that they were just being pedantic. But it's the kind of error that can burn up a $327.6 million project in minutes. It did in 1998, when the Mars Climate Orbiter built by NASA's Jet Propulsion Laboratory approached the Red Planet at the wrong angle. At this point, it could easily have been renamed the Mars Climate Bright Light in the Upper Atmosphere, and shortly afterward been renamed the Mars Climate Debris Drifting Through the Sky.
There were several problems with this spacecraft -- its uneven payload made it torque during flight, and its project managers neglected some important details during several stages of the mission. But the biggest problem was that different parts of the engineering team were using different units of measurement. One group working on the thrusters measured in English units of pounds-force seconds; the others used metric Newton-seconds. And whoever checked the numbers didn't use the red pen like a pedantic high-school teacher.
The result: The thrusters were 4.45 times more powerful than they should have been. If this goof had been spotted earlier, it could have been compensated for, but it wasn't, and the result of that inattention is now lost in space, possibly in pieces.
Mariner 1's five-minute flight
On July 22, 1962, the first spacecraft of NASA's Mariner program blasted off on a mission to fly by Venus. The booster did its job, taking the spacecraft from its Cape Canaveral launchpad, but after a few minutes, Mariner 1 began to yaw off course. The guidance system failed to correct the trajectory, and guidance commands failed to correct it manually.
As the rocket veered off toward North Atlantic shipping lanes, the range safety officer did the only thing he could do: blow the thing up. Four minutes and 55 seconds into the mission, the Mariner 1 exploded.
NASA was already suffering from Sputnik envy, and the Mariner I incident was another international embarrassment for the agency. The postmortem of this debacle revealed what NASA described as "improper operation of the Atlas airborne beacon equipment" -- though later it came out that the mistranscription of a single punctuation mark by an engineer caused the mission's fatal software error.
In his 1968 book The Promise of Space, Arthur C. Clarke described the mission as "wrecked by the most expensive hyphen in history."
That may not be strictly accurate. Although NASA did mention a hyphen in some of its reports of the incident, it appears that the agency was simplifying the story for a nontechnical audience.
A more widely accepted account is that the punctuation mark was a superscript bar over a radius symbol, handwritten in a notebook. In rocket science, the overbar signifies a smoothing function, so the formula should have calculated the smoothed value of the time derivative of a radius.
Without the smoothing function, even minor variations of speed would trigger the corrective boosters to kick in. The automobile driving equivalent would be to yank the steering wheel in the opposite direction of every obstacle in the driver's field of vision.
But few people know what an overbar is, and since it looks like a hyphen, that's how most people tell the story.
Moth in the machine: Debugging the origins of 'bug'
It's an oft-repeated tale that the grande dame of military computing, computer scientist and U.S. Navy Rear Admiral Grace Hopper, coined the terms bug and debug after an incident involving Harvard University's Mark II calculator.
The story goes like this:
On Sept. 9, 1945, a Harvard technical team looked at Panel F and found something unusual between points in Relay 70. It was a moth, which they promptly removed and taped in the log book. Grace Hopper added the caption, "First actual case of bug being found," and that's the first time anyone used the word bug to describe a computer glitch. Naturally, the term debugging followed.
Yes, it's an oft-repeated tale, but it's got more bugs in it than Relay 70 ever had.
For one thing, Harvard's Mark II came online in summer of 1947, two years after the date attributed to this story. For another thing, you don't use a line like "First actual case of bug being found" if the term bug isn't already in common use. The comment doesn't make sense in that context, except as an example of engineer humor. And although Hopper often talked about the moth in the relay, she did not make the discovery or the log entry.
The core facts of the story are true -- including the date of Sept. 9 and time of 15:45 hours -- but that's not how this meaning of the word bug entered the lexicon. Inventors and engineers had been talking about bugs for more than a century before the moth-in-the-relay incident. Even Thomas Edison used the word. Here's an excerpt from a letter he wrote in 1878 to Theodore Puskas, as cited in The Yale Book of Quotations (2006):
'Bugs' -- as such little faults and difficulties are called -- show themselves and months of intense watching, study and labor are requisite before commercial success or failure is certainly reached.
Word nerds trace the word bug to an old term for a monster -- it's a word that has survived in obscure terms like bugaboo and bugbear and in a mangled form in the word boogeyman. Like gremlins in machinery, system bugs are malicious. Anyone who spends time trying to get all the faults out of a system knows how it feels: After a few hours of debugging, any problems that remain are hellspawn, mocking attempts to get rid of them with a devilish glee.
And that's the real origin of the term bug. But the tale of the moth in the relay is worth retelling anyway.
Forty seconds of Ariane-5
The European Space Agency (ESA) has also suffered embarrassment on the software front. The inaugural flight of its fifth-generation Ariane launcher bested NASA's Mariner 1 score for unmanned spacecraft disaster: It took only 40 seconds to blow up.
On June 4, 1996, after the kind of dramatic vertical blastoff you'd expect from a high-profile European vehicle, cameras on the ground barely had time to focus on the Ariane-5 as it turned around and began to fall apart, before it completely exploded.
The Ariane Flight 501 disaster began with a loss of guidance and attitude information 30 seconds after liftoff. Once it veered completely off course, it automatically self-destructed.
The problem was that Ariane-5's inertial reference system dealt with 64-bit floating-point data and converted it into 16-bit signed integer values. The result of the data conversion was too large for a 16-bit signed integer, which caused an arithmetic overflow in the hardware. In the ESA's case, a software handler that could have dealt with the problem had been disabled, and so there was no levee to dam the cascade of system failures that led to the destruction.
Bad-PR bugs
Some bugs are noisy: They cause explosions that destroy machines. Others are subtler in their destructiveness: They cause severe embarrassment that turns companies' good names to "Mud" and sometimes threatens the bottom line.
Pentium chips fail math
In 1994, an entire line of CPUs by market leader Intel simply couldn't do their math. The Pentium floating-point flaw ensured that no matter what software you used, your results stood a chance of being inaccurate past the eighth decimal point. The problem lay in a faulty math coprocessor, also known as a floating-point unit. The result was a small possibility of tiny errors in hardcore calculations, but it was a costly PR debacle for Intel.
How did the first generation of Pentiums go wrong? Intel's laudable idea was to triple the execution speed of floating-point calculations by ditching the previous-generation 486 processor's clunky shift-and-subtract algorithm and substituting a lookup-table approach in the Pentium. So far, so smart. The lookup table consisted of 1,066 table entries, downloaded into the programmable logic array of the chip. But only 1,061 entries made it onto the first-generation Pentiums; five got lost on the way.
When the floating-point unit accessed any of the empty cells, it would get a zero response instead of the real answer. A zero response from one cell didn't actually return an answer of zero: A few obscure calculations returned slight errors typically around the tenth decimal digit, so the error passed by quality control and into production.
What did that mean for the lay user? Not much. With this kind of bug, there's a 1-in-360 billion chance that miscalculations could reach as high as the fourth decimal place. More likely, with odds of 1-to-9 billion against, was that any errors would happen in the 9th or 10th decimal digit.
More math bugs
Intel's Pentium flaw wasn't the only math-related bug to cause a PR disaster. These two had Microsoft execs red in the face:
Windows Calculator 3.x: In 1994, a bug in CALC.EXE came to light that had quietly been kicking around since Windows 3.x first appeared in 1990. Propellerheads had fun subtracting 2.1 from 2.11 in Windows Calculator and getting an answer of not 0.01, but 0.00.
Excel 2007: Ask people with calculators or slide rules to multiply 850 x 77.1, and they'll answer 65,535. But in September 2007, it was discovered that Excel 2007 answered 100,000. According to Microsoft, this bizarre rounding-up occurred only in calculations that resulted in 65,535 or 65,536. What's more, Excel actually calculated the correct answer, but a bug prevented it from displaying properly.
But wouldn't you know it? A Virginia-based math professor named Thomas Nicely needed that level of accuracy, found he wasn't getting it and figured out why.
In October 1994, he alerted Intel, then others, to the problem. Intel retorted with a response only marginally less tactful than "Oh, that thing? Yeah, we noticed that back in June."
Thus began an inexorable slide into PR hell and a costly mop-up bill. In January 1995, Intel announced a pretax charge of $475 million against earnings, most of which apparently stemmed from replacing flawed processors.
The bottom line in this arithmetic mess is this: In lookup-table and money calculations, 1,066 -- 5 = --$475,000,000. Any way you look at it, that's bad math.
Call waiting ... and waiting ... and waiting
On Jan. 15, 1990, around 60,000 AT&T long-distance customers tried to place long-distance calls as usual -- and got nothing. Behind the scenes, the company's 4ESS long-distance switches, all 114 of them, kept rebooting in sequence. AT&T assumed it was being hacked, and for nine hours, the company and law enforcement tried to work out what was happening. In the end, AT&T uncovered the culprit: an obscure fault in its new software.
Here's how the switches were supposed to work: If one switch gets congested, it sends a "do not disturb" message to the next switch, which picks up its traffic. The second switch resets itself to keep from disturbing the first switch. Switch 2 checks back on Switch 1, and if it detects activity, it does another reset to reflect that Switch 1 is back online. So far, so simple.