Avoiding the Meltdown

melted-animation-sculpture by Cecilia Fletcher, on Flickr
Image by Cecilia Fletcher, on Flickr

You need to learn from your mistakes in order to survive, but you need to learn from other people’s mistakes if you want to succeed. I recently had the occasion to research the nuclear power tragedies at Chernobyl and Fukushima Daiichi. Each disaster came about for very different reasons but both highlighted several common mistakes, both in preparation and in response. While very few organizations are going to have issues that are literally life and death, we can still take the key lessons to heart. These two nuclear catastrophes together have taken the lives of many dedicated and brave individuals and the communities and nations around the affected locations are still today trying to recover. Without callously pointing blame in any particular direction we will take a sober look at some of the commonly accepted missteps that occurred to produce these two terrible events and examine how the lessons that can be learned from them.

Prepare for Failure

When the number 4 reactor at Chernobyl began its meltdown in April 1986 the technicians and engineers on hand were performing a systems test designed to help resolve a fundamental problem with the reactor’s emergency response system. You see, most nuclear power plants require powerful hydraulic pumps to circulate water in order to maintain temperature in the reactor. Without constant supplies of cool water the fuel rods begin to heat up to the point that bad things begin to happen. When the plant is operational and generating power these pumps are powered by the plant itself, but in the case of a loss of power generation an external source is needed. In the case of Chernobyl this external power came from several diesel generators designed to automatically start up in the event of a power failure. The problem was that the generators would take over a minute to power up to the point where they can begin to power the pumps. When dealing with nuclear power a minute can be a very long time, so a new process was designed to try to minimize the power gap by capturing residual  steam energy and using it to power the pumps in the case of a power failure. On the night of the meltdown technicians were attempting to test this process – the three previous tests had failed – and it was a combination of their actions during the test and other design failures that led to the disaster.

Lesson: Identify points of failure before implementation, preferably during the analysis or design phase and build in redundant systems that can handle the load.

Constantly Re-Evaluate Based on New Information

Despite being several years older than Chernobyl, the reactors at the Fukushima I Nuclear Power Plant were considered to be a much safer design. They included a number of more capable safety features and unlike Chernobyl human error played very little part in the direct cause of the failure. Yet arguably, the Japanese disaster is at least as catastrophic as Chernobyl. Ultimately if any major human factor contributed to the scale of the problem it was in the inability to properly re-evaluate the safety systems when new information was identified. After the devastating effects of the 2004 Indian Ocean earthquake and tsunami several sources warned the managers of the plant that the nuclear power station was vulnerable, but the warnings went unheeded. The Fukushima I plant was particularly hit hard because:

  • Most of its emergency generators were located in basement areas that were flooded by the waters of the tsunami
  • The protective tsunami barrier was less than half the height of the tsunami that actually hit
  • While a few generators were located on high ground, their control links still terminated in the flooded basement areas so while they worked, they could not send power to the areas that needed it
  • The earthquake had knocked out all connections to the community power grid so the technicians could not draw power from other sources and the roads were made impassable enough that mobile generators could not be brought in fast enough

Lesson: Be vigilant about issues, constantly re-evaluate your rules, resources, and procedures when new information comes along. Dismiss nothing without full consideration, not just of the probability, but also the ramifications.

Don’t Be Afraid to Sit Out a Hand

Leonid Toptunov was the night shift operator of Chernobyl’s reactor number 4. He had been a senior engineer for less than three months which is why he was working nights. The test that would ultimately generate the conditions making a meltdown possible wasn’t supposed to occur on his shift at all and he had not been part of the extensive and comprehensive planning for the testing as his team was originally only intended to perform some minor clean up. The test was intended to occur during the day shift. However, the plant director postponed the test because of a need to satisfy some particularly strong peak demand. Rather than wait till the next day, they moved forward that night. Toptunov ended up accidentally inserted the control rods too far at the beginning of the test and began a series of events that would set the stage for the final trigger – a failure of the control rod mechanism itself – that would ultimately cause the first of two explosions and the eventual meltdown. At numerous times between this first error and the failure the technicians had the opportunity to recognize the dangers and identify key signs that they were headed for disaster, such as:

  • While the test plan called for power levels to be between 700 MW and 800 MW for the test, ultimately the decision was made to move forward at 200 MW. This particular type of power plant, it should be noted, actually became more unstable as it powered down.
  • Measurements of the reactor showed an unstable core temperature and coolant flow even leading to various alarms going off. These were ignored and the test continued
  • Emergency alarms concerning the reactor emergency protection system caused two turbine generators to trip off prior to the start of the test

Lesson: Don’t waste all of the effort you put into creating a quality plan by ignoring it when the situation changes. Rather stop, regroup, and re-evaluate before charging ahead.

Follow Your Procedures

After the earthquake and tsunami hit Unit 1 of the Fukushima reactor and external power failed was initially cooled by two unpowered isolation condenser systems. While not as efficient as a powered system these might have given the technicians more time before a meltdown to try to resolve the power issues. However, workers on site removed both isolation condensers and switched to battery-powered pumps despite procedures stating that at least one unpowered condenser should have been left in case of total power failure. Half an hour later, the batteries failed and the isolation condensers could not be restarted. The unit began to build up an excess of hydrogen which would eventually explode further damaging the reactor.

Meanwhile engineers throughout the other reactor units were working frantically to maintain cooling in their areas with varying levels of success. A little less than a day after Unit 1 exploded, Unit 3 exploded for the same reason, this time injuring eleven people. It turns out that those in charge of Unit 3 had failed to follow procedures and notify their workers that a hydrogen explosion was likely and so they entered the dangerous area unprepared.

Lesson: When under stress it is a natural reaction to want to do something very quickly or to get overwhelmed by all that needs to get done. If your organization has done its job well there are procedures to follow and guidelines to use, don’t forget about them.

Don’t Try to Force Your Expectations Onto Reality

Surprisingly the initial explosions at Chernobyl were not sufficient to cause the crew chief on duty to announce a containment failure. The two dosimeters capable of detecting the huge amounts of radiation that was present were either damaged by the explosion or inoperable. When a new dosimeter was finally brought it its results were dismissed with the claim that it must have been defective. Chunks of reactor fuel and burning graphite from the containment vessel lying about the facility were also ignored. Workers continued to try to pump water into the reactor and fire fighters brought in to combat the flames continued to work for hours without any protective gear. Many of them would die from radiation exposure, including the crew chief.

Similarly, the Soviet government responsible for notifying and evacuating the public consistently took an optomistic view of the disaster. The local town of Prypiat was not evacuated until two days after the incident when the public was first informed of the problem. Even when the evacuation began the townspeople were told it would only last a few days. Today, there is still a large exclusion zone around the facility that includes the entire town of Prypiat.

Lesson: Turning your back on things you don’t want to hear is a good way to get kicked in the butt. Remember, in order to properly re-evaluate the circumstances you need to first recognize that they have changed.

Clear Communication Is Critical

One of the biggest complaints about the handling of the Fukushima I disaster involve how the company that managed the plant (TEPCO) and the related Japanese government agencies communicated with the public. In the days and weeks after the incident the Japanese government changed the evacuation zone or the policies of enforcement within the evacuation zone six times. Additionally, differing and confusing radiation measurements were provided to the public, including a mathematical error telling the public that radiation levels were more than 10 million times normal, when in fact they were 100,000 times normal. Furthermore, the government consistently fought with international agencies that were monitoring the problems regarding the scale of the disaster. The International Nuclear Event Scale has categories from 1 to 7 with 7 being the worst. Chernobyl was a 7. Japanese officials only grudgingly upped the rating from the initial 4 to 5 and only increased it to 7 after it became clear the international community would rate it that way regardless of their internal stance.

Lesson: Provide a consistent, yet accurate message. If you don’t have all the fact, say so, rather than claiming knowledge you don’t possess. Being confident in your message doesn’t help if you

Don’t Play the Blame Game

Investigations into the Fukushima I nuclear disaster are still underway and are likely to be for some time. Chernobyl however has had two major investigations completed with startlingly different results. The first report, probably as a result of political pressure, placed almost all of the blame on the managers and technicians at the plant. It claimed they failed to follow regulations, preformed their duties irresponsibly, and in general ignored the clear signs of potential disaster. Because of this many safety procedures that could have been implemented at other similar facilities were marginalized. The second report, which has gained far more acceptance, while it did not completely remove human error from the causes did significantly minimize it. The second report noted such issues as:

  • A roof that was improperly built with flammable materials
  • A cost cutting measure that allowed the plant to run off of unrefined uranium, but sacrificed safety and stability
  • A design flaw in the control rods which caused them to actually be counterproductive in some instances when attempting to minimize an out-of-control reaction
  • An inadequate “culture of safety” from the top down that did not stress the needs for proper procedures and backups.

Lesson: Deal with the problem and the actual causes. Pointing fingers only keeps you from properly setting yourself up to prevent similar situations. You can’t learn from your mistakes if you refuse to see them.

The Wikipedia entries for the Chernobyl and Fukushima Daiichi nuclear disasters were extremely helpful in compiling this blog entry.

Follow me on Twitter or Add me to a Google+ Circle.

Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: