NASA finds ‘fundamental’ software problems in Boeing’s Starliner spacecraft

WASHINGTON D.C., Feb 8, 2020, WP. Investigators probing the botched flight of Boeing’s Starliner spacecraft in December have found widespread and “fundamental” problems with the company’s software that could have led to a disastrous outcome more grievous than previously known, the agency said Friday, The Washington Post reported.

Boeing is now reviewing all 1 million lines of code in the capsule’s computer systems, officials said. How long that review will take is uncertain, Boeing officials said.

The discovery of widespread software problems in the Starliner spacecraft is reminiscent of the issues that surfaced in the aftermath of the crashes of two Boeing 737 Max airplanes that killed 346 people and led to the plane’s grounding since early last year. Doug Loverro, the head of human exploration for NASA, said he could not speak to what, if any, connection there might be between the Starliner’s software problems and the issues with the 737 Max.

But he said the discovery of widespread issues with the Starliner software indicated “we have a real breakdown of the software process.”

“We don’t know how many software errors we have — if we have just two or many hundreds,” Loverro said. In an interview, he added that the “bottom line is that industry is very bad at doing software.” Boeing, he said, very well may have had “a good program, but it was not executed correctly.”

Speaking in unusually blunt language, NASA officials also acknowledged that the space agency had failed to properly police Boeing’s work and that the checks that were supposed to discover such problems failed repeatedly.

No astronauts were aboard the Starliner capsule during its test flight in December, but the software malfunctions could have caused what a safety official called a “catastrophic spacecraft failure.”

In the days after the flight, NASA and Boeing officials repeatedly sought to emphasize the things that went right with the mission, even though the spacecraft failed to dock with the space station, one of its main objectives.

But speaking Friday, a day after a NASA safety advisory panel warned that the problems were far more severe than previously known, they acknowledged the severity of the issues would require a wholesale review of Boeing, its safety procedures and the way NASA oversees Boeing’s work as it prepares to fly humans to space for the first time since the space shuttle fleet was retired in 2011.

Officials at NASA and Boeing said they were able to fix the problems that plagued the maiden flight of the Starliner, which landed safely two days after it lifted off from Cape Canaveral. But they said there were multiple failures along the way that, if not caught in the nick of time, could have led to a disastrous outcome.

In a blog post Friday, NASA pointed the finger at Boeing, saying, “there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects.” It added that those problems could have had serious consequences and “led to risk of spacecraft loss.”

During the call, however, Loverro said that NASA had failed as well in policing a contractor that many have said has grown too cozy with the agency.

“NASA oversight was insufficient — that’s obvious,” he said. “And we recognize that.”

Initially, Boeing said that a software issue caused the spacecraft’s timer to be off by 11 hours. That, in turn, meant the thrusters that were to put the Starliner on a trajectory to the space station failed to fire.

The timing issue was caused because the spacecraft’s computer system was supposed to sync with that of the rocket booster — but only after the countdown to launch began. For some reason, however, the two computers synced before the countdown, resulting in the error, said Jim Chilton, the head of Boeing’s space division.

The revelation this week of yet another software problem indicates the issue was not an isolated one.

After discovering the issue with the timer, Boeing officials “went hunting” for any other software problems while the Starliner was in flight, Chilton said.

They found one — a big one that Chilton said would have gone undiscovered if they had not had the earlier problem.

In the second case, the issue would have caused the wrong thrusters to fire during the separation of what’s known as the service module from the crew module. Company officials grew concerned that if the wrong thrusters fired the two capsules could have collided, leading to an array of significant problems, from damaging the capsule’s heat shield, to sending it tumbling off course.

While it was not clear what exactly would have happened, “nothing good could come from those two spacecraft bumping,” Chilton said.

Boeing officials were able to quickly come up with a software patch, beam it up to the spacecraft, which prevented the wrong thrusters from firing. The spacecraft then landed safely the next day.

NASA has repeatedly said it has been working side-by-side with both Boeing and SpaceX, the other company under contract to develop a spacecraft to carry humans into space, to ensure their spacecrafts meet requirements and are safe to fly. But a series of problems have plagued the $6.8 billion program, which is now three years behind schedule.

Last year, during the test of the Starliner’s abort motors, one of the main parachutes failed to deploy because a pin was not properly secured to a smaller drag chute. The pin was underneath a protective sheath and out of sight. Boeing has said it intends in the future to verify that the pin is secure by pulling on it.

NASA announced this week that it plans to reverse itself and conduct a more thorough review of Boeing’s safety culture. In 2018, NASA announced it was launching “invasive” probes of both Boeing and SpaceX. The reviews were prompted after SpaceX CEO Elon Musk took a puff of marijuana on a show streamed on the Internet. NASA confirmed it had proceeded with a full review of SpaceX, but that it had decided to perform a far more limited review of Boeing, which it had worked alongside for decades.

On Friday, NASA officials said that while they had been planning on a full review of Boeing even before the flight, that review took on a greater sense of urgency after the mission was marred by the software problems.

Investigators have identified the cause of the problems and have a sense of how to fix them, NASA said. But the agency said it was “still too early for us to definitely share” the causes and corrections. Still, it said that investigators have identified 11 remedies that will need to be implemented.

It is unclear whether NASA will force Boeing to redo the test flight without astronauts before allowing crews to fly on Starliner.

The agency said it hopes to have an answer on that within a few weeks.

Christian Davenport covers the defense and space industries for The Washington Post’s Financial desk. He joined The Post in 2000 and has served as an editor on the Metro desk and as a reporter covering military affairs. He is the author of “The Space Barons: Elon Musk, Jeff Bezos and the Quest to Colonize the Cosmos” (PublicAffairs, 2018).