Two years following one of the worst cyber-espionage attacks on the USA, we sat down with Thomas LaRock from SolarWinds to learn about how they managed the SUNBURST crisis and came away stronger
Thomas LaRock is the Senior Technical Product Marketing Manager–or as they like to put it ‘Head Geek’–at SolarWinds. After many years as a production database administrator (DBA), Thomas joined a startup building DB performance monitoring tools as a sales engineer. That company was acquired by SolarWinds in 2013, where he has been for ten years and counting.
David Leichner, Cybellum’s CMO, and Shlomi Ashkenazi, their Head of Brand, invited LaRock to share his invaluable experience from the SUNBURST attack on the Left to Our Own Devices podcast, and discuss how it has shaped the product security discipline.
A Four Pronged Response to SUNBURST
For those who aren’t familiar, the SUNBURST breach was a recent software supply chain attack which embedded malware early in upstream components to avoid detection. Through a series of successive steps, bad actors gained access to secret data of the US federal government, NATO, UK government, EU parliament and others. One of those upstream components was SolarWinds’ Orion software.
LaRock recalled those first moments vividly: “[Our] initial response was to ask … ‘What steps do we need to take to protect customers immediately?’ We’re talking 200,000 plus customers at the time, and so the first thing we needed to do is [figure out] who’s affected…and what versions of the software [are exposed] – we had 60 something products at the time.”
The company worked quickly on four parallel tracks:
- Figuring out who was affected and mapping out what could be done to help.
- Discerning which software versions were affected, then prioritizing and addressing them.
- Actively engaging with customers to help with remediation.
- Updating and notifying authorities like the SEC and the general public.
A core tenet of SolarWinds’ incident response strategy was to not go at it alone, and to not leave customers alone. They partnered with vendors such as CrowdStrike, who initially recognized the vulnerability, Chris Krebs and Alex Stamos from KSG, and Loop One, all of whom helped work with customers to discover, patch and ultimately smooth things over.
“We engaged with the best of the best, immediately!”, said LaRock. “You absolutely need a third party to come in and have that extra set of eyes to make sure that you haven’t missed anything.”
LaRock also commended the extraordinary leadership displayed by SolarWinds’ CISO who knew exactly what to do and mobilized everyone to execute quickly and effectively. Not all companies facing an incident are as lucky.
Most Providers Are in Denial About Their True Incident Response Abilities
A recent survey Cybellum conducted in the medical device community revealed that despite the vast majority of providers believing they are ready to address a breach, when questioned about specific incident response practices and frameworks in place, they came up empty handed.
“[Many] people … haven’t really gone through a breach,” says LaRock. “They just think they know what they would do. It was Mike Tyson who said: ‘Everybody has a plan ‘till you get punched in the mouth.’”
That is especially true when a nation state is involved, like in the SUNBURST attack: “The stakes are extremely high, pressure is immense, and in the middle of an active investigation there are limits on what one can share.”
Thomas reflected on the most important lesson he would give to his younger self: “Develop a sense of empathy.” Whether it’s a database failure or a major product security breach “Nobody knows what’s going on…they’re screaming for help, but they’re also kind of screaming at you.” In the end, he explains, one is providing a service to people in dire need, don’t judge them when they’re down, don’t take it personally, just do everything you can to help.
Going Forward with ‘Secure by Design’
SUNBURST was an eye-opener for the industry and SolarWinds particularly. Within four weeks, Sudhakar Ramakrishna (President & CEO of SolarWinds) rolled out a companywide Secure by Design Initiative aimed at minimizing risk going forward. This includes, for example, running three parallel builds prior to release with stringent access and authentication principles.
The company also decided to make everything they learn public and open source so that anyone who might be susceptible to a supply chain attack could assimilate.
SolarWinds were ushered into a Secure by Design framework by a vulnerability exploit, but that doesn’t need to be the case. “You can’t wait to be secure by design– you just gotta’ get started,” urges LaRock. “[Ask yourselves] what pieces of this can we start with today, what’s the timeline, and what does that look like? It’s gonna be hard to try to do everything all at once, but what pieces can you bite off this elephant, one meal at a time?”
Ensuring security becomes part of the development process is paramount for all product developers, and that takes time. Customers need to know that even if it sometimes delays a feature release, the infrastructure is critical for making sure a product is safe, doesn’t expose them or fall apart in their hands. In the end, that’s what builds trust.
How SolarWinds Regained Market Trust
For SolarWinds, the breach wasn’t the end, but rather an opportunity to build even more confidence in their team and their product.
“The fact that a specific company had an incident doesn’t mean that company doesn’t take security seriously.” notes Shlomi. “It’s sometimes just a Russian roulette and one company can be extremely ready and still have something happen to it. You never have 100% security!”
LaRock agrees. The effort they made “to do right for customers” was what helped build confidence so quickly. “Our response is what regained their trust to turn things back on as soon as they knew if they were truly affected. There were so many things that had to fall into place for a person to be vulnerable. If you were not one of those, you could turn us back on and update your software. [It was also] our engagement with other vendors to come in for free, at no charge, and help get you back up and running.”
In addition, SolarWinds has set up a Trust Center where everything related to their products’ security is detailed.
There’s a big lesson to be learned here about openness and communication. “We’ve seen other companies in the market and in the world”, summarizes David, ”that have not been as forthcoming. And I think their reputation suffered greatly from that.”
Listen to the full interview here.
