Recognized as a Leader on Gartner Magic Quadrant for BCMP Solutions, Worldwide

ITDR Communications: Lessons from a Middleware Bug

Oct 3, 2019

Back to Veoci BlogITDR Communications: Lessons from a Middleware Bug

When MuleSoft uncovered a scary security flaw in their software, they helmed responses for the IT teams of their entire customer base. Anyone with the will and skill could crack it, which spelled serious danger for MuleSoft’s users and those users’ customers. MuleSoft needed to patch the bug immediately. And they did, all while making their customers aware of an issue that could’ve flown under the radar.

MuleSoft’s reaction was prompt, and their customers appreciated the actions they took. Not only was their plan considerate, but it was thorough. Communications were the real hero of this response, and the streamlined plan MuleSoft had in place ensured success.

Communications Strategies for ITDR

The MuleSoft Bug

When MuleSoft discovered this flaw in August 2019, they reacted to protect their customers. The patch was immediately applied to Mule engines and API gateways. But those with on-premise systems still ran the bad code. This proved to be a hurdle. MuleSoft needed to ensure these customers installed the fix.

The directory traversal bug enabled any file to be uploaded to a system using the middleware and planted anywhere within. Again, this was already a non-issue for off-premise customers as the patch was immediately applied. On-premise customers were at serious risk still. Many of these customers also handle very sensitive data, amplifying the urgency of the issue.

MuleSoft staff called meetings with IT teams across this segment of their customers, explained the issue, and what to do. But what was remarkable about their response? 

Reporters at ZDNet caught wind of the issue after a leaked email circulated. When they reached out to MuleSoft for comment, they received an immediate, thought-out, and pleasant response from the middleware producer, asking them to delay their publishing to limit the risk of attacks on MuleSoft customers.

The ZDNet team wasn’t pushed aside, but instead received an audience with company leadership and the go-ahead to publicize the issue, albeit when it was responsible to do so. This coincided with MuleSoft’s initial plan to go public in a month, which was how much time they estimated it would take their on-premise customers to install the patch.

The “Why” Behind Planning

While talking to ZDNet, MuleSoft Chief Technical Officer Uri Sarid stated that while smaller vulnerabilities can be patched through a regular deployment schedule, “the ones that call for urgent action that are being disclosed before being publicly disclosed, those ones there aren’t any standard procedures that companies can follow.”

But there was a basic understanding of what needed to be done here. The MuleSoft team was aware of the danger this backdoor presented, and sailed ahead with a particular course of action. They offered persistent customer support and followed up with their customers to ensure they were taking action. Then, once MuleSoft was sure the vulnerability was moot through the patch they provided, they became fully transparent through a webpage anyone could find.

Predicting a bug is impossible. And trying to guess what issues a bug will introduce is a pure waste of time. But a team can lay groundwork before one crops up. Generally, for a major security flaw, a company can control the chaos and fallout through a plan by pushing adequate communications and providing needed resources.

What We Learn for ITDR Communications

So, as we can see, trying to predict bugs is useless. There is something to be said for general plans, however, and communications plans in particular. 

For software providers, clear communication with customers should be a top priority, only trailing the actual fix itself. The best understanding of the issue lies with your team, who should then be able to pass it on. At the heart of this is taking the charge on the fix, and being as transparent as a bug will allow. Add a proactive effort to ensure the safety version of a software to guarantee some form of success in a response.

Something else MuleSoft’s response teaches us is the importance of having pre-existing relationships. If the best lines of communication sit with account executives, work them into your communications plans. Make them aware of the issue, the best way to explain it, and what steps for remediation they can deliver. Also, rope in business continuity, as there will certainly be some type of impact on the business if something with this much gravity happens.

A security flaw isn’t something that should be taken lightly. MuleSoft, when they recognized a significant vulnerability in their software, jumped immediately to action to ensure the issue left as small of a mark as possible.

Their example is an important one, and it’s one those involved in ITDR for any organization should internalize. Bugs and flaws always come out of the blue, but their general impacts can be anticipated. This isn’t advising that you have a fix on-deck at all times, but that communication pathways are carved out, allowing the information you parse from the issue to travel to the right people.

Despite the initial, justified, silence around the MuleSoft bug, it’s now totally public. Read more about the story at ZDNet and apply the takeaways to your ITDR processes.

[Image credit: Fatos Bytyqi via Unsplash.com]

Subscribe to the Veoci Blog

Receive all the latest emergency, crisis, and continuity management news, tips, and advice

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Related Posts

ITDR Communications: Lessons from a Middleware Bug

Anyone with the will and skill could crack the bug, which spelled serious danger for MuleSoft’s users and those users’ customers. MuleSoft needed to patch the bug immediately. And they did, all while making their customers aware of an issue that could’ve flown under the radar.

Continue reading
The “Magic” of Veoci

As the IT market is replete with contenders, it’s important for companies to stand out from the crowd - getting named by Gartner in the Magic Quadrant is a big boost.

Continue reading
Business Continuity vs ITDR: What are the Differences?

In the past, you may have heard the terms “business continuity” and “disaster recovery” used in conjunction, or even interchangeably, but what do they really mean? You probably won’t be surprised to discover that they have many similar goals when it comes to recovering from an unplanned incident and restoring essential functions, but it is the nuances of their differences that are really crucial to understand.

Continue reading

Connect with us on Social Media

Join us on our journey to improve emergency, operations, and continuity management!

Veoci Facebook PageVeoci Twitter AccountVeoci Linkedin Company Page

Face crisis and continuity challenges with expert solutions designed for you and your teams.

Learn how Veoci puts you in control