When MuleSoft uncovered a scary security flaw in their software, they helmed responses for the IT teams of their entire customer base. Anyone with the will and skill could crack it, which spelled serious danger for MuleSoft’s users and those users’ customers. MuleSoft needed to patch the bug immediately. And they did, all while making their customers aware of an issue that could’ve flown under the radar.
MuleSoft’s reaction was prompt, and their customers appreciated the actions they took. Not only was their plan considerate, but it was thorough. Communications were the real hero of this response, and the streamlined plan MuleSoft had in place ensured success.
Communications Strategies for ITDR
â€
The MuleSoft Bug
When MuleSoft discovered this flaw in August 2019, they reacted to protect their customers. The patch was immediately applied to Mule engines and API gateways. But those with on-premise systems still ran the bad code. This proved to be a hurdle. MuleSoft needed to ensure these customers installed the fix.
The directory traversal bug enabled any file to be uploaded to a system using the middleware and planted anywhere within. Again, this was already a non-issue for off-premise customers as the patch was immediately applied. On-premise customers were at serious risk still. Many of these customers also handle very sensitive data, amplifying the urgency of the issue.
MuleSoft staff called meetings with IT teams across this segment of their customers, explained the issue, and what to do. But what was remarkable about their response?
Reporters at ZDNet caught wind of the issue after a leaked email circulated. When they reached out to MuleSoft for comment, they received an immediate, thought-out, and pleasant response from the middleware producer, asking them to delay their publishing to limit the risk of attacks on MuleSoft customers.
The ZDNet team wasn’t pushed aside, but instead received an audience with company leadership and the go-ahead to publicize the issue, albeit when it was responsible to do so. This coincided with MuleSoft’s initial plan to go public in a month, which was how much time they estimated it would take their on-premise customers to install the patch.
The “Why” Behind Planning
While talking to ZDNet, MuleSoft Chief Technical Officer Uri Sarid stated that while smaller vulnerabilities can be patched through a regular deployment schedule, “the ones that call for urgent action that are being disclosed before being publicly disclosed, those ones there aren’t any standard procedures that companies can follow.”
But there was a basic understanding of what needed to be done here. The MuleSoft team was aware of the danger this backdoor presented, and sailed ahead with a particular course of action. They offered persistent customer support and followed up with their customers to ensure they were taking action. Then, once MuleSoft was sure the vulnerability was moot through the patch they provided, they became fully transparent through a webpage anyone could find.
Predicting a bug is impossible. And trying to guess what issues a bug will introduce is a pure waste of time. But a team can lay groundwork before one crops up. Generally, for a major security flaw, a company can control the chaos and fallout through a plan by pushing adequate communications and providing needed resources.
â€
What We Learn for ITDR Communications
So, as we can see, trying to predict bugs is useless. There is something to be said for general plans, however, and communications plans in particular.
For software providers, clear communication with customers should be a top priority, only trailing the actual fix itself. The best understanding of the issue lies with your team, who should then be able to pass it on. At the heart of this is taking the charge on the fix, and being as transparent as a bug will allow. Add a proactive effort to ensure the safety version of a software to guarantee some form of success in a response.
Something else MuleSoft’s response teaches us is the importance of having pre-existing relationships. If the best lines of communication sit with account executives, work them into your communications plans. Make them aware of the issue, the best way to explain it, and what steps for remediation they can deliver. Also, rope in business continuity, as there will certainly be some type of impact on the business if something with this much gravity happens.
A security flaw isn’t something that should be taken lightly. MuleSoft, when they recognized a significant vulnerability in their software, jumped immediately to action to ensure the issue left as small of a mark as possible.
Their example is an important one, and it’s one those involved in ITDR for any organization should internalize. Bugs and flaws always come out of the blue, but their general impacts can be anticipated. This isn’t advising that you have a fix on-deck at all times, but that communication pathways are carved out, allowing the information you parse from the issue to travel to the right people.
Despite the initial, justified, silence around the MuleSoft bug, it’s now totally public. Read more about the story at ZDNet and apply the takeaways to your ITDR processes.
