Bloomberg News has reported the widespread discovery of Chinese-install microchips that were inserted, secretly, onto the motherboards of one of the largest computer motherboard manufacturers, Supermicro. According to the report, servers at Amazon, the DoD, Apple, the CIA, the Navy, and more; 30 in all, were compromised. It is amazing how much press this has received.
Again, according to Bloomberg, the discovery was made after Amazon had hired an independent security firm to look into the rumor, which originally was discovered about four years ago. What exactly was Amazon’s purpose for this action is unknown and why has it taken four years to get eyeballs (including the year Bloomberg put into it)?
The issue is around server mainboards, containing compromised chips, which were made by Supermicro, one of the world’s largest suppliers of motherboards. Stories vary. Some say it is a “spy chip” others say it was an on-board chip compromise. Few details are known about this since the government has been very tight-lipped since the investigation began. However, a friend of mine in the cybersecurity space said it is more likely to be an onboard chip hack than a spy chip. I will go into that reasoning a bit further on. However, first some details about what the hack does.
The most plausible scenario presented, so far, points to the baseboard management controller (BMC). This is a standard chip on all computer mainboards that has complete control of it. So compromising it has huge implications. Bloomberg says it was a chip, but I do not believe they have the expertise to understand the difference between a spy chip or a chip hack. However, either scenario is possible; the backdoor is just more likely.
Considering this breach has gotten so much attention, it seems like a good time to drill down on such compromises. What this incident does bring up is something that has been going on in chip compromises for a long time – backdoors. I am going to discuss these, in this special missive, because the implication for wireless devices are significant, especially with the move to OTA device access.
There is a bit of confusion around this. Why the backdoor hack makes more sense is because the BMC is part of the remote management system – the perfect place to enter the hardware. The BMC is a standard component on such hardware and a way to hack the board is to, clandestinely, replace it with a counterfeit one. Was this the case, or was there actually a rogue chip installed on the motherboard? I will talk about this in a bit more depth as this missive unfolds because inserting rouge chips is a extremely difficult and monumental task, logistically.
It is much more likely that the BMC controller, itself, was hacked. Such chip hacking is often done through backdoors because it is nearly impossible to detect. Whereas unauthorized or counterfeit components are much harder to insert, considering modern manufacturing processes. To insert (or replace) a chip would require compromise of hardware, software, testing, and QC, among others elements. I have a hard time believing a company as global as Supermicro would not be able to detect a rogue chip inserted, or replaced, somewhere along the manufacturing process. Therefore, I am going with the BMC chip hack theory.
Backdoors are legitimate code access points put there to allow for chip for testing, upgrading, downloading data, etc. Virtually all chips have backdoors and they have been around for decades. They are chip manufacturers’ direct access points. In most cases, they can be accessed without leaving any footprints.
Unfortunately, the dark side has discovered that as well. Backdoor breaches, caused by exploiting, precisely the definition of their design, are fairly well known and difficult to detect. Typically, hacks like this are accomplished by using an existing, or inserting a new, backdoor in the chip during the manufacturing process.
Compromised backdoors involve either modifying the chip design or adding one during the design process. Some exist for years before they are discovered. Others are inserted, without being detected, with a specific malfeasant goal in mind. Either way, they circumvent the “front” door, i.e. the normal functionality of the chip. Backdoors bypass all of the chips’ normal functionality, software, hardware, and security.
However, accessing these backdoors is not as simple as it once was. Early on, they were, generally, unsecured. However, in the last few years, manufacturers have seen the error of their ways and have begun locking them down. In most cases, today, it is a well-guarded gate. Accessing it requires intimate knowledge of chip architecture, as well as knowing the security protocol the manufacturer has layered over it and the manufacturing process.
As I noted above, these types of backdoor hardware attacks are very difficult to pull off. They require a security breach in the chip design somewhere in the supply chain to be successful. What that means is that, somewhere along the design and manufacturing process, a nefarious entity gleaned knowledge about the backdoor. This can happen when an employee on the manufacturer’s payroll, leaks design plans, or by a “double agent” employee who, literally, alters the design and open the backdoor (or in some cases, actually installs one).
Incidentally, this can happen in transit as well. After shipping, the boards are diverted to an intermediate stop, modified, and sent on their merry way with no one being any wiser. However, this takes a lot of orchestrated coordination among a number of entities, all being part of the dark side. With modern supply chain checks and balances, this is much more difficult to pull off. Thusly, it is uncommon, today.
Now, armed with some decent knowledge of what may have happened, technologically, let us move on to some fallout from this hack. So how is it that this occurred, if, indeed the story is true? I say that because there is a vehement denial by Amazon and others that this did not occur…hmmm.
There are a couple of issues here. One is that such a claim against the Chinese feeds the fires of national security, unfair trade practices and such. This gives support to the current administration’s mentality about tariffs and punishing them and is seen as a move to keep the current momentum going to support its platforms. That can be profitable both politically and economically, especially with what is at stake in the upcoming midterm elections. Another angle is simply to hurt companies and affect their bottom line, perhaps to manipulate their stock. However, I really cannot see Bloomberg being an accomplice to either of these.
There is, shockingly, strong pushback against Bloomberg for some reason. This makes me fall back on an old Hamlet adage (slightly reworked) “the victims do protest too much.” If this were true, one would assume that the companies would just take their lumps and move on – as is the usual case. Yet, Amazon Web Services’ chief information security officer, Steve Schmidt blogged, in a 400-word diatribe, that the story was filled with “so many inaccuracies.”
Apple is quoted as saying, “At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards.” Yet, according to Bloomberg, Apple found out about the problem in 2015 and essentially removed all of its Supermicro-based servers in just a few weeks. Apple has denied this allegation. Nevertheless, one has to ask why it cut relations with Supermicro in 2016?
Bloomberg is a trusted media outlet with a stellar reputation. Why would it publish such a story without extensive investigation? They claim that there were 17 sources that confirmed the hack, including six former, and current, senior national security officials as well as Apple and AWS insiders. I just cannot see Bloomberg not doing a boatload of fact-checking and legal review. Bloomberg is not noted for fabricating stories or poor/no research.
Supermicro also got on the soapbox with, “Supermicro has never found any malicious chips, nor been informed by any customer that such chips have been found… Supermicro has never been contacted by any government agencies either domestic or foreign regarding the alleged claims.”
I can see why the affected companies would panic over this. If it was found out to be true, and the statements made by Amazon, Apple, and the like were found to be lying to investors, market and regulatory ramifications would be tremendous.
This is an interesting scenario. I find it hard to believe that Bloomberg, one of the most reputable media outlets would fabricate this. On the other hand, with so much at stake for the victims, it does not make sense that they are not being honest either.
I do not really have a strong gut feeling about who is telling the truth. However, I am going to pad that feeling a bit by saying that the victims of this hack are not known for honesty or being the epitome of a model, transparent business. Therefore, it is not that this is infeasible especially in the Chinese supply chain where there is less oversight and the integrity of the chip manufacturing process is less secure (at least it was four years ago). There is also the question if it was, actually, a secret chip or the compromise of a standard component.
If I were a gambling man, I would bet that, in the end, the Bloomberg article will stand up to scrutiny and the victims will have to man-up. I just get the feeling that there is something else going on here. But then, I am not always right.