64 / 88
Even as Amazon, Apple, and U.S.
o
icials were investigating
malicious microchips embedded in
Supermicro server motherboards,
Supermicro was the target of at
least two other possible forms of
attack, people familiar with multiple
corporate probes say.
The first of the other two prongs
involved a Supermicro online portal
that customers used to get critical
software updates, and that was
breached by China-based attackers
in 2015. The problem, which was
never made public, was identified
after at least two Supermicro
customers downloaded firmware—
software installed in hardware
components—meant to update their
motherboards’ network cards, key
components that control communi-
cations between servers running in
a data center. The code had been
altered, allowing the attackers to
secretly take over a server’s
communications, according to
samples passed around at the time
among a small group of Supermicro
customers. One of these customers
was Facebook Inc.
“In 2015, we were made aware of
malicious manipulation of software
related to Supermicro hardware
from industry partners through our
threat intelligence industry sharing
programs,” Facebook said in an
emailed statement. “While
Facebook has purchased a limited
number of Supermicro hardware for
testing purposes confined to our
labs, our investigations reveal that it
has not been used in production,
and we are in the process of
removing them.”
The victims considered the faulty
code a serious breach. Firmware
updates obtained directly from the
manufacturer are usually assumed
to be secure. Firmware is tailored to
specific types of computer
hardware and embedded directly
into those parts, where it provides a
narrow set of operating instructions.
Detecting attacks at this level
requires specialized security
programs, so the code is rarely
scanned for bugs. By corrupting
Supermicro’s update mechanism,
the attackers were trying to get
customers to infect themselves.
While the chip and software attacks
could be efective on their own,
security experts say these
approaches could also be used in
concert, with the corrupted network
cards amplifying the capabilities of
the embedded chips.
In its denial that a chip attack
had reached its server network,
Apple did acknowledge to
Bloomberg Businessweek
that it
had encountered malware
downloaded from Supermicro’s
customer portal. Apple said the
infection occurred in 2016, months
after the events described by
Facebook, and involved a single
Windows-based server in one of
the company’s labs. The malware
was on a network card driver,
which is distinct from firmware and
allows an operating system and a
piece of hardware to communicate.
This was the reason Apple gave for
dropping Supermicro as a supplier
later that year. “As a matter of
practice, before servers are put
into production at Apple they are
inspected for security vulnerabili-
ties and we update all firmware
and software with the latest
protections,” Apple said in its
statement to
Businessweek
. “We
did not uncover any unusual
vulnerabilities in the servers we
purchased from Super Micro when
we updated the firmware and
software according to our
standard procedures.”
However, a person familiar with
Apple’s investigation says that
around the time the company
discovered malicious chips, it also
found a more serious problem with
network cards on Supermicro
motherboards. Some Supermicro
servers had network cards that
came with outdated firmware, so
the machines that were delivered
to customers contained a critical
security vulnerability that had
been fixed in newer versions. This
was potentially a third avenue of
attack. Security experts say
attackers could take advantage of
a known firmware vulnerability in
the same way they would use a
more traditional software exploit.
Once inside a target network,
hackers could seek out servers
with the dated code and easily
infect them.
More Elements of the Attack
Bloomberg Businessweek
October 8, 2018
57
further instructions, operatives could hack those comput-
ers to identify others who’d been afected. Although the
investigators couldn’t be sure they’d found every victim, a
person familiar with the U.S. probe says they ultimately con-
cluded that the number was almost 30 companies.
That left the question of whom to notify and how.
U.S. oicials had been warning for years that hardware
made by two Chinese telecommunications giants, Huawei
Corp. and ZTE Corp., was subject to Chinese govern-
ment manipulation. (Both Huawei and ZTE have said
no such tampering has occurred.) But a similar pub-
lic alert regarding a U.S. company was out of the ques-
tion. Instead, oicials reached out to a small number
of important Supermicro customers. One executive of
a large web-hosting company says the message he took
away from the exchange was clear: Supermicro’s hard-
ware couldn’t be trusted. “That’s been the nudge to
everyone—get that crap out,” the person says.
Amazon, for its part, began acquisition talks with an
Elemental competitor, but according to one person famil-
iar with Amazon’s deliberations, it reversed course in the
summer of 2015 after learning that Elemental’s board was
nearing a deal with another buyer. Amazon announced
its acquisition of Elemental in September 2015, in a trans-
action whose value one person familiar with the deal
places at $350 million. Multiple sources say that Amazon
intended to move Elemental’s software to AWS’s cloud,
whose chips, motherboards, and servers are typically
designed in-house and built by factories that Amazon
contracts from directly.
A notable exception was AWS’s data centers inside
China, which were illed with Supermicro-built servers,
according to two people with knowledge of AWS’s opera-
tions there. Mindful of the Elemental indings, Amazon’s
security team conducted its own investigation into AWS’s
Beijing facilities and found altered motherboards there
as well, including more sophisticated designs than they’d
previously encountered. In one case, the malicious chips
were thin enough that they’d been embedded between
the layers of iberglass onto which the other components
were attached, according to one person who saw pic-
tures of the chips. That generation of chips was smaller
than a sharpened pencil tip, the person says. (Amazon
denies that AWS knew of servers found in China contain-
ing malicious chips.)
China has long been known to monitor banks, man-
ufacturers, and ordinary citizens on its own soil, and
the main customers of AWS’s China cloud were domes-
tic companies or foreign entities with operations there.
Still, the fact that the country appeared to be conducting
those operations inside Amazon’s cloud presented the
company with a Gordian knot. Its security team deter-
mined that it would be diicult to quietly remove the
equipment and that, even if they could devise a way,
doing so would alert the attackers that the chips had