The Huawei Security Problem Isn’t the Hardware, it’s Engineers Fixing the Bugs.

I’ve been thinking about the security issues of working with Huawei equipment and Huawei the company. I’ve spoken with a number of people who, off the record, talk of working with Huawei as customers and their experiences of the product have been less than excellent but the price is low. What I’ve realised is concerning. The security risks are not in the hardware, it’s the tech support engineers that are the security risk.

Some of largest telcos in Europe have chosen to take on Huawei as a hardware supplier of telecoms equipment because the price is low. Often less than half the cost of competitors. Almost immediately after deployment  these buyers have discovered bugs in the equipment. Some of these bugs are hardware, some of them are software. Recently, Recurity Labs performed some basic security scanning (PDF) on the Huawei gear, and widely reported “The code quality is pretty much from the ’90s”. It’s mostly well-known among engineers that the Huawei software architecture is very poor, lacks robustness and quite a few vulnerabilities (although it’s improving).

This is a business security risk. Many of these companies have now built testing and validation divisions to validate Huawei code and products against a baseline. It’s still cheaper to invest in a massive testing operation and lab than to change strategy to buy Cisco or Juniper equipment.

Technical Support as a Security Risk

The actual information security risk or breach occurs when Huawei acts to solve the customer problems. Huawei typical response for large customers is a team of engineers shipped out from China to cut code and fix the problems. I’ve been told anything from 20 to 100 engineers at a time, at Huawei’s expense, will work through the identification of the bugs, make code changes and commit them back to the main team and receive new code overnight for testing.

I’ve confirmed this has occurred with three large service providers in Europe. Lets leave aside the fact that your products should be tested, have good software quality, and be reasonably bug free and even that Cisco and Juniper did similar actions in mid 1990’s when networking was in its infancy.

The security risk is the team of engineers. They will be physically located in the middle of the telecommunications infrastructure for your country. To identify and resolve the issues, they will gain full access to network architecture and design  – a security risk for network reconnaissance. They are likely to  have full access to installed equipment, including the existing equipment in addition to the Huawei assets – a security risk since they will have full knowledge of the current configuration. Those engineers will be integrated into the corporate structure  where they can  measure and map the tactical response to security events and the overall security resilience of the network.

Knowledge is the key to attacking a network.

What Security Actions are Possible ?

It seems unlikely that the Huawei equipment is insecure because that’s hard to hide. Far easier for a foreign government to slip a few key people into the support team and then export documents, diagrams and network configurations. In practice, there is very little security inside a telco or service provider to prevent such data theft from trusted personnel.

With this information, an attacker could easily map out the telecommunications network. As an attacker, knowing weak points, physical locations, logical layouts, what the target response plan is, and what equipment is all just marvellous intelligence. Many external hackers spend vast resources researching targets to find vulnerabilities and weak points in telco networks. Having internal knowledge would be invaluable.

Keep in mind, that ongoing support for the network and equipment means that the Huawei  also continually updated as to the network design and deployment. The tech support team needs to know the data to keep the close support on an ongoing basis.

Having full knowledge of the internal network for a national carrier, its configuration, the corporate support, the security regime and tools seems quite close to a major compromise of national security to me.

Mitigations ?

All development is performed in China and any background check of Chinese nationals from Huawei’s offices would obviously be suspect. And note that performing background checks on US nationals in China would also be hard to validate.

It would be very difficult to build information barriers to prevent data leakage to the your equipment provider. They simply cannot help you without that information.

The EtherealMind View

The hype around Huawei having back doors in their network equipment is fanciful. If I was the CTO for a telco, I’d be much more concerned about service integrity and operational cost of poor software quality since Huawei product quality is widely discussed among engineers as poor or sub-standard.

People are right to be concerned about National Security but only because Huawei engineers could gain access to a lot of data about the telecommunications infrastructure that underpins modern life. This is the same a security reconnaissance for “cyber attack”. But I’d also note companies with Cisco products will equally have the same risk. I imagine that sales of Cisco equipment to Chinese government is limited for exactly the same reasons but we just aren’t hearing about that in the major press in the Western world.

To measure the national security risk, consider that for recent project for a very large Point of Sale network, I was informed that three days without electronic transactions would lead massive economic failure and probably rioting and potential societal collapse within five days since nothing could be paid for. Commercial services are critical infrastructure like electricity and water.

As an attacker, knowing where to focus to create an event like this is exactly the information I want. And that’s the sort of information that  engineers could easily obtain when located at customer sites and resolving their problems. On this basis, we are right to be concerned about security and ensure that oversight is applied to Huawei as a business but not as a product. And, conversely, US network vendors like Cisco & Juniper should have the same security applied to their engineers in certain geographies.

The issue is Information Security not physical security.

References: Risky Business #250 — Hack it like it’s 1999

  • chrismarget

    It wasn’t stated explicitly, but I got the impression from that episode of Risky Business that the 90’s vintage bugs were actual Cisco bugs from 20 years ago.

    If that’s the case, is there an explanation for it other than stolen code? Replicating functionality is hard enough, but bugs too?

    • Etherealmind

      No. What he said was that these bugs are the same as for operating systems from the 90’s. That is, the software architecture is based on older operating system designs and that the software development is likely not applying good security practices to solve these problems. Importantly, these are ‘known’ types/families of vulnerabilities that are relatively easy to scan and detect.

      Have a read of the Recurity presentation which helps to make it clearer.

      • chrismarget

        In the 90s and today, the only time that multiple platforms exhibit the same bugs is when they use the same code.

        Lots of *NIX systems had bugs in sendmail, bind, etc… because they were all running the same code.

        To exhibit the same bugs as an old IOS seems strange, unless Huawei and Cisco were both using some ancient free implementation of (say) in.fingerd.

        I’m off to find the presentation…

      • chrismarget

        Okay, so I’ve just read the presentation, thanks for the pointer.

        On the one hand, it explicitly says that some versions contain actual Cisco code, including Cisco bugs. Bummer that.

        On the other hand, the bulk of the message is about 90’s style sloppy coding practices: handling calls to sprintf(), heap overflows, direct object references in the web UI, etc…, so it’s not that the specific bugs are from the 90’s (as I misunderstood), but that the code contains sloppy programming practice that should have been phased out 15 years ago.

  • Mrs. Y.

    I think the most important takeaway from this is the part of the CIA triad that everyone in security seems to forget: availability. If critical equipment isn’t stable, then #FAIL. Good business continuity practices continue to be a pipe dream in most organizations.

  • Lindsay Hill

    It’s not just a case of bringing in engineers to resolve a specific problem – the Huawei model seems to be based around pricing their hardware low, but basically requiring their engineers to implement it. Then they pay their engineers less than the going rate for local engineers.

    From my experiences working with the equipment, getting access to knowledge bases was a pain – and then many articles are only in Chinese – and getting updated code was almost impossible. For some reason, even though we had a large installation of Huawei kit, we couldn’t just go to a webpage for a specific product, see what the current release was, and download it. No, we had put a request through a Huawei engineer to get an updated image, and that could take some time. Plus you only knew there was updated code because you’d seen a passing reference to it in a KB article. Getting simple stuff like release notes could be a big challenge.

    This was a couple of years ago, and it’s possible they’ve addressed some of these issues, but it seemed clear that their model did not involve non-Huawei engineers configuring code.

    Don’t get me started on their other practices too – e.g. Telco networks that would use random public allocations (NOT Huawei’s, and NOT the customer’s) within internal networks. Deploying Windows 2000 systems in 2008 – with no plan to upgrade, etc.

    I think what these recent findings are showing is that culturally Huawei has got a long way to go before they can really deliver. I’m sure it could be turned around, but these seem like the sorts of issues that take years to turn around – c.f. Microsoft and their changing security practices over the last decade.

    But hey, it’s cheap, right?

  • Pat Cable

    I’ve always suspected that the concern with Huawei were in the form of:
    – Suspect hardware (Sure, give the code, but what’s going on in hardware? They’re not going to give away any of their IC-related VHDL, etc.)
    – Classified information that we’ll never know of (The presence of classified data can’t be discussed, and some .gov agency probably has some Huawei kit installed, and coming out and saying “we’ve got this hardware that we’ve determined it’s not safe” is likely enough to at least put it at collateral low level of classification)

    You bet other countries worry about this same thing coming from us, too. It’s part of the game.

  • Will Hogan

    “for recent project for a very large Point of Sale network, I was informed that three days without electronic transactions would lead massive economic failure and probably rioting and potential societal collapse within five days since nothing could be paid for”

    That must have been cool to be a part of. No sweat?

  • cjinfantino

    And now, I’m scared.

  • Peter Glock

    you missed one detail – Recurity Labs report was not based on Huawei routers, but on H3C routers AR18 and AR29, which in the past were sold as OEM by Huawei.
    Right now those routers are still being sold by US company Hewlett-Packard, as HP MSR-series.

    The system used on this routers is not the same VRP which is used on Huawei own routers.
    Other than that I agree that Chinese should not buy Cisco and Americans should not buy Huawei and HP. But not because of security, rather becuase of politics and US inability to coexist peacefully with another superpower.

    • sternhead

      What a stupid conclusion. So China is now coexisting peacefully with the neighbors, and has an interest in the free flow of information among its own populace so nationalism won’t get out of hand? No, China is turning downright belligerent and remains untrustworthy as ever. So it is about security after all.

  • thema27

    I did not know the magnitude of the threat until I read the second-to-last paragraph…
    Also, thanks for posting the source/reference at the end of the article.