Firstly, thanks to GitHub using HSTS on github.com (although not www.github.com), the certificate error will be fatal in Chrome and (I believe, but haven't checked) Firefox as long as you have visited GitHub previously.
(It's not preloaded HSTS so it would have to be learnt from a previous, unattacked connection.)
I know that the unbypassable errors for some sites upset the more technically minded people, but I think that incidents like this show its value.
The CloudShark trace shows what appears to be Firefox connecting to the GitHub IP address, but the server clearly isn't GitHub from the config. The server appears to be configured to accept the client's ciphersuite preference, but doesn't support DHE nor ECDHE.
The server is also only 9ms from the client - that's clearly not crossing any oceans. I'd also guess that the server is overloaded at the time because the ServerHello (which doesn't take significant processing to generate in this case) takes 900ms to come back.
Sadly, it appears to show the user overriding the certificate error and talking to the server anyway :( Hopefully that was a fresh FF install just to see what would happen (which would explain why HSTS didn't prevent the override).
Lastly, the certificate appears to be self-signed, but the Authority Key Id doesn't match. One assumes, based on "OpenSSL Generated Certificate" that OpenSSL was used, but the person may have had some trouble. I'd guess that they generated a CA certificate first (with the same Subject) and then signed the certificate in question as a leaf. Many of the tutorials that you'll find online are for that sort of setup so perhaps they weren't very familiar with X.509 certificates.
It seems even github is susceptible to this. That is, for people who type www.github.com into their browser rather than github.com. They both did the redirect wrong, as well as left off HSTS of https://www.github.com.
So your browser will warn you that you are not making a secure connection. Firefox users, for instance, will have to make 5 clicks to get through that warning and visit the page.
I think "China turns off SSL access to GitHub" might be a more appropriate title.
SSL infrastructure was already known to be best treated as globally compromised.
If this is real, then the Chinese may have collected some interesting statistics regarding the percentage of developers who don't care about certificate issues.
I didn't see the cert appear over the wire myself, but I'm in China at the moment and spent part of this week in communication with the Gentoo and Debian release engineering teams suggesting they revisit perceived issues in their respective key distribution processes and documentation (issues focused on automated validation of install media; not individual packages). Gentoo was already working on it and Debian got back to me pretty quick. I don't feel I was wasting my time now.
Really, we need a key distribution and trust anchor solution for the masses, as Moxie has spoken about, that includes 'trust agility'. IIRC the latest iteration of his proposed solution there is http://tack.io/
It's worth pointing out that many governments have MITM and warrantless surveillance systems, not only the Chinese. For more background see http://wikileaks.org/spyfiles/ which summarizes "Mass interception of entire populations is not only a reality, it is a secret new industry spanning 25 countries." and Jacob Applebaum's keynote at 29C3, https://www.youtube.com/watch?v=QNsePZj_Yks (Youtube is also banned in China).
(edit: posted beneath this thread as moxie is one of the community's most respected parties in this area and just posted, but had some extra info in here! :)
Agreed there's nothing in the data to directly suggest government involvement.
It's only "sloppy," though, when conceptualized as a MITM. China does have an extensive history of censoring access to sites, and recently censored access to GitHub entirely IIRC. It could be that they decided to block SSL access, but allow HTTP access, and this is how they implemented that.
Everything in my bones (25 years, 中文研究, China research) tells me the China government is directly involved with this. China is corrupt beyond belief, and any smaller destabilization can lead to further problems.
I agree that this may be a further extending of the "New Years train ticket" block on Github.
It may also be new toying after the recent "experiment". Leaving Github without SSL inside China still makes trouble - China's insidious corruption at the very top is subtle, incremental small steps, all designed for the "long game".
It may also be raw mercantilism ... as with Google, Twitter, and Facebook long before this.
As an sort of old china hand, china is corrupt but not beyond belief, there are plenty of countries that are much more corrupt, even India is worse than china and they even have democracy.
The level of sophistication that the GFW seems to be achieving is disturbing. We've had certificate attacks before, perhaps they are testing something out that will be deployed more broadly to solve there "gmail" problem?
It appears that my contrived scenario is incorrect. The incident seems to have occurred at several locations across the country. Assuming this is correct, it does appear to be a government action.
And this only lasted for a few hours. Users connecting to github saw perfect tlsv1 session without github sending its real cert. This means github was routed to a reverse proxy for the mitm test.
This reminds me the Firefox certificate "bug"[1] two years ago. A China certificate root server was added into trusted servers in Firefox and Chinese hackers started to submit bug report regarding this, since people don't trust certificate servers run by China government. Man-in-the-middle attack was exact what Chinese hackers worried about.
If they put this fake certificate in a certificate root server that's in the trusted server list, they can easily get anyone's account who's using affected browsers.
It's weird that they start with Github. It's not a website that's popular among human activists or any other people that China government might be interested in. Instead, it's popular among programmers and hackers, who are the main group and forces in China to help people bypass GFW to access blocked content. I suppose this attack might be what the government uses to test reaction and capability of hackers.
Spoofing GitHub's SSL certificate is a step in the direction of inserting espionage-style backdoors, as GitHub permits HTTPS read-only checkouts of repositories.
I'm not suggesting that this will be free of problems, given how particular Git is about checksums, nor am I certain what methods they would use to acquire SSH commit access for altering repository contents to affect the rest of the world.
Still, it's absolutely a necessary prerequisite to altering data from GitHub - and, if they acquire SSH keys, altering data at GitHub. Both possibilities are terrifying.
GitHub now [allows HTTPS pushes][1], so a man-in-the-middle attack on a single connection would be sufficient to push backdoors. In any case, can't the intermediary just add another SSH key to the account?
If CNNIC is complicit in a MITM attack there will be a paper trail (namely a certificate signed by them) proving their involvement. To this day nobody has produced a cert signed by CNNIC that was used for a MITM attack. There's no reason to believe that CNNIC is bad/evil/whatever other than their affiliation with the PRC.
Disclosure: I work for Mozilla, but not on security.
Thanks for the reply. Glad to see some guy from Mozilla here!
You are right there's no reason to believe that CNNIC has to be bad/evil/whatever, but we cannot assume CNNIC would never be bad/evil either. We have to look at its history, and what's behind it.
When something turns bad, it won't throw out an announcement beforehand. And we need something that can handle this in time.
I'm thinking about something like this: in addition to the trusted root server list, keep another list, e.g., less trusted root servers; besides normal update program, put a special piece of code in the browser that allows firefox team to delete certain root server from less trusted root servers on the air. By "on the air", I mean not needing a full version update, or restarting the browser. It can warn the user, but it should not be possible to canceled the deletion.
That's something that we've considered doing. I'm not sure what the current state is. I suspect there's a bug somewhere on bugzilla.mozilla.org for implementing a cert blocklist that does not require a full update.
Having been in the internet industry in China for more than 10 years before I decided to move to the US I would say CNNIC is just as evil as any government organization in China if not worse. There are some well known rules regarding CNNIC: don't buy a .cn domain, don't install any software from them and don't trust what they say.
BTW non government organizations are technically banned in China. You need to apply for a permit and you seldom get one.
If you actually look at the certificate, it is self-signed, not signed by a malicious root server. Your anecdote has nothing to do with this current attack.
You are right. This one has nothing to do with root servers. But they are already able to fake certificate national wide. It's only one step from adding it into root server.
My sense is that there is a correlation between geeks and activists, so I think it's a "logical" group to target for further "attention"/persecution. It also makes sense to want to inject flaws in project code. (Even though git has hashes, the hash can be fine if the injection happens on commit.)
I have always imagined that github would be a huge target for the chinese... for industrial espionage. they could get the source for private repositories for many startups and successful companies.
Don't forget the ability to inject code into repos on behalf of someone else.
If you've been logged into github and you told your browser to ignore the warnings then better keep a really good eye on your commit log, lest something pops up that you didn't actually put in there.
And change your password at the earliest opportunity.
Uncounted open source projects that end up being run with root privileges are on github. It all depends on how you determine value.
If value is defined as the number of machines that can be backdoored, communications intercepted and so on then it may very well be that that other form of value will be realized in good time as well.
Many developers have faster computers and more computers per household than others that aren't in IT/development.
Many developers, though they pretend to be security minded and focused at times, often have a more lax attitude in practice than system administrators, especially at home or working on personal projects.
Some developers have more usernames/passwords and certs to other servers available than non-developers would.
Some developers have access to other confidential data or newer technology that could be of practical use.
FWIW, I always try to remove the CNNIC root cert from my setups. It's not that hard to do (except for iOS devices) and I have never had an issue since I don't frequent sites that would be signed by CNNIC.
Fortunately, it looks like Chrome prevents users from clicking past the ssl errors. This Chinese chrome screenshot[1] shows that there's only a back button (for English versions of this page, see [2]). Unfortunately, it appears that IE allows users to click past the errors[3]. I'm also interested in how the 360 browser[4], which has been gaining market share in China, handles the error.
Does anyone have any theories why China would use a self signed certificate when it's very indiscreet? A lot of users will just click through if given the opportunity (around 60% in chrome before new security measures prevented it[2]), but I doubt many users with truly sensitive private repos would do this.
What bugs me about stuff like this is that there will always be mercenaries, guys just like you and me that will do anything as long as it pays. The Chinese government wouldn't stand a chance if they had to do this stuff themselves. Mercenary coders are nothing new, we have them in every country (and sysadmins, companies and so on).
But you have to wonder what goes on in their heads, what mindset would prompt you to sell out like that.
We should be worried about mercenaries when it comes to malware, Russian botnets, and the like.
In the case of the Chinese government, I believe the bigger problem is patriots. They honestly believe that they are doing the Right Thing. So did the people who wrote Stuxnet.
I kind of get what you're saying, but I think there's a lot of hubris in thinking "my profession is special, only good guys (can learn to) do it". Craftsmen have been complaining about the wrong people getting into their trade for as long as there have been crafts.
We should deny all those companies that have developed tools of censorship access to government contracts and deny them the right to work in the United States ... oops, there goes almost every single major networking manufacturer!
What exactly is he right about? China hasn't done anything that you can't do on a local WiFi connection. They grab the connection and put a self-signed cert to it. Security is still intact. The only way this hurts anyone is if they blindly trust all certificates, in which case they're screwed anyways.
There is nothing wrong in insisting the repository to be hosted on the client metal. The way things were done in the ancient times before the cloud to become the favorite buzzword of the month.
I agree. The cloud isn't new. VPN's have existed for ever as well as client-mainframe.
The perception that the cloud is somehow re-invented is in fact the saddening thing. It's just more accessible and faster than in the fast, but ultimately you can't manage infrastructure by abdication and farming it out.
There's always an uneasy balance between security and convenience.
My client would never allow their source code to be hosted in the cloud either. There's really no point fighting them on this, since they employ security and legal teams for just this purpose. The external marketing partner is not going to trump that. Ever. And there's nothing wrong with this.
There's always a precarious balance between security and convenience.
The customers own their code. They have been around longer for more oscillations between the "cloud" and bare metal than most people who have an opinion on it.
They're forward thinking enough to not embrace the cloud, or bare metal, and instead are exploring hybrid cloud technology that can run on a combination, and more importantly, move dev ops between different environments.
One reality is many people who blindly think it's ok to put everything on in the cloud, don't always know of the reality of liability, or have a relationship with any codebase and the associated IP for more than a few years.
It's more likely capacity limit than A/B testing. Returning arbitrary response is consuming more resource compared to cutting off connection (sending TCP RST).
I guess it was a planned public test or technical verification of certain larger project about live traffic decryption on national level. There really isn't a plausible political reason of github being targeted. Maybe targeting github can get data about user response or verify scalability of the infrastructure given recent high https traffic volume from China to github.
It seems strange that they would target a site whose users are exactly the people most likely to detect and understand (and make a fuss about) internet peculiarities.
I was wondering when Github was going to start supporting HSTS and 2-Factor Auth. I'm betting that it gets bumped in priority after this event. Nothing like an incident to move along security requirements!
I don't think either would help in this situation. HSTS helps prevent SSL stripping attacks, which whoever is denying SSL access to GitHub doesn't need to bother with.
2FA serves as an annoyance to phishers, but whoever is doing this network attack has direct access to your session cookie.
How would this fake cert work? It doesn't seem to originate with a root or have any associated information. Do browsers actually accept certs that look like this?
Censorship in China is a big business, for the companies that have connections to the officials, for the universities that rely on the funds, and for the officials who use this as a stepstone for their career. It is so common that you can even find many fresh graduates who worked for the GFW in the job market.
(It's not preloaded HSTS so it would have to be learnt from a previous, unattacked connection.)
I know that the unbypassable errors for some sites upset the more technically minded people, but I think that incidents like this show its value.
The CloudShark trace shows what appears to be Firefox connecting to the GitHub IP address, but the server clearly isn't GitHub from the config. The server appears to be configured to accept the client's ciphersuite preference, but doesn't support DHE nor ECDHE.
The server is also only 9ms from the client - that's clearly not crossing any oceans. I'd also guess that the server is overloaded at the time because the ServerHello (which doesn't take significant processing to generate in this case) takes 900ms to come back.
Sadly, it appears to show the user overriding the certificate error and talking to the server anyway :( Hopefully that was a fresh FF install just to see what would happen (which would explain why HSTS didn't prevent the override).
Lastly, the certificate appears to be self-signed, but the Authority Key Id doesn't match. One assumes, based on "OpenSSL Generated Certificate" that OpenSSL was used, but the person may have had some trouble. I'd guess that they generated a CA certificate first (with the same Subject) and then signed the certificate in question as a leaf. Many of the tutorials that you'll find online are for that sort of setup so perhaps they weren't very familiar with X.509 certificates.