— Laurie Voss in Melbourne (@seldo) June 20, 2019
There are 11 million JS devs, 99% of whom use NPM. Not only do we use JS more, we are building bigger things; and 97% of the code comes from NPM. The average app uses over 2000 packages!
Open source JS is massive and a huge win, but it is not free. While NPM started out as a package manager, it has become a security company – because there was so much insecure stuff going on they had to do something about it.
What do devs do to ensure the security of their code? Most do code review, many do automated scans, a few get third party audits, and 23% do NOTHING.
How does Laurie define an incident?
Anything that can allow a malicious act to hurt users is a security failure. Even if nobody gets hurt.
Writing secure code is essentially the same in any language, including knowing where third party comes from.
— Ben Ritthichai (@BenRchai) June 20, 2019
- Angry bears ? – literally an angry animal that bursts into your office. This would be exceptionally severe, but has a very low likelihood.
- Denial of service – NPM has 99.98% uptime, people put a lot of effort into the downtime scenario even though it is rare.
- Malicious packages – the amount of malware is going up, but the numbers are very low (less than 0.1% of NPM publishes are malicious; and nearly all of those are detected and deleted automatically before they even go live)
- Accidental vulnerabilities – this is worth a lot of attention, because accidental vulnerabilities happen relatively frequently AND people don’t update their dependencies. 33% of installs include vulnerable packages despite warnings!
- Social engineering – this is a huge problem, and hard to guard against
- Compliance failures – eg. using packages with prohibited licenses (many simply unrecognised like WTFPL)… this is at epidemic levels
Case study: financial industry. 8 of the world’s largest banks were analysed and discovered to have vulnerabilities in 3% of the packages they were downloading. They also use licenses they claim they prohibit.
If banks have tons of money and still get it wrong, what about the rest of us? JS snuck up on enterprises, they didn’t realise how much JS they were creating and using. Also the way security works in JS has to be different – you cannot use manual processes for 25k packages. Blacklists and whitelists don’t work either, you just can’t keep up.
The scale and nature of the JS ecosystem demands automation of security and compliance.
Security case studies:
left-pad: the number of people who were directly affected is actually incredibly low! It’s famous but the impact wasn’t as big as people imagine. What went wrong? The package name ‘kik’ was given to the chat client with millions of users, instead of the existing owner, Azer, whose ‘kik’ had zero downloads. Kik tried asking nicely (Azer told them very literally to ‘fuck off’); so when asking didn’t work Kik they tried vague legal threats; and when that didn’t work either they eventually emailed NPM with their case. NPM made a huge mistake and gave them the name. In protest Azer removed every package he had, including left-pad… and thousands of packages broke, including some really big ones. In the end someone just re-published left-pad, because it was WTFPL licensed and that was fine. Nodejs now has
String.prototype.padStart()and yet people still use left-pad. NPM no longer allows unpublishing packages after they’ve been up for 24 hours.
event-stream: This one was much sneakier. It would only execute at runtime inside a specific application, a cryptocurrency wallet. The attacker got in by gaining trust as a legitimate contributor, before taking over the package from a very busy maintainer – then they injected their attack. On the plus side, 11 million users are a pretty good detection system and it was found. But we need to remember that open source maintainer burnout is an attack vector. The maintainer was too busy and exhausted to do a deep background check on the attacker, they just gratefully took a useful contribution. This also surfaces an issue where a huge proportion of packages are maintained by a very small group of prolific publishers.
electron-native-notify: This one was detected by the NPM security team using internal tooling. Their approach was similar to event-stream, they first submitted a useful package; then added a malicious payload in a minor update. Sadly this means we really can’t assume open source contributors have good intentions, the rate of attack is going up.
Maintainer burnout is an attack vector: millions of people depend on exhausted open source developers. Hackers offer to help, earn trust, get publishing access, and release deliberate vulnerabilities. #code19 @seldo pic.twitter.com/zqRLDAfOXi
— Jason O'Neil (at #code19) (@jasonaoneil) June 20, 2019
So what is NPM doing about it?
- In response to left-pad they changed the unpublish policy. Barring extreme cases (active security threats and legitimate legal takedowns) you can no longer take things down.
- They also employ lawyers to ward off vague and dodgy legal threats, which might have defused left-pad.
- They’ve added
npm-auditwhich automatically scans for vulnerabilities; and
npm audit fixwill automatically update to secure versions where possible, based on SemVer. There is a force option to ignore SemVer.
- You can set npm to fail tests if audit detects failures; and the severity level that should cause that failure.
- 2FA support – although just 7% of authors use it, over 50% of packages are covered by those highly-prolific authors (most of those use 2FA).
- Automatic token revocation – if you accidentally publish your token in a public repo, it gets revoked.
- Registry-signed packages
- Simple as it sounds, the “Report Vulnerability” button is on every NPM page. Reports are actively reviewed.
- The NPM Security Team
- Automated threat detection
- SSO support in the NPM Enterprise product
Future steps? The goal is zero malware. They will continue to grow their security team, machine learning is being used. They won’t be doing static code analysis however, as it is provably impossible to detect security issues that way.
Addressing social engineering is a big challenge. The community is too big to rely on “knowing each other”; and that also reduces diversity anyway. NPM is looking for ways to improve social signals without turning it all into a stupid game.
Maintainer burnout is a problem that everyone can fix. We need to find a way to compensate maintainers appropriately; and prevent them becoming so tired they make significant mistakes.
Security is hard, the world is scary, your paranoia is justified. But you can help by reporting vulnerabilities, or simply contributing and taking the load off maintainers.