JavaScript debugging the hard way

Error on line 1, column 6532112 of bundle.js? Out of memory error trying to load a CPU profile into the Chrome debugger? Two minutes to see wait and see if a change you made fixed a bug?

While upgrading our complex web application from Webpack 3 to Webpack 4, we ran into these of challenges and more, that required adapting my use of debugging tools and techniques to deal with the scale of the problem.

As your codebase grows the debugging techniques you apply need to adapt – things you take for granted like setting a quick breakpoint, reproducing a bug in seconds, or loading a CPU profile into the Chrome Dev Tools start to break down.

This talk dives into the different issues we encountered, and how we debugged and fixed them – providing practical examples, and tips, for debugging JavaScript in the Browser and Node as your codebase grows, that provide a valuable addition to any developer’s toolbox.

JavaScript debugging the hard way

Marcin Szczepanski, Principal Developer Atlassian

While this talk is about debugging, it’s mostly the story of a particularly difficult Webpack upgrade. While that might seem an unusual thing to have trouble with, the sheer size of the JIRA codebase turns it into a challenge.

JIRA’s codebase had 2,018,981 lines of code as at 2020.07.09 (not including external dependencies); and most of this goes through the webpack build.

This produces 200megs of total JavaScript assets, and 300megs of sourcemaps. Thankfully this is served in a very granular way, but the total is pretty big. The build took about 20 minutes.

So they needed to upgrade from Webpack 3 to 4 and scheduled a week to do it, based on previous experience. As you can guess from the fact Marcin is here talking to us… it did not take a week.

Normally you’d start with a minimal local build to see if the config is working, then push up to the CI environment for a full build.

But while it worked locally, it was timing out in CI. The main difference was that CI was not just emitting code, it was generating sourcemaps and running minification as well.

But to complicate matters there was a problem that made local builds run for over two hours, making repeated test runs impractical.

So they turned to profiling to work out what was going on in CI.

The most common types of profiling are memory and CPU; this talk will focus on CPU profiling, using flame charts and tree profiling.

Usually they’d do this locally by running webpack via node and using the node inspector:

node --inspect-brk ./node_modules/.bin/webpack

…but it was running out of memory. Eventually they found the cpuprofile-webpack-plugin which allowed them to generate a CPU profile in CI.

Comparing a typical flame chart for a normal build, they found the sourcemap stage was taking far longer than normal. The tree table view confirmed that the split function was the source of the problem.

They tracked that back to the sourcemap library, finding it was calling a regular expression a lot and triggering a great deal of garbage collection.

Then they discovered they were not the first people to find the problem; and they’d fixed it in sourcemap – but that hadn’t made it into the Webpack codebase due to breaking changes. So why not backport it? Someone tried that too and the fix wasn’t accepted.

So what do you do when one of your upstream dependencies won’t update and you need the new version?

Forking was impractical in this case as you’d have to fork everything that referenced it. So instead they used patch-package to install the fixes during bundling.

So happy days, the build was working again and back to a normal build time. Time to get this in front of internal customers and that worked, but as soon as it went live to a small initial cohort people started reporting that the navigation wasn’t loading.

Browser dev tools to the rescue? They discovered an undefined error related to the broken feature; and after some digging through minified code, tracked the problem back to the webpack runtime.

Debugging this is difficult. A normal breakpoint is impractical as it’s called too many times, so you have to use a conditional breakpoint. They use a type check to only trigger the breakpoint when the undefined condition is met. This revealed which module was failling to be found.

There’s also a log point, which is a conditional console.log, which would also have located this issue.

The problem ultimately is that the component wasn’t being bundled in Webpack 4, when it was in Webpack 3. They were able to bundled up all the information they had and provide it in a Webpack issue; and ultimately a new version of Webpack fixed the problem.

Key lessons:

  • understand the tools available to you
  • find out if others have hit the same problem before you
  • raise issues (just do your homework first!)

@marcins