WebWorker Performance Benchmarks

When Nolan Lawson published his article on High Performance WebWorker Messages, I was immediately stuck with how disparate his conclusions and data were from my own experience.

However, I've mostly used WebWorkers in a mobile context (Cordova applications), which by and large means my experience has been with Safari. Could it really be true that Chrome and Firefox were so bad that no method of transfer is better than JSON.stringify?

This is something I needed to know, because I'm working on Skyrocket, a build system and interface to make WebWorkers stupid simple to use in Ember Applications.

Lawson's benchmarks were incomplete in a few areas, so I set out to write my own. One of the areas I wanted to test was the MessageChannel API.

MessageChannel is a way to open a dedicated message port between two iframes, windows, or a window and a worker. Generally speaking, using MessageChannel allows you to differentiate communication with separate processes running in a single worker, but shouldn't have a noticeable overhead when compared directly against postMessage.

My test also looks at and prints which message transfer options are available to the browser being tested.

Note: If transfer is true and JSON is true then Cloning is likely actually true as well, but the only way to know for sure that it is true is if JSON is true and Cloning is false. This means that for most browsers, the json transfer test is testing structured cloning. Some browsers allow json transfer but utilize stringification in their implementation, not cloning.

Test Setup

I wanted to test "round trips" of various transfer methods, as that's the scenario I deal with more often. These benchmarks are completely invalid if your data flow is one directional or if you are dealing with a data type that is naturally a "blob" or binary in nature.

If your data is one directional, or a blob/binary in nature, you will very likely get the best results sticking to Transferable Objects, despite their poor performance here. Their performance here is subject to an expensive serialization/deserialization process required when you need to transfer JSON-like data.

I measured start and end time with window.performance.now, and averaged together the results of 10 test runs of 1000 trips for each message transfer method.

I tested two different forms of messages, a "simple" object consisting of a wrapper with a payload consisting of an array of 100 strings, and a "complex" object using a similar wrapper but containing an array of 100 objects.

The idea behind these payloads was to test the sorts of JSON payload that might be expected from a server fetch or as part of a "record array".

Each test measures the total amount of time it takes to make a round trip beginning on the main thread. This means time it takes to serialize the payload into something the transfer method can use, deserialize it on the other end (ostensibly to use it), re-serialize it for transfer back, and de-serialize it again on the main thread. E.G. this measure the time it takes to complete a full request-response cycle.

This setup biases the test in several meaningful ways that won't be true for many forms of worker use. Scenarios for which this is NOT a good benchmark include:

  • operations that don't expect a response
  • unidirectional operations (serialization cost only paid once, likely only by worker)
  • operations that are triggered by a simple signal from the main thread but for which the worker will return a heavy payload (serialization cost mostly within the worker).

This benchmark also does not currently delineate how much of the consumed trip time is spent in processing versus transfer, and in all likelihood that is something useful to benchmark as it would help to determine which transfer method is best for unidirectional or signal based work.

Results

Chrome 47

The first browser I tested was Chrome.

(SIMPLE) strings: 0.19966599999999943ms
(SIMPLE) json: 0.3174949999999798ms
(SIMPLE) transfer: 0.39632850000004594ms
(SIMPLE) strings-channel: 0.43924600000006253ms
(SIMPLE) json-channel: 0.5812129999999701ms
(COMPLEX) strings: 0.36659750000003766ms
(COMPLEX) json: 1.2953260000000375ms
(COMPLEX) transfer: 0.8302744999999488ms
(COMPLEX) strings-channel: 0.597058499999988ms
(COMPLEX) json-channel: 1.6076915000000294ms

One thing to notice with Chrome is there is no result for transfer-channel. It turns out Chrome 47 does not support transfer within Channels, although a note in the developer console says that the behavior exists behind a feature flag, so support is coming.

The main takeaway from these results is that in accordance with Nolan Lawson's results, JSON.stringify is significantly faster for Chrome, especially as complexity increases.

The surprising takeaway is that there is a significant performance degradation when using the MessageChannel API.

Firefox 44

(SIMPLE) strings: 0.2347229999999692ms
(SIMPLE) json: 0.2727284999999937ms
(SIMPLE) transfer: 0.4038820000000481ms
(SIMPLE) strings-channel: 0.351715000000041ms
(SIMPLE) json-channel: 0.40030199999998484ms
(SIMPLE) transfer-channel: 0.5252480000000416ms
(COMPLEX) strings: 0.38572549999997763ms (COMPLEX) json: 0.8824694999999622ms
(COMPLEX) transfer: 0.7006129999999862ms
(COMPLEX) strings-channel: 0.5283189999999833ms
(COMPLEX) json-channel: 1.0413520000000143ms
(COMPLEX) transfer-channel: 0.9039295000000458ms

With Firefox the performance difference is less drastic, and the MessageChannel degradation less severe but still large. JSON.stringify smokes json for more complex data though, and thus continues to be the recommendation.

Safari 9.0.3

This takes us to Safari, where I've done most my work in the past. Nolan chalked his Safari results up to "Safari JS perf is usually much faster." And while this is true, if that were all the difference were we would expect to see a similar ranking of methods and degradation when using channels.

(SIMPLE) strings: 0.20990550000000194ms
(SIMPLE) json: 0.20715250000002347ms
(SIMPLE) transfer: 0.256632000000005ms
(SIMPLE) strings-channel: 0.20892950000000982ms
(SIMPLE) json-channel: 0.1936350000000046ms
(SIMPLE) transfer-channel: 0.25974350000002744ms
(COMPLEX) strings: 0.4001925000000006ms
(COMPLEX) json: 0.3805344999999939ms
(COMPLEX) transfer: 0.4923319999999921ms
(COMPLEX) strings-channel: 0.41156050000001115ms
(COMPLEX) json-channel: 0.40011600000000147ms
(COMPLEX) transfer-channel: 0.5130644999999946ms

Safari is faster across the board, with json edging out JSON.stringify and json-channel being fastest overall on simple cases and second fastest (to json) on complex cases. Safari's slowest times looks like some of the faster times from Chrome and Firefox.

It's safe to say that Safari has more going for it than just better Javascript performance, their MessageChannel, Structured Cloning and Transfer algorithms seem to all be implemented better.

Mobile Safari (iOS 9.2.1)

And just for good measure:

(SIMPLE) strings: 0.500940999999994ms
(SIMPLE) json: 0.42557500000004406ms
(SIMPLE) transfer: 0.5197065000000133ms
(SIMPLE) strings-channel: 0.4607855000000189ms
(SIMPLE) json-channel: 0.4255719999999947ms
(SIMPLE) transfer-channel: 0.5051360000000369ms
(COMPLEX) strings: 0.7861170000000368ms
(COMPLEX) json: 0.7694750000000201ms
(COMPLEX) transfer: 0.8752450000000256ms
(COMPLEX) strings-channel: 0.8238915000000049ms
(COMPLEX) json-channel: 0.7953659999999528ms
(COMPLEX) transfer-channel: 0.9154860000000141ms

The Mobile Safari are in line with the desktop Safari albeit 2x slower (expected, this is mobile after all). And yeah, you are reading this right. Mobile Safari is close to or beats Desktop Chrome in performance in most of these comparisons.

Conclusion

With these benchmarks in hand, I'd like to amend Nolan Lawson's conclusions.

  1. Don't use MessageChannel when speed matters. The overhead is likely much more significant than doing some of your own delegation.

  2. If you are focusing on mobile Safari (eg. a Cordova app), just directly transfer your json. Since these benchmarks are likely to change over time, and since it's already a good idea to pass a few messages back and forth when launching a worker to warm it up and do some feature detection, potentially you could time these early messages and from them determine the best transfer method while setting up your connection.

Chris Thoburn

Chris (@runspired) is a Javascript Engineer specializing in mobile performance. He's also an outdoor enthusiast, and a runner.

About Isle of Code:

We’re a Javascript/Ember firm in Toronto + Chicago, est. 2012. We provide on-demand development for Mobile Apps, Websites, iBeacons/ hardware, existing codebases and team augmentation / Ember leadership.

read more