V8 plan for Node.js LTS Carbon (A potential path to TurboFan + Ignition)

[MylesBorins]

EDIT: CTC voting tally for this issue

Problem

V8 5.9 will be the first version with TurboFan + Ignition (TF+I) turned on by default. As parts of the Node.js codebase have been tuned to CrankShaft, there will be a non trivial amount of churn to adapt to the new pipeline. This also creates a security risk as CrankShaft and FullCodeGen are no longer maintained by the V8 team or tested by the Chrome security team.

If TF + I lands in Node.js 9.x backporting any changes to Node.js 8.x is going to prove extremely difficult and time consuming.

Below are three proposals of how we can approach this problem. To anyone in @nodejs/collaborators, and the community at large, we would love to hear your opinions on this. Further we really want to get some real world benchmarks, so if you have a way of testing builds to get non micro benchmarks please chime in and I'll get you binaries to work with.

Three Proposals:

1. Target 5.7 or 5.8 in 8.x release with standard APi / ABI:

• Do not change anything.
• Pros:
  • Uses classic pipeline based on FullCodeGen and CrankShaft which has an execution profile that is well known
• Cons:
  • V8 will no longer be maintaining FullCodeGen+Crankshaft pipeline
    • Backports from V8 will be much harder
    • Chrome security group no longer be testing Crankshaft pipeline
  • With Turbo Fan + Ignition landing in 9.x there will be a lot of churn. This will make backporting much harder
    • Backports will require different performance profiling due to different pipeline

---

2. Target 5.8 in 8.x release with forward compatible ABI to 5.9, upgrade to TF+I as semver minor:

• Land 5.8 with 5.9 forward compatible API / ABI in 8.x beta
  • We could potentially do so with TF+I turned on, this has been tested by V8 team
  • 5.8 will go stable April 25
• Upgrade to 5.9 with TF+I when it goes stable in early June.
• Backport from upstream to 5.9 as changes land
• Pros:
  • Uses Ignition+TurboFan which will be the supported pipeline from V8 for the foreseeable future
  • Lower risk of unknown security exploits
  • Easier to backport
• Cons:
  • TF + I will be a newer pipeline and the performance profile is not yet clear
  • There are potentially 10’s of thousands of lines of churn between 5.9 -> 6.0 which may make future backports more difficult
  • Potential for large churn in core in an LTS release to fix performance issues and subtle changes to runtime

---

3. Target 5.8 in 8.x release. Delay release 3 - 4 weeks to allow forward compatible ABI to 6.0. Upgrade to TF+I as semver minor

• Land 5.8 with 6.0 forward compatible API / ABI in 8.x beta
  • We could potentially do so with TF+I turned on, this has been tested by V8 team
  • 5.8 will go stable April 25
  • 6.0 API / ABI should be relatively stable between May 17 - 25
• Upgrade to 5.9 with TF+I when it goes stable in early June
• Upgrade to 6.0 when it goes stable in early August
• Pros:
  • Uses Ignition+TurboFan which will be the supported pipeline from V8 for the foreseeable future
  • Lower risk of unknown security exploits
  • Easier to backport
• Cons:
  • TF + I will be a newer pipeline and the performance profile is not yet clear
  • Release delayed by 3-4 weeks
  • Potential for large churn in core in an LTS release to fix performance issues and subtle changes to runtime

What needs to be done?

• We need to get the opinions of our community. What would they prefer?
• We need to do more benchmarking. While microbenchmarks can be helpful it would be good to test using real life code.

Testing

Test build of 8.x including V8 5.9 (TF + I turned on)

• https://nodejs.org/download/test/v8.0.0-test201704119b43f9c487

Install with nvm

$ NVM_NODEJS_ORG_MIRROR=https://nodejs.org/download/test  nvm install v8.0.0-test201704119b43f9c487

EDIT: CTC voting tally for this issue

[vsemozhetbyt]

Some refs:
nodejs/node#11851
nodejs/node#12257 (comment)

[chicoxyzzy]

Is this version of Node built from sources of vee-eight-last-lkgr branch from v8/node repo?

[vsemozhetbyt]

Can Node.js a test suite be considered as a real life code?

https://twitter.com/evanhlucas/status/851950523490160640

[evanlucas]

@vsemozhetbyt that wasn't the core test suite. That was a test suite for one of my services at help.com. :]

[MylesBorins]

@chicoxyzzy the build above is indeed vee-eight-last-lkgr with a single commit to stub out some docs stuff breaking our build infra

[Qard]

Personally I'd lean toward being prepared for 6.0, but I know a lot of companies that can be very strict about predictable performance. I imagine there'd be a few companies that'd avoid the 8.x line until after TF+I lands so they can do a proper performance audit and have a more clear idea what they can expect.

[aheckmann]

I'm +1 on option 3.

[vsemozhetbyt]

An example of a real life case. Run ESLint (v4.0.0-alpha.0) check on Node.js code base.

Test script:

'use strict';

const execSync = require('child_process').execSync;

const command =
  'eslint' +
  ' --rule "indent: 0, no-multi-spaces: 0, space-before-function-paren: 0"' +
  ' --rulesdir tools/eslint-rules/' +
  ' benchmark lib test';

console.time('eslint1');
try { execSync(command, { encoding: 'utf8' }); }
catch (err) { console.error(err.stdout); }
console.timeEnd('eslint1');

console.time('eslint2');
try { execSync(command, { encoding: 'utf8' }); }
catch (err) { console.error(err.stdout); }
console.timeEnd('eslint2');

console.time('eslint3');
try { execSync(command, { encoding: 'utf8' }); }
catch (err) { console.error(err.stdout); }
console.timeEnd('eslint3');

Results:

// Node.js 8.0.0-nightly20170411b8f416023d (v8 5.7.492.69)
eslint1: 25632.138ms
eslint2: 24990.877ms
eslint3: 24390.442ms

// Node.js 8.0.0-test201704119b43f9c487 (v8 5.9.0 candidate)
eslint1: 25644.981ms
eslint2: 25228.094ms
eslint3: 24891.385ms

[AdriVanHoudt]

Would running CITGM with the new v8 produce any meaningful results?

[joyeecheung]

Another real world benchmark suite: https://github.com/eggjs/benchmark (benchmark of an enterprise Web framework egg.js), this includes real-world stuff like security & authentication and needs minimal external dependencies to run (no DB, wrk is required though, simply npm install && npm test and it will start running) . It also uses generators & async/await heavily so the new pipeline would make a huge difference.

[targos]

CITGM run (V8 5.9.213): https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/711/

[joyeecheung]

Results from egg's benchmark, the test build defintely outperforms the nightly

8.0.0-nightly20170411b8f416023d (v8 5.7.492.69)

------- koa hello -------
Running 10s test @ http://127.0.0.1:7002/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     5.61ms    8.84ms 236.41ms   98.30%

Req/Sec     1.24k   229.45     1.62k    71.39%

98433 requests in 10.01s, 14.83MB read

Requests/sec:   9832.27

Transfer/sec:      1.48MB
------- toa hello -------
Running 10s test @ http://127.0.0.1:7003/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     5.63ms    5.20ms  92.20ms   89.29%

Req/Sec     1.30k   161.83     1.66k    76.75%

103731 requests in 10.02s, 17.51MB read

Requests/sec:  10348.31

Transfer/sec:      1.75MB
------- egg hello -------
Running 10s test @ http://127.0.0.1:7001/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     9.02ms   10.37ms 252.42ms   98.64%

Req/Sec   736.79     89.38     0.89k    81.64%

58394 requests in 10.01s, 19.60MB read

Requests/sec:   5832.80

Transfer/sec:      1.96MB
------- egg hello (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     8.48ms    3.87ms 135.44ms   96.26%

Req/Sec   725.35     57.84     1.10k    86.61%

57742 requests in 10.02s, 19.38MB read

Requests/sec:   5762.01

Transfer/sec:      1.93MB
------- koa view -------
Running 10s test @ http://127.0.0.1:7002/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     8.67ms   10.26ms 243.42ms   96.96%

Req/Sec   782.52    126.78     1.00k    81.28%

62071 requests in 10.02s, 155.15MB read

Requests/sec:   6192.35

Transfer/sec:     15.48MB
------- toa view -------
Running 10s test @ http://127.0.0.1:7003/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     9.41ms    7.93ms 165.57ms   88.63%

Req/Sec   725.59     91.40     0.92k    72.81%

57704 requests in 10.03s, 145.28MB read

Requests/sec:   5751.99

Transfer/sec:     14.48MB
------- egg nunjucks view -------
Running 10s test @ http://127.0.0.1:7001/nunjucks

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    20.29ms   29.04ms 443.84ms   96.13%

Req/Sec   379.83     71.00   575.00     86.62%

29916 requests in 10.09s, 80.57MB read

Requests/sec:   2965.36

Transfer/sec:      7.99MB
------- egg ejs view -------
Running 10s test @ http://127.0.0.1:7001/ejs

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    13.74ms    8.63ms 242.56ms   87.65%

Req/Sec   453.48     55.75   790.00     79.60%

36095 requests in 10.02s, 97.93MB read

Requests/sec:   3602.51

Transfer/sec:      9.77MB
------- egg nunjucks view (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/nunjucks-aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    16.44ms   10.45ms 243.80ms   86.53%

Req/Sec   380.81     50.96   770.00     84.84%

30279 requests in 10.02s, 81.72MB read

Requests/sec:   3021.16

Transfer/sec:      8.15MB
------- egg ejs view (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/ejs-aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    13.69ms    8.50ms 197.92ms   87.97%

Req/Sec   456.37     58.07     0.87k    77.82%

36312 requests in 10.02s, 98.73MB read

Requests/sec:   3625.08

Transfer/sec:      9.86MB
------- egg passport -------
Running 10s test @ http://127.0.0.1:7001/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     9.94ms   12.80ms 290.00ms   98.30%

Req/Sec   685.20    110.37     1.32k    81.38%

54283 requests in 10.03s, 18.12MB read

Requests/sec:   5414.16

Transfer/sec:      1.81MB
------- egg passport (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     8.03ms    2.86ms 108.64ms   94.12%

Req/Sec   759.24    168.18     5.27k    99.13%

60554 requests in 10.10s, 20.44MB read

Requests/sec:   5996.82

Transfer/sec:      2.02MB

8.0.0-test201704119b43f9c487 (v8 5.9.0 candidate)

------- koa hello -------
Running 10s test @ http://127.0.0.1:7002/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     5.17ms    4.23ms 139.91ms   95.02%

Req/Sec     1.25k   253.08     1.83k    81.48%

99458 requests in 10.02s, 14.99MB read

Requests/sec:   9923.77

Transfer/sec:      1.50MB
------- toa hello -------
Running 10s test @ http://127.0.0.1:7003/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     5.89ms    6.32ms 112.00ms   88.49%

Req/Sec     1.34k   248.16     1.93k    70.00%

106674 requests in 10.02s, 18.01MB read

Requests/sec:  10640.94

Transfer/sec:      1.80MB
------- egg hello -------
Running 10s test @ http://127.0.0.1:7001/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     7.74ms    4.76ms 160.21ms   94.06%

Req/Sec   808.03    130.07     1.66k    83.85%

64295 requests in 10.03s, 21.58MB read

Requests/sec:   6409.31

Transfer/sec:      2.15MB
------- egg hello (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     7.37ms    4.70ms 159.82ms   95.19%

Req/Sec   844.56     66.80     1.44k    92.24%

67233 requests in 10.02s, 22.57MB read

Requests/sec:   6710.81

Transfer/sec:      2.25MBiew example</title>
------- koa view -------
Running 10s test @ http://127.0.0.1:7002/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     9.18ms    7.66ms 182.34ms   88.82%

Req/Sec   727.27    153.48     1.14k    67.67%

57838 requests in 10.02s, 144.57MB read

Requests/sec:   5772.13

Transfer/sec:     14.43MB
------- toa view -------
Running 10s test @ http://127.0.0.1:7003/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     9.64ms    9.29ms 171.58ms   87.36%

Req/Sec   756.45    189.22     1.39k    69.71%

60239 requests in 10.02s, 151.66MB read

Requests/sec:   6009.90

Transfer/sec:     15.13MB
------- egg nunjucks view -------
Running 10s test @ http://127.0.0.1:7001/nunjucks

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    17.02ms   13.60ms 282.96ms   87.93%

Req/Sec   381.47     69.48   553.00     75.75%

30276 requests in 10.02s, 81.54MB read

Requests/sec:   3020.98

Transfer/sec:      8.14MB
------- egg ejs view -------
Running 10s test @ http://127.0.0.1:7001/ejs

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    12.61ms    7.43ms 187.54ms   86.77%

Req/Sec   496.42     61.09     1.04k    80.08%

39494 requests in 10.02s, 107.16MB read

Requests/sec:   3941.07

Transfer/sec:     10.69MB
------- egg nunjucks view (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/nunjucks-aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    16.16ms   10.62ms 240.12ms   85.22%

Req/Sec   393.36     60.61     1.09k    83.98%

31315 requests in 10.02s, 84.52MB read

Requests/sec:   3124.77

Transfer/sec:      8.43MB
------- egg ejs view (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/ejs-aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency    12.56ms    7.65ms 187.92ms   86.36%

Req/Sec   499.39    102.21     2.78k    92.37%

39756 requests in 10.09s, 108.09MB read

Requests/sec:   3939.64

Transfer/sec:     10.71MB
------- egg passport -------
Running 10s test @ http://127.0.0.1:7001/

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     9.09ms    7.55ms 204.95ms   94.42%

Req/Sec   710.58    190.52     0.99k    69.55%

56504 requests in 10.02s, 18.86MB read

Requests/sec:   5639.99

Transfer/sec:      1.88MB
------- egg passport (Async Await) -------
Running 10s test @ http://127.0.0.1:7001/aa

8 threads and 50 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev

Latency     6.83ms    3.24ms 127.36ms   95.58%

Req/Sec     0.90k    75.49     1.38k    91.75%

71777 requests in 10.02s, 24.23MB read

Requests/sec:   7160.32

Transfer/sec:      2.42MB

(Trying to write a script to output structured data with it..UPDATE: PR opened in eggjs/benchmark#6)

[jasnell]

Sitting here with @mcollina running a number of module benchmarks are we're not yet seeing a significant performance delta between 8-with-5.9 and master or 6.10.x. yet. We will be running a number of additional benchmarks over the coming couple of days to prove that out.

If there is not going to be a significant difference between the two, then I'm seeing less of a need to delay the release for 5.9 but I'm also not seeing a significant technical reason not to. It's a bit up in the air, I think. We'll definitely need to run additional tests but if the performance profile is not significantly different, then it should actually be ok to ship 8.0.0 with 5.7 or 5.8 on time this month (option 1) and move up to 5.9 later in 8.x assuming the ABI compatibility is preserved without any issue.

[YurySolovyov]

It may also be nice to see some typical front-end workflows comparison

1. clear npm cache and run time npm i
2. build project (the bigger the better) with webpack with a lot of loaders (babel, typescript, etc.)
3. run eslint without cache.