Expanding tests run in coverage job

[Trott]

I changed the configuration for https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage-daily/ to also run pummel and benchmark jobs.

Under "Execute Shell", I changed the last line from this:

NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python COVTESTS="test-ci test-internet" make coverage -j $(getconf _NPROCESSORS_ONLN)

...to this:

NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python COVTESTS="test-ci test-internet test-pummel test-benchmark" make coverage -j $(getconf _NPROCESSORS_ONLN)

Once nodejs/node#25799 lands, we can replace the list of suites with test-all-suites and all tests under the test directory will run no matter where they are.

[Trott]

Guess I'll leave this open until we can change it to test-all-suites.

[Trott]

Hmmm...my change somehow added code in the benchmark directory to the measurements, which is definitely not what I wanted to do. Any idea how to exclude it or why it got included? @bcoe

[Trott]

Oh, I see, I need to add it as an --exclude in the call to c8 in the Makefile.

[Trott]

(In the meantime, I'll remove the benchmark tests from the coverage job.)

[mhdawson]

@Trott, how much run time does adding those 2 make?

Working on #1669 and just want to understand how that will affect the run time there as well. We will also have to sync that job.

[Trott]

@Trott, how much run time does adding those 2 make?

Working on #1669 and just want to understand how that will affect the run time there as well. We will also have to sync that job.

The whole job, start-to-finish, takes around 20 minutes give-or-take 5 minutes. My run took 19 minutes and the previous run (without these additions) was 21 minutes, so whatever additional time there is, it is negligible. On the other hand, that is odd because I believe two pummel tests timed out, so I would expect that to add 4 minutes right there. (Maybe there's some Jenkins magic there? Does this job run things in parallel even though they're not parallel-safe somehow?)

[mhdawson]

@Trott key thanks, if its not significantly longer that's good.

I doubt there is "magic".

[mhdawson]

Something strange is going on. I'm seeing failures in the sanity coverage job when trying to add pummel. Looks like the different types are running in parallel which seems wrong so I'm not sure adding these until we figure that out is a good idea as it seems to be causing tests to fail that previously passed because they are running in parallel.

The coverage job does not intentionally try to run anything in parallel that is not normally run that way.

[refack]

Looks like the different types are running in parallel which seems wrong

The change in nodejs/node#25799 will fix that. Parallelism will only be done by test.py not by make.
(We should make sure all make test* run with -j anyway)

[mhdawson]

I don't see how 25799 is going to fix that for the coverage jobs. @refack is it a known problem that the suites are running in parallel?

[Trott]

@mhdawson I'd guess it will fix it because the coverage job will run only one make target (test-all-suites) whereas currently, make runs multiple targets (test-pummel and test-internet and so on) and ends up running multiple test.py instances.