coverage is only being applied by running specific .exe test files

[midorishibukawa]

hello!

I'm trying to get test coverage for a project of mine using bisect_ppx, but unfortunately I haven't been successful yet.

in order to figure out whether it's a problem with my project/system or with bisect_ppx itself, I followed the dune starter instructions, and managed to make it work by explicitly running the tester executable with the following command:

dune exec --instrument-with bisect_ppx ./tester.exe

however, when I try to run tests by using dune's runtest command as provided by the readme, no coverage file is generated. for the sake of completeness, the command I'm trying to run is as follows:

dune runtest --instrument-with bisect_ppx --force

on my actual project, I'm using ppx_inline_test, so I tried adding a simple inline test to the starter project to see if that would work, but to no avail. I forked the starter project and added my changes to it in case anyone wants to try to reproduce the problem.

the inline test file is composed of a single line:

let%test "inline" = Hello.run (); true

and the following lines were added to the dune file:

(library
 (name inline_tester)
 (modules inline_tester)
 (inline_tests)
 (libraries hello)
 (preprocess (pps ppx_inline_test))
 (instrumentation (backend bisect_ppx)))

I wasn't quite sure of whether I should add the instrumentation stanza to the testing library or not, but I tried both ways with no apparent changes in behavior.

I'm running nixos, and all dependencies are provided by opam-nix, as the nix package manager doesn't really play well with opam, so I suspect that might have something to do with it.

any help would be greatly appreciated!

[aantron]

Thank you! I haven't tried Bisect_ppx with ppx_inline_test. Will probably have to take a look at the details of how it works.

[Blue-Berry]

I was also struggling to get coverage files to be generated for ppx_inline_test, I found if I rolled back my version of dune to between 2.7 and 3.4 it worked perfectly.

[mbarbin]

@midorishibukawa I may have a related issue. Would you mind having a look at #411 and let me know if BISECT_FILE does anything for you? I'm preparing to use bisect_ppx more and would be grateful for any insights into the setup. Thanks!

[mooreryan]

tl; dr: the setup in the fork works pretty much as expected with dune 3.12.1 and bisect_ppx 2.8.3.

---

I downloaded the fork linked by the OP (here), but I could not reproduce the issues with using dune 3.12.1 and bisect_ppx 2.8.3. (Could be that newer versions fixed the issues.)

Here are commands I ran to check it.

dune exec --instrument-with bisect_ppx ./tester.exe

^ I would expect this to generate coverage files in the main directory, which it does.

dune exec ./tester.exe

^ I would expect this to generate no coverage report, since it is not run with instrumentation, and that is what happens.

dune runtest

^ This generates no coverage, which is correct as it is not being run with instrumentation.

dune runtest --instrument-with bisect_ppx --force

^ generates coverage in the _build directory, which also is correct given the commands that dune is actually running:

Running[1]: (cd _build/default && ./tester.exe)
Running[2]: (cd _build/default && .inline_tester.inline-tests/inline_test_runner_inline_tester.exe inline-test-runner inline_tester -partition inline_tester.ml -source-tree-root . -diff-cmd -)

Now, it also also generates coverage for the Inline_tester module as well, which I normally would not want, but it happens because the dune file in the forked project includes the (instrumentation (backend bisect_ppx)) stanza in the inline_tester as well.

bisect-ppx-report html

^ This generates coverage report correctly regardless of where the coverage files are.

One final note is that I would generally add the instrumentation to the executable file as well, and that way any cram tests or whatever will be picked up and you will get coverage of the executable if you run under instrumentation.

(cc @mbarbin)

[mooreryan]

Oh and it looks like another reason why this works okay is that it has aliased the runtest to be:

(rule
 (alias runtest)
 (action (run %{exe:tester.exe})))

So it appears to not be running in the sandbox.

[mooreryan]

See this issue comment for more info.

[midorishibukawa]

Thank you so much, @mooreryan and @mbarbin!

After reading @mooreryan's thorough explanation (both here and on the mentioned issue, I was able to verify that my problem was in fact caused by the lack of a BISECT_FILE environment variable. Once the variable was properly set, everything worked as intended!

Reading everything was very enlightening, and also made me realise that I had misunterstood the usage of the instrumentation stanza.

I agree with @mbarbin's suggestion of adding that information to bisect_ppx's documentation.

I'll be closing this issue, as the problem has been solved.

[aantron]

Thank you @midorishibukawa, @mooreryan, @mbarbin, @Blue-Berry! I used ppx_expect with Bisect in Dream without having to use BiSECT_FILE, but per @Blue-Berry's comment it appears that more recent versions of Dune have changed sandbox behavior, and Bisect's docs need to be updated to reflect this. I'm going to reopen this issue to keep it as a reminder for myself in addition to #411.