Speed up slow tests

[msokolov]

Description

The goal should be tests that run in < 1 second.

We've got some that are consistently slow:

  166.32s TestRescoreTopNQuery.testLateInteractionQuery (:lucene:core)
  50.10s TestFullPrecisionFloatVectorSimilarityValuesSource.testFullPrecisionVectorSimilarityDVS (:lucene:core)
  39.02s TestPriorityQueue.testRemovalsAndInsertions (:lucene:core)
  23.61s TestIndexOrDocValuesQuery.testUseIndexForSelectiveMultiValueQueries (:lucene:core)
  16.98s TestIndexedDISI.testRandom (:lucene:core)
  12.86s TestLateInteractionRescorer.testMissingLateIValues (:lucene:core)
  10.10s TestBlockMaxConjunction.testRandom (:lucene:core)
   8.98s TestIndexWriterExceptions.testRandomExceptions (:lucene:core)
   7.91s TestTermInSetQuery.testDuel (:lucene:core)
   7.22s TestLateInteractionFloatValuesSource.testValues (:lucene:core)

some (like TestIndexedDISI.testRandom) are just occasionally slow, but the others on this list seem to consistently show up as the worst offenders

Gradle command to reproduce

No response

[benwtrent]

TestLateInteractionFloatValuesSource should use tiny arrays and very few IMO. I am not sure there is significant benefit on having vector sizes larger than 8 or more than 4 vectors per doc and much fewer docs.

Looking at the code, it is already at 16 dims, but it will test 1k+ docs (nightly as much as 2k), which seems unnecessary to me.

Maybe we can reduce that doc count a whole magnitude.

[msokolov]

I tried running TestRescoreTopNQuery.testLateInteractionQuery by itself and it ony took 4-12 secs. Not sure what bloats it up to 166 sec, but anyway it has some potential to get big multiples through dimension, num docs, and num segments: I think we can reduce all of these

[rmuir]

Look out for tests that use too many threads. its often the case of why they run fast by themselves (because they have a ton of cpus to monopolize). But if test takes minutes in the suite, it is using too many resources.

I'd recommend to @Nightly the long-tail of bad tests from the "bad list" printed by gradle, and open separate issues to fix them so they can be un-@Nightly'ed?

[msokolov]

TestPriorityQueue.testRemovalsAndInsertions has this:

    // Try many random additions to existing entries - we should always see
    // increasing scores in the lowest entry in the PQ
    for (int p = 0; p < 500000; p++) {

which seems excessive to me, but it has been this way for > 10 years, so I'm reluctant to wade in with a brushcutter. Does anyone know why we need such a large iteration count here?

[rmuir]

Also creation of tons of garbage can cause the effect of being "fast in isolation" but slow in the test suite. Same issue as threads, if you've got other idle cpus for the parallel GC algorithms to clean up, then it "seems to be fast" but in fact it is monopolizing too many resources.

[msokolov]

After the "late interaction" vector tests are reduced, test run shows:

  32.84s TestIndexOrDocValuesQuery.testUseIndexForSelectiveMultiValueQueries (:lucene:core)
  21.89s TestPriorityQueue.testRemovalsAndInsertions (:lucene:core)
  16.42s TestIndexedDISI.testRandom (:lucene:core)
  14.05s TestIndexWriterExceptions.testRandomExceptions (:lucene:core)
  13.29s TestTermInSetQuery.testDuel (:lucene:core)
  12.15s TestRescoreTopNQuery.testRescoreField (:lucene:core)
  10.86s TestPointQueries.testRandomLongsBig (:lucene:core)
   7.43s TestLucene104PostingsFormat.testDocIDRunEnd (:lucene:core)
   6.54s TestPerFieldPostingsFormat.testDocIDRunEnd (:lucene:core)
   5.95s TestNRTThreads.testNRTThreads (:lucene:core)

I see TestIndexedDISI.testRandom has this comment (from @rmuir):

   // TODO: can this be toned down?
  @Nightly

not sure why it is running for me when I do gradlew -p lucene/core test even though it is annotated as nightly?

[rmuir]

I don't see any @Nightly on testRandom(). are we looking at the same codebase?

lucene/lucene/core/src/test/org/apache/lucene/codecs/lucene90/TestIndexedDISI.java

Lines 493 to 501 in 7271b5f

	public void testRandom() throws IOException {
	try (Directory dir = newDirectory()) {
	int numIters = atLeast(3);
	for (int i = 0; i < numIters; ++i) {
	doTestRandom(dir);
	}
	}
	}

[msokolov]

Oh, that is correct, I got distracted by testRandomBlocks

[rmuir]

IMO that one should be made @Nightly. I didn't profile it or anything, but its gonna write bitsets up to size 2 billion to memory/disk.

Seems like it is good to validate in this way, but i'd do it at nightly. as followup we could consider a toned down version when running locally.

[msokolov]

OK, I made that change. I'll leave this issue open in case someone has more knowledge about the scope of TestIndexOrDocValuesQuery.testUseIndexForSelectiveMultiValueQueries and/or TestPriorityQueue.testRemovalsAndInsertions or any of the other tests on the list above

[dweiss]

Thanks for tackling this, @msokolov ! When looking at test times you should also be careful because the reported time is a wall-clock time and many things run in parallel (like Robert mentioned).

You could get a more realistic estimate of individual tests if you run with a single test jvm (and one gradle worker) but in general what's printed on github workflows is a fairly good estimate.