Incremental probing for `IndexJoin` and `IndexHashJoin` to reduce memory usage

[gengliqi]

Enhancement

tidb/pkg/executor/join/index_lookup_join.go

Lines 731 to 748 in 22580a3

	for {
	select {
	case <-ctx.Done():
	return ctx.Err()
	default:
	}
	err := exec.Next(ctx, innerExec, iw.executorChk)
	failpoint.Inject("ConsumeRandomPanic", nil)
	if err != nil {
	return err
	}
	if iw.executorChk.NumRows() == 0 {
	break
	}
	innerResult.Add(iw.executorChk)
	iw.executorChk = exec.TryNewCacheChunk(innerExec)
	}
	task.innerResult = innerResult

In the current probing process of IndexJoin and IndexHashJoin, all matched inner rows for the join keys of an outer chunk are fetched at once. This can lead to excessive memory usage and potentially cause the query to exceed the memory limit.

To address this, the probing process should fetch and consume inner rows incrementally, in a streaming fashion similar to hash join.

[gengliqi]

/sig execution