TiDB OOM at merge sort step when using distributed framework to add index

[fzzf678]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb 8c16g
set @@global.tidb_ddl_reorg_worker_cnt = 8;
use failpoint to enable merge sort
add index on an ordered bigint column, total 8 billion rows.

2. What did you expect to see? (Required)

No OOM

3. What did you see instead (Required)

OOM at merge sort step

4. What is your TiDB version? (Required)

master

[fzzf678]

The heap:

[fzzf678]

The reason is too much memory is allocated for concurrent reader in merge sort step.

tidb/pkg/lightning/backend/external/iter.go

Lines 523 to 525 in 4230cf3

	concurrentReaderConcurrency := max(256/outerConcurrency, 8)
	// TODO: merge-sort step passes outerConcurrency=0, so this bufSize might be
	// too large when checkHotspot = true(add-index).

As the TODO said, in the merge sort step the concurrent reader concurrency is 256, when the reader meet hotspot and switch to concurrent mode, it will ask for 256*8MiB=2GiB memory, If all 8 file groups need to be switched, memory is exhausted.

tidb/pkg/lightning/backend/external/byte_reader.go

Lines 200 to 206 in 4230cf3

	readerFields.largeBuf = make([][]byte, readerFields.concurrency)
	for i := range readerFields.largeBuf {
	readerFields.largeBuf[i] = readerFields.largeBufferPool.AllocBytes(readerFields.bufSizePerConc)
	if readerFields.largeBuf[i] == nil {
	return errors.Errorf("alloc large buffer failed, size %d", readerFields.bufSizePerConc)
	}
	}