gdalwarp ~40% (?) slower with Ubuntu 26.04 than with Ubuntu 24.04

[pjonsson]

What is the bug?

This is a repeat of #12170, but for the master branch built early this morning comparing the performance against GDAL 3.12.3.

Pytest has gotten the feature to print test timings so this time around I can provide a bit more information than last time. Here is the timing with a GDAL 3.12.3-based image:

tests/.../test_ol2wfr PASSED             1m 11s

and here is the timing with a GDAL 3.13.0-based image:

tests/.../test_ol2wfr PASSED             1m 34s

The test copies some data and writes a STAC file, but it's a fast SSD so that time should be measured in seconds, the rest of the time is spent on running gdalwarp. The machine is not starved on memory, but looking at top while running the tests on an otherwise (mostly) idle machine shows the 3.12.3 gdalwarp to go up to 1.4G of resident memory and taking 18.99s for the longest running process, and the 3.13.0 gdalwarp goes up to 1.8G of resident memory and taking 26.47s for the longest running process.

The image sets GDAL_CACHEMAX=256, which seemed like a good choice the last time I checked (sometime in 2025, so that was probably with GDAL 3.10).

Looking at the NEWS file for 3.13.0, I don't see anything indicating that a slowdown is expected, so I'm filing this as a bug. Feel free to close this if the slowdown is expected, we're still better off than we were in 3.10.

Steps to reproduce the issue

It's the same image and steps as #10809, but using Float32 for latitude.vrt and longitude.vrt.

Versions and provenance

The ubuntu-full image built on latest master this morning, so just before or after commit 2a4ac2d.

Additional context

No response

[rouault]

I don't reproduce this. on ubuntu 24.04 with RelWithDebInfo buillds

with 3.12.4:

$ time gdalwarp  -geoloc -r cubic -of COG -co COMPRESS=DEFLATE -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 oa02.vrt oa02.tif -overwrite --config GDAL_CACHEMAX=256
Creating output file that is 6260P x 2280L.
Using internal nodata values (e.g. 65535) for image oa02.vrt.
Copying nodata values from source oa02.vrt to destination ./oa02.tif_1015539_7.
0...10...20...30...40...50...60...70...80...90...100 - done.

real    0m20,067s
user    0m19,353s
sys     0m0,704s

with 3.13.0beta2:

$ time gdalwarp  -geoloc -r cubic -of COG -co COMPRESS=DEFLATE -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 oa02.vrt oa02.tif -overwrite --config GDAL_CACHEMAX=256
Creating output file that is 6260P x 2280L.
Using internal nodata values (e.g. 65535) for image oa02.vrt.
Copying nodata values from source oa02.vrt to destination ./oa02.tif_1015577_7.
0...10...20...30...40...50...60...70...80...90...100 - done.

real   0m19,964s
user   0m19,246s
sys    0m0,704s

[rouault]

can't repeat either using Docker images

$ time docker run --rm -it -v $PWD:$PWD ghcr.io/osgeo/gdal:ubuntu-full-3.12.3 sh -c 'cd /home/even/gdal/gdal/build_cmake; gdalwarp  -geoloc -r cubic -of COG -co COMPRESS=DEFLATE -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 oa02.vrt oa02.tif -overwrite --config GDAL_CACHEMAX=256'
Creating output file that is 6260P x 2280L.
Using internal nodata values (e.g. 65535) for image oa02.vrt.
Copying nodata values from source oa02.vrt to destination ./oa02.tif_7_7.
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m17,735s
user	0m0,011s
sys	0m0,019s

$ time docker run --rm -it -v $PWD:$PWD ghcr.io/osgeo/gdal:ubuntu-full-3.13.0beta2 sh -c 'cd /home/even/gdal/gdal/build_cmake; gdalwarp  -geoloc -r cubic -of COG -co COMPRESS=DEFLATE -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 oa02.vrt oa02.tif -overwrite --config GDAL_CACHEMAX=256'
Creating output file that is 6260P x 2280L.
Using internal nodata values (e.g. 65535) for image oa02.vrt.
Copying nodata values from source oa02.vrt to destination ./oa02.tif_7_7.
0...10...20...30...40...50...60...70...80...90...100 - done.

real	0m16,786s
user	0m0,011s
sys	0m0,020s

[pjonsson]

I just remembered that you mentioned there are nightly docker images in the previous ticket, so I tested again with that. Running the tests inside the image based on ghcr.io/osgeo/gdal:ubuntu-full-latest gives identical timings to what I reported:

tests/../test_ol2wfr PASSED             1m 34s

Just to make sure I didn't mix things up I ran gdalinfo --version inside the container and that gives GDAL 3.13.0beta1, released 2026/04/22, so it's definitely the GDAL 3.13-based image that I'm running things from. I have some experience getting cache pollution in Docker, so I have run docker system prune -af --volumes to clear the Docker cache (and everything else) before building the image, both before writing this and before I initially filed this issue.

Looking at top during the test, the gdalwarp processes take 25-26 CPU seconds each like I reported. I started a giant terminal with atop in the background and set its refresh interval to 1 second and stared at that during the run, and all the cores are going at 5.4-5.5GHz, which is fairly similar to 3.12.3.

Since you can't reproduce the issue it's clearly something on my side, but right now I'm at a loss of what that might be since my computer seems to be working. I will sleep on the matter and continue to investigate this tomorrow.

Edit: apparently I read your response too fast, you mention the beta2 image in the second response. I just tried that image as well and for better or worse I still get the same result. Puzzling, so I will sleep now and look into it again tomorrow.

[pjonsson]

Having slept on it, the issue is still reproducible for me. After some digging, I have one correction to the initial report and that is the code was updated to pass -co COMPRESS=ZSTD -co LEVEL=21 instead of -co COMPRESS=DEFLATE.

Measuring the time for a single run of gdalwarp takes ~11.6s on my machine, and the timing is the same for both 3.12.3 and the 3.13.0beta2 image. If I spawn gdalwarp on more cores, the timing for the longest running processes changes:

GDAL Version	1 core	16 cores
3.12.3	11.6s	19-20s
3.13.0beta2	11.5s	28-29s

I also tried 8 and 12 cores, but I didn't write the numbers down. But the runtimes when not utilizing all cores were between the two extremes from the table.

The time measured is just using

def local_check_call(command):
    before = datetime.now(UTC)
    subprocess.check_call(command)
    print(f"{datetime.now(UTC) - before} seconds for {command}")

and the machine has my web browser and other things open so there isn't zero noise in the measurement, but I kept my terminal window focused during the testing so I don't believe the noise can fully explain 8-10 seconds.

The spawning of gdalwarp is done with

    pool = multiprocess.Pool(processes=num_procs)
    multiple_results = [pool.apply_async(local_check_call, (cmd,)) for cmd in cmds]
    for res in multiple_results:
        ret = res.get()

which I believe should avoid the Python GIL.

I have a Ryzen 9950X3D so there is one CCD with a lot more cache, but when I ran the 1-threaded version the processes seemed to take more or less the same amount of time despite some processes starting on the CCD with more cache. I have disabled hyperthreading in the bios, so there are 16 real cores that can all do useful work at the same time.

My immediate guess for what would explain the difference would be memory bandwidth, having more than enough of it when running on a single core, but something changed with 3.13 so gdalwarp uses more memory bandwidth and that shows up when trying to go flat out on more cores.

[rouault]

@pjonsson Please provide a fully isolated reproducer (python script) as I'm getting lost at what you do exactly

[pjonsson]

Here's a Python script that runs gdalwarp on all available cores and reproduces the issue I'm seeing. Execution time for each gdalwarp is printed when it completes along with the command that was run. If I change num_procs in the configurable section to 1, the execution time for each gdalwarp looks very similar between 3.12.3 and 3.13.0beta2.

To run, change product_dir to the directory that contains S3B_OL_2_WFR____20230309T115352_20230309T115652_20230310T233800_0180_077_066_1800_MAR_O_NT_003.SEN3 and run the script. The script copies the product to a temporary directory under /tmp/tmpfs and uses that directory for other temporary files and the output as well, so it can be run as many times as desired without filling up /tmp.

If you are using a Docker container, you might want to pass --mount type=tmpfs,destination=/tmp/tmpfs,tmpfs-mode=1777 to docker run to get /tmp/tmpfs on a tmpfs-backed storage instead of the default.

The script starts with running cp -a in a shell for copying the product so it probably only runs on Linux. The script also requires lxml and xarray (I have recent versions, but I think any version should work).

I'm not sure how much memory each gdalwarp takes, but if it's around 2GB per instance and you have many cores the total amount of memory required can be significant.

Running time -p python3 gdal-313-repro.py on my machine gives these total timings:

3.12.3:
real 72.01
user 844.77
sys 49.19

3.13.0beta2:
real 101.00
user 3.20
sys 0.25

I'm guessing something changed with multiprocessing implementation in 3.14 since the user time disappeared, but it's the increased "real" that is the issue (and that stems from each gdalwarp taking longer).

import os.path
from datetime import UTC, datetime
from multiprocessing import Pool
from os import sched_getaffinity
from pathlib import Path
from subprocess import check_call
from tempfile import TemporaryDirectory

import lxml.etree
import xarray as xr

# REQUIRED CONFIGURATION
product_dir = "/home/ubuntu/testdata"

# OPTIONAL CONFIGURATION.
num_procs = len(sched_getaffinity(0))
product_name = "S3B_OL_2_WFR____20230309T115352_20230309T115652_20230310T233800_0180_077_066_1800_MAR_O_NT_003.SEN3"  # noqa: E501
tmpdir = "/tmp/tmpfs"


def local_check_call(command: list[str]):
    before = datetime.now(UTC)
    check_call(command)
    print(f"Time: {datetime.now(UTC) - before} seconds for command '{command}'")  # noqa: T201


def check_calls(pool, cmds: list[list[str]]) -> None:
    multiple_results = [pool.apply_async(local_check_call, (cmd,)) for cmd in cmds]
    for res in multiple_results:
        _ = res.get()


def vrt_name(name: str) -> str:
    # Always generate _unc files instead of _err files, even for old products.
    return name.replace("_err", "_unc") + ".vrt"


def generate_cogs(
    pool,
    measurements: list[tuple[str, str, str]],
    input_path: Path,
    output_path: Path,
    temp_path: Path,
) -> None:
    cmds = []
    for m in measurements:
        input_file = (temp_path if m[0] == "wqsf_temp.nc" else input_path) / m[0]
        command = [
            "gdal_translate",
            "-q",
            "-ot",
            m[2],
            "-unscale",
            f"NETCDF:{input_file}:{m[1]}",
            os.path.join(temp_path, vrt_name(m[1])),
        ]
        cmds.append(command)
    check_calls(pool, cmds)

    # https://gis.stackexchange.com/questions/103116/map-project-a-raster-having-separate-latitude-and-longitude-raster-bands
    geocoding = lxml.etree.fromstring(f"""
        <metadata domain="GEOLOCATION">
          <mdi key="SRS">
            GEOGCS["WGS 84",
                DATUM["WGS_1984",
                    SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],
                    AUTHORITY["EPSG","6326"]],
                PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],
                UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],
                AUTHORITY["EPSG","4326"]]
          </mdi>
          <mdi key="X_DATASET">{temp_path.absolute()!s}/longitude.vrt</mdi>
          <mdi key="X_BAND">1</mdi>
          <mdi key="Y_DATASET">{temp_path.absolute()!s}/latitude.vrt</mdi>
          <mdi key="Y_BAND">1</mdi>
          <mdi key="PIXEL_OFFSET">0</mdi>
          <mdi key="LINE_OFFSET">0</mdi>
          <mdi key="PIXEL_STEP">1</mdi>
          <mdi key="LINE_STEP">1</mdi>
        </metadata>
      """)
    cmds = []
    for m in measurements:
        if m[1] in ["latitude", "longitude"]:  # Internal helpers.
            continue
        # Replace NetCDF metadata with geocoding information.
        file = temp_path / vrt_name(m[1])
        with open(file) as f:
            root = lxml.etree.fromstring(f.read())
        for metadata in root.xpath("//Metadata"):
            metadata.getparent().remove(metadata)
        root.insert(0, geocoding)
        with open(file, "wb") as f:
            f.write(lxml.etree.tostring(root))

        command = [
            "gdalwarp",
            "-q",
            "-geoloc",
            "-r",
            "cubic" if m[2].startswith("Float") else "near",
            "-of",
            "COG",
            "-co",
            "COMPRESS=ZSTD",
            "-co",
            "LEVEL=21",
            *["-co", "NUM_THREADS=1", "--config", "GDAL_NUM_THREADS", "1"],
            str(file),
            str(file.with_segments(output_path, file.name).with_suffix(".tif")),
        ]
        cmds.append(command)
    # Generate COGs.
    check_calls(pool, cmds)


def main():
    measurements = [
        ("geo_coordinates.nc", "latitude", "Float32"),
        ("geo_coordinates.nc", "longitude", "Float32"),
        ("chl_nn.nc", "CHL_NN", "Float32"),
        ("chl_nn.nc", "CHL_NN_err", "Float32"),
        ("chl_oc4me.nc", "CHL_OC4ME", "Float32"),
        ("chl_oc4me.nc", "CHL_OC4ME_err", "Float32"),
        ("iop_nn.nc", "ADG443_NN", "Float32"),
        ("iop_nn.nc", "ADG443_NN_err", "Float32"),
        ("iwv.nc", "IWV", "Float32"),
        ("iwv.nc", "IWV_err", "Float32"),
        ("Oa01_reflectance.nc", "Oa01_reflectance", "Float32"),
        ("Oa01_reflectance.nc", "Oa01_reflectance_err", "Float32"),
        ("Oa02_reflectance.nc", "Oa02_reflectance", "Float32"),
        ("Oa02_reflectance.nc", "Oa02_reflectance_err", "Float32"),
        ("Oa03_reflectance.nc", "Oa03_reflectance", "Float32"),
        ("Oa03_reflectance.nc", "Oa03_reflectance_err", "Float32"),
        ("Oa04_reflectance.nc", "Oa04_reflectance", "Float32"),
        ("Oa04_reflectance.nc", "Oa04_reflectance_err", "Float32"),
        ("Oa05_reflectance.nc", "Oa05_reflectance", "Float32"),
        ("Oa05_reflectance.nc", "Oa05_reflectance_err", "Float32"),
        ("Oa06_reflectance.nc", "Oa06_reflectance", "Float32"),
        ("Oa06_reflectance.nc", "Oa06_reflectance_err", "Float32"),
        ("Oa07_reflectance.nc", "Oa07_reflectance", "Float32"),
        ("Oa07_reflectance.nc", "Oa07_reflectance_err", "Float32"),
        ("Oa08_reflectance.nc", "Oa08_reflectance", "Float32"),
        ("Oa08_reflectance.nc", "Oa08_reflectance_err", "Float32"),
        ("Oa09_reflectance.nc", "Oa09_reflectance", "Float32"),
        ("Oa09_reflectance.nc", "Oa09_reflectance_err", "Float32"),
        ("Oa10_reflectance.nc", "Oa10_reflectance", "Float32"),
        ("Oa10_reflectance.nc", "Oa10_reflectance_err", "Float32"),
        ("Oa11_reflectance.nc", "Oa11_reflectance", "Float32"),
        ("Oa11_reflectance.nc", "Oa11_reflectance_err", "Float32"),
        ("Oa12_reflectance.nc", "Oa12_reflectance", "Float32"),
        ("Oa12_reflectance.nc", "Oa12_reflectance_err", "Float32"),
        ("Oa16_reflectance.nc", "Oa16_reflectance", "Float32"),
        ("Oa16_reflectance.nc", "Oa16_reflectance_err", "Float32"),
        ("Oa17_reflectance.nc", "Oa17_reflectance", "Float32"),
        ("Oa17_reflectance.nc", "Oa17_reflectance_err", "Float32"),
        ("Oa18_reflectance.nc", "Oa18_reflectance", "Float32"),
        ("Oa18_reflectance.nc", "Oa18_reflectance_err", "Float32"),
        ("Oa21_reflectance.nc", "Oa21_reflectance", "Float32"),
        ("Oa21_reflectance.nc", "Oa21_reflectance_err", "Float32"),
        ("par.nc", "PAR", "Float32"),
        ("par.nc", "PAR_err", "Float32"),
        ("trsp.nc", "KD490_M07", "Float32"),
        ("trsp.nc", "KD490_M07_err", "Float32"),
        ("tsm_nn.nc", "TSM_NN", "Float32"),
        ("tsm_nn.nc", "TSM_NN_err", "Float32"),
        ("w_aer.nc", "A865", "Float32"),
        ("w_aer.nc", "A865_err", "Float32"),
        ("w_aer.nc", "T865", "Float32"),
        ("w_aer.nc", "T865_err", "Float32"),
        ("wqsf_temp.nc", "WQSF_LSB", "UInt32"),
        ("wqsf_temp.nc", "WQSF_MSB", "UInt32"),
    ]
    pool = Pool(processes=num_procs)
    try:
        with (
            TemporaryDirectory(prefix="local_", dir=tmpdir) as local_dir,
            TemporaryDirectory(prefix="output_", dir=tmpdir) as output_dir,
            TemporaryDirectory(prefix="tmp_", dir=tmpdir) as temp_dir,
        ):
            check_call(f"cp -a {product_dir}/{product_name} {local_dir}", shell=True)
            input_path = Path(local_dir, product_name)
            with xr.open_dataset(input_path / "wqsf.nc") as wqsf:
                wqsf["WQSF_LSB"] = wqsf.WQSF.astype("uint32")
                wqsf["WQSF_MSB"] = wqsf.WQSF
                wqsf["WQSF_MSB"].values = wqsf.WQSF.to_numpy() >> 32
                wqsf["WQSF_MSB"] = wqsf.WQSF_MSB.astype("uint32")
                wqsf.to_netcdf(path=Path(temp_dir, "wqsf_temp.nc"))
            generate_cogs(
                pool, measurements, input_path, Path(output_dir), Path(temp_dir)
            )
    finally:
        pool.close()
        pool.join()


if __name__ == "__main__":
    main()

[rouault]

@pjonsson I don't reproduce with your python script on my machine (i7-10750H CPU @ 2.60GHz, 6 core hyper threaded, so 12 vCPUs)

master:
real 373,64
user 2674,50
sys 1205,12

3.12.4:
real 394,09
user 2627,63
sys 1224,40

to be fair I was slighly active during 3.12.4 run (emails, browsing etc, but nothing supposedly computation or I/O intensive), so timings might be a bit misleading, but likely not masking that master would be significantly smaller than 3.12.4

I'm afraid you'll have to go for a git bisect section to spot the offending commit (even if I could reproduce, given the complexity of the scenario, I would have suggested that to you as I don't have time for such investigation)

[pjonsson]

Checking out 36d4d60 and applying this patch:

diff --git a/docker/ubuntu-small/Dockerfile b/docker/ubuntu-small/Dockerfile
index 578bb153bb..42b1f68fd5 100644
--- a/docker/ubuntu-small/Dockerfile
+++ b/docker/ubuntu-small/Dockerfile
@@ -72,7 +72,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
        libpq-dev${APT_ARCH_SUFFIX} libssl-dev${APT_ARCH_SUFFIX} libopenjp2-7-dev${APT_ARCH_SUFFIX} \
        libspatialite-dev${APT_ARCH_SUFFIX} \
        libmuparser-dev${APT_ARCH_SUFFIX} \
-       autoconf automake sqlite3 bash-completion swig
+       autoconf automake sqlite3 libnetcdf-dev${APT_ARCH_SUFFIX} libnetcdf-c++4-1 netcdf-bin bash-completion swig

 RUN mkdir proj gdal

@@ -243,7 +243,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
         bash-completion python3-numpy libpython3.12 \
         libjpeg-turbo8 libgeos3.12.1 libgeos-c1v5 \
         libexpat1 \
-        libxerces-c3.2 \
+        libxerces-c3.2 libnetcdf-c++4-1 netcdf-bin \
         libwebp7 libpng16-16 \
         libdeflate0 \
         libzstd1 bash libpq5 libssl3 libopenjp2-7 libspatialite8 \

followed by running ./build.sh --platform linux/amd64 --gdal master --proj 9.8.1 in the ubuntu-small docker directory and using the built image results in these timings:

real 76.53
user 877.43
sys 52.25

so the issue seems to be related to the switch to Ubuntu 26.04. I also ran ./build.sh --paltform linux/amd64 --gdal 313a78e65f569351092b378b192aa3325866401b --proj 9.8.1 and that resulted in similar timings, so there appear to be no changes in GDAL itself between something close to the GDAL 3.12.0 release and latest master.

As part of my testing yesterday, I switched to using Python 3.14 in the GDAL 3.12.3-based image, and that gave consistent timings with Python 3.12, so I don't believe the difference is caused by different Python versions.

Checking out 44534a4 and applying this patch:

diff --git a/docker/ubuntu-small/Dockerfile b/docker/ubuntu-small/Dockerfile
index 0b1a6f7cf7..87593d5837 100644
--- a/docker/ubuntu-small/Dockerfile
+++ b/docker/ubuntu-small/Dockerfile
@@ -72,7 +72,7 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
        libpq-dev${APT_ARCH_SUFFIX} libssl-dev${APT_ARCH_SUFFIX} libopenjp2-7-dev${APT_ARCH_SUFFIX} \
        libspatialite-dev${APT_ARCH_SUFFIX} \
        libmuparser-dev${APT_ARCH_SUFFIX} \
-       autoconf automake sqlite3 bash-completion swig
+       autoconf automake sqlite3 libnetcdf-dev${APT_ARCH_SUFFIX} libnetcdf-c++4-1 netcdf-bin bash-completion swig

 RUN mkdir proj gdal

@@ -243,9 +243,9 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
         bash-completion python3-numpy libpython3.14 \
         libjpeg-turbo8 libgeos3.14.1 libgeos-c1v5 \
         libexpat1 \
-        libxerces-c3.2 \
+        libxerces-c3.2 libnetcdf-c++4-1 netcdf-bin \
         libwebp7 libpng16-16 \
-        libdeflate0 \
+        libdeflate0 libaec0 \
         libzstd1 bash libpq5 libssl3 libopenjp2-7 libspatialite8 \
         libmuparser2v5 \
         libtcmalloc-minimal4 \

followed by running ./build.sh --platform linux/amd64 --gdal master --proj 9.8.1 in the ubuntu-small docker directory and using the built image results in these timings:

real 103.67
user 3.29
sys 0.25

and I get similar timings if I build an image with 313a78e instead.

My bisection was done by hand so I stepped through some other commits as well, and those show the same results. Just to make sure the Docker cache wasn't causing anything, I did docker system prune -af --volumes before I built the images that were used for the measurements reported in this comment.

Looking at the release notes for Ubuntu 26.04, the updated development tools are:

GCC 🐄 has been updated from version 14 to 15.2, binutils from 2.42 to 2.46, and glibc from 2.39 to 2.43.

but I'm not sure what the reference point for that is, my Ubuntu 24.04-based image contains GCC 13.3.0. Both GCC 14 and 15 changes mention vectorizer improvements, which I'm guessing might come in play in GDAL.

I haven't looked at C++ performance in this decade, is there some easy way to get the relevant CPU performance counters from a run?

[rouault]

Both GCC 14 and 15 changes mention vectorizer improvements, which I'm guessing might come in play in GDAL.

Interesting. We have code manually optimized for SIMD with the final loop for the remaining iterations using scalar code. If an optimizer is too aggressive, it might try to optimize those loops, which could be detrimental. Although I would anticipate that to result in useless code never executed bloating a bit the binary rather than a measurable runtime slowdown.

I haven't looked at C++ performance in this decade, is there some easy way to get the relevant CPU performance counters from a run?

there's the "perf" utility, but I'm not familiar with it. Otherwise I've used sysprof or valgrind --tool=cachegrind + kcachegrind to visualize the results.

[pjonsson]

Interesting. We have code manually optimized for SIMD with the final loop for the remaining iterations using scalar code. If an optimizer is too aggressive, it might try to optimize those loops, which could be detrimental. Although I would anticipate that to result in useless code never executed bloating a bit the binary rather than a measurable runtime slowdown.

I'm not saying it's the root cause of this issue, but getting better use of the instruction cache can definitely affect performance.

there's the "perf" utility, but I'm not familiar with it. Otherwise I've used sysprof or valgrind --tool=cachegrind + kcachegrind to visualize the results.

I had a look at perf and it looks quite competent, so that is the route I would consider most promising. Unfortunately it's coupled with the kernel version, so since I'm running Ubuntu 24.04 locally, I can only get perf measurements of that. We want to make a comparison between the two, so that doesn't fly.

My next attempt was Valgrind, but that runs for 5 minutes and then ends with this message:

==669== Cachegrind, a high-precision tracing profiler
==669== Copyright (C) 2002-2017, and GNU GPL'd, by Nicholas Nethercote et al.
==669== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==669== Command: gdalwarp -q -geoloc -r cubic -of COG -co COMPRESS=ZSTD -co LEVEL=21 -co NUM_THREADS=1 --config GDAL_NUM_THREADS 1 /tmp/tmpfs/tmp__690pmmb/CHL_NN.vrt /tmp/tmpfs/output_pd41fk9s/CHL_NN.tif
==669==
==669== brk segment overflow in thread #1: can't grow to 0x498a000
==669== (see section Limitations in user manual)
==669== NOTE: further instances of this message will not be shown
==669==
==669== I refs:        269,724,645,069

so apparently I have to compile my own version of Valgrind to be able to use that.

Third attempt was sysprof, and sysprof-cli --no-debuginfod --no-sysprofd --command=... gdalwarp.cap in the Ubuntu 26.04 image gave me a trace file that I can view in sysprof. Trying the same thing in the 24.04 image revealed that the version of sysprof there didn't support the --no- parameters, so I would need to install a more recent version of sysprof to be able to profile in a 24.04 container.

Since I have sysprof output for one gdalwarp under 26.04, here's what it looks like:

The cost model seems to be how large percentage of the run that is spent in each function and that is useful for realizing what to optimize to make things go faster, but I'm not sure that information is useful here unless there's a major performance regression in GCC 15.2, and that doesn't seem too likely.

Almost 70% of the time is spent in GDALCreateGenImgProjTransformer2::{lambda ...}, so if that function is called repeatedly and C++ compiles lambdas as an indirect jump, there's probably some performance to be had from passing an enum and then switching on that enum value to call methods with a known names instead (and that could trigger further compiler optimizations on those code paths). This was measured inside an image built with --with-debug-symbols, so it's also possible the regular build without debug symbols manages to optimize this away.

My gut feeling is that the underlying issue is related to memory bandwidth since the performance is fine when running a single gdalwarp and you can't reproduce the problem on your machine (fewer cores, and Intel has more memory bandwidth than AMD). If I'm right about that, there should be differences in the number of loads/stores or cache hit rates when running a single gdalwarp on 24.04/26.04 as well.

[pjonsson]

I figured it was worth a try to see if memcheck would emit any diagnostics for the number of loads/stores, so I tried using valgrind -v --tool=memcheck instead, and that completed the run.

I didn't get the information I was looking for, but there are hundreds of mismatched free() / delete / delete[] reported, is that something you care about so I should create an issue and attach the log to that? It doesn't seem to be in the functions from the Valgrind suppression file used for the tests in this repository. Here's the end of the log:

==2205== 18882 errors in context 296 of 296:
==2205== Mismatched free() / delete / delete []
==2205==    at 0x484C45D: operator delete[](void*, unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2205==    by 0x5F70196: GDALArrayBandBlockCache::FlushBlock(int, int, int) (gdalarraybandblockcache.cpp:443)
==2205==    by 0x5F703DA: GDALArrayBandBlockCache::FlushCache() (gdalarraybandblockcache.cpp:269)
==2205==    by 0x5EDA043: GDALDataset::FlushCache(bool) (gdaldataset.cpp:736)
==2205==    by 0x5EF3345: GDALPamDataset::FlushCache(bool) (gdalpamdataset.cpp:189)
==2205==    by 0x53E622C: GTiffDataset::FlushCacheInternal(bool, bool) [clone .part.0] (gtiffdataset_write.cpp:2170)
==2205==    by 0x53D8DFC: UnknownInlinedFun (gtiffdataset_write.cpp:2167)
==2205==    by 0x53D8DFC: GTiffDataset::Finalize() (gtiffdataset.cpp:201)
==2205==    by 0x53D99D2: GTiffDataset::Close(int (*)(double, char const*, void*), void*) [clone .part.0] (gtiffdataset.cpp:138)
==2205==    by 0x53D9DFC: UnknownInlinedFun (shared_ptr_base.h:348)
==2205==    by 0x53D9DFC: GTiffDataset::~GTiffDataset() (gtiffdataset.cpp:127)
==2205==    by 0x53D9E5C: GTiffDataset::~GTiffDataset() (gtiffdataset.cpp:128)
==2205==    by 0x525F428: UnknownInlinedFun (unique_ptr.h:99)
==2205==    by 0x525F428: UnknownInlinedFun (unique_ptr.h:404)
==2205==    by 0x525F428: GDALFillNodata (rasterfill.cpp:1008)
==2205==    by 0x51FF0A1: GDALGeoLoc<GDALGeoLocCArrayAccessors>::GenerateBackMap(GDALGeoLocTransformInfo*) (gdalgeoloc.cpp:1494)
==2205==  Address 0x3441a0f0 is 0 bytes inside a block of size 72 alloc'd
==2205==    at 0x48479E8: operator new(unsigned long, std::nothrow_t const&) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2205==    by 0x5F10C35: UnknownInlinedFun (gdalabstractbandblockcache.cpp:183)
==2205==    by 0x5F10C35: GDALRasterBand::GetLockedBlockRef(int, int, int) [clone .part.0] (gdalrasterband.cpp:2016)
==2205==    by 0x5FABE0B: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) (rasterio.cpp:308)
==2205==    by 0x53ED828: GTiffRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) (gtiffrasterband.cpp:491)
==2205==    by 0x5ECF634: GDALRasterBand::RasterIOInternal(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) (gdalrasterband.cpp:472)
==2205==    by 0x5ECF838: GDALRasterBand::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, long long, long long, GDALRasterIOExtraArg*) (gdalrasterband.cpp:445)
==2205==    by 0x5ED1696: GDALRasterIO (gdalrasterband.cpp:505)
==2205==    by 0x52608B9: UnknownInlinedFun (rasterfill.cpp:226)
==2205==    by 0x52608B9: GDALFillNodata (rasterfill.cpp:986)
==2205==    by 0x51FF0A1: GDALGeoLoc<GDALGeoLocCArrayAccessors>::GenerateBackMap(GDALGeoLocTransformInfo*) (gdalgeoloc.cpp:1494)
==2205==    by 0x5204557: UnknownInlinedFun (gdalgeoloc_carray_accessor.h:188)
==2205==    by 0x5204557: GDALCreateGeoLocTransformerEx(void*, char const* const*, int, char const*, char const* const*) (gdalgeoloc.cpp:2044)
==2205==    by 0x62F7DF4: GDALCreateGenImgProjTransformer2::{lambda(GDALGenImgProjTransformPart&, void*, char const*, OGRSpatialReference&, bool&)#1}::operator()(GDALGenImgProjTransformPart&, void*, char const*, OGRSpatialReference&, bool&) const [clone .constprop.0] (gdaltransformer.cpp:2568)
==2205==    by 0x522C7AD: GDALCreateGenImgProjTransformer2 (gdaltransformer.cpp:2646)
==2205==
==2205== ERROR SUMMARY: 22655 errors from 296 contexts (suppressed: 0 from 0)

[rouault]

I didn't get the information I was looking for, but there are hundreds of mismatched free() / delete / delete[] reported

I believe this is either a false positive, or an ultra subtle pedantic stuff hopefully without harmful consequence. https://stackoverflow.com/questions/4005601/how-do-i-invoke-the-nothrow-version-of-delete is slightly related. We allocate the raster block with "new (std::nothrow)" and delete with "delete". To the best of my knowledge, this is the correct way of proceeding

[rouault]

Unfortunately it's coupled with the kernel version, so since I'm running Ubuntu 24.04 locally, I can only get perf measurements of that.

I believe that whatever your host OS you'll need to hand compile the GCC version of the other one (or find some packaging in alternate repositories). Compiling gcc is not that difficult . Last time I tried was very much in the line of what https://stackoverflow.com/questions/14728652/how-to-make-a-light-build-of-gcc-with-language-supports-etc-pruned suggests: ./configure --enable-languages=c,c++ --prefix=/some/install-prefix && make -j$(nproc) && make install