ras_dec.c: 3 signed integer overflows in RAS decoder (UBSan)

[JorgeBarredo14]

Describe the bug

Three UBSan signed-int-overflow sites in the RAS decoder. All reached from normal jas_image_decode on crafted RAS files with large width/depth/maplength. Found by libFuzzer + UBSan, reproducing against master (src/libjasper/ras/ras_dec.c).

1) Bit accumulator overflow at ras_dec.c:352

// ras_dec.c, inside the sample-decoding loop:
while (nz < hdr->depth) {
    if ((c = jas_stream_getc(in)) == EOF) goto error;
    z = (z << 8) | c;          // <-- UBSan: left shift of int
    nz += 8;
}

z is declared as int. With hdr->depth == 32 the loop shifts z left repeatedly and z << 8 overflows signed int as soon as the high byte has bit 23 set. UBSan:

ras_dec.c:352:12: runtime error: left shift of 167772160 by 8 places cannot be represented in type 'int'

2) Header length computation overflow at ras_dec.c:206

if (hdr.type == RAS_TYPE_OLD) {
    hdr.length = RAS_ROWSIZE(&hdr) * hdr.height;   // <-- signed int * int
}

With RAS_ROWSIZE = ((width*depth + 15)/16)*2, width=32768, depth=32, height=32768, the product is ~2^36 and wraps signed int.

ras_dec.c:206:34: runtime error: signed integer overflow

3) Palette size shift at ras_dec.c:412

case RAS_MT_EQUALRGB: {
    jas_logwarnf("warning: palettized images not fully supported\n");
    numcolors = 1 << hdr->depth;           // <-- 1 << 32 is UB
    if (numcolors > RAS_CMAP_MAXSIZ) {
        return -1;
    }
}

hdr->depth comes straight from the file. Values >= 32 make the shift UB. The existing bound check on numcolors runs after the UB.

ras_dec.c:412:17: runtime error: shift exponent 32 is too large for 32-bit type 'int'

To Reproduce

Found by fuzzing with libFuzzer + ASan + UBSan. Several hundred crash inputs for all three sites from a 12h campaign. Any RAS file with

• depth = 32, or
• depth + width combination making RAS_ROWSIZE * height > 2^31, or
• depth > 24 combined with any non-zero sample bytes

triggers at least one of them. Happy to attach a minimal PoC for each if it helps.

Expected behavior

Reject RAS files with out-of-range depth/width/height before the arithmetic.

Fix idea

--- a/src/libjasper/ras/ras_dec.c
+++ b/src/libjasper/ras/ras_dec.c
@@ around the header validation block (~line 180-200)
     /* Validate header fields before use. */
+    if (hdr.depth <= 0 || hdr.depth > 24) {
+        goto error;  /* RAS format supports up to 24 bits per pixel */
+    }
+    if (hdr.width <= 0 || hdr.height <= 0) {
+        goto error;
+    }
+    if (RAS_ROWSIZE(&hdr) > INT_MAX / hdr.height) {
+        goto error;
+    }
     if (hdr.type == RAS_TYPE_OLD) {
         hdr.length = RAS_ROWSIZE(&hdr) * hdr.height;
     }
@@ ras_dec.c:~352
-                z = (z << 8) | c;
+                z = (uint32_t)((uint32_t)z << 8) | (uint32_t)c;
@@ ras_dec.c:~412
-        numcolors = 1 << hdr->depth;
+        if (hdr->depth > 24) return -1;  /* belt+braces after earlier check */
+        numcolors = 1 << hdr->depth;

The depth bound also helps the shift at line 206 and the accumulator at line 352.