Assertion failure in calcstepsizes (unfixed #57)

[asarubbo]

On 1.900.22

warning: trailing garbage in marker segment (9 bytes)
warning: trailing garbage in marker segment (28 bytes)
warning: trailing garbage in marker segment (40 bytes)
warning: ignoring unknown marker segment (0xffee)
type = 0xffee (UNKNOWN); len = 23;1f 32 ff ff ff 00 10 00 3d 4d 00 01 32 40 e4 e4 00 10 00 00 4f warning: trailing garbage in marker segment (12 bytes)
imginfo: /tmp/portage/media-libs/jasper-1.900.22/work/jasper-1.900.22/src/libjasper/jpc/jpc_dec.c:1650: void calcstepsizes(uint_fast16_t, int, uint_fast16_t *): Assertion `!((expn + (numrlvls - 1) - (numrlvls - 1 - ((bandno > 0) ? ((bandno + 2) / 3) : (0)))) & (~0x1f))' failed.

Testcase: https://github.com/asarubbo/poc/blob/master/00044-jasper-assert-calcstepsizes

[thoger]

There's no assertion on 1.900.26 and later due to max samples limit. The use of --max-samples 0 is required to reproduce with current jasper versions.

[thoger]

There was CVE-2016-9399 assigned for this via @asarubbo , see:

https://blogs.gentoo.org/ago/2016/11/16/jasper-multiple-assertion-failure/

There is now another report from @owl337 triggering the same assertion, and with other CVE - CVE-2017-13751. Reporters have not provided any reasoning to explain why separate CVE is justified.

They reported in Red Hat bugzilla and I can't find matching report here in the upstream issue tracker, so I'm attaching their test case here.

https://bugzilla.redhat.com/show_bug.cgi?id=1485283

Test case: POC6.zip

(Note that this POC6 triggers the issue without needing --max-samples 0.)

[thoger]

The assertion is not triggered for either of the reproducers after applying patch from @MaxKellermann's pull request #158.

[MaxKellermann]

.. now if only this project wasn't dead! All of these well-known vulnerabilities have been reported long ago, but were never fixed, nor were my pull requests considered.

[D0x17]

The provided PoC in Bugzilla #1485283 triggered assertions in both the versions 1.900.22 and 2.0.12
@thoger, why do you think the version 1.900.26 do not have an assertion?

$ jasper --input POC6 --output pocdup.jp2
warning: trailing garbage in marker segment (14 bytes)
warning: trailing garbage in marker segment (2 bytes)
assertion "!((expn + (numrlvls - 1) - (numrlvls - 1 - ((bandno > 0) ? ((bandno + 2) / 3) : (0)))) & (~0x1f))" failed: file "jpc_dec.c", line 1650, function: calcstepsizes
Aborted (core dumped)

I've applied the patch from @MaxKellermann PR 158, seems like it solves the issue. Still we are not sure why the reporter requested two CVE's for similar issue
CVE-2016-9399 and CVE-2017-13751, didn't they run the fuzzer on RHEL earlier, is that the reason?

[thoger]

The provided PoC in Bugzilla #1485283 triggered assertions in both the versions 1.900.22 and 2.0.12
@thoger, why do you think the version 1.900.26 do not have an assertion?

You're not reading my comments correctly (likely because of poor wording of my first comment). I mentioned that the reproducer 00044-jasper-assert-calcstepsizes that is in the first comment of this issue does not trigger assertion in 1.900.26 or later in imginfo by default, as its loading fails with:

maximum number of samples exceeded (116716467662613 > 67108864)

prior to reaching the assertion. I also noted that running it with --max-samples 0 makes it possibly to reproduce the assert with that reproducer in the later jasper versions.

The other reproducer - POC6 - trigger the assert without needing to override samples limit.

Still we are not sure why the reporter requested two CVE's for similar issue CVE-2016-9399 and CVE-2017-13751, didn't they run the fuzzer on RHEL earlier, is that the reason?

@owl337's report does not indicate which version they fuzzed, but AFAICT they did not use the version in Red Hat Enterprise Linux 7 they reported the bug against. At least one of the issues they reported in Red Hat Bugzilla did not affect the version of jasper as used in RHEL 7. I assume they used the upstream version that was the latest at the time, something around 2.0.12 is my guess.

[thoger]

Still we are not sure why the reporter requested two CVE's for similar issue CVE-2016-9399 and CVE-2017-13751, didn't they run the fuzzer on RHEL earlier, is that the reason?

Re-reading your question again - there is no single reporter behind those two CVEs. The 2016 one was requested by @asarubbo, while the 2017 one by @owl337.

[D0x17]

@thoger I ran the reproducer POC6 on the versions 2.0.14 and 1.900.22 upstream, I don't think the trigger paths are different for the CVE of 2016 & 2017. Did we make sure that the two trigger paths exactly the same as @owl337 cautioned in Bugzilla report

[jubalh]

Fixed in 84d00fb