[LibRaw] Initial integration

[jamiepinheiro]

Adding LibRaw fuzzing. The Google3 fork maintainers will be porting issues to GitHub.

[TravisBuddy]

Travis tests have failed

Hey @jamiepinheiro,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

9th Build

View build log

./infra/travis/travis_build.py

Traceback (most recent call last):
  File "./infra/travis/travis_build.py", line 151, in <module>
    sys.exit(main())
  File "./infra/travis/travis_build.py", line 139, in main
    build_project(project)
  File "./infra/travis/travis_build.py", line 109, in build_project
    project_yaml = yaml.safe_load(file_handle)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/constructor.py", line 41, in get_single_data
    node = self.get_single_node()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/parser.py", line 382, in parse_block_sequence_entry
    if self.check_token(BlockEntryToken):
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/yaml/scanner.py", line 579, in fetch_value
    self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in "/home/travis/build/google/oss-fuzz/projects/libraw/project.yaml", line 9, column 21

TravisBuddy Request Identifier: 438eddc0-a440-11ea-946e-418bdaa55b61

[alex]

Best of luck. We test this indirectly via imagemagick, and the libraw maintainer doesn't agree with certain types of undefined behavior being treated as bugs.

[evverx]

the libraw maintainer doesn't agree with certain types of undefined behavior being treated as bugs.

It's not the end of the world. It would probably make sense to just turn off UBSan (or a subset of the UBSan checks) then. I'd probably turn off MSan as well because I'm not exactly sure what is the point of unleashing all the sanitizers at the same time onto the project with only one maintainer.

To be fair, all the issues I was able to find where UB was mentioned were fixed except for one "left shift of negative value -1". Though I'm pretty sure that if instead of reporting the issue (with no way to reproduce it locally) someone had sent a PR fixing it it would probably have been merged.

[evverx]

FWIW since I took a look at the fuzz target, up until the point where it crashes with a stack overflow it prints a lot of the "unknown file: Unexpected end of file" lines which should probably be turned off when the code is fuzzed. Other than that, it doesn't seem to be guarded with if (size > ... ) return 0, which might lead to "timeout" issues that aren't really important. Plus, it looks like the fuzz target isn't linked with clang++ somewhere causing it to fail to compile with UBsan.

[evverx]

@jamiepinheiro just to clarify, I have nothing to do with LibRaw so whatever I'm saying should be taken with a pinch of salt :-) To judge from LibRaw/LibRaw#295 (comment), the LibRaw project doesn't have resources to maintain the integration and it's understandable because in its current form OSS-Fuzz requires much more manual intervention than advertised. I almost gave up after #3886 (comment) :-)

[jamiepinheiro]

@alex @evverx Thanks for the info, will make the appropriate changes! (We also add the size check once I figure out an appropriate one)

@evverx Yeaa unfortunately looks like it might not be accepted by him, however he does seem onboard for fixing problems found. I'm hoping that this configuration for fuzzing LibRaw doesn't require more manual work since its the same fuzzer we use on it internally. I've removed UBSan and MSan, hopefully that makes it more manageable!

[evverx]

however he does seem onboard for fixing problems found

@jamiepinheiro my understanding is that the LibRaw project would prefer to receive bug reports via GitHub and someone would have to kind of clone issues from Monorail to GitHub manually (which, in a sense, looks like a variation of #2985).

[jamiepinheiro]

@evverx I see, I'll provide him with the link to track that feature request as well. It's my understanding we don't open GitHub issues since they are immediate public and pose a security risk?

[evverx]

It's my understanding we don't open GitHub issues since they are immediate public and pose a security risk?

As far as I know, yes, the idea is to hide vulnerabilities and give maintainers some time to fix and patch them according to the standard disclosure policy. Though some project might want to choose to make some (or maybe all) bug reports public right away: #2582

[inferno-chromium]

@alex @evverx Thanks for the info, will make the appropriate changes! (We also add the size check once I figure out an appropriate one)

@evverx Yeaa unfortunately looks like it might not be accepted by him, however he does seem onboard for fixing problems found. I'm hoping that this configuration for fuzzing LibRaw doesn't require more manual work since its the same fuzzer we use on it internally. I've removed UBSan and MSan, hopefully that makes it more manageable!

Lets not remove MSan and UBSan. People usually forget reenabling them and libraw has been fuzzed before indirectly, so i dont expect a whole flow of bugs.

[evverx]

libraw has been fuzzed before indirectly, so i dont expect a whole flow of bugs.

The fuzz target crashes with a stack overflow in less than a minute so it doesn't look like the code paths it covers have been fuzzed extensively.

[TravisBuddy]

Travis tests have failed

Hey @jamiepinheiro,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

TravisBuddy Request Identifier: 8a67d220-a4e4-11ea-af7d-f32b9e1b312f

[TravisBuddy]

Hey @jamiepinheiro,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 01226390-a502-11ea-af7d-f32b9e1b312f

[TravisBuddy]

Hey @jamiepinheiro,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 011d5a80-a502-11ea-af7d-f32b9e1b312f