GCC badge requirements should be more than "-Wall"

[adamshostack]

There are two issues which underly this recommendation.

1. -Wall does not warn for everything. As the man page explains: "some warning flags are not implied by -Wall...Some of them are enabled by -Wextra but many of them must be enabled individually."

2. There is a set of security functionality available in GCC. This Stackexchange article seems to be a super-set of recommendations on other pages (rather than a review of GCC options or consulting with experts) http://security.stackexchange.com/questions/24444/what-is-the-most-hardened-set-of-options-for-gcc-compiling-c-c

[david-a-wheeler]

The "passing" badge requirements do not require -Wall; that is simply used as as an example. I'm happy to chat about possibly changing that, but there are a lot problems even getting projects to simply the "passing" level. I expect that our higher-level badges, once created, will have much more stringent warning flag (and test) requirements. Below are some details. I'll first describe the current badge criteria state - then why that's the current state.

The current criterion "warnings" requires: "The project MUST enable one or more compiler warning flags, a "safe" language mode, or use a separate "linter" tool to look for code quality errors or common simple mistakes, if there is at least one FLOSS tool that can implement this criterion in the selected language." Notice that the MUST includes "one or more compiler warning flags" without specifying a particular one.

The details say, "Examples of compiler warning flags include gcc/clang "-Wall". Examples of a "safe" language mode include Javascript "use strict" and perl5's "use warnings". A separate "linter" tool is simply a tool that examines the source code to look for code quality errors or common simple mistakes." Notice that this is stated as an example - you don't actually have to use that particular compiler warning flag.

All criteria lists have potential problems. The potential problem with the current set is that someone could drop in 1 or more warning flags that are nearly useless, and not enable useful ones. Noted. However, once you enable any warning flags, you have to set up your infrastructure to handle warning flags... which makes it easier to add more. We have a similar stance about tests: we don't mandate a specific level of test coverage, but we require that you do some testing, because it's much easier to get better at something once you get started.

But why is it this way? Unfortunately, even -Wall is too difficult or counterproductive for many projects if you use gcc or clang. Never mind -Wextra, which includes a number of checks that the gcc developers expect to be too harsh for many projects (or they'd be in -Wall). Even the advocate on StackExchange noted that those flags "Some of these will not work for all circumstances." The "passing" level is supposed to work for all projects and all circumstances. In addition, we're trying to cover lots of different technologies - people who only do Javascript won't know (or care) about -Wall or -Wextra, and not all C compilers are gcc or clang.

If you want to advocate a different requirement at the "passing" level, that's great, propose away. However, given how few projects have even managed "passing" so far, I don't think that's a good thing to do. We developed the criteria by looking at what well-run projects tend to do, but requiring projects to do the whole set turns out to be harder even though each one is individually common.

But don't stop there. At the next higher level ("gold"?) this is definitely a criterion we'd like to add. I'm currently thinking that at least test coverage criteria, hardening criteria, and warning criteria should be increased or added at the higher badge level. The trick will be to deal with the variety of technologies. I'm okay with having a few general guidelines, and then specific requirements for specific common cases (e.g., C/C++ on gcc or clang; Node.js; etc.). Please reply with a proposal that way, if you'd like.

[david-a-wheeler]

Note: This is highly related to issue #453 ; that issue recommends that projects support the standard ways to add warning and hardening flags.

[david-a-wheeler]

I agree that creating a set of "suggested settings" is a good idea. It would probably be technology-specific, and there would also probably need to have different levels.

While they aren't compiler settings, many organizations have suggested settings for security. For example:

• Mozilla's Security/Server Side TLS, which provides recommended settings for web servers. They have 3 setting levels: Modern, Intermediate, and Old.
• DISA STIGs provide configurations settings for various devices used by the US DoD.

My slides on developing secure software mention some flags, too.

• Tools mention -Wall, -Wextra, and -pedantic:
• Error handling mention something not noted in your sources, namely, consider using compiler flags to change undefined behavior into a safer C dialect. C compilers by default presume that authors never make mistakes, and rush to turn code into something insecure if possible. The Linux kernel folks intentionally turn on some flags to reduce the risk of an error becoming a vulnerability after painful experience. Some potentially-relevant flags (gcc or clang) include -fwrapv (wrap signed integer overflow), -fno-strict-overflow, -fno-strict-aliasing flag, and -fno-delete-null-pointer-checks

[adamshostack]

I hadn't considered badge leveling. It makes sense that the base badge not require things which are too hard. (I know there are complex issues around naming, and was confused by the name 'best practices' ) When you ask for a proposal for the 'next level', are you thinking that there will be a base badge and a gold, or a base, gold, platinum, diamond and adamantium?

Are you including things like -fstack-protector-all in your error handling category?

[david-a-wheeler]

For passing+1 we currently plan to have a "hardening" criterion.

Here's the current draft text:

• (Future criterion) Hardening mechanisms SHOULD be used so software defects are less likely to result in security vulnerabilities.
  ^[hardening]

  Details:

  Hardening mechanisms may include HTTP headers like Content Security Policy (CSP), compiler flags to mitigate attacks (such as -fstack-protector), or compiler flags to eliminate undefined behavior. For our purposes least privilege is not considered a hardening mechanism (least privilege is important, but separate).

It'd be nice to provide some sort of baseline for warning flags, but it becomes way too specific to particular languages and compilers, and in at least some cases the warning flags are too noisy for normal use.

[david-a-wheeler]

In addition, at higher levels we plan to change "warnings_strict" to a SHOULD or MUST, so it'd become "Projects SHOULD/MUST be maximally strict with warnings, where practical."

We'll put this in the draft higher-level criteria, and close this comment for now. I expect we'll hear more once we get the draft out :-).

Thanks for the feedback!