Updating data type constraints

[floydtree]

Related Issue: Discussed in the Weekly Call Sep 10, 2024.

Description of changes:

1. Removing max limits from file_hash_t, resource_uid_t & string_t
2. Fixing datetime_t regex, to allow t/T , z/Z in the timestamp string, allowing supported variations as defined in the RFC3339 - https://www.rfc-editor.org/rfc/rfc3339#section-5.6

[rmouritzen-splunk]

What's the rationale for removing the max lengths for these types? I don't mind them, but I'm curious why. (I missed most of today's weekly call. But this should be captured anyway.)

I suspect these were defined as sanity checks. Also, the string_t limit of 65535 seems like another Java-ism. Java has a 65535 "character" (UTF-16 code unit) limit for string constants (just constants, not arbitrary strings).

[floydtree]

What's the rationale for removing the max lengths for these types? I don't mind them, but I'm curious why. (I missed most of today's weekly call. But this should be captured anyway.)
I suspect these were defined as sanity checks. Also, the string_t limit of 65535 seems like another Java-ism. Java has a 65535 "character" (UTF-16 code unit) limit for string constants (just constants, not arbitrary strings).

Sure, good to recap and have it documented.

Current max length limits for file_hash_t was 64 chars, doesn't make sense even if you consider existing hashing algorithms like SHA-512, which yields 128 chars. Similarly for resource_uid_t, the limit of 64 is arbitrary, that type was added to capture various different uids, e.g. AWS ARNs, 64 chars is not a suitable max length. Likewise, 65535 char limit for string, again is arbitrary for an implementer of the framework with the most recent example coming from cisco, where they are looking to add env var strings, which can very well go beyond 65k chars. @pagbabian-splunk feel free to add more context if necessary.

[rmouritzen-splunk]

Excellent.

Minor point:

... SHA-512, which yield 128 chars.

SHA-512 is 64 characters (512 bits / 8 bits per byte = 64 bytes). (Edit: This is incorrect -- 64 bytes becomes 128 characters.)

The point stands though. There may well be longer hashes out there now or in the future.

[floydtree]

SHA-512 is 64 characters (512 bits / 8 bits per byte = 64 bytes). The point stands though. There may well be longer hashes out there now or in the future.

Right 64bytes, wouldn't that be 128 hexadec chars?

[rmouritzen-splunk]

Right 64bytes, wouldn't that be 128 hexadec chars?

Heh. Yup. I was thinking bytes, not hexadecimal characters, which double the length.

[pagbabian-splunk]

Right 64bytes, wouldn't that be 128 hexadec chars?

Heh. Yup. I was thinking bytes, not hexadecimal characters, which double the length.

Are you assuming Unicode characters? UTF-8 is usually what is considered, and yes, the limits may have been Java oriented but Java strings are actually not limited to 65K. I read somewhere they can be 2B characters.

[rmouritzen-splunk]

Are you assuming Unicode characters? UTF-8 is usually what is considered, and yes, the limits may have been Java oriented but Java strings are actually not limited to 65K. I read somewhere they can be 2B characters.

This is getting in the weeds... but hey, we are all engineers and we like getting technical.

I was just saying that the string_t limit of 65,535 seemed suspiciously like Java's limit on string constants (string literals in source files). Java max string length is as you mention limited by its use of a signed 32-bit int for length, so 2 some billion. I also mentioned that Java string lengths are in terms of its UTF-16 code units (where a Unicode characters require 1 or 2 UTF-16 units), which makes checking string lengths messy in Java anyway.

But this is all moot. The main point is that the 65,535 length limit for OCSF's string_t should be removed as it was a bit arbitrary and is too short for some known cases.