Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement lexer int128 support #11571

Merged
merged 10 commits into from
Dec 18, 2021
Merged

Implement lexer int128 support #11571

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
128d1ac
Implement lexer int128 support
BlobCodes Dec 11, 2021
043f9f1
format code
BlobCodes Dec 11, 2021
3e5399b
Show integer not fitting in U/Int128 on failed conversion
BlobCodes Dec 12, 2021
6e1f0c3
Try conversion to decimal using u64 before using u128
BlobCodes Dec 12, 2021
7c58a92
Add i128 support in MathInterpreter
BlobCodes Dec 12, 2021
7e7e172
Remove everything not lexer-related
BlobCodes Dec 12, 2021
e25b754
Apply rules from discussion in #8373
BlobCodes Dec 12, 2021
92728ef
Add comment about reasoning behind using u64 for base 10 conversion f…
BlobCodes Dec 13, 2021
a9dc57c
Add old specs
BlobCodes Dec 13, 2021
36707d0
Merge branch 'crystal-lang:master' into feature/lexer-int128-support
BlobCodes Dec 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 28 additions & 16 deletions spec/compiler/lexer/lexer_spec.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,16 +302,28 @@ describe "Lexer" do
assert_syntax_error "18446744073709551616_i32", "18446744073709551616 doesn't fit in an Int32"
assert_syntax_error "9999999999999999999_i32", "9999999999999999999 doesn't fit in an Int32"

assert_syntax_error "-9999999999999999999", "-9999999999999999999 doesn't fit in an Int64"
assert_syntax_error "-99999999999999999999", "-99999999999999999999 doesn't fit in an Int64"
assert_syntax_error "-11111111111111111111", "-11111111111111111111 doesn't fit in an Int64"
assert_syntax_error "-9223372036854775809", "-9223372036854775809 doesn't fit in an Int64"
assert_syntax_error "18446744073709551616", "18446744073709551616 doesn't fit in an UInt64"

assert_syntax_error "9223372036854775808_i128", "9223372036854775808 doesn't fit in an Int64. Int128 literals that don't fit in an Int64 are currently not supported"
assert_syntax_error "-9223372036854775809_i128", "-9223372036854775809 doesn't fit in an Int64. Int128 literals that don't fit in an Int64 are currently not supported"
assert_syntax_error "118446744073709551616_u128", "118446744073709551616 doesn't fit in an UInt64. UInt128 literals that don't fit in an UInt64 are currently not supported"
assert_syntax_error "18446744073709551616_u128", "18446744073709551616 doesn't fit in an UInt64. UInt128 literals that don't fit in an UInt64 are currently not supported"
assert_syntax_error "-9999999999999999999_i64", "-9999999999999999999 doesn't fit in an Int64"
assert_syntax_error "-99999999999999999999_i64", "-99999999999999999999 doesn't fit in an Int64"
assert_syntax_error "-11111111111111111111_i64", "-11111111111111111111 doesn't fit in an Int64"
assert_syntax_error "-9223372036854775809_i64", "-9223372036854775809 doesn't fit in an Int64"
assert_syntax_error "18446744073709551616_u64", "18446744073709551616 doesn't fit in an UInt64"

assert_syntax_error "340282366920938463463374607431768211456", "340282366920938463463374607431768211456 doesn't fit in an UInt128"
assert_syntax_error "-170141183460469231731687303715884105729", "-170141183460469231731687303715884105729 doesn't fit in an Int128"
assert_syntax_error "-999999999999999999999999999999999999999", "-999999999999999999999999999999999999999 doesn't fit in an Int128"

it_lexes_number :i128, ["9223372036854775808_i128", "9223372036854775808"]
it_lexes_number :i128, ["-9223372036854775809_i128", "-9223372036854775809"]
it_lexes_number :u128, ["118446744073709551616_u128", "118446744073709551616"]
it_lexes_number :u128, ["18446744073709551616_u128", "18446744073709551616"]
it_lexes_number :i128, "-9223372036854775809"
it_lexes_number :i128, "118446744073709551616"
it_lexes_number :i128, "18446744073709551616"
it_lexes_number :i128, "170141183460469231731687303715884105727"
it_lexes_number :u128, "170141183460469231731687303715884105728"
it_lexes_number :u128, "340282366920938463463374607431768211455"
it_lexes_number :u128, ["0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF", "340282366920938463463374607431768211455"]
it_lexes_number :i128, ["-0x80000000000000000000000000000000", "-170141183460469231731687303715884105728"]
assert_syntax_error "-1_u128", "Invalid negative value -1 for UInt128"

assert_syntax_error "1__1", "consecutive underscores in numbers aren't allowed"
Expand Down Expand Up @@ -351,17 +363,17 @@ describe "Lexer" do
it_lexes_u64 [["0xffff_ffff_ffff_ffff", "18446744073709551615"], ["0o177777_77777777_77777777", "18446744073709551615"], ["0b11111111_11111111_11111111_11111111_11111111_11111111_11111111_11111111", "18446744073709551615"]]
it_lexes_u64 [["0x00ffffffffffffffff", "18446744073709551615"], ["0o001777777777777777777777", "18446744073709551615"], ["0b001111111111111111111111111111111111111111111111111111111111111111", "18446744073709551615"]]
# 2**64
assert_syntax_error "0x10000_0000_0000_0000", "0x10000_0000_0000_0000 doesn't fit in an UInt64"
it_lexes_number :i128, ["0x10000_0000_0000_0000", "18446744073709551616"]
BlobCodes marked this conversation as resolved.
Show resolved Hide resolved
assert_syntax_error "0x10000_0000_0000_0000_u64", "0x10000_0000_0000_0000 doesn't fit in an UInt64"
assert_syntax_error "0xfffffffffffffffff_u64", "0xfffffffffffffffff doesn't fit in an UInt64"
assert_syntax_error "0o200000_00000000_00000000", "0o200000_00000000_00000000 doesn't fit in an UInt64"
assert_syntax_error "0b100000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000", "0b100000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000 doesn't fit in an UInt64"
assert_syntax_error "0o200000_00000000_00000000_u64", "0o200000_00000000_00000000 doesn't fit in an UInt64"
assert_syntax_error "0b100000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_u64", "0b100000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000 doesn't fit in an UInt64"
# Very large
assert_syntax_error "0x1afafafafafafafafafafaf", "0x1afafafafafafafafafafaf doesn't fit in an UInt64"
assert_syntax_error "0x1afafafafafafafafafafafu64", "0x1afafafafafafafafafafaf doesn't fit in an UInt64"
assert_syntax_error "0x1afafafafafafafafafafafi32", "0x1afafafafafafafafafafaf doesn't fit in an Int32"
assert_syntax_error "0o1234567123456712345671234567", "0o1234567123456712345671234567 doesn't fit in an UInt64"
assert_syntax_error "0o1234567123456712345671234567u64", "0o1234567123456712345671234567 doesn't fit in an UInt64"
assert_syntax_error "0o12345671234567_12345671234567_i8", "0o12345671234567_12345671234567 doesn't fit in an Int8"
assert_syntax_error "0b100000000000000000000000000000000000000000000000000000000000000000", "0b100000000000000000000000000000000000000000000000000000000000000000 doesn't fit in an UInt64"
assert_syntax_error "0b100000000000000000000000000000000000000000000000000000000000000000u64", "0b100000000000000000000000000000000000000000000000000000000000000000 doesn't fit in an UInt64"

it_lexes_i64 [["0o700000000000000000000", "8070450532247928832"]]
it_lexes_u64 [["0o1000000000000000000000", "9223372036854775808"]]
Expand Down
6 changes: 2 additions & 4 deletions src/compiler/crystal/codegen/codegen.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,11 +475,9 @@ module Crystal
when :u64
@last = int64(node.value.to_u64)
when :i128
# TODO: implement String#to_i128 and use it
@last = int128(node.value.to_i64)
@last = int128(node.value.to_i128)
when :u128
# TODO: implement String#to_u128 and use it
@last = int128(node.value.to_u64)
@last = int128(node.value.to_u128)
when :f32
@last = float32(node.value)
when :f64
Expand Down
22 changes: 12 additions & 10 deletions src/compiler/crystal/codegen/const.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,16 +194,18 @@ class Crystal::CodeGenVisitor
# We inline constants. Otherwise we use an LLVM const global.
@last =
case value = const.compile_time_value
when Bool then int1(value ? 1 : 0)
when Char then int32(value.ord)
when Int8 then int8(value)
when Int16 then int16(value)
when Int32 then int32(value)
when Int64 then int64(value)
when UInt8 then int8(value)
when UInt16 then int16(value)
when UInt32 then int32(value)
when UInt64 then int64(value)
when Bool then int1(value ? 1 : 0)
when Char then int32(value.ord)
when Int8 then int8(value)
when Int16 then int16(value)
when Int32 then int32(value)
when Int64 then int64(value)
when Int128 then int128(value)
when UInt8 then int8(value)
when UInt16 then int16(value)
when UInt32 then int32(value)
when UInt64 then int64(value)
when UInt128 then int128(value)
else
set_current_debug_location node if @debug.line_numbers?
last = read_const_pointer(const)
Expand Down
18 changes: 10 additions & 8 deletions src/compiler/crystal/syntax/ast.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,14 +249,16 @@ module Crystal

def integer_value
case kind
when :i8 then value.to_i8
when :i16 then value.to_i16
when :i32 then value.to_i32
when :i64 then value.to_i64
when :u8 then value.to_u8
when :u16 then value.to_u16
when :u32 then value.to_u32
when :u64 then value.to_u64
when :i8 then value.to_i8
when :i16 then value.to_i16
when :i32 then value.to_i32
when :i64 then value.to_i64
when :i128 then value.to_i128
when :u8 then value.to_u8
when :u16 then value.to_u16
when :u32 then value.to_u32
when :u64 then value.to_u64
when :u128 then value.to_u128
else
raise "Bug: called 'integer_value' for non-integer literal"
end
Expand Down
39 changes: 24 additions & 15 deletions src/compiler/crystal/syntax/lexer.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1450,17 +1450,13 @@ module Crystal
@token.raw = ":#{value}" if @wants_raw
end

macro gen_check_int_fits_in_size(type, method, size, number_size, raw_number_string, start, pos_before_suffix, negative, actual_type = nil)
macro gen_check_int_fits_in_size(type, method, size, number_size, raw_number_string, start, pos_before_suffix, negative)
{% if type.stringify.starts_with? "U" %}
raise "Invalid negative value #{string_range({{start}}, {{pos_before_suffix}})} for {{type}}", @token, (current_pos - {{start}}) if {{negative}}
{% end %}

if [email protected] || {{number_size}} > {{size}} || ({{number_size}} == {{size}} && {{raw_number_string}}.to_{{method.id}}? == nil)
{% if actual_type.nil? %}
raise_value_doesnt_fit_in "{{type}}", {{start}}, {{pos_before_suffix}}
{% else %}
raise("#{string_range({{start}}, {{pos_before_suffix}})} doesn't fit in an {{actual_type}}. {{type}} literals that don't fit in an {{actual_type}} are currently not supported", @token, current_pos - {{start}})
{% end %}
raise_value_doesnt_fit_in "{{type}}", {{start}}, {{pos_before_suffix}}
end
end

Expand Down Expand Up @@ -1548,7 +1544,7 @@ module Crystal
raw_number_string = raw_number_string.delete('_') if has_underscores
@token.value = raw_number_string
else
base10_number_string = raw_number_string.to_u64?(base: base, underscore: true).try &.to_s
base10_number_string = raw_number_string.to_u128?(base: base, underscore: true).try &.to_s
BlobCodes marked this conversation as resolved.
Show resolved Hide resolved
if base10_number_string
number_size = base10_number_string.size
first_byte = @reader.string.byte_at(start).chr
Expand All @@ -1574,17 +1570,30 @@ module Crystal
if raw_number_string.to_i64?
BlobCodes marked this conversation as resolved.
Show resolved Hide resolved
:i64
elsif negative
raise_value_doesnt_fit_in(Int64, start, pos_before_suffix)
:i128
else
:u64
end
when 20
raise_value_doesnt_fit_in(Int64, start, pos_before_suffix) if negative
raise_value_doesnt_fit_in(UInt64, start, pos_before_suffix) unless raw_number_string.to_u64?
:u64
if !negative && raw_number_string.to_u64?
:u64
else
:i128
end
when 21..38 then :i128
when 39
if raw_number_string.to_i128?
:i128
elsif negative
raise_value_doesnt_fit_in(Int128, start, pos_before_suffix)
elsif raw_number_string.to_u128?
:u128
else
raise_value_doesnt_fit_in(UInt128, start, pos_before_suffix)
end
else
raise_value_doesnt_fit_in(Int64, start, pos_before_suffix) if negative
raise_value_doesnt_fit_in(UInt64, start, pos_before_suffix)
raise_value_doesnt_fit_in(Int128, start, pos_before_suffix) if negative
raise_value_doesnt_fit_in(UInt128, start, pos_before_suffix)
end
else
case @token.number_kind
Expand All @@ -1596,8 +1605,8 @@ module Crystal
when :u32 then gen_check_int_fits_in_size(UInt32, :u32, 10, number_size, raw_number_string, start, pos_before_suffix, negative)
when :i64 then gen_check_int_fits_in_size(Int64, :i64, 19, number_size, raw_number_string, start, pos_before_suffix, negative)
when :u64 then gen_check_int_fits_in_size(UInt64, :u64, 20, number_size, raw_number_string, start, pos_before_suffix, negative)
when :i128 then gen_check_int_fits_in_size(Int128, :i64, 19, number_size, raw_number_string, start, pos_before_suffix, negative, Int64)
when :u128 then gen_check_int_fits_in_size(UInt128, :u64, 20, number_size, raw_number_string, start, pos_before_suffix, negative, UInt64)
when :i128 then gen_check_int_fits_in_size(Int128, :i128, 39, number_size, raw_number_string, start, pos_before_suffix, negative)
when :u128 then gen_check_int_fits_in_size(UInt128, :u128, 39, number_size, raw_number_string, start, pos_before_suffix, negative)
end
end
end
Expand Down