Extract platform-specifics from ENV to Crystal::System::Env and implement for win32 #6333

straight-shoota · 2018-07-04T23:11:46Z

Requires #5623 (for win32).

RX14 · 2018-07-04T23:36:50Z

src/crystal/system/unix/env.cr

+module Crystal::System::Env
+  # Sets an environment variable.
+  def self.set(key : String, value : String) : Nil
+    raise ArgumentError.new("Key contains null byte") if key.byte_index(0)


key.check_no_bull_byte and down the diff

That's what I initially thought, too. But check_no_null_byte loses the information which of the arguments was invalid (it's String contains null byte instead of Key contains null byte). Doesn't make much difference, so I'm not sure about it...

Then refactor it to take an argument

RX14 · 2018-07-04T23:40:43Z

src/crystal/system/win32/env.cr

+    System.retry_wstr_buffer do |buffer, small_buf|
+      length = LibC.GetEnvironmentVariableW(key.to_utf16, buffer, buffer.size)
+      if 0 < length < buffer.size
+        break String.from_utf16(buffer[0, length])


This is probably slightly clearer using return instead of break.

RX14 · 2018-07-04T23:43:55Z

src/env.cr

    value
  end

  # Returns `true` if the environment variable named *key* exists and `false`
  # if it doesn't.
  def self.has_key?(key : String) : Bool
-    !!LibC.getenv(key)
+    !!Crystal::System::Env.set?(key)


rename this to has_key? and remove the !!

RX14 · 2018-07-04T23:46:57Z

src/lib_c/x86_64-windows-msvc/c/winnt.cr

@@ -6,4 +6,6 @@ lib LibC
  alias WCHAR = UInt16
  alias LPSTR = CHAR*
  alias LPWSTR = WCHAR*
+  alias LPCWSTR = WCHAR*


LPCWSTR shouldn't be an alias since it's just constant which means nothing ABI-wise.

So... what should it be?

LPWSTR...

RX14 · 2018-07-04T23:49:17Z

src/lib_c/x86_64-windows-msvc/c/winbase.cr

@@ -5,6 +5,8 @@ require "c/int_safe"
 lib LibC
  fun GetLastError : DWORD

+  ERROR_ENVVAR_NOT_FOUND = 203_u32


https://github.com/crystal-lang/crystal/blob/master/src/winerror.cr#L253

straight-shoota · 2018-07-05T09:45:25Z

Rebased after #5623 was merged and ready for review 👍

RX14 · 2018-07-07T18:18:48Z

src/string/utf16.cr

+  #
+  # Invalid values are encoded using the unicode replacement char with
+  # codepoint `0xfffd`.
+  def self.from_utf16(pointer : Pointer(UInt16)) : {String, Pointer(UInt16)}


This is a breaking change and as such needs some more discussion.

Sure. When reading from a pointer it needs to advance the amount of UInt16 read. We can't determine that afterwards from String#bytesize because UTF-8 and UTF-16 have different bytesizes. We could rename the method with this changed behaviour (to maybe String.from_utf16_advancing_pointer). Or change the signature in some way. But I don't think it makes much sense to have both a version of this method (with pointer) that returns the advanced pointer and one that doesn't. It's easy to just ignore the returned pointer value if you don't need it.

asterite · 2018-07-08T12:24:12Z

This is why I recommended keeping the UTF-16 implementation internal until we got it working well in Windows, otherwise we are bound to make breaking changes.

I'd say for now don't worry about breaking changes here, I doubt anyone is using these methods, and if they are, well, it's a simple fix.

RX14 · 2018-07-08T13:29:57Z

I'm fine with this breaking change actually

Remove LPCWSTR

RX14 · 2018-07-08T19:20:11Z

src/string.cr

+  def check_no_null_byte(name = nil)
+    if byte_index(0)
+      name = "`#{name}` "
+      raise ArgumentError.new("String #{name}contains null byte")


Why is this on two lines??

Also, I'd rather the entire exception message was passed in, i.e. def check_no_null_byte(message = "String contains null byte").

I agree with the first comment, disagree with the second one.

The only thing you save by doing it the other way is a bit of typing, and the benefit of having fully customized error messages. For "Key contains null byte" vs "String `key` contains null byte" - the former is way better. Please, let's keep it simple, understandable and flexible.

Two lines because I forgot a if name in the first line. To avoid empty argument to be printed. That's why the specs fail.

Maybe full message is better. But actually, I don't think there is much benefit from it. Yes, it's more flexible, but I don't see any use for that. check_no_null_byte is only used to validate method arguments and for such it makes sense to declare the name of the argument if there are more than one. But apart from that, I don't see any need for customization.

Refactor String#check_no_null_byte

ysbaddaden · 2018-07-09T09:26:36Z

src/string.cr

-    raise ArgumentError.new("String contains null byte") if byte_index(0)
+  def check_no_null_byte(name = nil)
+    if byte_index(0)
+      name = "`#{name}` " if name


Why the backticks?

To show that this refers to a name. This formatting is used in many other compiler and stdlib error messages.

Weird, I never noticed that.

I think it's not that common

IMHO Key contains null byte reads way better than String 'key' contains null byte.

@asterite Do you mean name should be required?

Perhaps check_no_null_byte(what = "String") then just #{what} contains null byte?

Or just go with making it fully configurable.

I'm looking at the code, it always seems to be a check against an argument. To avoid repeating the argument name, maybe we can have:

class String macro check_no_null_byte(name) raise ArgumentError.new("Argument '{{name}}' contains null byte") if {{name}}.byte_index(0) {{name}} end end

Then it can be used like this:

def some_method(arg) String.check_no_null_byte(arg) # do something end

That way there's no string interpolation, and there's no need to repeat the argument name.

This is bikeshedding and overengineering. The diff is fine as-is.

@RX14 I was writing the same thing. All are valid options and it doesn't make a difference regarding ENV.
Anyone can send a PR later to refactor check_no_null_byte if you care enough.

Use String#check_no_null_byte

Add spec for empty environment variable. This fails on win32 because `GetEnvironmentVariableW` doesnt differentiate between unset variable and empty string. In contrast, `GetEnvironmentStringsW` contains empty variables.

straight-shoota · 2018-07-10T22:35:02Z

I discovered an issue on win32: GetEnvironmentVariableW doesn't differentiate between unset variable and empty string. Both have length == 0 and return value ERROR_ENVVAR_NOT_FOUND. The latter seems incorrect in case of an empty string value. In contrast, GetEnvironmentStringsW contains variables with empty values.

This could be used to identify empty values if length == 0 but will of course have a performance impact, and it's probably not really worth it for this edge case. There will however be a discrepancy between ENV[] and ENV.each regarding empty strings.

Unless someone has a idea how to prevent this, I don't think there is anything to be done.

RX14 · 2018-07-11T09:55:46Z

@straight-shoota i'm pretty skeptical that GetEnvironmentVariable is behaving this way. Can you print the result of GetLastError (pp WinError.new) before and directly after the API call?

RX14 · 2018-07-11T10:04:04Z

This stackoverflow answer seems to show the Win32 API correctly returning the value of an empty env var: https://stackoverflow.com/questions/26993642/call-to-environment-getenvironmentvariable-affects-subsequent-calls

But it does show there's definitely weirdness around this part of the API - perhaps GetEnvironmentVariable is failing to clear GetLastError correctly as all syscalls should.

Fix implementation for empty string on win32 using `SetLastError`.

straight-shoota · 2018-07-11T10:25:48Z

It actually seems to be the case that GetEnvironmentVariableW doesn't clear the last error. GetLastError is even ERROR_ENVVAR_NOT_FOUND if a positive length was returned.
And indeed, that's documented for SetLastError:

Most functions call SetLastError or SetLastErrorEx only when they fail. However, some system functions call SetLastError or SetLastErrorEx under conditions of success; those cases are noted in each function's documentation.

I fixed it by adding a call to SetLastError to set it to 0 before the call to GetEnvironmentVariableW. Now everything works 👍

RX14 · 2018-07-11T12:34:11Z

Oh actually, can you uncomment out the file/dir specs which were commented out due to using ENV?

straight-shoota · 2018-07-11T13:01:44Z

As far as I can see, they're all pending for other reasons.

RX14 · 2018-07-11T20:37:46Z

@straight-shoota these should go from macroed out to just pending

crystal/spec/std/file_spec.cr

Line 585 in 8d28cbb

# TODO: these specs don't compile on windows because ENV isn't ported

RX14 · 2018-07-11T22:15:55Z

spec/std/file_spec.cr

@@ -582,41 +582,38 @@ describe "File" do
      File.expand_path("~/a", "/tmp/gumby/ddd").should eq(File.join([home, "a"]))
    end

-    # TODO: these specs don't compile on windows because ENV isn't ported
    # TODO: remove /\A\/\// hack after this is removed from macros


please check the diff for #5623 and do this TODO too please

Sorry, I'm not sure what you mean... reinstating %r{\A//}?

yes, and remove the TODO comment

RX14 · 2018-07-24T11:15:41Z

ping, needs a second review

straight-shoota · 2018-07-26T18:18:26Z

@RX14 The commit message is a mess... it should've been squashed without all these fixups.

straight-shoota mentioned this pull request Jul 4, 2018

Coordinate porting to Windows #5430

Open

22 tasks

RX14 requested changes Jul 4, 2018

View reviewed changes

straight-shoota force-pushed the jm/feature/windows-env branch 2 times, most recently from ca2905f to 7efc63e Compare July 5, 2018 08:40

straight-shoota added 2 commits July 5, 2018 10:44

Extract Crystal::System::Env interface

05e9b91

Implement Crystal::System::Env for win32

be3115e

straight-shoota force-pushed the jm/feature/windows-env branch 3 times, most recently from 52da7cc to a68fd42 Compare July 5, 2018 09:45

Fix use correct bytesize of UTF-16 strings

9bf5cb8

straight-shoota force-pushed the jm/feature/windows-env branch from a68fd42 to 9bf5cb8 Compare July 5, 2018 10:38

RX14 reviewed Jul 7, 2018

View reviewed changes

RX14 added topic:stdlib breaking-change platform:windows Windows support based on the MSVC toolchain / Win32 API labels Jul 8, 2018

fixup! Implement Crystal::System::Env for win32

d2311e1

Remove LPCWSTR

RX14 requested changes Jul 8, 2018

View reviewed changes

fixup! Extract Crystal::System::Env interface

75715f7

Refactor String#check_no_null_byte

straight-shoota force-pushed the jm/feature/windows-env branch from ec4b780 to 5de7a7a Compare July 8, 2018 21:53

ysbaddaden reviewed Jul 9, 2018

View reviewed changes

straight-shoota mentioned this pull request Jul 9, 2018

Implement Path type #5635

Merged

straight-shoota force-pushed the jm/feature/windows-env branch from 5de7a7a to 2d01d82 Compare July 9, 2018 14:06

fixup! Implement Crystal::System::Env for win32

7adb2cc

Use String#check_no_null_byte

straight-shoota force-pushed the jm/feature/windows-env branch from 2d01d82 to 7adb2cc Compare July 9, 2018 14:24

RX14 approved these changes Jul 9, 2018

View reviewed changes

straight-shoota added 2 commits July 10, 2018 22:04

fixup! Fix use correct bytesize of UTF-16 strings

af0e517

fixup! Implement Crystal::System::Env for win32

7f6d3c1

Add spec for empty environment variable. This fails on win32 because `GetEnvironmentVariableW` doesnt differentiate between unset variable and empty string. In contrast, `GetEnvironmentStringsW` contains empty variables.

straight-shoota mentioned this pull request Jul 10, 2018

Enable Time::Location loader on win32 #6363

Merged

fixup! fixup! Implement Crystal::System::Env for win32

ca1967b

Fix implementation for empty string on win32 using `SetLastError`.

RX14 approved these changes Jul 11, 2018

View reviewed changes

Uncomment specs depending on ENV in file_specs

0b33619

straight-shoota force-pushed the jm/feature/windows-env branch from 66166cb to 0b33619 Compare July 11, 2018 21:05

RX14 reviewed Jul 11, 2018

View reviewed changes

fixup! Uncomment specs depending on ENV in file_specs

ce98927

RX14 approved these changes Jul 12, 2018

View reviewed changes

ysbaddaden approved these changes Jul 24, 2018

View reviewed changes

RX14 added this to the Next milestone Jul 24, 2018

RX14 merged commit 4f720ab into crystal-lang:master Jul 24, 2018

straight-shoota deleted the jm/feature/windows-env branch July 24, 2018 15:54

RX14 modified the milestones: Next, 0.26.0 Jul 30, 2018

straight-shoota mentioned this pull request Aug 1, 2018

Release 0.26.0 #6475

Merged

miketheman mentioned this pull request Aug 7, 2018

ICE: Module validation failed #6253

Open

straight-shoota mentioned this pull request Aug 7, 2018

Fix ENV#each #6499

Merged

Extract platform-specifics from ENV to Crystal::System::Env and implement for win32 #6333

Extract platform-specifics from ENV to Crystal::System::Env and implement for win32 #6333

Conversation

straight-shoota commented Jul 4, 2018

RX14 Jul 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

straight-shoota commented Jul 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asterite commented Jul 8, 2018

RX14 commented Jul 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RX14 Jul 8, 2018 • edited Loading

Choose a reason for hiding this comment

straight-shoota Jul 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RX14 Jul 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

straight-shoota commented Jul 10, 2018 • edited Loading

RX14 commented Jul 11, 2018

RX14 commented Jul 11, 2018

straight-shoota commented Jul 11, 2018

RX14 commented Jul 11, 2018

straight-shoota commented Jul 11, 2018

RX14 commented Jul 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RX14 Jul 12, 2018 • edited Loading

Choose a reason for hiding this comment

RX14 commented Jul 24, 2018

straight-shoota commented Jul 26, 2018

RX14 Jul 4, 2018 •

edited

Loading

RX14 Jul 8, 2018 •

edited

Loading

straight-shoota Jul 8, 2018 •

edited

Loading

RX14 Jul 9, 2018 •

edited

Loading

straight-shoota commented Jul 10, 2018 •

edited

Loading

RX14 Jul 12, 2018 •

edited

Loading