Implement WASI system calls #112

ospencer · 2020-05-02T17:48:03Z

This PR implements system calls for Grain using the WebAssembly System Interface. This includes things like accessing system clocks, reading environment variables, and reading/writing to files.

We use @wasmer/wasi to make WASI calls within Node.js. While writing this PR, Node.js 14 was released, which ships with built-in support for WASI, with a similar API. After this lands, I'll make an issue to use the built-in WASI implementation for our Node runtime, but we'll still need to rely on @wasmer/wasi for the browser runtime.

New Modules

sys.gr

This module is meant to contain fairly thin wrappers around the WASI calls. It's not expected for users to interact with these directly, but rather with libraries built on top of these.

Environment

# Access command line arguments
# @returns Array<String> The positional string arguments to the process
argv : () -> Array<String>

# Access environment variables
# @returns Array<String> The environment variables supplied to the process
env : () -> Array<String>

Clocks

# Get the current time, in nanoseconds.
# Time value 0 corresponds with 1970-01-01T00:00:00Z.
# @returns Int64 The current time
realTime : () -> Int64

# Get the time of the system's high-resolution clock, in nanoseconds.
# This system clock cannot be adjusted and cannot have negative time jumps.
# The epoch of this clock is undefined, and thus time value 0 is meaningless.
# Useful for calculation of precise time intervals.
# @returns Int64 The current time
monotonicTime : () -> Int64

# Get the number of nanoseconds elapsed since the process began
# @returns Int64
processCpuTime : () -> Int64

# Get the number of nanoseconds elapsed since the thread began
# @returns Int64
threadCpuTime : () -> Int64

Files

# Open a file or directory
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param lookupFlags: List<LookupFlag> Flags which affect path resolution
# @param path: String The path to the file or directory
# @param openFlags: List<OpenFlag> Flags that decide how the path will be opened
# @param rights: List<Rights> The rights that dictate what may be done with the returned file descriptor
# @param rightsInheriting: List<Rights> The rights that dictate what may be done with file descriptors derived from this file descriptor
# @param flags: List<FdFlag> Flags which affect read/write operations on this file descriptor
# @returns FileDescriptor The opened file or directory
pathOpen : (FileDescriptor, List<LookupFlag>, String, List<OpenFlag>, List<Rights>, List<Rights>, List<FdFlag>) -> FileDescriptor

# Read from a file descriptor
# @param fd: FileDescriptor The file descriptor to read from
# @param nBytes: Number The maximum number of bytes to read from the file descriptor
# @returns (String, Number) The bytes read and the number of bytes read
fdRead : (FileDescriptor, Number) -> (String, Number)

# Read from a file descriptor without updating the file descriptor's offset
# @param fd: FileDescriptor The file descriptor to read from
# @param offset: Int64 The position within the file to begin reading
# @param nBytes: Number The maximum number of bytes to read from the file descriptor
# @returns (String, Number) The bytes read and the number of bytes read
fdPread : (FileDescriptor, Int64, Number) -> (String, Number)

# Write to a file descriptor
# @param fd: FileDescriptor The file descriptor to which data will be written
# @param buf: String The data to be written
# @returns Number The number of bytes written
fdWrite : (FileDescriptor, String) -> Number

# Read from a file descriptor without updating the file descriptor's offset
# @param fd: FileDescriptor The file descriptor to which data will be written
# @param buf: String The data to be written
# @param offset: Int64 The position within the file to begin writing
# @returns Number The number of bytes written
fdPwrite : (FileDescriptor, String, Int64) -> Number

# Allocate space within a file
# @param fd: FileDescriptor The file descriptor in which space will be allocated
# @param offset: Int64 The position within the file to begin writing
# @param nBytes: Int64 The number of bytes to allocate
fdAllocate : (FileDescriptor, Int64, Int64) -> Void

# Close a file descriptor
# @param fd: FileDescriptor The file descriptor to close
fdClose : (FileDescriptor) -> Void

# Synchronize the data of a file to disk
# @param fd: FileDescriptor The file descriptor to synchronize
fdDatasync : (FileDescriptor) -> Void

# Synchronize the data and metadata of a file to disk
# @param fd: FileDescriptor The file descriptor to synchronize
fdSync : (FileDescriptor) -> Void

# Retrieve information about a file descriptor
# @param fd: FileDescriptor The file descriptor of which to retrieve information
# @returns Stats A record containing the filetype, flags, rights, and inheriting rights associated with the file descriptor
fdStats : (FileDescriptor) -> Stats

# Update the flags associated with a file descriptor
# @param fd: FileDescriptor The file descriptor to update flags
fdSetFlags : (FileDescriptor, List<FdFlag>) -> Void

# Update the rights associated with a file descriptor
# @param rights: List<Rights> Rights to apply to the file descriptor
# @param rightsInheriting: List<Rights> Inheriting rights to apply to the file descriptor
fdSetRights : (FileDescriptor, List<Rights>, List<Rights>) -> Void

# Retrieve information about a file
# @param fd: FileDescriptor The file descriptor of the file to retrieve information
# @returns Filestats A record containing the information about the file
fdFilestats : (FileDescriptor) -> Filestats

# Set (truncate) the size of a file
# @param fd: FileDescriptor The file descriptor of the file to truncate
# @param size: Int64 The number of bytes to retain in the file
fdSetSize : (FileDescriptor, Int64) -> Void

# Set the access (created) time of a file
# @param fd: FileDescriptor The file descriptor of the file to update
# @param timestamp: Int64 The time to set
fdSetAccessTime : (FileDescriptor, Int64) -> Void

# Set the access (created) time of a file to the current time
# @param fd: FileDescriptor The file descriptor of the file to update
fdSetAccessTimeNow : (FileDescriptor) -> Void

# Set the modified time of a file
# @param fd: FileDescriptor The file descriptor of the file to update
# @param timestamp: Int64 The time to set
fdSetModifiedTime : (FileDescriptor, Int64) -> Void

# Set the modified time of a file to the current time
# @param fd: FileDescriptor The file descriptor of the file to update
fdSetModifiedTimeNow : (FileDescriptor) -> Void

# Read the entires of a directory
# @param fd: FileDescriptor The directory to read
# @returns Array<DirectoryEntry> An array of records containing information about each entry in the directory
fdReaddir : (FileDescriptor) -> Array<DirectoryEntry>

# Atomically replace a file descriptor by renumbering another file descriptor
# @param from: FileDescriptor The file descriptor to renumber
# @param to: FileDescriptor The file descriptor to overwrite
fdRenumber : (FileDescriptor, FileDescriptor) -> Void

# Update a file descriptor's offset
# @param fd: FileDescriptor The file descriptor to operate on
# @param delta: Int64 The number of bytes to move the offset
# @param whence: The location from which the offset is relative
# @returns Int64 The new offset of the file descriptor, relative to the start of the file
fdSeek : (FileDescriptor, Int64, Whence) -> Int64

# Read a file descriptor's offset
# @param fd: FileDescriptor The file descriptor to read the offset
# @returns Int64 The offset of the file descriptor, relative to the start of the file
fdTell : (FileDescriptor) -> Int64


# Create a directory
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param path: String The path to the new directory
pathCreateDirectory : (FileDescriptor, String) -> Void

# Retrieve information about a file
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param flags: List<LookupFlag> Flags which affect path resolution
# @param path: String The path to retrieve information about
# @returns Filestats A record containing information about the file
pathFilestats : (FileDescriptor, List<LookupFlag>, String) -> Filestats

# Set the access (created) time of a file
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param flags: List<LookupFlag> Flags which affect path resolution
# @param path: String The path to set the time
# @param timestamp: Int64 The time to set
pathSetAccessTime : (FileDescriptor, List<LookupFlag>, String, Int64) -> Void

# Set the access (created) time of a file to the current time
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param flags: List<LookupFlag> Flags which affect path resolution
# @param path: String The path to set the time
pathSetAccessTimeNow : (FileDescriptor, List<LookupFlag>, String) -> Void

# Set the modified time of a file
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param flags: List<LookupFlag> Flags which affect path resolution
# @param path: String The path to set the time
# @param timestamp: Int64 The time to set
pathSetModifiedTime : (FileDescriptor, List<LookupFlag>, String, Int64) -> Void

# Set the modified time of a file to the current time
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param flags: List<LookupFlag> Flags which affect path resolution
# @param path: String The path to set the time
pathSetModifiedTimeNow : (FileDescriptor, List<LookupFlag>, String) -> Void

# Create a hard link
# @param sourceWorkingDirectory: FileDescriptor The directory in which the source path resolution starts
# @param flags: List<LookupFlag> Flags which affect path resolution
# @param sourcePath: String The path to the source of the link
# @param targetWorkingDirectory: FileDescriptor The directory in which the target path resolution starts
# @param targetPath: String The path to the target of the link
pathLink : (FileDescriptor, List<LookupFlag>, String, FileDescriptor, String) -> Void

# Create a symlink
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param sourcePath: String The path to the source of the link
# @param targetPath: String The path to the target of the link
pathSymlink : (FileDescriptor, String, String) -> Void

# Unlink a file
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param path: String The path of the file to unlink
pathUnlink : (FileDescriptor, String) -> Void

# Read the contents of a symbolic link
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param path: String The path to the symlink
# @param nBytes: The number of bytes to read
# @returns (String, Number) The bytes read and the number of bytes read
pathReadlink : (FileDescriptor, String, Number) -> (String, Number)

# Remove a directory
# @param workingDirectory: FileDescriptor The directory in which path resolution starts
# @param path: String The path to the directory to remove
pathRemoveDirectory : (FileDescriptor, String) -> Void

# Rename a file or directory
# @param sourceWorkingDirectory: FileDescriptor The directory in which the source path resolution starts
# @param sourcePath: String The path of the file to rename
# @param targetWorkingDirectory: FileDescriptor The directory in which the target path resolution starts
# @param targetPath: String The new path of the file
pathRename : (FileDescriptor, String, FileDescriptor, String) -> Void

Process

# Terminate the process normally
# @param code: Number The value to exit with. An exit code of 0 is considered normal, with other values having meaning depending on the platform
exit : (Number) -> Void

# Send a signal to the process of the calling thread
# @param signal: Signal The signal to send
sigRaise : (Signal) -> Void

# Yield execution to the calling thread
schedYield : () -> Void

Random

# Produce a random number. This function can be slow, so it's best to seed a generator if lots of random data is needed
# @returns Number
random : () -> Number

stdint64.gr

This module provides some basic operations over the Int64 type.

fromNumber : Number -> Int64
toNumber : Int64 -> Number

lnot : Int64 -> Int64

land : (Int64, Int64) -> Int64
lor : (Int64, Int64) -> Int64
lxor : (Int64, Int64) -> Int64

lsl : (Int64, Number) -> Int64
lsr : (Int64, Number) -> Int64
asr : (Int64, Number) -> Int64

gt : (Int64, Int64) -> Bool
gte : (Int64, Int64) -> Bool
lt : (Int64, Int64) -> Bool
lte : (Int64, Int64) -> Bool

Updated Modules

lists.gr

Added for_each and for_each_i.

arrays.gr

Added map.

Improvements

Hex, Octal, and Binary literals

0xffffff
0o77777
0b101010101

Visual Number Separators

Underscores are now allowed in number literals.

let oneMillion = 1_000_000

Int32 and Int64 Literals

42l
99999999999999999L

Void constant

void

Bug Fixes

One sided if statements now work
The files foo and bar make inconsistent assumptions over interface baz error while running tests no longer occurs. NOTE: This is not yet fully fixed, and could still happen when running compiling and running normal programs, though this will no longer happen while running the tests. A bug will be filed once this PR lands.
Printing would break for data types if not all data types in a file were exported, now fixed
Printing ADT variants relied on order defined, now fixed

Other Noteworthy Changes

To support running in WASI environments, GRAIN$MAIN has been renamed to _start.

memory is now exported from all Grain modules under the name memory.

Everything Else

While a fair number of these new system calls will work as-is, many of them rely on some bugfixes I made that have yet to be released. As soon as a new release is cut, we'll upgrade to the proper version. In the meantime, say, to develop libraries on top of these calls, the user will need to manually build and link in wasmer-js.

ospencer · 2020-05-02T20:54:37Z

sock_recv, sock_send, and sock_shutdown are all missing because the specification is still in flux. We'll implement them (maybe) once decisions are final.

peblair

Looks mostly good; just a couple of comments.

peblair · 2020-05-07T14:29:05Z

cli/bin/grain.js

@@ -1,4 +1,5 @@
-#!/usr/bin/env node
+#!/bin/sh
+// 2>/dev/null; exec /usr/bin/env node --experimental-wasm-bigint "$0" "$@"


Heh. Basically, we need the experimental-wasm-bigint flag for @wasmer/wasi to properly pass int64s to/from JS, but Linux machines don't allow you to pass more than one argument to a shebang.

https://unix.stackexchange.com/questions/65235/universal-node-js-shebang/65295#65295

This is temporary until it's not experimental anymore.

Gotcha, that makes sense. Maybe a comment to explain so that we don't forget?

peblair · 2020-05-07T14:31:12Z

compiler/src/codegen/compcore.ml

@@ -129,6 +129,9 @@ let wrap_int64 n = add_dummy_loc (Values.I64Value.to_value n)
 let wrap_float32 n = add_dummy_loc (Values.F32Value.to_value n)
 let wrap_float64 n = add_dummy_loc (Values.F64Value.to_value n)

+let grain_number_max = 0x3fffffff
+let grain_number_min = -0x3fffffff


Is it possible to write this without the negative? Or at least a comment with the full hex of the negative

The native numbers in OCaml extend past 32 bits, so we can't write out the literal just here, but we could do Int32.to_int 0xC0000001l if you'd like that better?

Hmm. Maybe just putting (* 0xC0000001 *) next to it would be sufficient. I mostly just want it so that we can quickly identify what the memory layout of the number is, so it's not important that the actual source has the proper constant.

peblair · 2020-05-07T14:35:06Z

compiler/src/parsing/lexer.mll

@@ -96,6 +108,8 @@ rule token = parse
  | comment { process_newlines lexbuf; token lexbuf }
  | blank { token lexbuf }
  | newline_chars { process_newlines lexbuf; EOL }
+  | (signed_int as x) 'l' { INT32 (Int32.of_string x) }
+  | (signed_int as x) 'L' { INT64 (Int64.of_string x) }


Where's the inspiration for this syntax from?

It's the same syntax as C/C++/Java/OCaml. JavaScript uses the n suffix for BigInt but that doesn't feel quite right since other languages use n for whatever the system's native int size is.

Some references:
https://docs.microsoft.com/en-us/cpp/c-language/c-floating-point-constants?view=vs-2019
https://en.wikiversity.org/wiki/Advanced_Java/Declaring_float_and_long_Literals

Okay, cool. Looks good.

peblair · 2020-05-07T14:35:19Z

compiler/src/parsing/lexer.mll

@@ -105,6 +119,7 @@ rule token = parse
  | "else" { ELSE }
  | "true" { TRUE }
  | "false" { FALSE }
+  | "void" { VOID }


ospencer added 30 commits April 29, 2020 23:50

Rename GRAIN -> _start

ccff60c

Add wasi bindings to instantiation

f655af2

Support wasi in the runtime

5229196

Add grain runtime imports to stdlib-external

489894c

Add initial args_get implementation

248d21c

Add memory export

e175eb9

Have explicit allocations

c7ea4a1

Handle errors for argv call

7c03e12

Better tagging

24b5693

Create sys stdlib

aa8702c

Add Sys.env

2322c73

Add Int64 type

ae51fc3

Use big ints in the runtime

1b410c3

Reorganize stdlib-external

b239b3c

Use consistent runtime type ids

53abfc9

Implement pathOpen

7600e14

Implement fdRead and fdWrite

7498563

Add basic to/from number functions for Int64

97fb08e

Only increment module IDs for grain modules

ff94e76

Implement fdDatasync and fdSync

7c40e9a

Add one-sided if statements

406fd42

Support dec, hex, oct, and bin ints

4df3702

Support int32 and int64 literals

3131e56

Support uppercase 0x, 0b, 0o

ed6c554

Implement int64 bitwise operators

6db06ce

Implement int32 and int64 equals

9f81f86

Add void constant

8711f1c

Bump the initial table size

748ea54

Add for_each and for_each_i functions

2953660

Fix int64 types

579e7f9

ospencer added 20 commits April 29, 2020 23:50

Implement pathRemoveDirectory

f765a55

Implement pathRename

1db432e

Implement proc methods and random

af16b33

Better documentation for rights

2893e9a

Document Sys functions

669f1d7

Update the runner to use wasi start

695159c

Specify node version

7015be2

Fix shebang

40bd0fb

Add basic int64 tests

1974596

Add tests for hex, oct, and bin literals

7e949c1

Add tests for int32 and int64 literals

7bcaedb

Add smoke test for reading from a file

0ad43d3

Don't need to do anything fancy for builtin idents

83c1680

Make more consistent assumptions during tests

f606150

Reset compiler before compiling

9d8b2bb

Rename int64.gr -> int-64.gr

92a4a76

Add documentation for stdlib argv, env, and clocks

94566de

Rename int-64 -> stdint64

29a7ec1

Use /sandbox as preopened dir name

b532da2

Add one-sided if tests

00b3795

ospencer requested a review from peblair May 2, 2020 17:48

ospencer self-assigned this May 2, 2020

peblair requested changes May 7, 2020

View reviewed changes

Add comment about 32-bit literal

9983ccb

ospencer requested a review from peblair May 7, 2020 17:00

peblair approved these changes May 7, 2020

View reviewed changes

Add link to issue about experimental-wasm-bigint

f2d6acb

ospencer merged commit 81510d0 into master May 8, 2020

ospencer deleted the wasi branch May 8, 2020 00:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement WASI system calls #112

Implement WASI system calls #112

ospencer commented May 2, 2020

ospencer commented May 2, 2020

peblair left a comment

peblair May 7, 2020

ospencer May 7, 2020

peblair May 7, 2020

peblair May 7, 2020

ospencer May 7, 2020

peblair May 7, 2020

peblair May 7, 2020

ospencer May 7, 2020

peblair May 7, 2020

peblair May 7, 2020

Implement WASI system calls #112

Implement WASI system calls #112

Conversation

ospencer commented May 2, 2020

New Modules

sys.gr

Environment

Clocks

Files

Process

Random

stdint64.gr

Updated Modules

lists.gr

arrays.gr

Improvements

Hex, Octal, and Binary literals

Visual Number Separators

Int32 and Int64 Literals

Void constant

Bug Fixes

Other Noteworthy Changes

Everything Else

ospencer commented May 2, 2020

peblair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment