Overhaul of namespaces

.github/workflows/elixir.yml

@@ -128,6 +128,7 @@ jobs:
       - name: Run dev build
         run: |
           mix
+          mix credo
           mix elixir_make.checksum --only-local --ignore-unavailable --print
           mix hex.build
       - name: Run overhead profiling

CONTRIBUTING.md

@@ -0,0 +1,141 @@
+# Contributing to Beaver
+This document describes how to contribute to Beaver by introducing the idea behind how Beaver group its functionalities and how to set up the development environment.
+
+## Beaver's functionalities
+
+There are three main parts in Beaver. Here is a brief introduction to each part:
+- DSL: The syntax of Beaver SSA expression.
+- Utilities: Work with MLIR in Elixir way.
+- Bindings: Thin wrapper of MLIR CAPI.
+
+### DSL
+Modules including `Beaver`, `Beaver.Env`.
+
+DSL is the core part of Beaver. It uses Elixir's syntax to express MLIR semantics.
+
+### Utilities
+Modules including `Beaver.Walker`, `Beaver.Composer`
+
+Utilities are the helper functions that help to generate or manipulate MLIR IR. They are implemented in Elixir and is designed to be used in the DSL part to further enhance it and improve ergonomics.
+
+### Bindings
+Modules including `Beaver.MLIR`, `Beaver.MLIR.Dialect`, `Beaver.MLIR.Pass`, `Beaver.MLIR.Transform`, `Beaver.MLIR.ExecutionEngine`
+
+Bindings are the part that provides the interface to the MLIR CAPIs. It is implemented in Zig and is responsible for calling MLIR functions. Note that Beaver's bindings will try not to use `TableGen` and instead try to make use Elixir and Zig's meta-programming features to generate the bindings.
+
+## Development
+
+1. Install Elixir, [see installation guide](https://elixir-lang.org/install.html)
+2. Install Zig, [see installation guide](https://ziglang.org/learn/getting-started/#installing-zig)
+3. Install LLVM/MLIR
+
+- Option 1: Install with pip
+
+  ```bash
+  python3 -m pip install -r dev-requirements.txt
+  export LLVM_CONFIG_PATH=$(python3 -c 'import mlir;print(mlir.__path__[0])')/bin/llvm-config
+  ```
+
+- Option 2: Build from source https://mlir.llvm.org/getting_started/
+  Recommended install commands:
+
+  ```bash
+  cmake -B build -S llvm -G Ninja -DLLVM_ENABLE_PROJECTS=mlir \
+    -DLLVM_TARGETS_TO_BUILD="host" \
+    -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
+    -DLLVM_ENABLE_ASSERTIONS=ON \
+    -DLLVM_ENABLE_OCAMLDOC=OFF \
+    -DLLVM_ENABLE_BINDINGS=OFF \
+    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
+    -DCMAKE_INSTALL_PREFIX=${HOME}/llvm-install
+  cmake --build build -t install
+  export LLVM_CONFIG_PATH=$HOME/llvm-install/bin/llvm-config
+  ```
+
+  (Optional) To use Vulkan:
+
+  - Install Vulkan SDK (global installation is required), reference: https://vulkan.lunarg.com/sdk/home
+  - Setting environment variable by adding commands these to your bash/zsh profile:
+
+    ```bash
+    # you might need to change the version here
+    cd $HOME/VulkanSDK/1.3.216.0/
+    source setup-env.sh
+    cd -
+    ```
+
+  - Use `vulkaninfo` and `vkvia` to verify Vulkan is working
+  - Add `-DMLIR_ENABLE_VULKAN_RUNNER=ON` in LLVM CMake config command
+
+4. Develop and run tests
+- Clone this repo and `kinda` in the same directory
+  ```bash
+  git clone https://github.com/beaver-lodge/beaver.git
+  git clone https://github.com/beaver-lodge/kinda.git
+  ```
+- Make sure LLVM environment variable is set properly, as otherwise it might fail to build
+
+  ```bash
+  echo $LLVM_CONFIG_PATH
+  ```
+
+- Build and run Elixir tests
+  ```bash
+  mix deps.get
+  BEAVER_BUILD_CMAKE=1 mix test
+  # run tests with filters
+  mix test --exclude vulkan # use this to skip vulkan tests
+  mix test --only smoke
+  mix test --only nx
+  ```
+
+5. debug
+
+- setting environment variable to control Erlang scheduler number, `ERL_AFLAGS="+S 10:5"`
+- run mix test under LLDB, `scripts/lldb-mix-test`
+
+## Release a new version
+
+### Update Elixir source
+
+- Bump versions in [`README.md`](README.md) and [`mix.exs`](/mix.exs)
+
+### Linux
+
+- Run CI, which generates the new GitHub release uploaded to https://github.com/beaver-lodge/beaver-prebuilt/releases.
+- Update release url in [`mix.exs`](/mix.exs)
+
+### Mac
+
+- Run macOS build with:
+
+  ```bash
+  rm -rf _build/prod
+  bash scripts/build-for-publish.sh
+  ```
+
+- Upload the `beaver-nif-[xxx].tar.gz` file to release
+
+### Generate `checksum.exs`
+
+```
+rm checksum.exs
+mix clean
+mix
+mix elixir_make.checksum --all --ignore-unavailable --print
+```
+
+Check the version in the output is correct.
+
+### Publish to Hex
+
+```
+BEAVER_BUILD_CMAKE=1 mix hex.publish
+```
+
+## Format CMake files
+
+```bash
+python3 -m pip install cmake-format
+cmake-format -i native/**/CMakeLists.txt native/**/*.cmake
+```

README.md

@@ -65,12 +65,17 @@ defmodule ToyPass do
   def run(%MLIR.Operation{} = operation) do
     with 1 <- Beaver.Walker.regions(operation) |> Enum.count(),
          {:ok, _} <-
-           MLIR.Pattern.apply_(MLIR.Module.from_operation(operation), [replace_add_op(benefit: 2)]) do
-      MLIR.dump!(operation)
+           MLIR.apply_(MLIR.Module.from_operation(operation), [replace_add_op(benefit: 2)]) do
+      :ok
+    else
+      _ -> raise "unreachable"
     end
   end
 end
 
+use Beaver
+import MLIR.Transform
+ctx = MLIR.Context.create()
 ~m"""
 module {
   func.func @tosa_add(%arg0: tensor<1x3xf32>, %arg1: tensor<2x1xf32>) -> tensor<2x3xf32> {
@@ -79,9 +84,9 @@ module {
   }
 }
 """.(ctx)
-|> MLIR.Pass.Composer.append(ToyPass)
+|> Beaver.Composer.append(ToyPass)
 |> canonicalize
-|> MLIR.Pass.Composer.run!()
+|> Beaver.Composer.run!()
 ```
 
 ## Goals
@@ -151,26 +156,7 @@ import_deps: [:beaver],
 ### Projects built on top of Beaver
 
 - [Charm](https://github.com/beaver-lodge/charms): Compile a subset of Elixir to native targets.
-- [MLIR Accelerated Nx](https://github.com/beaver-lodge/manx): A backend for [Nx](https://github.com/elixir-nx/nx/tree/main/nx#readme).
-
-## Erlang apps in Beaver
-
-LLVM/MLIR is a giant project, and built around that Beaver have thousands of functions. To properly ship LLVM/MLIR and streamline the development process, we need to carefully break the functionalities at different level into different Erlang apps under the same umbrella.
-
-- `:beaver`: Elixir and C/C++ hybrid.
-  - Top level app ships the high level functionalities including IR generation and pattern definition.
-  - MLIR CAPI wrappers built by parsing LLVM/MLIR CAPI C headers and some middle level helper functions to hide the C pointer related operations. This app will add the loaded MLIR C library and managed MLIR context to Erlang supervisor tree. Rust is also used in this app, but mainly for LLVM/MLIR CMake integration.
-  - All the Ops defined in stock MLIR dialects, built by querying the registry. This app will ship MLIR Ops with Erlang idiomatic practices like behavior compliance.
-- `:kinda`: Elixir and Zig hybrid, generating NIFs from MLIR C headers. Repo: https://github.com/beaver-lodge/kinda
-
-### Notes on consuming and development
-
-- Only `:beaver` and `:kinda` are designed to be used as stand-alone app being directly consumed by other apps.
-- `:manx` could only work with Nx.
-- Although `:kinda` is built for Beaver, any Erlang/Elixir app with interest bundling some C API could take advantage of it as well.
-- The namespace `Beaver.MLIR` is for standard features are generally expected in any MLIR tools.
-- The namespace `Beaver` is for concepts and practice only exists in Beaver, which are mostly in a DSL provided as a set of macros (including `mlir/0`, `block/1`, `defpat/2`, etc). The implementations are usually under `Beaver.DSL` namespace.
-- In Beaver, there is no strict requirements on the consistency between the Erlang app name and Elixir module name. Two modules with same namespace prefix could locate in different Erlang apps (this happens a lot to the `Beaver.MLIR` namespace). Of course redefinition of Elixir modules with an identical name should be avoided.
+- [MLIR Accelerated Nx](https://github.com/beaver-lodge/manx): A backend for [Nx](https://github.com/elixir-nx/nx/).
 
 ## How it works?
 
@@ -280,117 +266,4 @@ Usually a function accepting a MLIR context to create an operation or type is ca
 
 ## Development
 
-1. Install Elixir, https://elixir-lang.org/install.html
-2. Install Zig, https://ziglang.org/learn/getting-started/#installing-zig
-3. Install LLVM/MLIR
-
-- Option 1: Install with pip
-
-  ```bash
-  python3 -m pip install -r dev-requirements.txt
-  export LLVM_CONFIG_PATH=$(python3 -c 'import mlir;print(mlir.__path__[0])')/bin/llvm-config
-  ```
-
-- Option 2: Build from source https://mlir.llvm.org/getting_started/
-  Recommended install commands:
-
-  ```bash
-  cmake -B build -S llvm -G Ninja -DLLVM_ENABLE_PROJECTS=mlir \
-    -DLLVM_TARGETS_TO_BUILD="host" \
-    -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-    -DLLVM_ENABLE_ASSERTIONS=ON \
-    -DLLVM_ENABLE_OCAMLDOC=OFF \
-    -DLLVM_ENABLE_BINDINGS=OFF \
-    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-    -DCMAKE_INSTALL_PREFIX=${HOME}/llvm-install
-  cmake --build build -t install
-  export LLVM_CONFIG_PATH=$HOME/llvm-install/bin/llvm-config
-  ```
-
-  (Optional) To use Vulkan:
-
-  - Install Vulkan SDK (global installation is required), reference: https://vulkan.lunarg.com/sdk/home
-  - Setting environment variable by adding commands these to your bash/zsh profile:
-
-    ```
-    # you might need to change the version here
-    cd $HOME/VulkanSDK/1.3.216.0/
-    source setup-env.sh
-    cd -
-    ```
-
-  - Use `vulkaninfo` and `vkvia` to verify Vulkan is working
-  - Add `-DMLIR_ENABLE_VULKAN_RUNNER=ON` in LLVM CMake config command
-
-4. Develop and run tests
-- Clone this repo and `kinda` in the same directory
-  ```bash
-  git clone https://github.com/beaver-lodge/beaver.git
-  git clone https://github.com/beaver-lodge/kinda.git
-  ```
-- Make sure LLVM environment variable is set properly, otherwise it might fail to build
-
-  ```bash
-  echo $LLVM_CONFIG_PATH
-  ```
-
-- Build and run Elixir tests
-  ```bash
-  mix deps.get
-  BEAVER_BUILD_CMAKE=1 mix test
-  # run tests with filters
-  mix test --exclude vulkan # use this to skip vulkan tests
-  mix test --only smoke
-  mix test --only nx
-  ```
-
-5. debug
-
-- setting environment variable to control Erlang scheduler number, `ERL_AFLAGS="+S 10:5"`
-- run mix test under LLDB, `scripts/lldb-mix-test`
-
-## Release a new version
-
-### Update Elixir source
-
-- Bump versions in [`README.md`](README.md) and [`mix.exs`](/mix.exs)
-
-### Linux
-
-- Run CI, which generates the new GitHub release uploaded to https://github.com/beaver-lodge/beaver-prebuilt/releases.
-- Update release url in [`mix.exs`](/mix.exs)
-
-### Mac
-
-- Run macOS build with:
-
-  ```bash
-  rm -rf _build/prod
-  bash scripts/build-for-publish.sh
-  ```
-
-- Upload the `beaver-nif-[xxx].tar.gz` file to release
-
-### Generate `checksum.exs`
-
-```
-rm checksum.exs
-mix clean
-mix
-mix elixir_make.checksum --all --ignore-unavailable --print
-```
-
-Check the version in the output is correct.
-
-### Publish to Hex
-
-```
-BEAVER_BUILD_CMAKE=1 mix hex.publish
-```
-
-## (Optional) Format CMake files
-
-```bash
-python3 -m pip install cmake-format
-cmake-format -i native/**/CMakeLists.txt native/**/*.cmake
-```
+Please refer to [Beaver's contributing guide](/CONTRIBUTING.md)

bench/enif_add.ex

@@ -10,18 +10,20 @@ defmodule AddENIF do
 
   @impl ENIFSupport
   def after_verification(op) do
-    s_table = op |> MLIR.Operation.from_module() |> MLIR.CAPI.mlirSymbolTableCreate()
-    found = MLIR.CAPI.mlirSymbolTableLookup(s_table, MLIR.StringRef.create("enif_make_int64"))
+    op
+    |> MLIR.Operation.from_module()
+    |> MLIR.Operation.with_symbol_table(fn s_table ->
+      found = MLIR.CAPI.mlirSymbolTableLookup(s_table, MLIR.StringRef.create("enif_make_int64"))
 
-    if MLIR.is_null(found) do
-      raise "Function not found"
-    end
+      if MLIR.null?(found) do
+        raise "Function not found"
+      end
 
-    if not MLIR.Dialect.Func.is_external(found) do
-      raise "Function is not external"
-    end
+      if not MLIR.Dialect.Func.external?(found) do
+        raise "Function is not external"
+      end
+    end)
 
-    MLIR.CAPI.mlirSymbolTableDestroy(s_table)
     op
   end
 

bench/enif_support.ex

@@ -15,7 +15,7 @@ defmodule ENIFSupport do
       @behaviour ENIFSupport
       def init(ctx) do
         create(ctx)
-        |> MLIR.Operation.verify!()
+        |> MLIR.verify!()
         |> after_verification()
         |> ENIFSupport.lower()
       end
@@ -25,17 +25,17 @@ defmodule ENIFSupport do
   @print_flag "ENIF_SUPPORT_PRINT_IR"
   def lower(op) do
     op
-    |> then(&if(System.get_env(@print_flag), do: MLIR.Transforms.print_ir(&1), else: &1))
-    |> MLIR.Pass.Composer.nested("func.func", "llvm-request-c-wrappers")
+    |> then(&if(System.get_env(@print_flag), do: MLIR.Transform.print_ir(&1), else: &1))
+    |> Beaver.Composer.nested("func.func", "llvm-request-c-wrappers")
     |> convert_scf_to_cf
     |> convert_arith_to_llvm()
     |> convert_index_to_llvm()
     |> convert_func_to_llvm()
-    |> MLIR.Pass.Composer.append("convert-vector-to-llvm{reassociate-fp-reductions}")
-    |> MLIR.Pass.Composer.append("finalize-memref-to-llvm")
-    |> then(&if(System.get_env(@print_flag), do: MLIR.Transforms.print_ir(&1), else: &1))
+    |> Beaver.Composer.append("convert-vector-to-llvm{reassociate-fp-reductions}")
+    |> Beaver.Composer.append("finalize-memref-to-llvm")
+    |> then(&if(System.get_env(@print_flag), do: MLIR.Transform.print_ir(&1), else: &1))
     |> reconcile_unrealized_casts
-    |> MLIR.Pass.Composer.run!()
+    |> Beaver.Composer.run!()
     |> then(fn m ->
       e = MLIR.ExecutionEngine.create!(m, opt_level: 3) |> Beaver.ENIF.register_symbols()
       %ENIFSupport{mod: m, engine: e}