Deliver STL together with compiler 🎉

Also includes minor work on including types in the LSP

CHANGELOG.md

@@ -0,0 +1,2 @@
+# Changelog
+- Standard library is shipped with the compiler now

README.md

@@ -68,9 +68,9 @@ There's a few categories of HDLs as I see it nowadays. I shall visit them in tur
 These languages were originally designed as Hardware *Description* Languages, meant to describe exactly how hand-drawn hardware components function. Later on a "Synthesizeable Subset" was created from these languages to actually create hardware from them. The issue is, these old languages still carry this simulation-first core design. The feature-set that's actually available for synthesis is rather small, and common constructs like pipelining routinely introduce bugs. Even things like what inputs and outputs mean are left vague. 
 
 #### High Level Synthesis: [BlueSpec](https://bluespec.com), [Intel OneAPI](https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html), [Xilinx Vitis](https://www.xilinx.com/products/design-tools/vitis.html)
-This approach attempts to generate hardware from an imperative description in an existing software language, usually C++. They rely on control flow analysis and a clever compiler to turn this description into hardware that actually performs the operation. The core issue with these is an over-reliance on such compiler smarts. This usually means fiddling with compiler directives until the compiler actually outputs the hardware you originally had in mind. In some cases, it may not even be possible to express the hardware you intend because the compiler designers didn't provide it. The final nail is that optimization on such generated hardware is nigh-impossible. The powerful synthesis tools like Intel Quartus and Vivado with their timing analyzers are unuseable. The tradeoff is inefficient use of resources and lower clock speeds.
+This approach attempts to generate hardware from an imperative description in an existing software language, usually C++. They rely on control flow analysis and a clever compiler to turn this description into hardware that actually performs the operation. The core issue with these is an over-reliance on such compiler smarts. This usually means fiddling with compiler directives until the compiler actually outputs the hardware you originally had in mind. In some cases, it may not even be possible to express the hardware you intend because the compiler designers didn't provide it. This means ptimization on such generated hardware is often impossible. The powerful synthesis tools like Intel Quartus and Vivado with their timing analyzers are unuseable. The tradeoff is inefficient use of resources and lower clock speeds.
 
-One final thing that must be said about the corporate HLS flows, is that the promise of 'portability' is absolute bogus. These systems are simply more attempts to build walled gardens around their respective platforms. This is evident from Intel's deprecation of the more open OpenCL frontend they used to have, in favor of their platform-locked Intel OneAPI. (Which, funnily enough, is just a thin wrapper around the old OpenCL codebase). If I sound salty, it is because I am. 
+A final thing that must be said about the corporate HLS flows, is that the promise of 'portability' is absolute bogus. These systems are simply more attempts to build walled gardens around their respective platforms. This is evident from Intel's deprecation of the more open OpenCL frontend they used to have, in favor of their platform-locked Intel OneAPI. (Which, funnily enough, is just a thin wrapper around the old OpenCL codebase). If I sound salty, it is because I am. 
 
 #### Embedded Languages such as [Chisel](https://www.chisel-lang.org/) and [SpinalHDL](https://github.com/SpinalHDL/SpinalHDL):
 If one is being pedantic, they actually shouldn't actually be called "languages" per se, but rather hardware construction libraries within an existing software language; usually Scala. There is a solid argument to be made for this style though. Why invent a new meta-language for the generation of hardware when there's widely-used software languages already out there? My main arguments against this approach are written below, but they can be summed up as the language designers having made the tradeoff of reducing development time on the compiler sacrificing the useability of the final product. 
@@ -181,8 +181,9 @@ In this example, we create a memory block with a read port and a write port. Thi
 - [x] Intrinsic Modules
 - [x] Can Parse FIFO implementation
 - [ ] Clock Domain Crossings
-- [ ] Rhythm Syntax
+- ~~[ ] Rhythm Syntax~~
 - [ ] Interface Generator Syntax
+- [ ] Standard Library Bundled with compiler
 
 ### Performance, Linking and Name Resolution
 - [ ] Namespaces
@@ -233,6 +234,7 @@ In this example, we create a memory block with a read port and a write port. Thi
 - [x] Renaming
 - [x] Basic code completion
 - [ ] Port code completion
+- [ ] Struct field code completion
 - [ ] Per-Line Resource Utilization Reporting
 
 ### Code Generation
@@ -248,6 +250,8 @@ In this example, we create a memory block with a read port and a write port. Thi
 - [ ] Dedekind Kernel Port
 - [ ] Sparse Matrix Multiply
 - [ ] RISC-V CPU
+- [ ] Enigma Machine
+- [ ] Enigma Code Breaking
 
 ### Safety through Interface Asserts (PDL-style asserts)
 - [ ] btor2?

build.rs

@@ -0,0 +1,36 @@
+use std::env;
+use std::fs;
+use std::path::PathBuf;
+
+fn main() {
+    let out_dir = env::var("OUT_DIR").unwrap();
+    let install_dir = PathBuf::from(out_dir)
+        .join("../../..")
+        .join("share")
+        .join("sus_compiler")
+        .join("std");
+
+    fs::create_dir_all(&install_dir).expect("Failed to create std_lib directory");
+
+    copy_dir("stl", &install_dir).expect("Failed to copy STL folder");
+
+    // Print the path to make it available during the build
+    println!("cargo:rustc-env=SUS_COMPILER_STD_LIB_PATH={}", install_dir.display());
+}
+
+// Helper function to copy a directory and its contents recursively
+fn copy_dir(src: &str, dst: &PathBuf) -> std::io::Result<()> {
+    for entry in fs::read_dir(src)? {
+        let entry = entry?;
+        let path = entry.path();
+        let dest_path = dst.join(entry.file_name());
+
+        if path.is_dir() {
+            fs::create_dir_all(&dest_path)?;
+            copy_dir(path.to_str().unwrap(), &dest_path)?;
+        } else {
+            fs::copy(&path, &dest_path)?;
+        }
+    }
+    Ok(())
+}

src/compiler_top.rs

@@ -1,3 +1,7 @@
+use std::ffi::OsStr;
+use std::path::PathBuf;
+use std::str::FromStr;
+
 use crate::prelude::*;
 
 use tree_sitter::Parser;
@@ -11,8 +15,39 @@ use crate::flattening::{
     flatten_all_modules, gather_initial_file_data, typecheck_all_modules, Module,
 };
 
+const STD_LIB_PATH: &str = env!("SUS_COMPILER_STD_LIB_PATH");
+
+pub trait LinkerExtraFileInfoManager {
+    /// This is there to give an acceptable identifier that can be printed
+    fn convert_filename(&self, path : &PathBuf) -> String {
+        path.to_string_lossy().into_owned()
+    }
+    fn on_file_added(&mut self, _file_id : FileUUID, _linker : &Linker) {}
+    fn on_file_updated(&mut self, _file_id : FileUUID, _linker : &Linker) {}
+    fn before_file_remove(&mut self, _file_id : FileUUID, _linker : &Linker) {}
+}
+
+impl LinkerExtraFileInfoManager for () {}
+
 impl Linker {
-    pub fn add_file(&mut self, file_identifier: String, text: String) -> FileUUID {
+    pub fn add_standard_library<ExtraInfoManager : LinkerExtraFileInfoManager>(&mut self, info_mngr : &mut ExtraInfoManager) {
+        println!("Standard Library Directory: {STD_LIB_PATH}");
+        let stl_path = PathBuf::from_str(STD_LIB_PATH).expect("Standard library directory is not a valid path?");
+        self.add_all_files_in_directory(&stl_path, info_mngr);
+    }
+
+    pub fn add_all_files_in_directory<ExtraInfoManager : LinkerExtraFileInfoManager>(&mut self, directory : &PathBuf, info_mngr : &mut ExtraInfoManager) {
+        for file in std::fs::read_dir(directory).unwrap() {
+            let file_path = file.unwrap().path().canonicalize().unwrap();
+            if file_path.is_file() && file_path.extension() == Some(OsStr::new("sus")) {
+                let file_text = std::fs::read_to_string(&file_path).unwrap();
+                let file_identifier : String = info_mngr.convert_filename(&file_path);
+                self.add_file(file_identifier, file_text, info_mngr);
+            }
+        }
+    }
+
+    pub fn add_file<ExtraInfoManager : LinkerExtraFileInfoManager>(&mut self, file_identifier: String, text: String, info_mngr : &mut ExtraInfoManager) -> FileUUID {
         // File doesn't yet exist
         assert!(!self.files.iter().any(|fd| fd.1.file_identifier == file_identifier));
 
@@ -38,29 +73,41 @@ impl Linker {
             gather_initial_file_data(builder);
             span_debugger.defuse();
         });
-
+
+        info_mngr.on_file_added(file_id, self);
+
         file_id
     }
 
     // When lsp is not used, this gives a warning
     #[allow(dead_code)]
-    pub fn update_file(&mut self, text: String, file_id: FileUUID) {
-        let file_data = self.remove_everything_in_file(file_id);
-
-        let mut parser = Parser::new();
-        parser.set_language(&tree_sitter_sus::language()).unwrap();
-        let tree = parser.parse(&text, None).unwrap();
+    pub fn add_or_update_file<ExtraInfoManager : LinkerExtraFileInfoManager>(&mut self, file_identifier: &str, text: String, info_mngr : &mut ExtraInfoManager) {
+        if let Some(file_id) = self.find_file(file_identifier) {
+            let file_data = self.remove_everything_in_file(file_id);
+
+            let mut parser = Parser::new();
+            parser.set_language(&tree_sitter_sus::language()).unwrap();
+            let tree = parser.parse(&text, None).unwrap();
+
+            file_data.parsing_errors = ErrorStore::new();
+            file_data.file_text = FileText::new(text);
+            file_data.tree = tree;
+
+            self.with_file_builder(file_id, |builder| {
+                let mut span_debugger =
+                SpanDebugger::new("gather_initial_file_data in update_file", builder.file_data);
+                gather_initial_file_data(builder);
+                span_debugger.defuse();
+            });
 
-        file_data.parsing_errors = ErrorStore::new();
-        file_data.file_text = FileText::new(text);
-        file_data.tree = tree;
+            info_mngr.on_file_updated(file_id, self);
+        } else {
+            self.add_file(file_identifier.to_owned(), text, info_mngr);
+        }
+    }
 
-        self.with_file_builder(file_id, |builder| {
-            let mut span_debugger =
-                SpanDebugger::new("gather_initial_file_data in update_file", builder.file_data);
-            gather_initial_file_data(builder);
-            span_debugger.defuse();
-        });
+    pub fn find_file(&self, file_identifier: &str) -> Option<FileUUID> {
+        self.files.find(|_id, f| f.file_identifier == file_identifier)
     }
 
     pub fn recompile_all(&mut self) {

src/dev_aid/ariadne_interface.rs

@@ -1,5 +1,6 @@
 use std::{ops::Range, path::PathBuf};
 
+use crate::compiler_top::LinkerExtraFileInfoManager;
 use crate::linker::FileData;
 use crate::prelude::*;
 
@@ -39,11 +40,41 @@ impl Cache<()> for NamedSource<'_> {
     }
 }
 
+pub struct FileSourcesManager {
+    pub file_sources: ArenaVector<Source, FileUUIDMarker>
+}
+
+impl LinkerExtraFileInfoManager for FileSourcesManager {
+    fn convert_filename(&self, path : &PathBuf) -> String {
+        path.to_string_lossy().into_owned()
+    }
+
+    fn on_file_added(&mut self, file_id : FileUUID, linker : &Linker) {
+        let source = Source::from(linker.files[file_id].file_text.file_text.clone());
+
+        self.file_sources.insert(file_id, source);
+    }
+
+    fn on_file_updated(&mut self, file_id : FileUUID, linker : &Linker) {
+        let source = Source::from(linker.files[file_id].file_text.file_text.clone());
+
+        self.file_sources[file_id] = source;
+    }
+
+    fn before_file_remove(&mut self, file_id : FileUUID, _linker : &Linker) {
+        self.file_sources.remove(file_id)
+    }
+}
+
 pub fn compile_all(
     file_paths: Vec<PathBuf>,
-) -> (Linker, ArenaVector<Source, FileUUIDMarker>) {
+) -> (Linker, FileSourcesManager) {
     let mut linker = Linker::new();
-    let mut file_sources: ArenaVector<Source, FileUUIDMarker> = ArenaVector::new();
+    let mut file_source_manager = FileSourcesManager{
+        file_sources: ArenaVector::new()
+    };
+    linker.add_standard_library(&mut file_source_manager);
+
     for file_path in file_paths {
         let file_text = match std::fs::read_to_string(&file_path) {
             Ok(file_text) => file_text,
@@ -53,15 +84,12 @@ pub fn compile_all(
             }
         };
 
-        let source = Source::from(file_text.clone());
-        let uuid = linker.add_file(file_path.to_string_lossy().into_owned(), file_text);
-
-        file_sources.insert(uuid, source);
+        linker.add_file(file_path.to_string_lossy().into_owned(), file_text, &mut file_source_manager);
     }
 
     linker.recompile_all();
 
-    (linker, file_sources)
+    (linker, file_source_manager)
 }
 
 pub fn pretty_print_error<AriadneCache: Cache<FileUUID>>(