feat: Markdown documentation generator (#963)

Closes #663 ### Summary of Changes * Add a service to generate Markdown documentation for Safe-DS files * Add the `document` CLI command to invoke this generator * Create Markdown documentation for the builtin stubs --------- Co-authored-by: megalinter-bot <[email protected]>
Safe-DS · Apr 1, 2024 · 77d04de · 77d04de
1 parent dc5bf3b
commit 77d04de
Show file tree

Hide file tree

Showing 408 changed files with 13,118 additions and 568 deletions.
diff --git a/docs/development/generation-testing.md b/docs/development/generation-testing.md
@@ -1,14 +1,17 @@
 # Generation Testing
 
-Generation tests are data-driven instead of being specified explicitly. This document explains how to add a new
-generation test.
+Safe-DS has two generation targets, [Python](#python-generation) (for execution) and [Markdown](#markdown-generation)
+(for documentation). In either case, the generation tests are data-driven instead of being specified explicitly. This
+document explains how to add a new generation test.
 
-## Adding a generation test
+## Python Generation
 
-1. Create a new **folder** (not just a file!) in the `tests/resources/generation` directory or any subdirectory. Give
-   the folder a descriptive name, since the folder name becomes part of the test name. By default, the runner
+### Adding a generation test
+
+1. Create a new **folder** (not just a file!) in the `tests/resources/generation/python` directory or any subdirectory.
+   Give the folder a descriptive name, since the folder name becomes part of the test name. By default, the runner
    integration is disabled. If you want to test the runner integration, place the folder in
-   `tests/resources/generation/runner integration` or any subdirectory instead.
+   `tests/resources/generation/python/runner integration` or any subdirectory instead.
 
     !!! tip "Skipping a test"
 
@@ -30,6 +33,31 @@ generation test.
    and the contents of the Python files will be compared to the actual generation output.
 6. Run the tests. The test runner will automatically pick up the new test.
 
-## Updating the expected output
+### Updating the expected output
+
+To quickly update the expected output after changes to the code generator, run `vitest` with the `--update` flag.
+
+## Markdown Generation
+
+### Adding a generation test
+
+1. Create a new **folder** (not just a file!) in the `tests/resources/generation/markdown` directory or any
+   subdirectory. Give the folder a descriptive name, since the folder name becomes part of the test name.
+
+    !!! tip "Skipping a test"
+
+        If you want to skip a test, add the prefix `skip-` to the folder name.
+
+2. Add files with the extension `.sdstest`, `.sdspipe`, or `.sdsstub` **directly inside the folder**. All files in a
+   folder will be loaded into the same workspace, so they can reference each other. Files in different folders are
+   loaded into different workspaces, so they cannot reference each other. Generation will be triggered for all files in
+   the folder.
+3. Add the Safe-DS code that you want to test to the files.
+4. Add another folder called `generated` inside the folder that you created in step 1. Place folders and Python files
+   inside the `generated` folder to specify the expected output of the program. The relative paths to the Python files
+   and the contents of the Python files will be compared to the actual generation output.
+5. Run the tests. The test runner will automatically pick up the new test.
+
+### Updating the expected output
 
 To quickly update the expected output after changes to the code generator, run `vitest` with the `--update` flag.
diff --git a/docs/javascript/mathjax.js b/docs/javascript/mathjax.js
@@ -12,5 +12,8 @@ window.MathJax = {
 };
 
 document$.subscribe(() => {
+    MathJax.startup.output.clearCache();
+    MathJax.typesetClear();
+    MathJax.texReset();
     MathJax.typesetPromise();
 });
diff --git a/docs/language/stub-language/annotations.md b/docs/language/stub-language/annotations.md
@@ -53,7 +53,7 @@ To attach metainformation to a declaration, the annotation must be called on tha
 - [Type parameters][type-parameters]
 - [Pipelines][pipelines]
 
-The valid targets of an annotation can be restricted with the [`Target`][safeds-lang-target] annotation. By default all targets are allowed. Likewise, an annotation can only be called once on the same declaration by default, unless the annotation is marked as[Repeatable][safeds-lang-repeatable].
+The valid targets of an annotation can be restricted with the [`Target`][safeds.lang.Target] annotation. By default all targets are allowed. Likewise, an annotation can only be called once on the same declaration by default, unless the annotation is marked as[Repeatable][safeds.lang.Repeatable].
 
 Annotation calls are always located right in front of their target. Exception: In the case of compilations units they are located at the very top of the file. Here is an example that demonstrates how to call the annotation `#!sds OnlyForExperts` that we defined above on a [class][classes]:
 
@@ -87,10 +87,10 @@ The same [restrictions to arguments][argument-restrictions] as for [calls][calls
 
 ## Built-in Annotations
 
-The package `#!sds safeds.lang` contains several annotations that are processed by Safe-DS. Refer to the [API documentation][safeds-lang] for more details. Particularly important are the annotations
+The package `#!sds safeds.lang` contains several annotations that are processed by Safe-DS. Particularly important are the annotations
 
-- [`Target`][safeds-lang-target], which can restrict the possible targets of an annotation, and
-- [`Repeatable`][safeds-lang-repeatable], which allows an annotation to be called multiple times on the same declaration.
+- [`Target`][safeds.lang.Target], which can restrict the possible targets of an annotation, and
+- [`Repeatable`][safeds.lang.Repeatable], which allows an annotation to be called multiple times on the same declaration.
 
 [parameters]: ../common/parameters.md
 [types]: ../common/types.md
@@ -104,8 +104,5 @@ The package `#!sds safeds.lang` contains several annotations that are processed
 [segments]: ../pipeline-language/segments.md
 [type-parameters]: type-parameters.md
 [pipelines]: ../pipeline-language/pipelines.md
-[safeds-lang]: ../../stdlib/safeds_lang.md
-[safeds-lang-repeatable]: ../../stdlib/safeds_lang.md#annotation-repeatable
-[safeds-lang-target]: ../../stdlib/safeds_lang.md#annotation-target
 [calls]: ../pipeline-language/expressions.md#calls
 [argument-restrictions]: ../pipeline-language/expressions.md#restrictions-for-arguments
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -36,9 +36,7 @@ nav:
           - Enumerations: language/stub-language/enumerations.md
           - Annotations: language/stub-language/annotations.md
           - Type Parameters: language/stub-language/type-parameters.md
-  - Standard Library:
-      - stdlib/README.md
-      - safeds.lang: stdlib/safeds_lang.md
+  - Standard Library: stdlib/
   - Development:
       - Call Graph Testing: development/call-graph-testing.md
       - Formatting Testing: development/formatting-testing.md
@@ -80,6 +78,9 @@ theme:
     - navigation.top
 
 plugins:
+  - literate-nav:
+      nav_file: SUMMARY.md
+  - autorefs
   - glightbox
   - same-dir
   - search
@@ -130,6 +131,9 @@ markdown_extensions:
   - toc:
       permalink: true
 
+extra_css:
+  - stylesheets/extra.css
+
 extra_javascript:
   - javascript/mathjax.js
   - https://polyfill.io/v3/polyfill.min.js?features=es6

diff --git a/docs/poetry.lock b/docs/poetry.lock
diff --git a/docs/pyproject.toml b/docs/pyproject.toml
@@ -12,9 +12,11 @@ packages = [
 python = "^3.11"
 mkdocs = "^1.5.3"
 mkdocs-glightbox = "^0.3.7"
+mkdocs-literate-nav = "^0.6.1"
 mkdocs-material = "^9.5.12"
 mkdocs-same-dir = "^0.1.3"
 pygments = "^2.17.2"
+mkdocs-autorefs = "^1.0.1"
 
 [tool.poetry.plugins."pygments.lexers"]
 sds = "safe_ds_lexer:SafeDsLexer"

diff --git a/docs/stdlib/README.md b/docs/stdlib/README.md
diff --git a/docs/stdlib/SUMMARY.md b/docs/stdlib/SUMMARY.md
@@ -0,0 +1,73 @@
+- safeds
+    - data
+        - image
+            - containers
+                - [Image](safeds/data/image/containers/Image.md)
+        - tabular
+            - containers
+                - [Column](safeds/data/tabular/containers/Column.md)
+                - [Row](safeds/data/tabular/containers/Row.md)
+                - [Table](safeds/data/tabular/containers/Table.md)
+                - [TaggedTable](safeds/data/tabular/containers/TaggedTable.md)
+                - [TimeSeries](safeds/data/tabular/containers/TimeSeries.md)
+            - transformation
+                - [Discretizer](safeds/data/tabular/transformation/Discretizer.md)
+                - [Imputer](safeds/data/tabular/transformation/Imputer.md)
+                - [InvertibleTableTransformer](safeds/data/tabular/transformation/InvertibleTableTransformer.md)
+                - [LabelEncoder](safeds/data/tabular/transformation/LabelEncoder.md)
+                - [OneHotEncoder](safeds/data/tabular/transformation/OneHotEncoder.md)
+                - [RangeScaler](safeds/data/tabular/transformation/RangeScaler.md)
+                - [StandardScaler](safeds/data/tabular/transformation/StandardScaler.md)
+                - [TableTransformer](safeds/data/tabular/transformation/TableTransformer.md)
+            - typing
+                - [ColumnType](safeds/data/tabular/typing/ColumnType.md)
+                - [Schema](safeds/data/tabular/typing/Schema.md)
+    - lang
+        - [AnnotationTarget](safeds/lang/AnnotationTarget.md)
+        - [Any](safeds/lang/Any.md)
+        - [Boolean](safeds/lang/Boolean.md)
+        - [Deprecated](safeds/lang/Deprecated.md)
+        - [Experimental](safeds/lang/Experimental.md)
+        - [Expert](safeds/lang/Expert.md)
+        - [Float](safeds/lang/Float.md)
+        - [Impure](safeds/lang/Impure.md)
+        - [ImpurityReason](safeds/lang/ImpurityReason.md)
+        - [Int](safeds/lang/Int.md)
+        - [List](safeds/lang/List.md)
+        - [Map](safeds/lang/Map.md)
+        - [Nothing](safeds/lang/Nothing.md)
+        - [Number](safeds/lang/Number.md)
+        - [Pure](safeds/lang/Pure.md)
+        - [PythonCall](safeds/lang/PythonCall.md)
+        - [PythonModule](safeds/lang/PythonModule.md)
+        - [PythonName](safeds/lang/PythonName.md)
+        - [Repeatable](safeds/lang/Repeatable.md)
+        - [String](safeds/lang/String.md)
+        - [Target](safeds/lang/Target.md)
+    - ml
+        - classical
+            - classification
+                - [AdaBoostClassifier](safeds/ml/classical/classification/AdaBoostClassifier.md)
+                - [Classifier](safeds/ml/classical/classification/Classifier.md)
+                - [DecisionTreeClassifier](safeds/ml/classical/classification/DecisionTreeClassifier.md)
+                - [GradientBoostingClassifier](safeds/ml/classical/classification/GradientBoostingClassifier.md)
+                - [KNearestNeighborsClassifier](safeds/ml/classical/classification/KNearestNeighborsClassifier.md)
+                - [LogisticRegressionClassifier](safeds/ml/classical/classification/LogisticRegressionClassifier.md)
+                - [RandomForestClassifier](safeds/ml/classical/classification/RandomForestClassifier.md)
+                - [SupportVectorMachineClassifier](safeds/ml/classical/classification/SupportVectorMachineClassifier.md)
+            - regression
+                - [AdaBoostRegressor](safeds/ml/classical/regression/AdaBoostRegressor.md)
+                - [DecisionTreeRegressor](safeds/ml/classical/regression/DecisionTreeRegressor.md)
+                - [ElasticNetRegressor](safeds/ml/classical/regression/ElasticNetRegressor.md)
+                - [GradientBoostingRegressor](safeds/ml/classical/regression/GradientBoostingRegressor.md)
+                - [KNearestNeighborsRegressor](safeds/ml/classical/regression/KNearestNeighborsRegressor.md)
+                - [LassoRegressor](safeds/ml/classical/regression/LassoRegressor.md)
+                - [LinearRegressionRegressor](safeds/ml/classical/regression/LinearRegressionRegressor.md)
+                - [RandomForestRegressor](safeds/ml/classical/regression/RandomForestRegressor.md)
+                - [Regressor](safeds/ml/classical/regression/Regressor.md)
+                - [RidgeRegressor](safeds/ml/classical/regression/RidgeRegressor.md)
+                - [SupportVectorMachineRegressor](safeds/ml/classical/regression/SupportVectorMachineRegressor.md)
+        - nn
+            - [FNNLayer](safeds/ml/nn/FNNLayer.md)
+            - [NeuralNetworkClassifier](safeds/ml/nn/NeuralNetworkClassifier.md)
+            - [NeuralNetworkRegressor](safeds/ml/nn/NeuralNetworkRegressor.md)