Adds changes for setting up integration roundtrip tests for generated code #94

desaikd · 2024-04-03T22:30:19Z

Issue #88:

Description of changes:

This PR works on adding set-up changes for integration roundtrip tests on generated code

List of changes:

Adds some changes for struct templates, that were resulting in specific errors with roundtrip testing
- Replaces to_string()? with to_str()?.to_string(). (to_string() would return a raw string e.g. ""hello"" whereas to_str() returns a string value as "hello")
- Adds isl_type_name property in Field this helps store symbol as Rust/Java string but when writing it as Ion it will use write_symbol using write_{{ field.isl_type_name}}.
Add a namespace option for java code generation (This option is only required when --language option is set as java).
Adds changes to use only a single module with all generated data models in Rust. Generated module is named as ion_generated_code.rs. This will help in importing entire file using include! macro and also simplifies attaching a single file as a module in any Rust crate. More information can be found in Rust guide on code generation: https://doc.rust-lang.org/cargo/reference/build-script-examples.html#code-generation
Adds changes to generate code for all schema files in a directory/authority.
Adds changes to current tests and clippy changes.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

* Add ISL type name in the Field to call read/write APIs for symbol/string or blob/clob accordingly. * Adds manifest directory in the path for the templates

* Adds `CodeGenerator::generate_code_for_authorities` which generates code for each schema in given authorities * Makes `--schema` optional for code generation

* Removed Import statements from template as those were used for reference to other generated data models * Removes mod.templ as the generated code now uses single module * Removes generating `ion_data_model` module for generated code instead uses a single file `ion_generated_code`

desaikd · 2024-04-03T22:33:49Z

src/bin/ion/commands/beta/generate/generator.rs

@@ -19,27 +21,72 @@ pub(crate) struct CodeGenerator<'a, L: Language> {
    // more information: https://docs.rs/tera/latest/tera/
    pub(crate) tera: Tera,
    output: &'a Path,
+    // This field is used by Java code generation to get the namespace for generated code.
+    // For Rust code generation, this will be set to None.
+    namespace: Option<&'a str>,


(🗺️ PR tour) This adds a new namespace property to code generator which will only be used while Java code generation to specify namespace of generated code and is collected from --namespace option from CLI command.

desaikd · 2024-04-03T22:35:30Z

src/bin/ion/commands/beta/generate/generator.rs

    // Represents a counter for naming anonymous type definitions
    pub(crate) anonymous_type_counter: usize,
    phantom: PhantomData<L>,
 }

 impl<'a> CodeGenerator<'a, RustLanguage> {
    pub fn new(output: &'a Path) -> CodeGenerator<RustLanguage> {
+        let tera = Tera::new(&format!(
+            "{}/src/bin/ion/commands/beta/generate/templates/rust/*.templ",
+            env!("CARGO_MANIFEST_DIR")


(🗺️ PR tour) Tera is the templating engine sued for generating code and this sets the path to all template files using CARGO_MANIFEST_DIR so that it can be found from anywhere in a cargo project.

desaikd · 2024-04-03T22:36:49Z

src/bin/ion/commands/beta/generate/generator.rs

+        .unwrap();
+
+        // Render the imports into output file
+        let rendered = tera.render("import.templ", &Context::new()).unwrap();


(🗺️ PR tour) This separates the import statement which will be common for all generated data models and since Rust code generation has single file being generated with all the data models, the next step here writes this import to that file ion_generated_code.

desaikd · 2024-04-03T22:38:03Z

src/bin/ion/commands/beta/generate/generator.rs

            phantom: PhantomData,
        }
    }

+    /// Generates code in Rust for all the schemas in given authorities
+    pub fn generate_code_for_authorities(


(🗺️ PR tour) This API is used to generate code for all schema files in given authorities, it uses generate() to generate code for a single schema file.

desaikd · 2024-04-03T22:38:41Z

src/bin/ion/commands/beta/generate/generator.rs

-        // this will be used for Rust to create mod.rs which lists all the generated modules
-        let mut modules = vec![];
-        let mut module_context = tera::Context::new();
+    pub fn generate_code_for_schema(


(🗺️ PR tour) Similar to generate_code_for_authorities but only generates code for given schema file using schema_id.

desaikd · 2024-04-03T22:39:54Z

src/bin/ion/commands/beta/generate/utils.rs

@@ -121,8 +115,8 @@ impl Language for RustLanguage {
        "rust".to_string()
    }

-    fn file_name_for_type(name: &str) -> String {
-        name.to_case(Case::Snake)
+    fn file_name_for_type(_name: &str) -> String {


(🗺️ PR tour) Since Rust code generation uses a single file for all generated data models this method outputs a constant file name ion_generated_code

desaikd · 2024-04-03T22:40:51Z

src/bin/ion/commands/beta/generate/templates/rust/struct.templ

                                        {% else %}
-                                            reader.read_{{ target_kind_name }}()?;
+                                            reader.read_{% if fields[0].isl_type_name == "symbol" %}symbol()?.text().unwrap(){% else %}{{ fields[0].value | replace(from="string", to ="str") }}()?{% endif %}{% if fields[0].value | lower == "string" %} .to_string() {% endif %};


(🗺️ PR tour) Replaces to_string()? -> to_str()?.to_string() and uses isl_type_name to read symbol.

desaikd · 2024-04-03T22:41:33Z

src/bin/ion/commands/beta/generate/templates/rust/mod.templ

@@ -1,3 +0,0 @@
-{% for module in modules -%}


(🗺️ PR tour) This has been removed as its no longer needed due to single file being generated for Rust.

desaikd · 2024-04-03T22:42:30Z

src/bin/ion/commands/beta/generate/templates/rust/import.templ

@@ -0,0 +1 @@
+use ion_rs::{IonResult, IonReader, Reader, IonWriter, StreamItem};


(🗺️ PR tour) This template points to the common import statement for all generated data models which has imports for Ion reader, writer etc.

desaikd · 2024-04-03T22:44:04Z

src/bin/ion/commands/beta/generate/templates/java/class.templ

-{% if import %}
-import ion_data_model.{{ import.name | upper_camel }};
-{% endif %}
+package {{ namespace }};


(🗺️ PR tour) These changes were found during roundtrip testing locally.

Adds namespace declaration statement.

Adds ArrayList import.

Add changes to use camel casing for field names in Java.

src/bin/ion/commands/beta/generate/generator.rs

nirosys

Minor comments/questions, but otherwise looks good to me.

desaikd added 5 commits March 26, 2024 17:41

Adds changes to struct template

405c2c0

* Add ISL type name in the Field to call read/write APIs for symbol/string or blob/clob accordingly. * Adds manifest directory in the path for the templates

Adds support for generating code for all schemas in authorities

47f38d4

* Adds `CodeGenerator::generate_code_for_authorities` which generates code for each schema in given authorities * Makes `--schema` optional for code generation

Adds --namespace option for Java code generation

9b5e242

Adds changes for current test suite and clippy changes

9e73f65

desaikd commented Apr 3, 2024

View reviewed changes

desaikd requested a review from nirosys April 3, 2024 22:50

nirosys reviewed Apr 4, 2024

View reviewed changes

src/bin/ion/commands/beta/generate/generator.rs Outdated Show resolved Hide resolved

src/bin/ion/commands/beta/generate/generator.rs Outdated Show resolved Hide resolved

src/bin/ion/commands/beta/generate/generator.rs Outdated Show resolved Hide resolved

nirosys approved these changes Apr 4, 2024

View reviewed changes

Adds suggested changes

80b6b38

desaikd merged commit 425db12 into master Apr 5, 2024
7 checks passed

desaikd mentioned this pull request Apr 9, 2024

Define a test runner for roundtrip generated code tests #88

Closed

desaikd mentioned this pull request Apr 29, 2024

Modify template to dynamically add module names in import #70

Closed

desaikd deleted the desaikd-tests-set-up branch August 21, 2024 23:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds changes for setting up integration roundtrip tests for generated code #94

Adds changes for setting up integration roundtrip tests for generated code #94

desaikd commented Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

desaikd Apr 3, 2024

nirosys left a comment

		@@ -0,0 +1 @@
		use ion_rs::{IonResult, IonReader, Reader, IonWriter, StreamItem};

Adds changes for setting up integration roundtrip tests for generated code #94

Adds changes for setting up integration roundtrip tests for generated code #94

Conversation

desaikd commented Apr 3, 2024

Issue #88:

Description of changes:

List of changes:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirosys left a comment

Choose a reason for hiding this comment