Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds changes for setting up integration roundtrip tests for generated code #94

Merged
merged 6 commits into from
Apr 5, 2024
Merged

Adds changes for setting up integration roundtrip tests for generated code #94

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
405c2c0
Adds changes to struct template
desaikd Mar 26, 2024
47f38d4
Adds support for generating code for all schemas in authorities
desaikd Mar 26, 2024
9b5e242
Adds `--namespace` option for Java code generation
desaikd Mar 26, 2024
e172f3f
Adds changes to generate code in single module for Rust code generation
desaikd Mar 26, 2024
9e73f65
Adds changes for current test suite and clippy changes
desaikd Apr 3, 2024
80b6b38
Adds suggested changes
desaikd Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 128 additions & 69 deletions src/bin/ion/commands/beta/generate/generator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
use crate::commands::beta::generate::context::{AbstractDataType, CodeGenContext};
use crate::commands::beta::generate::result::{invalid_abstract_data_type_error, CodeGenResult};
use crate::commands::beta::generate::utils::{Field, Import, JavaLanguage, Language, RustLanguage};
use crate::commands::beta::generate::utils::{Field, JavaLanguage, Language, RustLanguage};
use crate::commands::beta::generate::utils::{IonSchemaType, Template};
use convert_case::{Case, Casing};
use ion_schema::isl::isl_constraint::{IslConstraint, IslConstraintValue};
use ion_schema::isl::isl_type::IslType;
use ion_schema::isl::isl_type_reference::IslTypeRef;
use ion_schema::isl::IslSchema;
use ion_schema::system::SchemaSystem;
use std::collections::HashMap;
use std::fs::File;
use std::fs;
use std::fs::OpenOptions;
use std::io::Write;
use std::marker::PhantomData;
use std::path::Path;
Expand All @@ -19,27 +21,72 @@ pub(crate) struct CodeGenerator<'a, L: Language> {
// more information: https://docs.rs/tera/latest/tera/
pub(crate) tera: Tera,
output: &'a Path,
// This field is used by Java code generation to get the namespace for generated code.
// For Rust code generation, this will be set to None.
namespace: Option<&'a str>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(🗺️ PR tour) This adds a new namespace property to code generator which will only be used while Java code generation to specify namespace of generated code and is collected from --namespace option from CLI command.

// Represents a counter for naming anonymous type definitions
pub(crate) anonymous_type_counter: usize,
phantom: PhantomData<L>,
}

impl<'a> CodeGenerator<'a, RustLanguage> {
pub fn new(output: &'a Path) -> CodeGenerator<RustLanguage> {
let tera = Tera::new(&format!(
"{}/src/bin/ion/commands/beta/generate/templates/rust/*.templ",
env!("CARGO_MANIFEST_DIR")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(🗺️ PR tour) Tera is the templating engine sued for generating code and this sets the path to all template files using CARGO_MANIFEST_DIR so that it can be found from anywhere in a cargo project.

))
.unwrap();

// Render the imports into output file
let rendered = tera.render("import.templ", &Context::new()).unwrap();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(🗺️ PR tour) This separates the import statement which will be common for all generated data models and since Rust code generation has single file being generated with all the data models, the next step here writes this import to that file ion_generated_code.

let mut file = OpenOptions::new()
.create(true)
.append(true)
desaikd marked this conversation as resolved.
Show resolved Hide resolved
.open(output.join("ion_generated_code.rs"))
.unwrap();
file.write_all(rendered.as_bytes()).unwrap();

Self {
output,
namespace: None,
anonymous_type_counter: 0,
tera: Tera::new("src/bin/ion/commands/beta/generate/templates/rust/*.templ").unwrap(),
tera,
phantom: PhantomData,
}
}

/// Generates code in Rust for all the schemas in given authorities
pub fn generate_code_for_authorities(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(🗺️ PR tour) This API is used to generate code for all schema files in given authorities, it uses generate() to generate code for a single schema file.

desaikd marked this conversation as resolved.
Show resolved Hide resolved
&mut self,
authorities: &Vec<&String>,
schema_system: &mut SchemaSystem,
) -> CodeGenResult<()> {
for authority in authorities {
let paths = fs::read_dir(authority)?;

for schema_file in paths {
let schema_file_path = schema_file?.path();
let schema_id = schema_file_path.file_name().unwrap().to_str().unwrap();

let schema = schema_system.load_isl_schema(schema_id).unwrap();

self.generate(schema)?;
}
}
Ok(())
}

/// Generates code in Rust for given Ion Schema
pub fn generate(&mut self, schema: IslSchema) -> CodeGenResult<()> {
// this will be used for Rust to create mod.rs which lists all the generated modules
let mut modules = vec![];
let mut module_context = tera::Context::new();
pub fn generate_code_for_schema(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(🗺️ PR tour) Similar to generate_code_for_authorities but only generates code for given schema file using schema_id.

&mut self,
schema_system: &mut SchemaSystem,
schema_id: &str,
) -> CodeGenResult<()> {
let schema = schema_system.load_isl_schema(schema_id).unwrap();
self.generate(schema)
}

fn generate(&mut self, schema: IslSchema) -> CodeGenResult<()> {
desaikd marked this conversation as resolved.
Show resolved Hide resolved
// Register a tera filter that can be used to convert a string based on case
self.tera.register_filter("upper_camel", Self::upper_camel);
self.tera.register_filter("snake", Self::snake);
Expand All @@ -49,54 +96,76 @@ impl<'a> CodeGenerator<'a, RustLanguage> {
self.tera
.register_filter("is_built_in_type", Self::is_built_in_type);

// Iterate through the ISL types, generate an abstract data type for each
for isl_type in schema.types() {
// unwrap here is safe because all the top-level type definition always has a name
let isl_type_name = isl_type.name().clone().unwrap();
self.generate_abstract_data_type(&isl_type_name, &mut modules, isl_type)?;
self.generate_abstract_data_type(&isl_type_name, isl_type)?;
}

self.generate_modules(&mut modules, &mut module_context)?;

Ok(())
}

pub fn generate_modules(
&mut self,
modules: &mut Vec<String>,
module_context: &mut Context,
) -> CodeGenResult<()> {
module_context.insert("modules", &modules);
let rendered = self.tera.render("mod.templ", module_context)?;
let mut file = File::create(self.output.join("mod.rs"))?;
file.write_all(rendered.as_bytes())?;
Ok(())
}
}

impl<'a> CodeGenerator<'a, JavaLanguage> {
pub fn new(output: &'a Path) -> CodeGenerator<JavaLanguage> {
pub fn new(output: &'a Path, namespace: &'a str) -> CodeGenerator<'a, JavaLanguage> {
let tera = Tera::new(&format!(
"{}/src/bin/ion/commands/beta/generate/templates/java/*.templ",
env!("CARGO_MANIFEST_DIR")
))
.unwrap();

Self {
output,
namespace: Some(namespace),
anonymous_type_counter: 0,
tera: Tera::new("src/bin/ion/commands/beta/generate/templates/java/*.templ").unwrap(),
tera,
phantom: PhantomData,
}
}

/// Generates code in Java for all the schemas in given authorities
pub fn generate_code_for_authorities(
&mut self,
authorities: &Vec<&String>,
schema_system: &mut SchemaSystem,
) -> CodeGenResult<()> {
for authority in authorities {
let paths = fs::read_dir(authority)?;

for schema_file in paths {
let schema_file_path = schema_file?.path();
let schema_id = schema_file_path.file_name().unwrap().to_str().unwrap();

let schema = schema_system.load_isl_schema(schema_id).unwrap();

self.generate(schema)?;
}
}
Ok(())
}

/// Generates code in Java for given Ion Schema
pub fn generate(&mut self, schema: IslSchema) -> CodeGenResult<()> {
// this will be used for Rust to create mod.rs which lists all the generated modules
let mut modules = vec![];
pub fn generate_code_for_schema(
&mut self,
schema_system: &mut SchemaSystem,
schema_id: &str,
) -> CodeGenResult<()> {
let schema = schema_system.load_isl_schema(schema_id).unwrap();
self.generate(schema)
}

fn generate(&mut self, schema: IslSchema) -> CodeGenResult<()> {
// Register a tera filter that can be used to convert a string based on case
self.tera.register_filter("upper_camel", Self::upper_camel);
self.tera.register_filter("snake", Self::snake);
self.tera.register_filter("camel", Self::camel);

// Iterate through the ISL types, generate an abstract data type for each
for isl_type in schema.types() {
// unwrap here is safe because all the top-level type definition always has a name
let isl_type_name = isl_type.name().clone().unwrap();
self.generate_abstract_data_type(&isl_type_name, &mut modules, isl_type)?;
self.generate_abstract_data_type(&isl_type_name, isl_type)?;
}

Ok(())
Expand Down Expand Up @@ -182,12 +251,10 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
fn generate_abstract_data_type(
&mut self,
isl_type_name: &String,
modules: &mut Vec<String>,
isl_type: &IslType,
) -> CodeGenResult<()> {
let mut context = Context::new();
let mut tera_fields = vec![];
let mut imports: Vec<Import> = vec![];
let mut code_gen_context = CodeGenContext::new();

// Set the ISL type name for the generated abstract data type
Expand All @@ -196,17 +263,12 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
let constraints = isl_type.constraints();
for constraint in constraints {
self.map_constraint_to_abstract_data_type(
modules,
&mut tera_fields,
&mut imports,
constraint,
&mut code_gen_context,
)?;
}

// add imports for the template
context.insert("imports", &imports);

// add fields for template
// TODO: verify the `occurs` value within a field, by default the fields are optional.
if let Some(abstract_data_type) = &code_gen_context.abstract_data_type {
Expand All @@ -218,47 +280,41 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
);
}

self.render_generated_code(modules, &isl_type_name, &mut context, &mut code_gen_context)
self.render_generated_code(isl_type_name, &mut context, &mut code_gen_context)
}

fn render_generated_code(
&mut self,
modules: &mut Vec<String>,
abstract_data_type_name: &str,
type_name: &str,
context: &mut Context,
code_gen_context: &mut CodeGenContext,
) -> CodeGenResult<()> {
modules.push(L::file_name_for_type(abstract_data_type_name));

// Add namespace to tera context
if let Some(namespace) = self.namespace {
context.insert("namespace", namespace);
}
// Render or generate file for the template with the given context
let template: &Template = &code_gen_context.abstract_data_type.as_ref().try_into()?;
let rendered = self
.tera
.render(&format!("{}.templ", L::template_name(template)), context)
.unwrap();
let mut file = File::create(self.output.join(format!(
"{}.{}",
L::file_name_for_type(abstract_data_type_name),
L::file_extension()
)))?;
let mut file = OpenOptions::new()
.create(true)
.append(true)
.open(self.output.join(format!(
"{}.{}",
L::file_name_for_type(type_name),
L::file_extension()
)))?;
file.write_all(rendered.as_bytes())?;
Ok(())
}

/// Provides name of the type reference that will be used for generated abstract data type
fn type_reference_name(
&mut self,
isl_type_ref: &IslTypeRef,
modules: &mut Vec<String>,
imports: &mut Vec<Import>,
) -> CodeGenResult<String> {
fn type_reference_name(&mut self, isl_type_ref: &IslTypeRef) -> CodeGenResult<String> {
Ok(match isl_type_ref {
IslTypeRef::Named(name, _) => {
if !L::is_built_in_type(name) {
imports.push(Import {
name: name.to_string(),
});
}
let schema_type: IonSchemaType = name.into();
L::target_type(&schema_type)
}
Expand All @@ -267,10 +323,8 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
}
IslTypeRef::Anonymous(type_def, _) => {
let name = self.next_anonymous_type_name();
self.generate_abstract_data_type(&name, modules, type_def)?;
imports.push(Import {
name: name.to_string(),
});
self.generate_abstract_data_type(&name, type_def)?;

name
}
})
Expand All @@ -286,22 +340,21 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
/// Maps the given constraint value to an abstract data type
fn map_constraint_to_abstract_data_type(
&mut self,
modules: &mut Vec<String>,
tera_fields: &mut Vec<Field>,
imports: &mut Vec<Import>,
constraint: &IslConstraint,
code_gen_context: &mut CodeGenContext,
) -> CodeGenResult<()> {
match constraint.constraint() {
IslConstraintValue::Element(isl_type, _) => {
let type_name = self.type_reference_name(isl_type, modules, imports)?;
let type_name = self.type_reference_name(isl_type)?;
self.verify_abstract_data_type_consistency(
AbstractDataType::Sequence(type_name.to_owned()),
code_gen_context,
)?;
self.generate_struct_field(
tera_fields,
L::target_type_as_sequence(&type_name),
isl_type.name(),
"value",
)?;
}
Expand All @@ -312,20 +365,24 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
code_gen_context,
)?;
for (name, value) in fields.iter() {
let type_name =
self.type_reference_name(value.type_reference(), modules, imports)?;

self.generate_struct_field(tera_fields, type_name, name)?;
let type_name = self.type_reference_name(value.type_reference())?;

self.generate_struct_field(
tera_fields,
type_name,
value.type_reference().name(),
name,
)?;
}
}
IslConstraintValue::Type(isl_type) => {
let type_name = self.type_reference_name(isl_type, modules, imports)?;
let type_name = self.type_reference_name(isl_type)?;

self.verify_abstract_data_type_consistency(
AbstractDataType::Value,
code_gen_context,
)?;
self.generate_struct_field(tera_fields, type_name, "value")?;
self.generate_struct_field(tera_fields, type_name, isl_type.name(), "value")?;
}
_ => {}
}
Expand All @@ -337,11 +394,13 @@ impl<'a, L: Language> CodeGenerator<'a, L> {
&mut self,
tera_fields: &mut Vec<Field>,
abstract_data_type_name: String,
isl_type_name: String,
field_name: &str,
) -> CodeGenResult<()> {
tera_fields.push(Field {
name: field_name.to_string(),
value: abstract_data_type_name,
isl_type_name,
});
Ok(())
}
Expand Down
Loading
Loading