Best way to do circular templating?

[jeremych1000]

Heya - want some advice on this. I'm currently using gomplate to template out some YAML files. Each set of YAML files represents a new user's config.

The input with my data currently is in JSON, and comes from an upstream data source.

My problem is that whenever I make changes to the underlying template e.g. making whitespace changes, new items, etc., I don't have a way to regenerate all the old YAML files.

Crucially, I don't save the JSON blob (not feasible for me to do so).

I want to have it so that the generated YAML files can be fed back into the templates, to generate the newest version of those YAML files.

What's the best way to do this?

[jeremych1000]

Current:

JSON blob from upstream --> gomplate templates --> resultant YAML files

Wanted:

JSON blob from upstream --> revised gomplate templates --> resultant YAML files (always up to date)
                                     ^                        |
                                    |                           |
                                    |                          |
                                    |---------------- |

[hairyhenderson]

(I've converted to a discussion given that this is a request for advice, rather than a bug report or feature request)

Maybe you could clarify a bit so I can understand this better - is the upstream datasource not available when you need to re-generate? I understand that you don't save the JSON locally, but if it's available when you first generate the output, why is it not available the next time?

It seems to me that what you need is a different template to take the rendered output (the YAML) and re-render it.

If you're dealing with one input (the JSON blob) that's not equivalent to the output (by definition - the YAML files), that's a one-way operation.... It would seem strange to take the output and feed it into the same original templates and expect the same output...

[jeremych1000]

Sure. It's part of a piece of automation for onboarding.

• A user fills out a survey
• Survey data gets sent to my system for processing the onboarding request (in JSON)
• I take this JSON blob and generate resultant YAML files using gomplate
• These YAML templates are used by ArgoCD to spin up Kubernetes things

Now when I change the template, e.g. a default value, whitespace, adding a section etc., I can't regenerate the YAMLs for old users. I also can't get the upstream system to re-send the JSON blob as it's only sent when the user onboards the first time.

Yeah I guess two templates would work. One for JSON --> YAML, and one for YAML --> YAML? Would there be a lot of duplication?

How would the YAML --> YAML even work? I just use the same keys in the input YAML as the template? So if my input YAML is

foo: 
  bar: baz

I do {{ $data.foo.bar }} in the template?

[hairyhenderson]

I also can't get the upstream system to re-send the JSON blob as it's only sent when the user onboards the first time.

This makes sense - thanks for explaining the flow! If I were designing the system, I would be saving the raw data always, to prevent loss and allow for scenarios such as this.

One for JSON --> YAML, and one for YAML --> YAML? Would there be a lot of duplication?

Yes, and... Possibly? Knowing nothing about your specific data and template, I don't know.

How would the YAML --> YAML even work?

The input datasource format really isn't relevant. You can use any supported format as input - JSON, YAML, TOML, etc...

I just use the same keys in the input YAML as the template?

Essentially yes.

[jeremych1000]

If I were designing the system, I would be saving the raw data always, to prevent loss and allow for scenarios such as this.

I did think about this again and it wouldn't really work even if I saved the JSON. The resultant YAML files are representative of the current state of the client in the system, and may be changed by other processes (think the client's current infrastructure config for example). That means I can't guarantee that their current state in the YAML is representative of the initial values they put when onboarding.

2 templates it is! Will try my best. Need to think about validations as well. I have a separate folder containing gomplate validations, and require these to be able to be applied to both templates.

[jeremych1000]

I tried using two templates but it didn't feel right - the second template had tons of duplication and subtle key differences, so I'm going to try something else.

I'm going to write a template to create the JSON that I initially expect from the upstream system. This way it takes the most up to date / current values from the current set of templated YAML. Then I can then feed this JSON back into the 1st template.

so template 1: upstream JSON --> generate YAML
template 2: take generated YAML from template 1 --> JSON in the format of the upstream JSON --> template 1 --> most up to date templated YAML

[jheyduk]

Hi @jeremych1000,
It seems you could have an answer for my question #1803
Thanks in advance!
Joerg