[tools,topgen] Split topgen into creating complete topcfg and the rest #25920

matutem · 2025-01-17T09:30:26Z

Description

Topgen is a monolithic tool and is called multiple times to generate the full set of artifacts. Each time it generated the complete top configuration, and then performs a some subset of all the actions it is able to do. This is not efficient and is hard to maintain.

This calls for a change in the topgen flow so the complete top configuration is generated and written to disk by one tool. The actions that depend on the complete config can then be performed by smaller separate tools, and most importantly, they can be invoked only if the complete top configuration file has changed. This will allow a more incremental flow, and since most of the separate tasks depending on the complete config are independent, they can be done in parallel.

This is not only more efficient and simpler, but it simplifies the development of different tops, so can be very helpful for multi-top.

This is a big change in topgen. Some salient changes: - It always regenerates from scratch the ipgen in-memory configurations (as IpBlock objects) so it is truly incremental: the previous flow reused the pre-existing generated ipgen hjson if found. - It delays the actual ipgen file generation until the complete top config is generated. This is a step towards breaking topgen into smaller and mutually independent steps, thus enabling parallelizing them. - It converges with a single pass of process_top. In order to accomplish this some important changes are made: - Call some of the merge_top functions suitable for each specific ipgen prior to generating the in-memory hjson config. This means not all IpBlock objects are available, so these merge functions are instrumented to skip blocks missing. - The generation of ipgens are ordered according to dependencies. This order created from the expected block functionality, so is not derived from a constructed graph, but converging in one pass means it is correct. - Calling merge functions early means some of the objects in the in-memory top config are not plain dictionaries but are classes, so the code needs to handle either. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

The reset_requests top config entry should really be initialized from the original hjson file rather than explicitly in merge.py, since it ends up making assumptions about what reset_requests should have. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This is a big change in topgen. Some salient changes: - It always regenerates from scratch the ipgen in-memory configurations (as IpBlock objects) so it is truly incremental: the previous flow reused the pre-existing generated ipgen hjson if found. - It delays the actual ipgen file generation until the complete top config is generated. This is a step towards breaking topgen into smaller and mutually independent steps, thus enabling parallelizing them. - It converges with a single pass of process_top. In order to accomplish this some important changes are made: - Call some of the merge_top functions suitable for each specific ipgen prior to generating the in-memory hjson config. This means not all IpBlock objects are available, so these merge functions are instrumented to skip blocks missing. - The generation of ipgens are ordered according to dependencies. This order created from the expected block functionality, so is not derived from a constructed graph, but converging in one pass means it is correct. - Calling merge functions early means some of the objects in the in-memory top config are not plain dictionaries but are classes, so the code needs to handle either. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This makes the complete top config hjson have the same names for attributes. Previously the original name for this was "clk" but the complete config renames it to "clock" because of the Clocks class field names. The Clocks class is used quite a bit more in topgen, so this rename creates less canges, and it is arguably a better name. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This is a big change in topgen. Some salient changes: - It always regenerates from scratch the ipgen in-memory configurations (as IpBlock objects) so it is truly incremental: the previous flow reused the pre-existing generated ipgen hjson if found. - It delays the actual ipgen file generation until the complete top config is generated. This is a step towards breaking topgen into smaller and mutually independent steps, thus enabling parallelizing them. - On each pass it uses the most up-to-date top config, and detects convergence if the top configs from one pass and the previous one match. - It converges with a single pass of process_top (though a second pass is done to enable the comparison). - The generated RTL with and without these changes is identical. In order to accomplish this some important changes are made: - Call some of the merge_top functions suitable for each specific ipgen prior to generating the in-memory hjson config. This means not all IpBlock objects are available, so these merge functions are instrumented to skip missing blocks. - The generation of ipgens are ordered according to dependencies. This order is created from the expected block functionality, so is not derived from a constructed graph, but converging in one pass means it is correct. - Calling merge functions early means some of the objects in the in-memory top config are not plain dictionaries but are classes, so the code needs to handle either. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

- Split the xbar flow into updates to the top config and the actual RTL generation. - Place the xbar top config updates before all file generation - Dump the complete config after all updates and before other file generation Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

The reset_requests top config entry should really be initialized from the original hjson file rather than explicitly in merge.py, since it ends up making assumptions about what reset_requests should have. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This makes the complete top config hjson have the same names for attributes. Previously the original name for this was "clk" but the complete config renames it to "clock" because of the Clocks class field names. The Clocks class is used quite a bit more in topgen, so this rename creates less canges, and it is arguably a better name. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This is a big change in topgen. Some salient changes: - It always regenerates from scratch the ipgen in-memory configurations (as IpBlock objects) so it is truly incremental: the previous flow reused the pre-existing generated ipgen hjson if found. - It delays the actual ipgen file generation until the complete top config is generated. This is a step towards breaking topgen into smaller and mutually independent steps, thus enabling parallelizing them. - On each pass it uses the most up-to-date top config, and detects convergence if the top configs from one pass and the previous one match. - It converges with a single pass of process_top (though a second pass is done to enable the comparison). - The generated RTL with and without these changes is identical. In order to accomplish this some important changes are made: - Call some of the merge_top functions suitable for each specific ipgen prior to generating the in-memory hjson config. This means not all IpBlock objects are available, so these merge functions are instrumented to skip missing blocks. - The generation of ipgens are ordered according to dependencies. This order is created from the expected block functionality, so is not derived from a constructed graph, but converging in one pass means it is correct. - Calling merge functions early means some of the objects in the in-memory top config are not plain dictionaries but are classes, so the code needs to handle either. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

- Split the xbar flow into updates to the top config and the actual RTL generation. - Place the xbar top config updates before all file generation - Dump the complete config after all updates and before other file generation Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

The reset_requests top config entry should really be initialized from the original hjson file rather than explicitly in merge.py, since it ends up making assumptions about what reset_requests should have. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This makes the complete top config hjson have the same names for attributes. Previously the original name for this was "clk" but the complete config renames it to "clock" because of the Clocks class field names. The Clocks class is used quite a bit more in topgen, so this rename creates less canges, and it is arguably a better name. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

This is a big change in topgen. Some salient changes: - It always regenerates from scratch the ipgen in-memory configurations (as IpBlock objects) so it is truly incremental: the previous flow reused the pre-existing generated ipgen hjson if found. - It delays the actual ipgen file generation until the complete top config is generated. This is a step towards breaking topgen into smaller and mutually independent steps, thus enabling parallelizing them. - On each pass it uses the most up-to-date top config, and detects convergence if the top configs from one pass and the previous one match. - It converges with a single pass of process_top (though a second pass is done to enable the comparison). - The generated RTL with and without these changes is identical. In order to accomplish this some important changes are made: - Call some of the merge_top functions suitable for each specific ipgen prior to generating the in-memory hjson config. This means not all IpBlock objects are available, so these merge functions are instrumented to skip missing blocks. - The generation of ipgens are ordered according to dependencies. This order is created from the expected block functionality, so is not derived from a constructed graph, but converging in one pass means it is correct. - Calling merge functions early means some of the objects in the in-memory top config are not plain dictionaries but are classes, so the code needs to handle either. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

Add support for added top config attributes. Dive deeper into a few attributes. Left a few TODOs for some attributes, most of these are not yet populated. Part of lowRISC#25920 Signed-off-by: Guillermo Maturana <[email protected]>

matutem self-assigned this Jan 17, 2025

matutem added Component:Tooling Issues related to tooling, e.g. tools/scripts for doc, code generation (docgen, reggen), CSR Component:MultiTop Component:IntegratedFoundational For an issue that is considered general to all Integrated variants labels Jan 17, 2025

matutem mentioned this issue Jan 31, 2025

[tools,topgen] Enhance and simplify topgen #25933

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tools,topgen] Split topgen into creating complete topcfg and the rest #25920

[tools,topgen] Split topgen into creating complete topcfg and the rest #25920

matutem commented Jan 17, 2025

[tools,topgen] Split topgen into creating complete topcfg and the rest #25920

[tools,topgen] Split topgen into creating complete topcfg and the rest #25920

Comments

matutem commented Jan 17, 2025

Description