Expose dmi signals instead of jtag signals in earlgrey toplevel

[olofk]

By exposing the dmi signals instead of the JTAG signals, it's possible to create target-specific transports for devices without accessible JTAG signals. Primarily, this would allow using FPGA macros such as Xilinx BSCAN and Intel's sld_virtual_jtag to tunnel JTAG through their dedicated JTAG ports, I did something similar in the SweRVolf SoC here, but it would also allow other FPGA and ASIC targets to define more esoteric transports such as UART or Ethernet.

Existing targets can implement the dmi_jtag module in their toplevel instead for complete backwards compatibility (except perhaps for unit tests that might use these interfaces internally)

[imphil]

Yeah, we had the two modules separated before and only merged them to make verification easier. You bring up a good reason to keep them separate.

@msfschaffner @sriyerg there's nothing we need to do right now, but we should keep this in mind when redesigning the top-level hierarchy.

[msfschaffner]

Indeed, this makes sense, thanks @olofk for bringing it up. Let's discuss this in more detail once we continue our top-level reorg discussion #295.

@eunchan @tjaychen for visibility

[tjaychen]

@msfschaffner, this seems reasonable, but it probably has some implications on where we place things hierarchically, is that right? For example if earlgrey exposed dmi signals on its pins, that technically would require pinmux to move also doesn't it?

[msfschaffner]

Hm indeed, that could be a complication if we require the debug pins always go through the pinmux.

this ties also into the discussion what the "OT IP" should expose such that it can be instantiated elsewhere (either in a bigger system or in the toplevel wrapper for ASIC/FPGA). I honestly do not know yet what the best answer to this is. I.e. once you move the pinmux out of the top of the "OT IP", we need to expose the internal bus due to the config registers in the pinmux. Or we tackle that issue in a similar way as we did width padctrl, where we kept the config register inside the IP, but route out the config signals that the register file drives.

However, didn't we say at some point that the debug access via JTAG (or dmi for that matter) should not be modifiable via any software programmable component (like the pinmux)?

[asb]

I don't actually recall a discussion where we decided JTAG can't be over pinmuxed IO...doesn't mean it didn't happen though. Given debug is likely to be fused off and disabled, it feels like it would be pretty handy to be able to use those pins for something else. But it's been a long day, so I could be missing a good reason why people thought that might be a bad idea.

[tjaychen]

so let me clarify a bit...

JTAG MUST be pinmuxed in the long term. We cannot afford to burn 4 or 5 dedicated pins for any system..especially for a function that will not be used beyond manufacturing, limited debug and perhaps RMA.

However, the debug function needs to have a way of being disabled once a system enters 'production' mode. so those things are somewhat orthogonal.

To @msf point, I personally think exposing a bus out of OT_IP will be inevitable in the long term for various reasons. @msf, maybe you and I can get together and do some more whiteboarding sometime? Should be fun!

[eunchan]

My 2cents here. If anyone wants to expose DMI interface directy outside of the top, it should be separate top design not top_earlgrey unless everyone agree that top_earlgrey has DMI port always. If we want to have DMI for a variant (some FPGA target) and JTAG for others, It is hard to manage in a target in my opinion.

This leads to the deeper question. What should the wrappers, _verilator/_asic/_nexysvideo look like? _nexysvideo and _arty7 are FPGA board specific targets, other _asic is somewhat chip/package specific. _verilator is special thing, I don't think _verilator should be in the RTL directory. It seems like to be better separating those two concepts(board/chip). Board specific one should be outside of the chip specific.

For instance, NexysVideo has shared JTAG/SPI interface but others don't. It means NexysVideo is functionally different not on FPGA chip level, but board-specific level. We could make them are same with either 1) Add SPI to USB (FTDI) interposer to NexysVideo (+extra cost), or 2) every target to have shared JTAG/SPI.

I think all the variants in a hw/top_{name} should be functionally same. If not, it should be another top.

I don't have the answer how to manage these yet. Trying to figure out with @msfschaffner (@tjaychen you tagged wrong person :) ) and @tjaychen. Please suggest any idea!

[tjaychen]

@msfschaffner, @eunchan and I talked about a different hierarchical separation that has certain longer term benefits as well. We'll probably try to diagram this and share more widely. This proposal however does not really address @eunchan's question.

I think what @eunchan is bringing up is really what hw/top_* should represent. I have always viewed it as the encapsulation of a particular end product, with possible variants underneath where needed (for example the NexysVideo example described). This to me feels reasonable, but does create complications especially because pinmux differences between asic / fpga variants will mean we need to generate multiple versions of pinmuxes (there may be other situations like this also).

This worked better for projects where these types of collateral were generated on the fly, and we could simply manipulate the configuration input...but for OpenTitan since everything is checked in, we'd need to be a bit more careful.

@eunchan , if i understand you correctly, you're suggesting that potentially top_earlgrey and top_earlgrey_nexysvideo would be split into different tops, because of differences of pinmuxing and possible debug connections, is that right? For example if top_earlgrey wanted to pinmux JTAG with other pins (likely) but top_earlgrey_nexysvideo enforced JTAG/SPI muxing. I am assuming for now there is no separate FTDI.

[eunchan]

Thanks for the comment Tim! I've just added a few comments to ammend my previous thought.

On Thu, Nov 07, 2019 at 10:02:17AM -0800, tjaychen wrote: @msfschaffner, @eunchan and I talked about a different hierarchical separation that has certain longer term benefits as well. We'll probably try to diagram this and share more widely. This proposal however does not really address @eunchan's question. I think what @eunchan is bringing up is really what `hw/top_*` should represent. I have always viewed it as the encapsulation of a particular end product, with possible variants underneath where needed (for example the NexysVideo example described). This to me feels reasonable, but does create complications especially because pinmux differences between asic / fpga variants will mean we need to generate multiple versions of pinmuxes (there may be other situations like this also). This worked better for projects where these types of collateral were generated on the fly, and we could simply manipulate the configuration input...but for OpenTitan since everything is checked in, we'd need to be a bit more careful.

I think some contents like padring, could be auto-generated. The reason why I insist to check-in the generated file is to help understand the design without running the generators, which lower the barrier for the new comers. PADRING, CLKGEN could be somewhat special, anyway those depend on the process/board/ FPGA and the generated files also don't represent actual functions (ANALOG..meaning the behavioral model...).

@eunchan , if i understand you correctly, you're suggesting that potentially `top_earlgrey` and `top_earlgrey_nexysvideo` would be split into different tops, because of differences of pinmuxing and possible debug connections, is that right?

Yes, if we decide to make DMI interface exposed on FPGA and not on ASIC, also JTAG/SPI mux on NexysVideo not on ASIC. I understand some mismatches between FPGA and the silicon is inevitable. But should try to standardize the design as much as possible to avoid any unintended issues.

[tjaychen]

yeah i agree. We should think about this a bit more. Specifically for the debug adapter, it's not unreasonable for us to find a prim_* like solution that could allow things to switch on the fly.

The generated modules are my bigger concern. While I have worked with generated on the go, I do like it when things are checked in. Let @msfschaffner document out the other hierarchy, and we'll sync with you to see if it helps address any of these areas.

Another thought that occurred to me is that if our design ever gets big enough where certain FPGAs don't fit, we'll need even more flexibility to stub things in and out, and it's worth considering how that might fit into this scheme also.

[eunchan]

On Thu, Nov 07, 2019 at 10:44:12AM -0800, tjaychen wrote: yeah i agree. We should think about this a bit more. Specifically for the debug adapter, it's not unreasonable for us to find a `prim_*` like solution that could allow things to switch on the fly. The generated modules are my bigger concern. While I have worked with generated on the go, I do like it when things are checked in. Let @msfschaffner document out the other hierarchy, and we'll sync with you to see if it helps address any of these areas. Another thought that occurred to me is that if our design ever gets big enough where certain FPGAs don't fit, we'll need even more flexibility to stub things in and out, and it's worth considering how that might fit into this scheme also.

Right. That one should be considered also. When we move to the subsystem hierarchy, we should have a way to make a subsystem to be a FPGA chip (generating the standalone wrapper together). It isn't that efficient but easiest solution I think.

[msfschaffner]

This issue is related to the top-level reorg.

[rswarbrick]

@msfschaffner: If I understand correctly, we've "doubled down" on JTAG for all our TAPS, so the idea that this issue is tracking isn't really possible any more. Is that correct?

[msfschaffner]

Indeed, all TAPs use JTAG now, and the JTAG selection mux has been moved into the pinmux hierarchy. This has been done primarily so that debug access to different TAPs can be controlled based on the current the life cycle state.

Also, the DMI is a RISC-V-debug-module-specific protocol which cannot be used for all TAPs in the design. Aligning on JTAG makes things more uniform, which in turn makes it easier to multiplex TAPs through the same design pins and qualify them with the current life cycle state.

It will be difficult to switch to a different transport layer at this point.
I hence recommend to close this issue.
Please comment and reopen if you think otherwise.

---

As for the overall top-level structure: this has evolved quite a bit as well by now.
The main design top-level generated by topgen is still called top_<design name>.
However, the technology specific top-levels for a specific top_<design name> design are now also generated by topgen (as opposed to the hand coded top-levels for FPGA, ASIC etc that we had before). These hierarchies are called chip_<design name>_<target name>.

The concept of targets has been introduced to the top_<design name>.hjson configuration file, which allows to configure a base configuration for the pinout and pinmux, plus very limited modifications (e.g. additional clock pads) that can be different for each technology target.

See for example here:

opentitan/hw/top_earlgrey/data/top_earlgrey.hjson

Lines 1285 to 1297 in ccffc98

	// Implementation targets.
	// This defines the configuration of the target-specific chip-levels to
	// generate from the shared template. Each target uses the same base
	// configuration for the pinmux and pinout as defined above, and the
	// generated software constants for the pinmux DIF do not change among the
	// implementation targets. However, in order to accommodate slight
	// differences among the ASIC, FPGA emulation and simulation environments,
	// it is possible to make very limited pinout changes below. In particular,
	// it is possible to remove and tie-off specific pads, or add more 'manual'
	// pads that need to be manually connected in the template. It is also possible
	// to override the JTAG and strap locations indices, since the testing and DFT
	// setups may differ among the targets.
	targets: [

Corresponding issues:
#295 (comment)
#5221