VCU118 FPGA Updates + FireMarshal on Prototypes

docs/Prototyping/VCU118.rst

@@ -58,3 +58,107 @@ The TSI Host Widget is used to interact with the DUT from the prototype over a S
 .. Note:: Remember that since whenever a new test harness is created (or the config changes, or the config packages changes, or...), you need to modify the make invocation.
     For example, ``make SUB_PROJECT=vcu118 CONFIG=MyNewVCU118Config CONFIG_PACKAGE=this.is.my.scala.package bitstream``.
     See :ref:`Prototyping/General:Generating a Bitstream` for information on the various make variables.
+
+Running Linux with Basic and Bringup Platforms
+----------------------------------------------
+
+As mentioned above, the default VCU118 harness is setup with a UART and a SPI SDCard.
+These are utilized to both interact with the DUT (with the UART) and load in Linux (with the SDCard).
+The following steps describe how to build and run buildroot Linux on the prototype platform.
+
+Building Linux with FireMarshal
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since the prototype does not have a block device, we build Linux with the rootfs built into the binary (otherwise known as "initramfs" or "nodisk" version of Linux).
+To make building this type of Linux binary easy, we will use the FireMarshal platform (see :ref:`fire-marshal` for more information).
+
+1. Setup FireMarshal (see :ref:`fire-marshal` on the initial setup).
+2. By default, FireMarshal is setup to work with FireSim.
+   Instead, we want to target the prototype platform.
+   This is done by switching the FireMarshal "board" from "firechip" to "prototype" using ``marshal-config.yaml``:
+
+.. code-block:: shell
+
+    echo "board-dir : 'boards/prototype'" > $PATH_TO_FIREMARSHAL/marshal-config.yaml
+
+.. Note:: Refer to the FireMarshal docs on more ways to set the board differently through environment variables and more.
+
+3. Next, build the workload (a.k.a buildroot Linux) with nodisk with FireMarshal.
+   For the rest of these steps, we will assume you are using the base ``br-base.json`` workload.
+   This workload has basic support for GPIO and SPI drivers but you can build off it in different workloads (refer to FireMarshal docs on workload inheritance).
+
+.. code-block:: shell
+
+    ./marshal -v -d build br-base.json # here the -d indicates --nodisk or initramfs
+
+.. Note:: Using the "board" FireMarshal functionality allows any child workload depending on ``br-base.json`` to use the "prototype" ``br-base.json`` rather than the FireChip version.
+   Thus, you can re-use existing workloads that depend on ``br-base.json`` on the prototype platform by just changing the "board"!
+
+4. The last step to generate the proper binary is to flatten it.
+   This is done by using FireMarshal's install feature and will produce a ``*-flat`` binary in the ``$PATH_TO_FIREMARSHAL/images`` directory (in our case ``br-base-bin-nodisk-flat``).
+
+.. code-block:: shell
+
+   ./marshal -v -d install -t prototype br-base.json
+
+Setting up the SDCard
+~~~~~~~~~~~~~~~~~~~~~
+
+These instructions assume that you have a spare uSDCard that can be loaded with Linux and other files using two partitions.
+The 1st partition will be used to store the Linux binary (created with FireMarshal or other means) while the 2nd partition will be used to store miscellaneous files.
+Additionally, these instructions assume you are using Linux with ``sudo`` privileges and ``gdisk`` but you can follow a similar set of steps on Mac (using ``gpt`` or another similar program).
+
+1. Wipe the GPT on the card using ``gdisk``.
+   Use the `z` command to zap everything.
+   For rest of these instructions, we assume the SDCard path is ``/dev/sdc`` (replace this with the path to your SDCard).
+
+.. code-block:: shell
+
+    sudo gdisk /dev/sdc
+
+2. The VCU118 bootrom assumes that the Linux binary to load into memory will be located on sector 34 of the SDCard.
+   Change the default partition alignment to `1` so you can write to sector `34`.
+   Do this with the `l` command.
+
+3. Create the new GPT with `o`.
+   Click yes on all the prompts.
+
+4. Create a 512MiB partition to store the Linux binary (this can be smaller but it must be larger than the size of the Linux binary).
+   Use `n` and select sector 34, with size `+1048576` (corresponding to 512MiB).
+   For the type search for the `apfs` type and use the hex number given.
+
+5. Create a second partition to store any other files with the rest of the SDCard.
+   Use `n` and use the defaults for starting sector and overall size (expand the 2nd partition to the rest of the SDCard space).
+   For the type search for the `hfs` and use the hex number given.
+
+6. Write the changes using `w`.
+
+7. Setup the filesystem on the 2nd partition.
+   Note that the ``/dev/sdc2`` points to the 2nd partition.
+   Use the following command:
+
+.. code-block:: shell
+
+    sudo mkfs.hfs -v "PrototypeData" /dev/sdc2
+
+Transfer and Run Linux from the SDCard
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After you have a Linux boot binary and the SDCard is setup properly (1st partition at sector 34), you can transfer the binary to the 1st SDCard partition.
+In this example, we generated a ``br-base-bin-nodisk-flat`` from FireMarshal and we will load it using ``dd``.
+Note that ``sdc1`` points to the 1st partition (remember to change the ``sdc`` to your own SDCard path).
+
+.. code-block:: shell
+
+    sudo dd if=$PATH_TO_FIREMARSHAL/br-base-bin-nodisk-flat of=/dev/sdc1
+
+If you want to add files to the 2nd partition, you can also do this now.
+
+After loading the SDCard with Linux and potentially other files, you can program the FPGA and plug in the SDCard.
+To interact with Linux via the UART console, you can connect to the serial port (in this case called ``ttyUSB1``) using something like ``screen``:
+
+.. code-block:: shell
+
+   screen -S FPGA_UART_CONSOLE /dev/ttyUSB1 115200
+
+Once connected you should see the binary being loaded as well as Linux output (in some cases you might need to reset the DUT).

fpga/src/main/resources/vcu118/sdboot/sd.c

@@ -8,10 +8,12 @@
 #define DEBUG
 #include "kprintf.h"
 
-#define MAX_CORES 8
-
-// A sector is 512 bytes, so ((1 << 11) * 512) = 1 MiB
-#define PAYLOAD_SIZE	(16 << 11)
+// Total payload in B
+#define PAYLOAD_SIZE_B (30 << 20) // default: 30MiB
+// A sector is 512 bytes, so (1 << 11) * 512B = 1 MiB
+#define SECTOR_SIZE_B 512
+// Payload size in # of sectors
+#define PAYLOAD_SIZE (PAYLOAD_SIZE_B / SECTOR_SIZE_B)
 
 // The sector at which the BBL partition starts
 #define BBL_PARTITION_START_SECTOR 34
@@ -168,9 +170,11 @@ static int copy(void)
 	int rc = 0;
 
 	dputs("CMD18");
+
+	kprintf("LOADING 0x%xB PAYLOAD\r\n", PAYLOAD_SIZE_B);
 	kprintf("LOADING  ");
 
-    // John: Let's go slow until we get this working
+	// TODO: Speed up SPI freq. (breaks between these two values)
 	//REG32(spi, SPI_REG_SCKDIV) = (F_CLK / 16666666UL);
 	REG32(spi, SPI_REG_SCKDIV) = (F_CLK / 5000000UL);
 	if (sd_cmd(0x52, BBL_PARTITION_START_SECTOR, 0xE1) != 0x00) {
@@ -182,7 +186,7 @@ static int copy(void)
 		long n;
 
 		crc = 0;
-		n = 512;
+		n = SECTOR_SIZE_B;
 		while (sd_dummy() != 0xFE);
 		do {
 			uint8_t x = sd_dummy();

fpga/src/main/scala/vcu118/Configs.scala

@@ -17,7 +17,7 @@ import sifive.fpgashells.shell.xilinx.{VCU118ShellPMOD, VCU118DDRSize}
 
 import testchipip.{SerialTLKey}
 
-import chipyard.{BuildSystem, ExtTLMem}
+import chipyard.{BuildSystem, ExtTLMem, DefaultClockFrequencyKey}
 
 class WithDefaultPeripherals extends Config((site, here, up) => {
   case PeripheryUARTKey => List(UARTParams(address = BigInt(0x64000000L)))
@@ -26,11 +26,10 @@ class WithDefaultPeripherals extends Config((site, here, up) => {
 })
 
 class WithSystemModifications extends Config((site, here, up) => {
-  case PeripheryBusKey => up(PeripheryBusKey, site).copy(dtsFrequency = Some(site(FPGAFrequencyKey).toInt*1000000))
-  case DTSTimebase => BigInt(1000000)
+  case DTSTimebase => BigInt((1e6).toLong)
   case BootROMLocated(x) => up(BootROMLocated(x), site).map { p =>
     // invoke makefile for sdboot
-    val freqMHz = site(FPGAFrequencyKey).toInt * 1000000
+    val freqMHz = (site(DefaultClockFrequencyKey) * 1e6).toLong
     val make = s"make -C fpga/src/main/resources/vcu118/sdboot PBUS_CLK=${freqMHz} bin"
     require (make.! == 0, "Failed to build bootrom")
     p.copy(hang = 0x10000, contentFileName = s"./fpga/src/main/resources/vcu118/sdboot/build/sdboot.bin")
@@ -41,18 +40,23 @@ class WithSystemModifications extends Config((site, here, up) => {
 
 // DOC include start: AbstractVCU118 and Rocket
 class WithVCU118Tweaks extends Config(
+  // harness binders
   new WithUART ++
   new WithSPISDCard ++
   new WithDDRMem ++
+  // io binders
   new WithUARTIOPassthrough ++
   new WithSPIIOPassthrough ++
   new WithTLIOPassthrough ++
+  // other configuration
   new WithDefaultPeripherals ++
   new chipyard.config.WithTLBackingMemory ++ // use TL backing memory
   new WithSystemModifications ++ // setup busses, use sdboot bootrom, setup ext. mem. size
   new chipyard.config.WithNoDebug ++ // remove debug module
   new freechips.rocketchip.subsystem.WithoutTLMonitors ++
-  new freechips.rocketchip.subsystem.WithNMemoryChannels(1))
+  new freechips.rocketchip.subsystem.WithNMemoryChannels(1) ++
+  new WithFPGAFrequency(100) // default 100MHz freq
+)
 
 class RocketVCU118Config extends Config(
   new WithVCU118Tweaks ++
@@ -64,9 +68,10 @@ class BoomVCU118Config extends Config(
   new WithVCU118Tweaks ++
   new chipyard.MegaBoomConfig)
 
-class WithFPGAFrequency(MHz: Double) extends Config((site, here, up) => {
-  case FPGAFrequencyKey => MHz
-})
+class WithFPGAFrequency(fMHz: Double) extends Config(
+  new chipyard.config.WithPeripheryBusFrequency(fMHz) ++ // assumes using PBUS as default freq.
+  new chipyard.config.WithMemoryBusFrequency(fMHz)
+)
 
 class WithFPGAFreq25MHz extends WithFPGAFrequency(25)
 class WithFPGAFreq50MHz extends WithFPGAFrequency(50)

fpga/src/main/scala/vcu118/TestHarness.scala

@@ -17,12 +17,10 @@ import sifive.blocks.devices.uart._
 import sifive.blocks.devices.spi._
 import sifive.blocks.devices.gpio._
 
-import chipyard.{HasHarnessSignalReferences, HasTestHarnessFunctions, BuildTop, ChipTop, ExtTLMem, CanHaveMasterTLMemPort}
+import chipyard.{HasHarnessSignalReferences, HasTestHarnessFunctions, BuildTop, ChipTop, ExtTLMem, CanHaveMasterTLMemPort, DefaultClockFrequencyKey, HasReferenceClockFreq}
 import chipyard.iobinders.{HasIOBinders}
 import chipyard.harness.{ApplyHarnessBinders}
 
-case object FPGAFrequencyKey extends Field[Double](100.0)
-
 class VCU118FPGATestHarness(override implicit val p: Parameters) extends VCU118ShellBasicOverlays {
 
   def dp = designParameters
@@ -55,7 +53,8 @@ class VCU118FPGATestHarness(override implicit val p: Parameters) extends VCU118S
   harnessSysPLL := sysClkNode
 
   // create and connect to the dutClock
-  val dutClock = ClockSinkNode(freqMHz = dp(FPGAFrequencyKey))
+  println(s"VCU118 FPGA Base Clock Freq: ${dp(DefaultClockFrequencyKey)} MHz")
+  val dutClock = ClockSinkNode(freqMHz = dp(DefaultClockFrequencyKey))
   val dutWrangler = LazyModule(new ResetWrangler)
   val dutGroup = ClockGroup()
   dutClock := dutWrangler.node := dutGroup := harnessSysPLL
@@ -136,4 +135,11 @@ class VCU118FPGATestHarnessImp(_outer: VCU118FPGATestHarness) extends LazyRawMod
   _outer.topDesign match { case d: HasIOBinders =>
     ApplyHarnessBinders(this, d.lazySystem, d.portMap)
   }
+
+  // check the top-level reference clock is equal to the default
+  // non-exhaustive since you need all ChipTop clocks to equal the default
+  _outer.topDesign match {
+    case d: HasReferenceClockFreq => require(d.refClockFreqMHz == p(DefaultClockFrequencyKey))
+    case _ =>
+  }
 }