On 18 Dec 2017, at 07:28, Luke Kenneth Casson Leighton <lkcl(a)lkcl.net&gt; wrote: http://www.lowrisc.org/docs/minion-v0.4/overview/ i notice in that documentation that there is significant difficulty experienced in getting the core booted up. a limit of 60k ROM is mentioned, 5mhz card speeds set as a hard-coded limit, older cards not supported and so on. in commercial SoCs all these problems are solved with a multi-stage boot process. they only require a 16k executable to be loaded off of either NAND, eMMC, SPI or MicroSD (the loader itself being executed from a boot ROM, which itself can be incredibly small), where the first task of that program is: to initialise DDR3 RAM, PLLs and to begin executing a *larger* (2nd stage) boot process. the sub-16k boot process usually loads u-boot directly into the *DDR3* ram then executes that. usually the sub-16k boot process is actually a cut-down version *of* u-boot, called an "SPL loader". all of this is extremely well-known, particularly to the developers of u-boot who often have to merge proprietary SPL-loaders or write ones for various SoCs where the proprietary SoC companies couldn't be bothered to respect software licenses... such as the GPL. which brings me to the strange conflict going on in my mind: what have i missed? is the minion 0.4 boot process sufficiently different from a standard commercial SoC (the A20's eGON bootloader is only 32k in size and it *still* does NAND *and* SPI *and* MicroSD *and* eMMC booting http://linux-sunxi.org/BROM - that's 60% the size of the ROM mentioned at the page above), such that it's not possible to deploy the same technique... or was it an oversight on the part of the team that they were unaware of this tried-and-tested technique? what am i missing? tia, l.

Dear Luke, Allow me to defend the design choices if I can, preferably without starting a “my processor is better than your processor” debate.

Now when we consider LowRISC, the parameters are different, memory is relatively cheap,

it’s a 64-bit architecture so offsets are relatively large,

compilation technology is preliminary,

compressed instructions are not implemented in our version of Rocket,

and there is a desire to minimise ROM space to make more room for caches etc.

And I forgot to mention, our ROM, incorporates a full FAT-compatible filing system which in principle allows secure booting,

rather than a typical first stage boot loader which just reads a fixed area of the boot disk and hopes for the best.

Furthermore SOCs frequently incorporate proprietary peripheral host controllers which make booting easier, these would have to come out of our overall gate count budget, whereas our demonstrator is targeting low end educational FPGA boards.

> what am i missing? It's simply less developed. The first-stage bootloader hasn't had the development time to minimise code size, increase flexibility etc. The current booting procedure is something that works reasonably well for the current FPGA demonstrator, rather than being a fully developed end product.

Ideally we'd have the first stage bootloader load coreboot.

Re initialisation of DDR3 RAM: I've had some interesting discussions with people about this. The initialisation process is often very much undocumented,

but there are advantages to initialising RAM as late as possible and allowing caches to be configured as simple memories in the early boot process. This allows you to get a system you can meaningfully debug even if there are issues configuring the DDR controller properly.

Hi, On Mon, Dec 18, 2017 at 09:24:51AM +0000, Luke Kenneth Casson Leighton wrote: > On Mon, Dec 18, 2017 at 8:49 AM, Alex Bradbury <asb(a)lowrisc.org&gt; wrote: > >>> what am i missing? >> It's simply less developed. The first-stage bootloader hasn't had the >> development time to minimise code size, increase flexibility etc. The >> current booting procedure is something that works reasonably well for >> the current FPGA demonstrator, rather than being a fully developed end >> product. > that makes a lot of sense. > >> Ideally we'd have the first stage bootloader load coreboot. > yyeahh that sounds like it'd need a bit of work, particularly to > support the "SPL" concept which *already* exists in u-boot. and > u-boot's already up and running on risc-v. coreboot i have to say i'm > very nervous of, because it's designed for Monster Machines (x86 > primarily) where for boot purposes alone the cost of putting down a > 16MBYTE or 32MBYTE SPI NOR FLASH IC that's more expensive than an 32 MBit is still common, at least on Laptops; although with the Intel ME firmware and UEFI, we're seeing 16 MByte parts too, because they are so big. > *entire embedded/smart-style processor* such as the Allwinner A20, A64 > or an Ingenic jz4775 SoC, is considered "pocket change" because of the > insane cost of x86 and amd processors. [i'm looking to do a THREE > dollar RISC-V 64-bit SoC]. I don't know the unit price here, but coreboot runs on and is shipped with some ARM SoCs from Rockchip/Nvidia/Mediatek[4] because it's used in Chromebooks[5]. > ha! mind you, coreboot already has coreboot support, yay! > https://github.com/coreboot/coreboot/tree/master/src/arch/riscv > > actively developed, too. how about that. Coreboot on RISC-V even gets a conference talk every once in a while :) >> Re initialisation of DDR3 RAM: I've had some interesting discussions >> with people about this. The initialisation process is often very much >> undocumented, > eww, yuk, what the hell?? how is anyone using xilinx or other FPGAs > supposed to *use* them if the *initialisation* process of all > peripherals isn't damn well made public?? ngggggggh! :) I agree. DDR3 RAM init is hard enough when the controller is documented[1], so it's very much preferable to have documentation that's available to as many developers as possible. >> but there are advantages to initialising RAM as late as >> possible and allowing caches to be configured as simple memories in >> the early boot process. This allows you to get a system you can >> meaningfully debug even if there are issues configuring the DDR >> controller properly. > ahh ok. i take it this something specific to FPGAs? SRAMs start out > as SRAMs and get reconfigured into caches later? i've barely just > started (i've got a zc706, now, thanks to a sponsor, woo!) Coreboot on x86 has to use an ugly trick called "cache-as-RAM". Through some trickery (enable writeback caching, but *disable* writeback), it can use part of the cache as an SRAM buffer in order to have a stack and run C code. The model I've seen in various ARM SoCs seems much preferable: Have a few dozen KiB of SRAM that can be used before the DRAM controller is up, and let the caches be caches. > ok.. so... let me think... ah no: you may have misunderstood (or i > have... let me think/talk it through...) i wasn't talking about > putting the DDR3 initialisation code into the *boot* ROM, i'm > referring to the (common) practice of putting the DDR3 initialisation > into the u-boot 16k *SPL* loader program. that's entirely software, > nothing to do with the "compilation step and uploading to and > configuring the FPGA hardware" at all. That's good, because RAM init can be quite board specific. One some boards, you'll read the SPD EEPROMs on (SO-)DIMMs, on others a few resistors connected to GPIOs indicate which of the possible RAM configurations is present. Sometimes there is only one configuration. > so you could *choose* not to initialise the DDR3 controller, by > simply flipping a config option in u-boot, or instead of u-boot SPL > (or coreboot once it has SPL mode), just putting a debug / test > program onto the SD card. I'm not sure what SPL mode entails exactly, but coreboot can load U-Boot as a payload[2] on other architectures (it has not been tested on RISC-V). > i think i start to see what you mean. you'd want to upload as much > of a "debugging / test" program as you could, so as to be able to > investigate. yes that makes sense. Only somewhat related: To ease early debugging, there should not be too many configuration steps (PLL setup, clock gating, power gating, DMA-able RAM) in the way of having a UART. And whatever is done, don't go the x86 way of relying on EHCI debug[3]! Thanks, Jonathan Neuschäfer [1]: https://www.youtube.com/watch?v=h-Lkkg03Erk [2]: A "payload" in coreboot's terminology is any program that is directly loaded by coreboot. Often this is a boot loader (GRUB2, SeaBIOS, depthcharge, even Linux), because coreboot can't really load files from anything but its internal file system. [3]: https://www.coreboot.org/EHCI_Debug_Port [4]: https://review.coreboot.org/cgit/coreboot.git/tree/src/soc [5]: https://www.chromium.org/chromium-os/developer-information-for-chrome-os-...

> *entire embedded/smart-style processor* such as the Allwinner A20, A64 > or an Ingenic jz4775 SoC, is considered "pocket change" because of the > insane cost of x86 and amd processors. [i'm looking to do a THREE > dollar RISC-V 64-bit SoC]. I don't know the unit price here, but coreboot runs on and is shipped with some ARM SoCs from Rockchip/Nvidia/Mediatek[4] because it's used in Chromebooks[5].

Coreboot on RISC-V even gets a conference talk every once in a while :)

That's good, because RAM init can be quite board specific.

> so you could *choose* not to initialise the DDR3 controller, by > simply flipping a config option in u-boot, or instead of u-boot SPL > (or coreboot once it has SPL mode), just putting a debug / test > program onto the SD card. I'm not sure what SPL mode entails exactly,

but coreboot can load U-Boot as a payload[2] on other architectures (it has not been tested on RISC-V).

> i think i start to see what you mean. you'd want to upload as much > of a "debugging / test" program as you could, so as to be able to > investigate. yes that makes sense. Only somewhat related: To ease early debugging, there should not be too many configuration steps (PLL setup, clock gating, power gating, DMA-able RAM) in the way of having a UART.

And whatever is done, don't go the x86 way of relying on EHCI debug[3]!

On Mon, Dec 18, 2017 at 01:02:38PM +0000, Luke Kenneth Casson Leighton wrote: > On Mon, Dec 18, 2017 at 12:40 PM, Jonathan Neuschäfer <j.neuschaefer(a)gmx.net&gt; wrote: [...] >> That's good, because RAM init can be quite board specific. > i've never dealt with anything other than SoCs, so i'm used to code > (in u-boot) which probes the DDR3 interface, does all the > initialisation(s), cranks the memory up to a 200mhz (or other set bus > speed) and off it goes. some of the init stuff i've seen even does > speed grading testing. The way coreboot on newer chromebooks with soldered-on RAM chips usually does it, is this: 1. It reads a few GPIO to calulate the "board ID"[1] 2. It selects the right block of information, that's either compiled in or loaded from CBFS (the coreboot file system)[2]. This can be either direct settings for the DRAM controller[3], or the same data that would be stored on SPD EEPROMs[4]. 3. It applies these settings[5], which can also involve some calibration. I didn't know you could just "probe" the DDR3 interface, especially for parameters like chip size. (But then again, I haven't done RAM init myself.)

>> but coreboot can load U-Boot >> as a payload[2] on other architectures (it has not been tested on RISC-V). > is there a config option to cut the compiled target down to an (appx) > sub 16k raw binary? Coreboot internally already uses multiple stages, called bootblock, romstage (runs from "ROM" (flash), without DRAM), ramstage (runs from DRAM). The bootblock should easily fit into 16 KiB and will chain-load the next stage. One thing that might cause problems: Coreboot boots from SPI flash on most boards, so CBFS-in-MMC is probably not a well tested configuration. (Why? Because x86 only boots from memory-mapped flash, which usually means SPI these days, and parallel flash in the old days; The Chromebooks' boot security scheme depends on being able to write-lock part of the boot flash, I'm not sure if that's possible with eMMC; And, of course, "it's the default thing that coreboot does", so it's easier to do.) >> Only somewhat related: To ease early debugging, there should not be too >> many configuration steps (PLL setup, clock gating, power gating, >> DMA-able RAM) in the way of having a UART. > that's pretty much the definition of a SPI u-boot executable :) Well, yes, all these things are doable, but what do you do when they don't work and you have to debug them? An easy to initialize UART can help a lot in this case. >> And whatever is done, don't >> go the x86 way of relying on EHCI debug[3]! > :) oh is that the one where you can hack into any x86 system just > by plugging in a USB cable? No, it's not that kind of debug interface (that's Intel DCI, and it's not quite *that* drastic). EHCI Debug is a special mode of the EHCI controller that makes it possible to use it without DMA. You can only send small USB packets, but that's enough for simple logging. My point is: EHCI Debug plus a compatible adapter (some Linux computer in USB device mode, or FTDI FT232H, but not FT232R) is already a much more complex setup than necessary when you could have a simple two-pin serial interface. Thanks, Jonathan Neuschäfer [1]: https://review.coreboot.org/cgit/coreboot.git/tree/src/mainboard/google/o... [2]: https://review.coreboot.org/cgit/coreboot.git/tree/src/mainboard/google/o... [3]: https://review.coreboot.org/cgit/coreboot.git/tree/src/mainboard/google/o... https://review.coreboot.org/cgit/coreboot.git/tree/src/soc/mediatek/mt817... [4]: https://review.coreboot.org/cgit/coreboot.git/tree/src/mainboard/google/c... [5]: https://review.coreboot.org/cgit/coreboot.git/tree/src/soc/mediatek/mt817...

On 18/12/17 14:10, Jonathan Neuschäfer wrote:

> The way coreboot on newer chromebooks with soldered-on RAM chips usually > does it, is this: > > 1. It reads a few GPIO to calulate the "board ID"[1] > 2. It selects the right block of information, that's either compiled in > or loaded from CBFS (the coreboot file system)[2]. This can be either > direct settings for the DRAM controller[3], or the same data that would > be stored on SPD EEPROMs[4]. > 3. It applies these settings[5], which can also involve some > calibration. > > I didn't know you could just "probe" the DDR3 interface, especially for > parameters like chip size. (But then again, I haven't done RAM init > myself.) If you have a DIMM module it will come with a serial PROM that you can query.