Work document on lowRISC interrupt handling Version 04 January 2017 Author Reinoud Zandijk Pre-amble ========= This document assumes the SoC to be in the default split of a hypervisor minion CPU, some application CPUs and IO minion CPUs. A signalling or receiving domain can either be running on one of these minion CPUs or be multiplexed on the application CPUs. Summary ======= The proposed system for interrupt handling tries to ease interrupt delegation while allowing multiple concurrent domains to be scheduled. Changelog ========= Motivation ========== To allow each domain to be both a receptor and a sender/creator of interrupts, an traditional interrupt controller might not suffice. On the lowRISC application CPUs the only sending and receiving entities are domains. Domains that can be constructed and destructed at will. The motivation to leave out explicit interrupt source notification is based on the assumption that the act of taking an interrupt is costly for a CPU. Not only does the entire register set to be saved/restored but it can/will flush TLBs and all kinds of speculative execution caches. It would be nice if this cost would be estimated more precisely. Note that it necessarily globs all interrupts together in one go. It is expected for the handlers to probe all possible causes for a possible cause and deal with them in the priority the domain seems fit. Detailed design =============== Since the only entity scheduling domains on the CPUs is the hypervisor, it makes sense to delegate interrupt routing to it. Reception --------- For the reception of interrupts, the present MIP register is extended for a new set of an additional set of flags to facilitate an IRQ_IO addition: +++ b/riscv/encoding.h @@ -34,9 +34,13 @@ #define MIP_STIP 0x00000020 #define MIP_HTIP 0x00000040 #define MIP_MTIP 0x00000080 +#define MIP_SIIP 0x00000200 +#define MIP_HIIP 0x00000400 +#define MIP_MIIP 0x00000800 #define SIP_SSIP MIP_SSIP #define SIP_STIP MIP_STIP +#define SIP_SIIP MIP_SIIP #define PRV_U 0 #define PRV_S 1 @@ -58,6 +62,7 @@ #define IRQ_TIMER 1 #define IRQ_HOST 2 #define IRQ_COP 3 +#define IRQ_IO 4 By adding these extra bits to the CSR_MIP/CSR_SIP and CSR_MIE/CSR_SIE delegation to other processor modes is possible. The selected interrupt can be enabled if desired and can even be passed on directly to a usermode program on just one CPU. Note that there isn't an interrupt cause register (yet). The cause in this case could be the signalling domainID. Creation -------- To facilitate sending/creation of interrupts an additional CSR (s)hintr is added. Writing a domainID to this CSR causes the relevant domain to be signalled. It can be implemented as an associative array that holds (domainID, CPU_bitmap) pairs. On writing the CSR, the array is checked (in parallel?) on the domainID and on a hit, the CPUs marked active in its bitmap are signalled for an IRQ_IO; on a fail the hypervisor CPU is notified and can use this information at scheduling time. The size of the associative array determines how many domains can be scheduled in parallel; it should at least be the number of minion CPUs plus at least one or more entries for the application CPUs. The array is updated and maintained by the hypervisor at schedule time. The interface to this associative array as seen from the hypervisor minion is left out of this implementation sketch but most likely the entries could be mapped in memory space in an region accessible only by the hypervisor. Drawbacks ========= The drawback of not knowing which domain signalled might be inconvenient but allows the receiving domain to order the interrupts, in parallel even if its capable to. Suggestions =========== The interrupting domain ID could be stored in a register on the (s)hintr CSR write and communicated to all CPUs in the bitmap. If multiple domains interrupted it could be signalled as zero to indicate its a multiple. Alternatives ============ A generic IPI mechanism can be used but the interworking of possibly quite different and possibly untrustworthy domains might not be a good idea. Unresolved questions ==================== What is the exact cost of an interrupt in say the Rocket or the BOOM processor, next to the register save/restoring?