interrupt-handling-using-the-x86-apic/chap/listings.tex

\chapter{Listings}
\label{ch:listings}

This chapter contains concrete implementation examples for concepts mentioned in \autoref{ch:implementation}, demonstrated on the example of hhuOS\@.
The code is simplified and not necessarily identical to the actual implementation, to allow for a self-contained demonstration.
Names and namespaces were adjusted especially.

\clearpage

\section{Integration into HhuOS}
\label{sec:apxhhuos}

\subsection{Interrupt Handling in HhuOS}
\label{subsec:apxcurrenthhuos}

Devices that provide interrupts implement the \code{InterruptHandler} interface:

\begin{codeblock}{The InterruptHandler Interface (InterruptHandler.h)~\autocite{hhuos}}{C++}
  \cppfile{code/interrupthandler_def.cpp}
\end{codeblock}

These interrupt handlers have to be registered to the appropriate interrupt vector:

\begin{codeblock}{Assigning an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
  \cppfile{code/interruptdispatcher_assign.cpp}
\end{codeblock}

Multiple interrupt handlers can be registered to the same interrupt vector.
For every interrupt that arrives at a CPU, the first-stage interrupt handler (registered in the IDT for every vector) is called, which in turn calls the device-specific handlers:

\begin{codeblock}{Triggering an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
  \cppfile{code/interruptdispatcher_dispatch.cpp}
\end{codeblock}

The PIT interrupt handler manages the system time and the scheduler:

\begin{codeblock}{The PIT Interrupt Handler (Pit.cpp)~\autocite{hhuos}}{C++}
  \cppfile{code/pit_trigger.cpp}
\end{codeblock}

\clearpage

\subsection{Necessary Modifications to HhuOS}
\label{subsec:hhuosintegration}

To integrate the APIC implementation into hhuOS, some preexisting components of its interrupt infrastructure (see \autoref{sec:apxhhuos}) have to be modified:

\begin{enumerate}
  \item The \code{InterruptService} has to forward calls to the \code{Apic} class instead of the \code{Pic} class, depending on if the APIC system is enabled -- to keep the option of running the OS on hardware that does not support the APIC (see \autoref{lst:interruptserviceafter}).
  \item The \code{Pit} interrupt handler may no longer trigger the scheduler preemption if the APIC timer is enabled (see \autoref{lst:pithandlerafter}).
  \item The \code{System::initializeSystem} function needs to enable the APIC system if it is available (see \autoref{lst:systemafter}).
  \item The existing devices using interrupts need to use the new \code{InterruptVector} and \code{InterruptRequest} enums.
\end{enumerate}

\begin{codeblock}[label=lst:interruptserviceafter]{Using the Correct Interrupt Controller (InterruptService.cpp)}{C++}
  \cppfile{code/interruptservice_after.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:pithandlerafter]{Disabling PIT Preemption (Pit.cpp)}{C++}
  \cppfile{code/pit_after.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:systemafter]{Enabling the APIC System (System.cpp)}{C++}
  \cppfile{code/system_after.cpp}
\end{codeblock}

\subsection{Public Interface of the APIC Subsystem}
\label{subsec:hhuospublicinterface}

To interact with the APIC subsystem from the \textquote{outside}, the \code{Apic} class is used, which exposes the necessary functionality:

\begin{codeblock}{The APIC Public Interface (Apic.h)}{C++}
  \cppfile{code/apic_def.cpp}
\end{codeblock}

Enabling the APIC subsystem is done by the \code{enable} function (see \autoref{fig:apicenable}).

\clearpage

\section{Local APIC}
\label{sec:apxlocalapic}

\subsection{Accessing Local APIC Registers in xApic Mode}
\label{subsec:apxxapicregacc}

The xApic register access requires a single page of strong uncachable memory, but since this requires setting attributes in the \textquote{Page Attribute Table}\footnote{
  See \url{https://docs.kernel.org/x86/pat.html} (visited on 12/02/2023).},
\ this implementation only uses hhuOS' \code{mapIO} function, which maps a physical address to the kernel heap, with hhuOS' \textquote{CACHE\textunderscore{}DISABLE} flag set in the kernel page table:

\begin{codeblock}{Allocating the Local APIC's MMIO Region (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_mmio_alloc.cpp}
\end{codeblock}

A register can now be written as follows:

\begin{codeblock}[label=lst:lapicmmiowrite]{Writing a Local APIC MMIO Register (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_mmio_write.cpp}
\end{codeblock}

To reduce the usage of manual bit-shifting and -masking, this implementation provides structures for some commonly used registers, that implement conversion operators to the register format:

\begin{codeblock}[label=lst:msrentry]{The MSREntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_msr_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:svrentry]{The SVREntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_svr_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:lvtentry]{The LVTEntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_lvt_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:icrentry]{The ICREntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_icr_entry.cpp}
\end{codeblock}

These can be used in combination with some convenience functions:

\begin{codeblock}{Writing the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_msr.cpp}
\end{codeblock}

\begin{codeblock}{Writing the SVR (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_svr.cpp}
\end{codeblock}

\begin{codeblock}{Writing the LVT (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_lvt.cpp}
\end{codeblock}

\begin{codeblock}{Writing the ICR (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_icr.cpp}
\end{codeblock}

\subsection{Disabling PIC Mode}
\label{subsec:apxdisablepic}

Setting the IMCR\footnote{
  Writing the IMCR is detailed in the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.2.1]{mpspec}.}
\ using hhuOS' \code{IoPort} class:

\begin{codeblock}{Disabling PIC Mode (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_imcr.cpp}
\end{codeblock}

\subsection{Initializing the LVT}
\label{subsec:apxlvtinit}

The interrupt vectors are set as defined by the \code{InterruptVector} enum.
This implementation configures the LVT by using the \code{LVTEntry} struct (see \autoref{lst:lvtentry}):

\begin{codeblock}{Configuring the LINT0 Local Interrupt (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_lvt_example.cpp}
\end{codeblock}

This process is repeated for each local interrupt.

\subsection{Handling the Spurious Interrupt}
\label{subsec:apxsvr}

This implementation sets the SVR by using the \code{SVREntry} struct (see \autoref{lst:svrentry}):

\begin{codeblock}{Setting the Spurious Interrupt Vector (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_svr_example.cpp}
\end{codeblock}

Because hhuOS uses a two-stage interrupt handling approach (described in \autoref{sec:currenthhuos}), the spurious interrupt does not receive its own interrupt handler.
Instead, it is ignored in the \code{dispatch} function, hhuOS' \textquote{first-stage} interrupt handler:

\begin{codeblock}{Checking for Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
  \cppfile{code/interruptdispatcher_check_spurious.cpp}
\end{codeblock}

\begin{codeblock}{Ignoring Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
  \cppfile{code/interruptdispatcher_ignore_spurious.cpp}
\end{codeblock}

\subsection{Using the APIC Timer}
\label{subsec:apxapictimer}

The timer frequency is determined by counting the ticks in one millisecond, using the PIT as calibration source:

\begin{codeblock}{Calibrating the APIC Timer (ApicTimer.cpp)}{C++}
  \cppfile{code/apictimer_calibrate.cpp}
\end{codeblock}

This calibration is performed before the interrupts get enabled, so it is not possible to use hhuOS' \code{TimeService} for the delay.
Instead, the \code{PIT::earlyDelay} function is used, which configures the PIT for mode \textquote{Interrupt on Terminal Count} on channel zero and polls the channel's output status~\autocite{pit}:

\begin{codeblock}{Microsecond Delay without Interrupts (Pit.cpp)}{C++}
  \cppfile{code/pit_early_delay.cpp}
\end{codeblock}

Furthermore, the calibration is only performed once, even if multiple APIC timers are used.
This removes the possibility of using multiple timers with different dividers.

To handle the APIC timer interrupt on multiple cores, \(n\) \code{ApicTimer} instances are registered to the appropriate interrupt vector, where \(n\) is the number of CPUs.
Because this means, that each APIC timer interrupt on any CPU core triggers all \(n\) interrupt handlers, the handler has to determine if it belongs to the calling CPU\@.
The handler belonging to the BSP's APIC timer also triggers the scheduler preemption (instead of the PIT):

\begin{codeblock}{Handling the APIC Timer Interrupt (ApicTimer.cpp)}{C++}
  \cppfile{code/apictimer_trigger.cpp}
\end{codeblock}

\subsection{Handling Local APIC Errors}
\label{subsec:apxhandlingerror}

The error interrupt handler obtains the ESR's contents by writing to it first:

\begin{codeblock}{The Local APIC Error Interrupt Handler (ApicErrorHandler.cpp)}{C++}
  \cppfile{code/apicerror_trigger.cpp}
\end{codeblock}

Because every CPU core can only access its own local APIC's registers, a single instance of this interrupt handler can be used for each AP in the system.

\clearpage

\section{I/O APIC}
\label{sec:apxioapic}

\subsection{Accessing I/O APIC Registers}
\label{subsec:iolistings}

The I/O APIC's indirect register access requires two MMIO registers, that can be written similar to the local APIC's registers:

\begin{codeblock}{Writing an I/O APIC MMIO Register (IoApic.h)}{C++}
  \cppfile{code/ioapic_write_mmio.cpp}
\end{codeblock}

Using the \textquote{Index} and \textquote{Data} registers to access the I/O APIC's indirect registers:

\begin{codeblock}{Writing an I/O APIC Indirect Register (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_write_indirect.cpp}
\end{codeblock}

To reduce the manual bit-shifting and -masking, the same approach is used as for the local APIC\@:

\begin{codeblock}[label=lst:redtblentry]{The REDTBLEntry Structure (IoApicRegisters.h)}{C++}
  \cppfile{code/ioapic_redtbl_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:writeredtbl]{Writing the REDTBL (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_write_redtbl.cpp}
\end{codeblock}

\subsection{Interrupt Overrides}
\label{subsec:apxirqoverrides}

This implementation represents an interrupt override using the following structure:

\begin{codeblock}{The External Interrupt Override Structure (IoApic.h)}{C++}
  \cppfile{code/ioapic_irqoverride.cpp}
\end{codeblock}

During initialization, the overrides are used to set the correct interrupt vectors, polarities and trigger modes for each REDTBL entry.
The \code{REDTBLEntry} struct (see \autoref{lst:redtblentry}) is used to write the REDTBL\@:

\begin{codeblock}{Initializing the REDTBL (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_redtbl_example.cpp}
\end{codeblock}

During regular operation, the overrides are used to determine the correct REDTBL entries for e.g.\ unmasking an interrupt:

\begin{codeblock}{Unmasking an IRQ (Apic.cpp)}{C++}
  \cppfile{code/apic_allow.cpp}
\end{codeblock}

To convey this deviation between GSIs and PC/AT compatible IRQs very clearly, the public \code{Apic::allow} function accepts an \code{InterruptRequest} as an argument (that allows addressing the interrupt by name), while the internal \code{IoApic::allow} function only accepts a \code{GlobalSystemInterrupt} as argument:

\begin{codeblock}{Unmasking an IRQ Internally (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_allow.cpp}
\end{codeblock}

The internal \code{IoApic::allow} function is hidden (\code{private}) from the OS, and gets only called by the exposed \code{Apic::allow} function.
This prevents accidentally setting the wrong REDTBL entry by not taking possible interrupt overrides into account.
The same principle is applied to the other operations concerning GSIs.

\clearpage

\section{Symmetric Multiprocessing}
\label{sec:apxsymmetric}

An overview of the complete SMP startup process can be found in \autoref{fig:smpenable}.

\subsection{Issuing Inter-Processor Interrupts}
\label{subsec:apxipis}

To issue an IPI, the ICR is written by using the \code{ICREntry} struct (see \autoref{lst:icrentry}):

\begin{codeblock}{Issuing an INIT IPI (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_initipi_example.cpp}
\end{codeblock}

\begin{codeblock}{Issuing a SIPI (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_sipi_example.cpp}
\end{codeblock}

\subsection{Preparing Symmetric Multiprocessing Startup}
\label{subsec:apxpreparesmp}

Before executing the \textquote{Universal Startup Algorithm}, the boot code has to be relocated to physical lower memory.
The memory region used for this copy has to be identity-mapped to the virtual kernel address space, so the effective addresses do not change after enabling paging in protected mode.
To keep the required variables available to the startup code, these are located in the routines \textquote{TEXT} section and initialized during runtime.
This approach was taken from SerenityOS~\autocite[APIC.cpp]{serenity}:

\begin{codeblock}{Preparing the Boot Routine's Variables (ApicSmp.cpp)}{C++}
  \cppfile{code/ap_boot_variables.cpp}
\end{codeblock}

Now, the initialized startup routine can be copied to \code{0x8000}:

\begin{codeblock}{Relocating the Boot Routine (ApicSmp.cpp)}{C++}
  \cppfile{code/ap_boot_copy.cpp}
\end{codeblock}

The \code{boot\textunderscore{}ap} function is the entry of the startup routine, it is described further in \autoref{subsec:apxapboot}.

\subsection{Universal Startup Algorithm}
\label{subsec:apxmpusa}

The \textquote{INIT-SIPI-SIPI} sequence, or \textquote{Universal Startup Algorithm} is performed by issuing IPIs as described in \autoref{subsec:apxipis} and using the PIT as time source (see \autoref{subsec:apxapictimer}):

\begin{codeblock}{The Universal Startup Algorithm (ApicSmp.cpp)}{C++}
  \cppfile{code/ap_boot_usa.cpp}
\end{codeblock}

Boot completion is signaled in the AP's entry function:

\begin{codeblock}{Signaling AP Boot Completion (smp\textunderscore{}entry.cpp)}{C++}
  \cppfile{code/smp_entry.cpp}
\end{codeblock}

\subsection{Application Processor Boot Routine}
\label{subsec:apxapboot}

Because like the BSP, every AP is initialized in real mode, it has to be switched to protected mode first:

\begin{codeblock}{Preparing the Switch to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_real_prepare.asm}
\end{codeblock}

Then, by enabling protected mode, setting the segment registers and far-jumping to the \SI{32}{\bit} code segment, the switch is performed:

\begin{codeblock}{Switching from Real Mode to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_real.asm}
\end{codeblock}

In \SI{32}{\bit} protected mode, paging is enabled and interrupt processing is prepared by reusing the control register values and the IDT from the BSP\@:

\begin{codeblock}{Reusing Values from the BSP (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_protected_bsp.asm}
\end{codeblock}

Finally, to call the AP entry function, the AP requires its own stack. If hardware context switching is to be used, the AP additionally requires its own GDT and TSS\@:

\begin{codeblock}{Calling the Entry Function (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_protected_ap.asm}
\end{codeblock}

\subsection{Application Processor Post-Boot Routine}
\label{subsec:apxappostboot}

When the AP has booted, it initializes its own APIC, including the APIC timer and error handler:

\begin{codeblock}{Initializing the Core's Local APIC (LocalApic.cpp)}{C++}
  \cppfile{code/ap_boot_post.cpp}
\end{codeblock}

Because this added another possible recipient to internal APIC messages, the arbitration IDs are synchronized by issuing an \textquote{INIT-level-deassert IPI} using the \code{ICREntry} struct (see \autoref{lst:icrentry}):

\begin{codeblock}{Synchronizing the Arbitration IDs (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_apr_example.cpp}
\end{codeblock}

Because hhuOS' paging is not designed for multiple processors, the booted AP remains in a busy loop and does not enable its interrupts.

\cleardoublepage