christoph/interrupt-handling-using-the-x86-apic
1
Fork 0
Code Actions Activity

Sync with BSuSP

Browse Source
This commit is contained in:
Christoph Urlacher
2023-03-03 15:40:25 +01:00
parent 8bec9735e3
commit 5349bca520
27 changed files with 804 additions and 2523 deletions
Download Patch File Download Diff File Expand all files Collapse all files

619
chap/implementation.tex
View File

@ -1,9 +1,8 @@
\chapter{Implementation}
\label{ch:implementation}
This section will detail how and to what extent the APIC support was implemented.
The steps performed in this implementation are described in a generally applicable way, details on the implementation source code are found in \autoref{ch:listings}.
\autoref{subsec:hhuosintegration} deals with the modifications done to the hhuOS system to integrate the APIC implementation.
This chapter describes the implementation of the APIC interrupt controller for hhuOS and the source code produced during this thesis.
Code examples are simplified and do not necessarily match the actual implementation completely, to allow for self-contained examples.
\clearpage
@ -59,7 +58,7 @@ A summary of features that are outside the scope of this thesis:
\begin{itemize}
\item Operation with a discrete APIC or x2Apic.
\item Interrupts with logical destinations or custom priorities.
\item Interrupts with logical destinations (flat/clustered) or custom priorities.
\item Returning from APIC operation to PIC mode\footnote{
This would be theoretically possible with single-core hardware, but probably useless.}.
\item Relocation of a local APIC's MMIO memory region\footnote{
@ -70,7 +69,7 @@ A summary of features that are outside the scope of this thesis:
\item Handling of MSIs.
\end{itemize}
To be able to easily extend an APIC implementation for single-core systems to SMP systems, some things have to be taken into account:
To be able to easily extend an APIC implementation for single-core systems to SMP systems, some things are taken into account:
\begin{itemize}
\item SMP systems need to manage multiple \code{LocalApic} and \code{ApicTimer} instances.
@ -98,400 +97,394 @@ To be able to easily extend an APIC implementation for single-core systems to SM
% classes only selectively expose their interfaces (by using the \code{friend} declaration) for the
% same reason.
\clearpage
\section{Integration into HhuOS}
\label{sec:apxhhuos}
\subsection{Interrupt Handling in HhuOS}
\label{subsec:apxcurrenthhuos}
In hhuOS, external interrupts are handled in two stages (see \autoref{subsec:apxcurrenthhuos} for code examples):
\begin{enumerate}
\item After an IRQ is sent by an interrupt controller, the CPU looks up the interrupt handler address in the IDT\@.
In hhuOS, every IDT entry contains the address of the \code{dispatch} function, which is invoked with the vector number of the interrupt.
\item The \code{dispatch} function determines which interrupt handler will be called, based on the supplied vector number.
In hhuOS, interrupt handlers are implemented through the \code{InterruptHandler} interface, that provides the \code{trigger} function, which contains the interrupt handling routine.
To allow the \code{dispatch} function to find the desired interrupt handler, it has to be registered to a vector number by the OS beforehand.
This process is handled by the \code{plugin} function of the interrupt handler interface, which uses the interrupt dispatcher's \code{assign} function to register itself to the correct vector number.
HhuOS supports assigning multiple interrupt handlers to a single interrupt vector and cascading interrupts.
\end{enumerate}
To prevent the need of interacting with a specific interrupt controller implementation (e.g. \code{Pic} class) or the dispatcher, a system service (the \code{InterruptService}) is implemented to expose this functionality to other parts of the OS (it allows e.g.\ registering interrupt handlers or masking and unmasking interrupts).
Currently, hhuOS utilizes the PIT to manage the global system time, which in turn is used to trigger hhuOS' preemptive round-robin scheduler (the \code{Scheduler} class).
The PIT and other devices are initialized before the system entry point, in the \code{System::initializeSystem} function.
Devices that provide interrupts implement the \code{InterruptHandler} interface:
\begin{codeblock}{The InterruptHandler Interface (InterruptHandler.h)~\autocite{hhuos}}{C++}
\cppfile{code/interrupthandler_def.cpp}
\end{codeblock}
These interrupt handlers have to be registered to the appropriate interrupt vector:
\begin{codeblock}{Assigning an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
\cppfile{code/interruptdispatcher_assign.cpp}
\end{codeblock}
Multiple interrupt handlers can be registered to the same interrupt vector.
For every interrupt that arrives at a CPU, the first-stage interrupt handler (registered in the IDT for every vector) is called, which in turn calls the device-specific handlers:
\begin{codeblock}{Triggering an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
\cppfile{code/interruptdispatcher_dispatch.cpp}
\end{codeblock}
The PIT interrupt handler manages the system time and the scheduler:
\begin{codeblock}{The PIT Interrupt Handler (Pit.cpp)~\autocite{hhuos}}{C++}
\cppfile{code/pit_trigger.cpp}
\end{codeblock}
\subsection{Necessary Modifications to HhuOS}
\label{subsec:hhuosintegration}
To integrate the APIC implementation into hhuOS, some preexisting components of its interrupt infrastructure (see \autoref{sec:apxhhuos}) have to be modified:
\begin{enumerate}
\item The \code{InterruptService} has to forward calls to the \code{Apic} class instead of the \code{Pic} class, depending on if the APIC system is enabled -- to keep the option of running the OS on hardware that does not support the APIC (see \autoref{lst:interruptserviceafter}).
\item The \code{Pit} interrupt handler may no longer trigger the scheduler preemption if the APIC timer is enabled (see \autoref{lst:pithandlerafter}).
\item The \code{System::initializeSystem} function needs to enable the APIC system if it is available (see \autoref{lst:systemafter}).
\item The existing devices using interrupts need to use the new \code{InterruptVector} and \code{InterruptRequest} enums.
\end{enumerate}
\begin{codeblock}[label=lst:interruptserviceafter]{Using the Correct Interrupt Controller (InterruptService.cpp)}{C++}
\cppfile{code/interruptservice_after.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:pithandlerafter]{Disabling PIT Preemption (Pit.cpp)}{C++}
\cppfile{code/pit_after.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:systemafter]{Enabling the APIC System (System.cpp)}{C++}
\cppfile{code/system_after.cpp}
\end{codeblock}
\subsection{Public Interface of the APIC Subsystem}
\label{subsec:hhuospublicinterface}
To interact with the APIC subsystem from the \textquote{outside}, the \code{Apic} class is used, which exposes the necessary functionality:
\begin{codeblock}{The APIC Public Interface (Apic.h)}{C++}
\cppfile{code/apic_def.cpp}
\end{codeblock}
Enabling the APIC subsystem is done by the \code{enable} function (see \autoref{fig:apicenable}).
\section{Local APIC}
\label{sec:lapicinit}
\label{sec:apxlocalapic}
The local APIC block diagram (see \autoref{fig:localapicblock}) shows a number of registers that are required for initialization\footnote{
Registers like the \textquote{Trigger Mode Register}, \textquote{Logical Destination Register} or \textquote{Destination Format Register} remain unused in this thesis.}:
\subsection{Disabling PIC Mode}
\label{subsec:apxdisablepic}
\begin{itemize}
\item \textbf{\gls{lvt}}: Used to configure how local interrupts are handled.
\item \textbf{\gls{svr}}: Contains the software-enable flag and the spurious interrupt vector number.
\item \textbf{\gls{tpr}}: Decides the order in which interrupts are handled and a possible interrupt priority threshold, to ignore low priority interrupts.
\item \textbf{\gls{icr}}: Controls the sending of IPIs for starting up APs in SMP enabled systems.
\item \textbf{\gls{apr}}: Controls the priority in the APIC arbitration mechanism.
\end{itemize}
Setting the IMCR\footnote{
Writing the IMCR is detailed in the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.2.1]{mpspec}.}
\ using hhuOS' \code{IoPort} class:
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.9\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_manual_local_apic_blockdiagram.svg}
\end{subfigure}
\caption{The Local APIC Block Diagram~\autocite[sec.~3.11.4.1]{ia32}.}
\label{fig:localapicblock}
\end{figure}
There are multiple registers associated with the LVT, those belong to the different local interrupts the local APIC can handle.
Local interrupts this implementation is concerned about are listed below:
\begin{itemize}
\item \textit{LINT1}: The local APIC's NMI source.
\item \textit{Timer}: Periodic interrupt triggered by the APIC timer.
\item \textit{Error}: Triggered by errors in the APIC system (e.g.\ invalid vector numbers or corrupted messages in internal APIC communication).
\end{itemize}
The LINT0 interrupt is unlisted, because it is mainly important for virtual wire mode (it can be triggered by external interrupts from the PIC).
The performance and thermal monitoring interrupts also remain unused in this implementation.
Besides the local APIC's own registers, the IMCR and IA32\textunderscore{}APIC\textunderscore{}BASE MSR also requires initialization (described in \autoref{subsec:lapicenable}).
After system power-up, the local APIC is in the following state~\autocite[sec.~3.11.4.7]{ia32}:
\begin{itemize}
\item IRR, ISR and TPR are reset to \code{0x00000000}.
\item The LVT is reset to \code{0x00000000}, except for the masking bits (all local interrupts are masked on power-up).
\item The SVR is reset to \code{0x000000FF}.
\item The APIC is in xApic mode, even if x2Apic support is present.
\item Only the BSP is enabled, other APs have to be enabled by the BSP's local APIC\@.
\end{itemize}
The initialization sequence consists of these steps, all performed before interrupt handling is enabled (see \autoref{fig:apicenable}):
\begin{enumerate}
\item Enable symmetric I/O mode and set the APIC operating mode.
\item Initialize the LVT and NMI\@.
\item Initialize the SVR\@.
\item Initialize the TPR\@.
\item Initialize the APIC timer and error handler.
\item Startup the APs and initialize their respective local APICs.
\item Synchronize the APRs.
\item Enable interrupts for the BSP and APs.
\end{enumerate}
\begin{codeblock}{Disabling PIC Mode (LocalApic.cpp)}{C++}
\cppfile{code/lapic_imcr.cpp}
\end{codeblock}
\subsection{Accessing Local APIC Registers in xApic Mode}
\label{subsec:xapicregacc}
\label{subsec:apxxapicregacc}
Accessing registers in xApic mode is done via MMIO and requires a single page (\SI{4}{\kilo\byte}) of memory, mapped to the \textquote{APIC Base} address, which can be obtained by reading the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (see \autoref{fig:ia32apicbasemsr}/\autoref{tab:lapicregsmsr}) or parsing the MADT (see \autoref{tab:madt}).
The IA-32 manual specifies a caching strategy of \textquote{strong uncachable}\footnote{
See IA-32 manual for information on this caching strategy~\autocite[sec.~3.12.3]{ia32}.}
~\autocite[sec.~3.11.4.1]{ia32} for this region (see \autoref{subsec:apxxapicregacc} for the example implementation).
The xApic register access requires a single page of strong uncachable memory, but since this requires setting attributes in the \textquote{Page Attribute Table}\footnote{
See \url{https://docs.kernel.org/x86/pat.html} (visited on 12/02/2023).},
\ this implementation only uses hhuOS' \code{mapIO} function, which maps a physical address to the kernel heap, with hhuOS' \textquote{CACHE\textunderscore{}DISABLE} flag set in the kernel page table:
The address offsets (from the base address) for the local APIC registers are listed in the IA-32 manual~\autocite[sec.~3.11.4.1]{ia32} and in \autoref{tab:lapicregs}.
\begin{codeblock}{Allocating the Local APIC's MMIO Region (LocalApic.cpp)}{C++}
\cppfile{code/lapic_mmio_alloc.cpp}
\end{codeblock}
\subsection{Enabling the Local APIC}
\label{subsec:lapicenable}
A register can now be written as follows:
Because the system boots in PIC mode, \code{0x01} should be written to the IMCR~\autocite[sec.~3.6.2.1]{mpspec} to disconnect the PIC from the local APIC's LINT0 INTI (see \autoref{fig:integratedapic}, for the example implementation see \autoref{subsec:apxdisablepic}).
\begin{codeblock}[label=lst:lapicmmiowrite]{Writing a Local APIC MMIO Register (LocalApic.cpp)}{C++}
\cppfile{code/lapic_mmio_write.cpp}
\end{codeblock}
To set the operating mode, it is first determined which modes are supported by executing the \code{cpuid} instruction with \code{eax=1}: If bit 9 of the \code{edx} register is set, xApic mode is supported, if bit 21 of the \code{ecx} register is set, x2Apic mode is supported~\autocite[sec.~5.1.2]{cpuid}.
If xApic mode is supported (if the local APIC is an integrated APIC), it will be in that mode already.
The \textquote{global enable/disable} (\textquote{EN}) bit in the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (see \autoref{fig:ia32apicbasemsr}/\autoref{tab:lapicregsmsr}) allows disabling xApic mode, and thus the entire local APIC, globally\footnote{
If the system uses the discrete APIC bus, xApic mode cannot be re-enabled without a system reset~\autocite[sec.~3.11.4.3]{ia32}.}.
Enabling x2Apic mode is done by setting the \textquote{EXTD} bit (see \autoref{fig:ia32apicbasemsr}/\autoref{tab:lapicregsmsr}) of the IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\autocite[sec.~3.11.4.3]{ia32}.
To reduce the usage of manual bit-shifting and -masking, this implementation provides structures for some commonly used registers, that implement conversion operators to the register format\footnote{
The usual approach of \textquote{adding} the different required flags together to obtain the desired register content was intentionally not used for the sake of expressiveness: This approach does not highlight which flags are available and which flags are \textit{not} chosen.}:
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_apic_base_msr.svg}
\end{subfigure}
\caption{The IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\autocite[sec.~3.11.4.4]{ia32}.}
\label{fig:ia32apicbasemsr}
\end{figure}
\begin{codeblock}[label=lst:msrentry]{The MSREntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_msr_entry.cpp}
\end{codeblock}
Because QEMU does not support x2Apic mode via its TCG\footnote{
QEMU Tiny Code Generator transforms target instructions to instructions for the host CPU.},
\ this implementation only uses xApic mode.
\begin{codeblock}[label=lst:svrentry]{The SVREntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_svr_entry.cpp}
\end{codeblock}
Besides the \textquote{global enable/disable} flag, the APIC can also be enabled/disabled by using the \textquote{software enable/disable} flag in the SVR, see \autoref{subsec:lapicsoftenable}.
\begin{codeblock}[label=lst:lvtentry]{The LVTEntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_lvt_entry.cpp}
\end{codeblock}
\subsection{Handling Local Interrupts}
\label{subsec:lapiclvtinit}
\begin{codeblock}[label=lst:icrentry]{The ICREntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_icr_entry.cpp}
\end{codeblock}
To configure how the local APIC handles the different local interrupts, the LVT registers are written (see \autoref{fig:localapiclvt}).
These can be used in combination with some convenience functions:
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_lvt.svg}
\end{subfigure}
\caption{The Local Vector Table~\autocite[sec.~3.11.5.1]{ia32}.}
\label{fig:localapiclvt}
\end{figure}
\begin{codeblock}{Writing the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_msr.cpp}
\end{codeblock}
Local interrupts can be configured to use different delivery modes (excerpt)~\autocite[sec.~3.11.5.1]{ia32}:
\begin{codeblock}{Writing the SVR (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_svr.cpp}
\end{codeblock}
\begin{itemize}
\item \textit{Fixed}: Simply delivers the interrupt vector stored in the LVT register.
\item \textit{NMI}: Configures this interrupt as non-maskable, this ignores the stored vector number.
\item \textit{ExtINT}: The interrupt is treated as an external interrupt (instead of local interrupt), used e.g.\ in virtual wire mode for LINT0\@.
\end{itemize}
\begin{codeblock}{Writing the LVT (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_lvt.cpp}
\end{codeblock}
Initially, all local interrupts are initialized to PC/AT defaults: Masked, edge-triggered, active-high and with \textit{fixed} delivery mode.
Most importantly, the vector fields (bits 0:7 of each LVT register) are set to their corresponding interrupt vector (see \autoref{subsec:apxlvtinit} for an example implementation).
\begin{codeblock}{Writing the ICR (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_icr.cpp}
\end{codeblock}
After default-initializing the local interrupts, LINT1 has to be configured separately (see \autoref{tab:lapicregslvtlint}), because it is the local APIC's NMI source\footnote{
In older specifications~\autocite{mpspec}, LINT1 is defined as NMI source (see \autoref{fig:integratedapic}).
It is possible that this changed with newer architectures, so for increased compatibility this implementation configures the local APIC NMI source as reported by ACPI\@.
It is also possible that ACPI reports information on the NMI source just for future-proofing.}.
The delivery mode is set to \textquote{NMI} and the interrupt vector to \code{0x00}.
This information is also provided by ACPI in the MADT (see \autoref{tab:madtlnmi}).
Other local interrupts (APIC timer and the error interrupt) will be configured later (see \autoref{subsec:lapictimer} and \autoref{subsec:lapicerror}).
\subsection{Initializing the LVT}
\label{subsec:apxlvtinit}
\subsection{Allowing Interrupt Processing}
\label{subsec:lapicsoftenable}
The interrupt vectors are set as defined by the \code{InterruptVector} enum.
This implementation configures the LVT by using the \code{LVTEntry} struct (see \autoref{lst:lvtentry}):
To complete a minimal local APIC initialization, the \textquote{software enable/disable} flag and the spurious interrupt vector (see \autoref{fig:ia32apicsvr}/\autoref{tab:lapicregssvr}), are set.
It makes sense to choose \code{0xFF} for the spurious interrupt vector, as on P6 and Pentium processors, the lowest four bits must be set (see \autoref{fig:ia32apicsvr}).
\begin{codeblock}{Configuring the Local Error Interrupt (LocalApic.cpp)}{C++}
\cppfile{code/lapic_lvt_example.cpp}
\end{codeblock}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_apic_svr.svg}
\end{subfigure}
\caption{The Local APIC SVR Register~\autocite[sec.~3.11.9]{ia32}.}
\label{fig:ia32apicsvr}
\end{figure}
This process is repeated for each local interrupt, with variations for the NMI and timer interrupts.
Because the APIC's spurious interrupt has a dedicated interrupt vector (unlike the PIC's spurious interrupt), it can be ignored easily by registering a stub interrupt handler for the appropriate vector (see \autoref{subsec:apxsvr} for an implementation example).
\subsection{Handling the Spurious Interrupt}
\label{subsec:apxsvr}
The final step to initialize the BSP's local APIC is to allow the local APIC to receive interrupts of all priorities.
This is done by writing \code{0x00} to the TPR~\autocite[sec.~3.11.8.3]{ia32} (see \autoref{tab:lapicregstpr}).
By configuring the TPRs of different local APICs to different priorities or priority classes, distribution of external interrupts to CPUs can be controlled, but this is not used in this thesis.
This implementation sets the SVR by using the \code{SVREntry} struct (see \autoref{lst:svrentry}):
\subsection{Local Interrupt EOI}
\label{subsec:lapiceoi}
\begin{codeblock}{Setting the Spurious Interrupt Vector (LocalApic.cpp)}{C++}
\cppfile{code/lapic_svr_example.cpp}
\end{codeblock}
To notify the local APIC that a local interrupt is being handled, its EOI register (see \autoref{tab:lapicregseoi}) has to be written.
Not all local interrupts require EOIs: NMI, SMI, INIT, ExtINT, SIPI, or INIT-Deassert interrupts are excluded~\autocite[sec.~3.11.8.5]{ia32}.
Because hhuOS uses a two-stage interrupt handling approach (described in \autoref{subsec:apxcurrenthhuos}), the spurious interrupt does not receive its own interrupt handler.
Instead, it is ignored in the \code{dispatch} function, hhuOS' \textquote{first-stage} interrupt handler:
EOIs for external interrupts are also handled by the local APIC, further described in \autoref{subsec:ioapiceoi}.
\begin{codeblock}{Checking for Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
\cppfile{code/interruptdispatcher_check_spurious.cpp}
\end{codeblock}
\subsection{APIC Timer}
\label{subsec:lapictimer}
\begin{codeblock}{Ignoring Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
\cppfile{code/interruptdispatcher_ignore_spurious.cpp}
\end{codeblock}
The APIC timer is integrated into the local APIC, so it requires initialization of the latter.
Like the PIT, the APIC timer can generate periodic interrupts in a specified interval by using a counter, that is initialized with a starting value depending on the desired interval.
Because the APIC timer does not tick with a fixed frequency, but at bus frequency, the initial counter has to be determined at runtime by using an external time source.
In addition to the counter register, the APIC timer interval is influenced by a divider: Instead of decrementing the counter at every bus clock, it will be decremented every \(n\)-th bus clock, where \(n\) is the divider.
This is useful to allow for long intervals (with decreased precision), that would require a larger counter register otherwise.
\subsection{Using the APIC Timer}
\label{subsec:apxapictimer}
The APIC timer supports three different timer modes, that can be set in the timer's LVT register:
The timer frequency is determined by counting the ticks in one millisecond, using the PIT as calibration source:
\begin{enumerate}
\item \textit{Oneshot}: Trigger exactly one interrupt when the counter reaches zero.
\item \textit{Periodic}: Trigger an interrupt each time the counter reaches zero, on zero the counter reloads its initial value.
\item \textit{TSC-Deadline}: Trigger exactly one interrupt at an absolute time.
\end{enumerate}
\begin{codeblock}{Calibrating the APIC Timer (ApicTimer.cpp)}{C++}
\cppfile{code/apictimer_calibrate.cpp}
\end{codeblock}
This implementation uses the APIC timer in periodic mode, to trigger the scheduler preemption.
Initialization requires the following steps (order recommended by OSDev~\autocite{osdev}):
This calibration is performed before the interrupts get enabled, so it is not possible to use hhuOS' \code{TimeService} for the delay.
Instead, the \code{PIT::earlyDelay} function is used, which configures the PIT for mode \textquote{Interrupt on Terminal Count} on channel zero and polls the channel's output status~\autocite{pit}:
\begin{enumerate}
\item Measure the timer frequency with an external time source.
\item Configuration of the timer's divider register (see \autoref{tab:lapicregstimerdiv}).
\item Setting the timer mode to periodic (see \autoref{tab:lapicregslvtt}).
\item Initializing the counter register (see \autoref{tab:lapicregstimerinit}), depending on the measured timer frequency and the desired interval.
\end{enumerate}
\begin{codeblock}{Microsecond Delay without Interrupts (Pit.cpp)}{C++}
\cppfile{code/pit_early_delay.cpp}
\end{codeblock}
In this implementation, the APIC timer is calibrated by counting the amount of ticks in one millisecond using oneshot mode (see \autoref{subsec:apxapictimer} for an example implementation).
The measured amount of timer ticks can then be used to calculate the required counter for an arbitrary millisecond interval, although very large intervals could require the use of a larger divider, while very small intervals (in micro- or nanosecond scale) could require the opposite, to provide the necessary precision.
For this approach it is important that the timer is initialized with the same divider that was used during calibration.
Furthermore, the calibration is only performed once, even if multiple APIC timers are used.
This has to be taken into account if multiple timers with different dividers are to be used.
To use the timer, an interrupt handler has to be registered to its interrupt vector (see \autoref{subsec:apxapictimer} for an example implementation).
To handle the APIC timer interrupt on multiple cores, \(n\) \code{ApicTimer} instances are registered to the appropriate interrupt vector, where \(n\) is the number of CPUs.
Because this means, that each APIC timer interrupt on any CPU core triggers all \(n\) interrupt handlers, the handler has to determine if it belongs to the calling CPU\@.
The handler belonging to the BSP's APIC timer also triggers the scheduler preemption (instead of the PIT):
\subsection{APIC Error Interrupt}
\label{subsec:lapicerror}
\begin{codeblock}{Handling the APIC Timer Interrupt (ApicTimer.cpp)}{C++}
\cppfile{code/apictimer_trigger.cpp}
\end{codeblock}
Errors can occur e.g.\ when the local APIC receives an invalid vector number, or an APIC message gets corrupted on the system bus.
To handle these cases, the local APIC provides the local error interrupt, whose interrupt handler can read the error status from the local APIC's \textbf{\gls{esr}} (see \autoref{fig:ia32esr}/\autoref{tab:lapicregsesr}) and take appropriate action.
\subsection{Handling Local APIC Errors}
\label{subsec:apxhandlingerror}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_error_status_register.svg}
\end{subfigure}
\caption{Error Status Register~\autocite[sec.~3.11.5.3]{ia32}.}
\label{fig:ia32esr}
\end{figure}
The error interrupt handler obtains the ESR's contents by writing to it first:
The ESR is a \textquote{write/read} register: Before reading a value from the ESR, it has to be written, which updates the ESR's contents to the error status since the last write.
Writing the ESR also arms the local error interrupt, which doesn't happen automatically~\autocite[sec.~3.11.5.3]{ia32}.
\begin{codeblock}{The Local APIC Error Interrupt Handler (ApicErrorHandler.cpp)}{C++}
\cppfile{code/apicerror_trigger.cpp}
\end{codeblock}
Enabling the local error interrupt is now as simple as enabling it in the local APIC's LVT and registering an interrupt handler for the appropriate vector (see \autoref{subsec:apxhandlingerror} for an example implementation).
\clearpage
Because every CPU core can only access its own local APIC's registers, a single instance of this interrupt handler can be used for all AP's in the system.
\section{I/O APIC}
\label{sec:ioapicinit}
\label{sec:apxioapic}
To fully replace the PIC and handle external interrupts using the APIC, the I/O APIC, located in the system chipset, has to be initialized by setting its \textbf{\gls{redtbl}} registers~\autocite[sec.~9.5.8]{ich5} (see \autoref{tab:ioapicregsredtbl}).
Like the local APIC's LVT, the REDTBL allows configuration of interrupt vectors, masking bits, interrupt delivery modes, pin polarities and trigger modes (see \autoref{subsec:lapiclvtinit}).
\subsection{Accessing I/O APIC Registers}
\label{subsec:iolistings}
Additionally, for external interrupts a destination and destination mode can be specified.
This is required because the I/O APIC is able to forward external interrupts to different local APICs over the system bus (see \autoref{fig:integratedapic}).
SMP systems use this mechanism to distribute external interrupts to different CPU cores for performance benefits.
Because this implementation's focus is not on SMP, all external interrupts are default initialized to \textit{physical} destination mode\footnote{
The alternative is \textit{logical} destination mode, which allows addressing individual or clusters of local APIC's in a larger volume of processors~\autocite[sec.~3.11.6.2.2]{ia32}.}
~\autocite[sec.~3.11.6.2.1]{ia32} and are sent to the BSP for servicing, by using the BSP's local APIC ID as the destination.
The other fields are set to ISA bus defaults\footnote{
Edge-triggered, active-high.},
\ with \textit{fixed} delivery mode, masked, and the corresponding interrupt vector, as defined by the \code{InterruptVector} enum.
The I/O APIC's indirect register access requires two MMIO registers, that can be written similar to the local APIC's registers:
The I/O APIC does not have to be enabled explicitly, if the local APIC is enabled and the REDTBL is initialized correctly, external interrupts will be redirected to the local APIC and handled by the CPU\@.
\begin{codeblock}{Writing an I/O APIC MMIO Register (IoApic.h)}{C++}
\cppfile{code/ioapic_write_mmio.cpp}
\end{codeblock}
Unlike the local APIC's registers, the REDTBL registers are accessed indirectly: Two registers, the \textquote{Index} and \textquote{Data} register~\autocite[sec.~9.5.1]{ich5}, are mapped to the main memory and can be used analogous to the local APIC's registers.
The MMIO base address can be parsed from the MADT (see \autoref{tab:madtioapic}).
Writing an offset to the index register exposes an indirectly accessible I/O APIC register through the data register (see \autoref{subsec:iolistings} for an example implementation).
This indirect addressing scheme is useful, because the number of external interrupts an I/O APIC supports, and in turn the number of REDTBL registers, can vary\footnote{
Intel's consumer \textbf{\glspl{ich}} always support a fixed amount of 24 external interrupts though~\autocite[sec.~9.5.7]{ich5}.}.
Using the \textquote{Index} and \textquote{Data} registers to access the I/O APIC's indirect registers:
It is possible that one or multiple of the I/O APIC's interrupt inputs act as an NMI source.
If this is the case is reported in the MADT (see \autoref{tab:madtionmi}), so when necessary, the corresponding REDTBL entries are initialized like the local APIC's NMI source (see \autoref{subsec:lapiclvtinit}), and using these interrupt inputs for external interrupts is forbidden.
\begin{codeblock}{Writing an I/O APIC Indirect Register (IoApic.cpp)}{C++}
\cppfile{code/ioapic_write_indirect.cpp}
\end{codeblock}
To reduce the manual bit-shifting and -masking, the same approach is used as for the local APIC\@:
\begin{codeblock}[label=lst:redtblentry]{The REDTBLEntry Structure (IoApicRegisters.h)}{C++}
\cppfile{code/ioapic_redtbl_entry.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:writeredtbl]{Writing the REDTBL (IoApic.cpp)}{C++}
\cppfile{code/ioapic_write_redtbl.cpp}
\end{codeblock}
\subsection{Interrupt Overrides}
\label{subsec:ioapicpcat}
\label{subsec:apxirqoverrides}
In every PC/AT compatible system, external devices are hardwired to the PIC in the same order.
Because this is not the case for the I/O APIC, the INTIs used by each PC/AT compatible interrupt has to be determined by the OS at runtime, by using ACPI\@.
ACPI provides \textquote{Interrupt Source Override} structures~\autocite[sec.~5.2.8.3.1]{acpi1} inside the MADT (see \autoref{tab:madtirqoverride}) for each PC/AT compatible interrupt that is mapped differently to the I/O APIC than to the PIC\@.
In addition to the interrupt input mapping, these structures also allow customizing the pin polarity and trigger mode of PC/AT compatible interrupts.
This implementation represents an interrupt override using the following structure:
This information does not only apply to the REDTBL initialization, but it has to be taken into account every time an action is performed on a PC/AT compatible interrupt, like masking or unmasking: If \code{IRQ0} (PIT) should be unmasked, it has to be determined what GSI (or in other words, I/O APIC interrupt input) it belongs to.
In many systems \code{IRQ0} is mapped to \code{GSI2}, because the PC/AT compatible PICs are connected to \code{GSI0} (see \autoref{fig:integratedapic}).
Thus, to allow the PIT interrupt in those systems, the REDTBL entry belonging to \code{GSI2} instead of \code{GSI0} has to be written (see \autoref{subsec:apxirqoverrides} for an example implementation).
\begin{codeblock}{The External Interrupt Override Structure (IoApic.h)}{C++}
\cppfile{code/ioapic_irqoverride.cpp}
\end{codeblock}
\subsection{External Interrupt EOI}
\label{subsec:ioapiceoi}
During initialization, the overrides are used to set the correct interrupt vectors, polarities and trigger modes for each REDTBL entry.
The \code{REDTBLEntry} struct (see \autoref{lst:redtblentry}) is used to write the REDTBL\@:
Notifying the I/O APIC that an external interrupt has been handled differs depending on the interrupt trigger mode: Edge-triggered external interrupts are completed by writing the local APIC's EOI register (see \autoref{subsec:lapiceoi})\footnote{
Because external interrupts are forwarded to the local APIC, the local APIC is responsible for tracking them in its IRR and ISR.}.
Level-triggered interrupts are treated separately: Upon registering a level-triggered external interrupt, the I/O APIC sets an internal \textquote{Remote IRR} bit in the corresponding REDTBL entry~\autocite[sec.~9.5.8]{ich5} (see \autoref{tab:ioapicregsredtbl}).
\begin{codeblock}{Initializing the REDTBL (IoApic.cpp)}{C++}
\cppfile{code/ioapic_redtbl_example.cpp}
\end{codeblock}
There are three possible ways to signal completion of a level-triggered external interrupt to clear the remote IRR bit:
During regular operation, the overrides are used to determine the correct REDTBL entries for e.g.\ unmasking an interrupt:
\begin{enumerate}
\item Using the local APIC's EOI broadcasting feature: If EOI broadcasting is enabled, writing the local APIC's EOI register also triggers EOIs for each I/O APIC (for the appropriate interrupt), which clears the remote IRR bit.
\item Sending a directed EOI to an I/O APIC\@: I/O APICs with versions greater than \code{0x20} include an I/O EOI register~\autocite[sec.~9.5.5]{ich5}.
Writing the vector number of the handled interrupt to this register clears the remote IRR bit.
\item Simulating a directed EOI for I/O APICs with versions smaller than \code{0x20}: Temporarily masking and setting a completed interrupt as edge-triggered clears the remote IRR bit~\autocite[io\textunderscore{}apic.c]{linux}.
\end{enumerate}
\begin{codeblock}{Unmasking an IRQ (Apic.cpp)}{C++}
\cppfile{code/apic_allow.cpp}
\end{codeblock}
Because the first option is the only one supported by all APIC versions, it is used in this implementation\footnote{
Disabling EOI broadcasting is not supported by all local APICs~\autocite[sec.~3.11.8.5]{ia32}.}.
To convey this deviation between GSIs and PC/AT compatible IRQs very clearly, the public \code{Apic::allow} function accepts an \code{InterruptRequest} as an argument (that allows addressing the interrupt by name), while the internal \code{IoApic::allow} function only accepts a \code{GlobalSystemInterrupt} as argument:
At this point, after initializing the local and I/O APIC for the BSP, the APIC system is fully usable.
External interrupts now have to be enabled/disabled by writing the \textquote{masked} bit in these interrupts' REDTBL entries, interrupt handler completion is signaled by writing the local APIC's EOI register, and spurious interrupts are detected by using the local APIC's spurious interrupt vector.
\begin{codeblock}{Unmasking an IRQ Internally (IoApic.cpp)}{C++}
\cppfile{code/ioapic_allow.cpp}
\end{codeblock}
\subsection{Multiple I/O APICs}
\label{subsec:multiioapic}
Most consumer hardware, for example all IA processors~\autocite{ia32} and ICH hubs~\autocite{ich5}, only provide a single I/O APIC, although technically multiple I/O APICs are supported by the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.8]{mpspec}.
If ACPI reports multiple I/O APICs (by supplying multiple MADT I/O APIC structures, see \autoref{tab:madtioapic}), the previously described initialization has to be performed for each I/O APIC individually.
Additionally, the I/O APIC's ID, also reported by ACPI, has to be written to the corresponding I/O APIC's ID register (see \autoref{tab:ioapicregsid}), because this register is always initialized to zero~\autocite[sec.~9.5.6]{ich5}.
Using a variable number of I/O APICs requires determining the target I/O APIC for each operation that concerns a GSI, like masking or unmasking.
For this reason, ACPI provides the \textquote{GSI Base}~\autocite[sec.~5.2.8.2]{acpi1} for each available I/O APIC, the number of GSIs a single I/O APIC can handle can be determined by reading the I/O APIC's version register~\autocite[sec.~9.5.7]{ich5} (see \autoref{tab:ioapicregsver})\footnote{
This approach was previously used in this implementation, but removed for simplicity.}.
\clearpage
The internal \code{IoApic::allow} function is hidden (\code{private}) from the OS, and gets only called by the exposed \code{Apic::allow} function.
This prevents accidentally setting the wrong REDTBL entry by not taking possible interrupt overrides into account.
The same principle is applied to the other operations concerning GSIs.
\section{Symmetric Multiprocessing}
\label{sec:smpinit}
\label{sec:apxsymmetric}
Like single-core systems, SMP systems boot using only a single core, the BSP\@.
By using the APIC's capabilities to send IPIs between cores, additional APs can be put into startup state and booted for system use.
An overview of the complete SMP startup process can be found in \autoref{fig:smpenable}.
To determine the number of usable processors, the MADT is parsed (see \autoref{tab:madtlapic}).
Note, that some processors may be reported as disabled, those may not be used by the OS (see \autoref{tab:madtlapicflags}).
\subsection{Issuing Inter-Processor Interrupts}
\label{subsec:apxipis}
\subsection{Inter-Processor Interrupts}
\label{subsec:ipis}
To issue an IPI, the ICR is written by using the \code{ICREntry} struct (see \autoref{lst:icrentry}):
Issuing IPIs works by writing the local APIC's ICR (see \autoref{fig:ia32icr}/\autoref{tab:lapicregsicr}).
It allows specifying IPI type, destination (analogous to REDTBL destinations, see \autoref{sec:ioapicinit}) and vector (see \autoref{subsec:apxipis} for an example implementation).
\begin{codeblock}{Issuing an INIT IPI (LocalApic.cpp)}{C++}
\cppfile{code/lapic_initipi_example.cpp}
\end{codeblock}
Depending on the APIC architecture, two different IPIs are required: The \textit{INIT IPI} for systems using a discrete APIC, and the \textbf{\gls{sipi}} for systems using the xApic or x2Apic architectures:
\begin{codeblock}{Issuing a SIPI (LocalApic.cpp)}{C++}
\cppfile{code/lapic_sipi_example.cpp}
\end{codeblock}
\begin{itemize}
\item The INIT IPI causes an AP to reset its state and start executing at the address specified at its system reset vector.
If paired with a system warm-reset, the AP can be instructed to start executing the AP boot sequence by writing the appropriate address to the warm-reset vector~\autocite[sec.~B.4.1]{mpspec}.
\item Since the xApic architecture, the SIPI is used for AP startup: It causes the AP to start executing code in real mode, at a page specified in the IPIs interrupt vector~\autocite[sec.~B.4.2]{mpspec}.
By copying the AP boot routine to a page in lower physical memory, and sending the SIPI with the correct page number, an AP can be booted.
\end{itemize}
It is important to note that the SIPI does not receive the startup address, but the page number.
For this reason, the startup routine must be relocated starting at a page boundary.
The \code{page} argument can be determined by removing the page offset from the linear address, which is done by right-shifting the address by the offset length (\SI{12}{\bit} for \SI{4}{\kilo\byte} pages).
For the physical memory location used in this implementation (\code{0x8000}) this yields page number \code{8}.
To wait until the IPI is sent, the ICR's delivery status bit can be polled.
\subsection{Preparing Symmetric Multiprocessing Startup}
\label{subsec:apxpreparesmp}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_interrupt_command_register.svg}
\end{subfigure}
\caption{Interrupt Command Register~\autocite[sec.~3.11.6.1]{ia32}.}
\label{fig:ia32icr}
\end{figure}
Before executing the \textquote{Universal Startup Algorithm}, the boot code has to be relocated to physical lower memory.
The memory region used for this copy has to be identity-mapped to the virtual kernel address space, so the effective addresses do not change after enabling paging in protected mode.
To keep the required variables available to the startup code, these are located in the routines \textquote{TEXT} section:
\begin{codeblock}{The Boot Routine's Variables (smp\textunderscore{}boot.asm)}{nasm}
\cppfile{code/ap_boot_variables.asm}
\end{codeblock}
The variables are initialized during runtime:
\begin{codeblock}{Preparing the Boot Routine's Variables (ApicSmp.cpp)}{C++}
\cppfile{code/ap_boot_variables.cpp}
\end{codeblock}
This approach was taken from SerenityOS~\autocite[APIC.cpp]{serenity}:
Now, the initialized startup routine can be copied to \code{0x8000}:
\begin{codeblock}{Relocating the Boot Routine (ApicSmp.cpp)}{C++}
\cppfile{code/ap_boot_copy.cpp}
\end{codeblock}
The \code{boot\textunderscore{}ap} function is the entry of the startup routine, it is described further in \autoref{subsec:apxapboot}.
\subsection{Universal Startup Algorithm}
\label{subsec:apstartup}
\label{subsec:apxmpusa}
SMP initialization is performed differently on various processors.
Intel's \textquote{MultiProcessor Specification} defines a \textquote{Universal Startup Algorithm} for multiprocessor systems~\autocite[sec.~B.4]{mpspec}, which can be used to boot SMP systems with either discrete APIC, xApic or x2Apic, as it issues both, INIT IPI and SIPI\@.
The \textquote{INIT-SIPI-SIPI} sequence, or \textquote{Universal Startup Algorithm} is performed by issuing IPIs as described in \autoref{subsec:apxipis} and using the PIT as time source (see \autoref{subsec:apxapictimer}):
This algorithm has some prerequisites: It is required to copy the AP boot routine (detailed in \autoref{subsec:apboot}) to lower memory, where the APs will start their execution.
Also, the APs need allocated stack memory to call the entry function, and in case of a discrete APIC that uses the INIT IPI, the system needs to be configured for a warm-reset (by writing \code{0xAH} to the CMOS shutdown status byte, located at \code{0xF}~\autocite[sec.~B.4]{mpspec}), because the INIT IPI does not support supplying the address where AP execution should begin, unlike the SIPI\@.
The warm-reset vector (a \SI{32}{\bit} field, located at physical address \code{40:67}~\autocite[sec.~B.4]{mpspec}) needs to be set to the physical address the AP startup routine was copied to.
Additionally, the entire AP startup procedure has to be performed with all sources of interrupts disabled, which offers a small challenge, since some timings need to be taken into account\footnote{
This implementation uses the PIT's mode 0 on channel 0 for timekeeping, see \autoref{subsec:apxapictimer}.}.
\begin{codeblock}{The Universal Startup Algorithm (ApicSmp.cpp)}{C++}
\cppfile{code/ap_boot_usa.cpp}
\end{codeblock}
The usage of delays in the algorithm is quite specific, but the specification provides no further information on the importance of these timings or required precision.
The algorithm allowed for successful startup of additional APs when tested in QEMU (with and without KVM) and on certain real hardware, although for different processors or emulators (like Bochs), different timings might be required~\autocite[lapic.c]{xv6}.
Boot completion is signaled in the AP's entry function:
After preparation, the universal startup algorithm is now performed as follows, for each AP sequentially (see \autoref{subsec:apxmpusa} for an example implementation and \autoref{fig:smpenable}):
\begin{enumerate}
\item Assert and de-assert the level-triggered INIT IPI\@.
\item Delay for \SI{10}{\milli\second}.
\item Send the SIPI\@.
\item Delay for \SI{200}{\micro\second}.
\item Send the SIPI again.
\item Delay for \SI{200}{\micro\second} again.
\item Wait until the AP has signaled boot completion, then continue to the next.
\end{enumerate}
If the system uses a discrete APIC, the APs will reach the boot routine by starting execution at the location specified in the warm-reset vector, if the system uses the xApic or x2Apic architecture, the APs will reach the boot routine because its location was specified in the SIPI\@.
Signaling boot completion from the APs entry function can be done by using a global bitmap variable, where the \(n\)-th bit indicates the running state of the \(n\)-th processor.
This variable does not have to be synchronized across APs, because the startup is performed sequentially.
\begin{codeblock}{Signaling AP Boot Completion (smp\textunderscore{}entry.cpp)}{C++}
\cppfile{code/smp_entry.cpp}
\end{codeblock}
\subsection{Application Processor Boot Routine}
\label{subsec:apboot}
\label{subsec:apxapboot}
After executing the \textquote{INIT-SIPI-SIPI} sequence, the targeted AP will start executing its boot routine in real mode.
The general steps required are similar to those required when booting a single-core system, but since the BSP in SMP systems is already fully operational at this point, much can be recycled.
The AP boot routine this implementation uses can be roughly described as follows (see \autoref{subsec:apxapboot} for an example implementation):
Because like the BSP, every AP is initialized in real mode, it has to be switched to protected mode first:
\begin{enumerate}
\item Load a temporary \textbf{\gls{gdt}}, used for switching to protected mode.
\item Enable protected mode by writing \code{cr0}.
\item Far jump to switch to protected mode and reload the code-segment register, set up the other segments manually.
\item Load the \code{cr3}, \code{cr0} and \code{cr4} values used by the BSP to enable paging (in that order).
\item Load the IDT used by the BSP\@.
\item Determine the AP's APIC ID by using CPUID\@.
\item Load the GDT and \textbf{\gls{tss}} prepared for this AP\@.
\item Load the stack prepared for this AP\@.
\item Call the (C++) AP entry function.
\end{enumerate}
\begin{codeblock}{Preparing the Switch to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_real_prepare.asm}
\end{codeblock}
The APIC ID is used to determine which GDT and stack were prepared for a certain AP\@.
It is necessary for each AP to have its own GDT, because each processor needs its own TSS for hardware context switching, e.g.\ when interrupt-based system calls are used on all CPUs.
Then, by enabling protected mode, setting the segment registers and far-jumping to the \SI{32}{\bit} code segment, the switch is performed:
Because it is relocated into lower physical memory, this code has to be position independent.
For this reason, absolute physical addresses have to be used when jumping, loading the IDTR and GDTR, or referencing variables.
Also, any variables required during boot have to be available after relocation, this can be achieved by locating them inside the \textquote{TEXT} section of the routine, so they stay with the rest of the instructions when copying.
These variables have to be initialized during runtime, before the routine is copied (see \autoref{subsec:apxpreparesmp} for an example implementation).
\begin{codeblock}{Switching from Real Mode to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_real.asm}
\end{codeblock}
In \SI{32}{\bit} protected mode, paging is enabled and interrupt processing is prepared by reusing the control register values and the IDT from the BSP\@:
\begin{codeblock}{Reusing Values from the BSP (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_protected_bsp.asm}
\end{codeblock}
Finally, to call the AP entry function, the AP requires its own stack. If hardware context switching is to be used, the AP additionally requires its own GDT and TSS\@:
\begin{codeblock}{Calling the Entry Function (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_protected_ap.asm}
\end{codeblock}
\subsection{Application Processor Post-Boot Routine}
\label{subsec:apsystementry}
\label{subsec:apxappostboot}
In the entry function, called at the end of the boot routine, the AP signals boot completion as described in \autoref{subsec:apstartup} and initializes its local APIC by repeating the necessary steps from \autoref{subsec:lapiclvtinit}, \autoref{subsec:lapicsoftenable}, \autoref{subsec:lapictimer} and \autoref{subsec:lapicerror}\footnote{
MMIO memory does not have to be allocated again, as all local APICs use the same memory region in this implementation.
Also, the initial value for the APIC timer's counter can be reused, if already calibrated.}.
When the AP has booted, it initializes its own APIC, including the APIC timer and error handler:
Because multiple local APICs are present and active in the system now, the possibility arises that a certain local APIC receives multiple messages from different local APICs at a similar time.
To decide the order of handling these messages, an arbitration mechanism based on the local APIC's ID is used~\autocite[sec.~3.11.7]{ia32}.
To make sure the arbitration priority matches the local APIC's ID, the ARPs can be synchronized by issuing an INIT-level-deassert IPI\footnote{
This is not supported on Pentium 4 and Xeon processors.}
\ (see \autoref{subsec:apxappostboot} for an example implementation).
\begin{codeblock}{Initializing the Core's Local APIC (LocalApic.cpp)}{C++}
\cppfile{code/ap_boot_post.cpp}
\end{codeblock}
Because this added another possible recipient to internal APIC messages, the arbitration IDs are synchronized by issuing an \textquote{INIT-level-deassert IPI} using the \code{ICREntry} struct (see \autoref{lst:icrentry}):
\begin{codeblock}{Synchronizing the Arbitration IDs (LocalApic.cpp)}{C++}
\cppfile{code/lapic_apr_example.cpp}
\end{codeblock}
Because hhuOS' paging is not designed for multiple processors, the booted AP remains in a busy loop and does not enable its interrupts.
\cleardoublepage