\chapter{Implementation}
\label{ch:implementation}

This chapter describes the implementation of the APIC interrupt controller for hhuOS and the source code produced during this thesis.
Code examples are simplified and do not necessarily match the actual implementation completely, to allow for self-contained examples.

\clearpage

\section{Design Decisions and Scope}
\label{sec:design}

The APIC interrupt architecture is split into multiple hardware components and tasks: The (potentially multiple) local APICs, the (usually single) I/O APIC and the APIC timer (part of each local APIC).
Furthermore, the APIC system needs to interact with its memory mapped registers and the hhuOS ACPI subsystem, to gather information about the CPU topology and interrupt overrides.
Also, the OS should be able to interact with the APIC system in a simple and easy manner, without needing to know all of its individual parts.

To keep the whole system structured and simple, the implementation is split into the following main components (see \autoref{fig:implarch}):

\begin{itemize}
  \item \code{LocalApic}: Provides functionality to interact with the local APIC (masking and unmasking, register access, etc.).
  \item \code{IoApic}: Provides functionality to interact with the I/O APIC (masking and unmasking, register access, etc.)
  \item \code{ApicTimer}: Provides functionality to calibrate the APIC timer and handle its interrupts.
  \item \code{Apic}: Condenses all the functionality above and exposes it to other parts of the OS\@.
\end{itemize}

\begin{figure}[h]
  \centering
  \begin{subfigure}[b]{0.5\textwidth}
    \includesvg[width=1.0\linewidth]{img/Architecture.svg}
  \end{subfigure}
  \caption{Caller Hierarchy of the Main Components.}
  \label{fig:implarch}
\end{figure}

This implementation is targeted to support systems with a single I/O APIC\footnote{
  Operation with more than one I/O APIC is described further in the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.8]{mpspec}.},
\ because consumer hardware typically only uses a single one, and so does QEMU emulation.
General information on implementing multiple I/O APIC support can be found in \autoref{subsec:multiioapic}.

With the introduction of ACPI's GSIs to the OS, new types are introduced to clearly differentiate different representations of interrupts and prevent unintentional conversions:

\begin{itemize}
  \item \code{GlobalSystemInterrupt}: ACPI's interrupt abstraction, detached from the hardware interrupt lines.
  \item \code{InterruptRequest}: Represents an \textbf{\gls{isa}} IRQ, allowing the OS to address interrupts by the name of the device that triggers it.
        When the APIC is used, interrupt overrides map IRQs to GSIs.
  \item \code{InterruptVector}: Represents an interrupt's vector number, as used by the \code{InterruptDispatcher}.
        The dispatcher maps interrupt vectors to interrupt handlers.
\end{itemize}

Both BIOS and UEFI are supported by hhuOS, but the hhuOS ACPI subsystem is currently only available with BIOS\footnote{
  State from 11/02/2023.},
\ so when hhuOS is booted using UEFI, APIC support cannot be enabled.
Also, the APIC can handle MSIs, but they are not included in this implementation, as hhuOS currently does not utilize them.

SMP systems are partially supported: The APs are initialized, but only to a busy-looping state, as hhuOS currently is a single-core OS and lacks some required infrastructure.
All interrupts are handled using the BSP\@.

A summary of features that are outside the scope of this thesis:

\begin{itemize}
  \item Operation with a discrete APIC or x2Apic.
  \item Interrupts with logical destinations (flat/clustered) or custom priorities.
  \item Returning from APIC operation to PIC mode\footnote{
        This would be theoretically possible with single-core hardware, but probably useless.}.
  \item Relocation of a local APIC's MMIO memory region\footnote{
        Relocation is possible by writing a new physical APIC base address to the IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\autocite[sec.~3.11.4.5]{ia32}.}.
  \item Distributing external interrupts to different APs in SMP enabled systems.
  \item Usage of the system's performance counter or monitoring interrupts.
  \item Meaningful treatment of APIC errors.
  \item Handling of MSIs.
\end{itemize}

To be able to easily extend an APIC implementation for single-core systems to SMP systems, some things are taken into account:

\begin{itemize}
  \item SMP systems need to manage multiple \code{LocalApic} and \code{ApicTimer} instances.
        This is handled by the \code{Apic} class.
  \item Initialization of the different components can no longer happen at the same \textquote{location}: The local APICs and APIC timers of additional APs have to be initialized by the APs themselves, because the BSP can not access an AP's registers.
  \item APs are only allowed to access instances of APIC classes that belong to them.
  \item Interrupt handlers that get called on multiple APs may need to take the current processor into account (for example the APIC timer interrupt handler).
  \item Register access has to be synchronized, if it is performed in multiple operations on the same address space.
\end{itemize}

% \section{Code Style}
% \label{sec:codestyle}
%
% Individual state of local and I/O APICs is managed through instances of their respective classes.
% Because each CPU core can only access the local APIC contained in itself, this can create a
% misconception: It is not possible to (e.g.) allow an interrupt in a certain local APIC by calling a
% function on a certain \code{LocalApic} instance. This is communicated through code by declaring a
% range of functions as \code{static}. It is also in direct contrast to the \code{IoApic} class: I/O
% APICs can be addressed by instances, because they are not part of the CPU core: Each core can
% always access all I/O APICs (if there are multiple).
%
% Error checking is done to a small extent in this implementation: Publicly exposed functions (from
% the \code{Apic} class) do check for invalid invocations, but the internally used classes do not
% protect their invariants, because they are not used directly by other parts of the OS. These
% classes only selectively expose their interfaces (by using the \code{friend} declaration) for the
% same reason.

\section{Integration into HhuOS}
\label{sec:apxhhuos}

\subsection{Interrupt Handling in HhuOS}
\label{subsec:apxcurrenthhuos}

In hhuOS, external interrupts are handled in two stages:

\begin{enumerate}
  \item After an IRQ is sent by an interrupt controller, the CPU looks up the interrupt handler address in the IDT\@.
        In hhuOS, every IDT entry contains the address of the \code{dispatch} function, which is invoked with the vector number of the interrupt.
  \item The \code{dispatch} function determines which interrupt handler will be called, based on the supplied vector number.
        In hhuOS, interrupt handlers are implemented through the \code{InterruptHandler} interface, that provides the \code{trigger} function, which contains the interrupt handling routine.
        To allow the \code{dispatch} function to find the desired interrupt handler, it has to be registered to a vector number by the OS beforehand.
        This process is handled by the \code{plugin} function of the interrupt handler interface, which uses the interrupt dispatcher's \code{assign} function to register itself to the correct vector number.
        HhuOS supports assigning multiple interrupt handlers to a single interrupt vector and cascading interrupts.
\end{enumerate}

To prevent the need of interacting with a specific interrupt controller implementation (e.g. \code{Pic} class) or the dispatcher, a system service (the \code{InterruptService}) is implemented to expose this functionality to other parts of the OS (it allows e.g.\ registering interrupt handlers or masking and unmasking interrupts).

Currently, hhuOS utilizes the PIT to manage the global system time, which in turn is used to trigger hhuOS' preemptive round-robin scheduler (the \code{Scheduler} class).
The PIT and other devices are initialized before the system entry point, in the \code{System::initializeSystem} function.

Devices that provide interrupts implement the \code{InterruptHandler} interface:

\begin{codeblock}{The InterruptHandler Interface (InterruptHandler.h)~\autocite{hhuos}}{C++}
  \cppfile{code/interrupthandler_def.cpp}
\end{codeblock}

These interrupt handlers have to be registered to the appropriate interrupt vector:

\begin{codeblock}{Assigning an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
  \cppfile{code/interruptdispatcher_assign.cpp}
\end{codeblock}

Multiple interrupt handlers can be registered to the same interrupt vector.
For every interrupt that arrives at a CPU, the first-stage interrupt handler (registered in the IDT for every vector) is called, which in turn calls the device-specific handlers:

\begin{codeblock}{Triggering an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
  \cppfile{code/interruptdispatcher_dispatch.cpp}
\end{codeblock}

The PIT interrupt handler manages the system time and the scheduler:

\begin{codeblock}{The PIT Interrupt Handler (Pit.cpp)~\autocite{hhuos}}{C++}
  \cppfile{code/pit_trigger.cpp}
\end{codeblock}

\subsection{Necessary Modifications to HhuOS}
\label{subsec:hhuosintegration}

To integrate the APIC implementation into hhuOS, some preexisting components of its interrupt infrastructure (see \autoref{sec:apxhhuos}) have to be modified:

\begin{enumerate}
  \item The \code{InterruptService} has to forward calls to the \code{Apic} class instead of the \code{Pic} class, depending on if the APIC system is enabled -- to keep the option of running the OS on hardware that does not support the APIC (see \autoref{lst:interruptserviceafter}).
  \item The \code{Pit} interrupt handler may no longer trigger the scheduler preemption if the APIC timer is enabled (see \autoref{lst:pithandlerafter}).
  \item The \code{System::initializeSystem} function needs to enable the APIC system if it is available (see \autoref{lst:systemafter}).
  \item The existing devices using interrupts need to use the new \code{InterruptVector} and \code{InterruptRequest} enums.
\end{enumerate}

\begin{codeblock}[label=lst:interruptserviceafter]{Using the Correct Interrupt Controller (InterruptService.cpp)}{C++}
  \cppfile{code/interruptservice_after.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:pithandlerafter]{Disabling PIT Preemption (Pit.cpp)}{C++}
  \cppfile{code/pit_after.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:systemafter]{Enabling the APIC System (System.cpp)}{C++}
  \cppfile{code/system_after.cpp}
\end{codeblock}

\subsection{Public Interface of the APIC Subsystem}
\label{subsec:hhuospublicinterface}

To interact with the APIC subsystem from the \textquote{outside}, the \code{Apic} class is used, which exposes the necessary functionality:

\begin{codeblock}{The APIC Public Interface (Apic.h)}{C++}
  \cppfile{code/apic_def.cpp}
\end{codeblock}

Enabling the APIC subsystem is done by the \code{enable} function (see \autoref{fig:apicenable}).

\section{Local APIC}
\label{sec:apxlocalapic}

\subsection{Disabling PIC Mode}
\label{subsec:apxdisablepic}

Setting the IMCR\footnote{
  Writing the IMCR is detailed in the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.2.1]{mpspec}.}
\ using hhuOS' \code{IoPort} class:

\begin{codeblock}{Disabling PIC Mode (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_imcr.cpp}
\end{codeblock}

\subsection{Accessing Local APIC Registers in xApic Mode}
\label{subsec:apxxapicregacc}

The xApic register access requires a single page of strong uncachable memory, but since this requires setting attributes in the \textquote{Page Attribute Table}\footnote{
  See \url{https://docs.kernel.org/x86/pat.html} (visited on 12/02/2023).},
\ this implementation only uses hhuOS' \code{mapIO} function, which maps a physical address to the kernel heap, with hhuOS' \textquote{CACHE\textunderscore{}DISABLE} flag set in the kernel page table:

\begin{codeblock}{Allocating the Local APIC's MMIO Region (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_mmio_alloc.cpp}
\end{codeblock}

A register can now be written as follows:

\begin{codeblock}[label=lst:lapicmmiowrite]{Writing a Local APIC MMIO Register (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_mmio_write.cpp}
\end{codeblock}

To reduce the usage of manual bit-shifting and -masking, this implementation provides structures for some commonly used registers, that implement conversion operators to the register format\footnote{
  The usual approach of \textquote{adding} the different required flags together to obtain the desired register content was intentionally not used for the sake of expressiveness: This approach does not highlight which flags are available and which flags are \textit{not} chosen.}:

\begin{codeblock}[label=lst:msrentry]{The MSREntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_msr_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:svrentry]{The SVREntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_svr_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:lvtentry]{The LVTEntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_lvt_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:icrentry]{The ICREntry Structure (LocalApicRegisters.h)}{C++}
  \cppfile{code/lapic_icr_entry.cpp}
\end{codeblock}

These can be used in combination with some convenience functions:

\begin{codeblock}{Writing the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_msr.cpp}
\end{codeblock}

\begin{codeblock}{Writing the SVR (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_svr.cpp}
\end{codeblock}

\begin{codeblock}{Writing the LVT (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_lvt.cpp}
\end{codeblock}

\begin{codeblock}{Writing the ICR (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_write_icr.cpp}
\end{codeblock}

\subsection{Initializing the LVT}
\label{subsec:apxlvtinit}

The interrupt vectors are set as defined by the \code{InterruptVector} enum.
This implementation configures the LVT by using the \code{LVTEntry} struct (see \autoref{lst:lvtentry}):

\begin{codeblock}{Configuring the Local Error Interrupt (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_lvt_example.cpp}
\end{codeblock}

This process is repeated for each local interrupt, with variations for the NMI and timer interrupts.

\subsection{Handling the Spurious Interrupt}
\label{subsec:apxsvr}

This implementation sets the SVR by using the \code{SVREntry} struct (see \autoref{lst:svrentry}):

\begin{codeblock}{Setting the Spurious Interrupt Vector (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_svr_example.cpp}
\end{codeblock}

Because hhuOS uses a two-stage interrupt handling approach (described in \autoref{subsec:apxcurrenthhuos}), the spurious interrupt does not receive its own interrupt handler.
Instead, it is ignored in the \code{dispatch} function, hhuOS' \textquote{first-stage} interrupt handler:

\begin{codeblock}{Checking for Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
  \cppfile{code/interruptdispatcher_check_spurious.cpp}
\end{codeblock}

\begin{codeblock}{Ignoring Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
  \cppfile{code/interruptdispatcher_ignore_spurious.cpp}
\end{codeblock}

\subsection{Using the APIC Timer}
\label{subsec:apxapictimer}

The timer frequency is determined by counting the ticks in one millisecond, using the PIT as calibration source:

\begin{codeblock}{Calibrating the APIC Timer (ApicTimer.cpp)}{C++}
  \cppfile{code/apictimer_calibrate.cpp}
\end{codeblock}

This calibration is performed before the interrupts get enabled, so it is not possible to use hhuOS' \code{TimeService} for the delay.
Instead, the \code{PIT::earlyDelay} function is used, which configures the PIT for mode \textquote{Interrupt on Terminal Count} on channel zero and polls the channel's output status~\autocite{pit}:

\begin{codeblock}{Microsecond Delay without Interrupts (Pit.cpp)}{C++}
  \cppfile{code/pit_early_delay.cpp}
\end{codeblock}

Furthermore, the calibration is only performed once, even if multiple APIC timers are used.
This has to be taken into account if multiple timers with different dividers are to be used.

To handle the APIC timer interrupt on multiple cores, \(n\) \code{ApicTimer} instances are registered to the appropriate interrupt vector, where \(n\) is the number of CPUs.
Because this means, that each APIC timer interrupt on any CPU core triggers all \(n\) interrupt handlers, the handler has to determine if it belongs to the calling CPU\@.
The handler belonging to the BSP's APIC timer also triggers the scheduler preemption (instead of the PIT):

\begin{codeblock}{Handling the APIC Timer Interrupt (ApicTimer.cpp)}{C++}
  \cppfile{code/apictimer_trigger.cpp}
\end{codeblock}

\subsection{Handling Local APIC Errors}
\label{subsec:apxhandlingerror}

The error interrupt handler obtains the ESR's contents by writing to it first:

\begin{codeblock}{The Local APIC Error Interrupt Handler (ApicErrorHandler.cpp)}{C++}
  \cppfile{code/apicerror_trigger.cpp}
\end{codeblock}

Because every CPU core can only access its own local APIC's registers, a single instance of this interrupt handler can be used for all AP's in the system.

\section{I/O APIC}
\label{sec:apxioapic}

\subsection{Accessing I/O APIC Registers}
\label{subsec:iolistings}

The I/O APIC's indirect register access requires two MMIO registers, that can be written similar to the local APIC's registers:

\begin{codeblock}{Writing an I/O APIC MMIO Register (IoApic.h)}{C++}
  \cppfile{code/ioapic_write_mmio.cpp}
\end{codeblock}

Using the \textquote{Index} and \textquote{Data} registers to access the I/O APIC's indirect registers:

\begin{codeblock}{Writing an I/O APIC Indirect Register (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_write_indirect.cpp}
\end{codeblock}

To reduce the manual bit-shifting and -masking, the same approach is used as for the local APIC\@:

\begin{codeblock}[label=lst:redtblentry]{The REDTBLEntry Structure (IoApicRegisters.h)}{C++}
  \cppfile{code/ioapic_redtbl_entry.cpp}
\end{codeblock}

\begin{codeblock}[label=lst:writeredtbl]{Writing the REDTBL (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_write_redtbl.cpp}
\end{codeblock}

\subsection{Interrupt Overrides}
\label{subsec:apxirqoverrides}

This implementation represents an interrupt override using the following structure:

\begin{codeblock}{The External Interrupt Override Structure (IoApic.h)}{C++}
  \cppfile{code/ioapic_irqoverride.cpp}
\end{codeblock}

During initialization, the overrides are used to set the correct interrupt vectors, polarities and trigger modes for each REDTBL entry.
The \code{REDTBLEntry} struct (see \autoref{lst:redtblentry}) is used to write the REDTBL\@:

\begin{codeblock}{Initializing the REDTBL (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_redtbl_example.cpp}
\end{codeblock}

During regular operation, the overrides are used to determine the correct REDTBL entries for e.g.\ unmasking an interrupt:

\begin{codeblock}{Unmasking an IRQ (Apic.cpp)}{C++}
  \cppfile{code/apic_allow.cpp}
\end{codeblock}

To convey this deviation between GSIs and PC/AT compatible IRQs very clearly, the public \code{Apic::allow} function accepts an \code{InterruptRequest} as an argument (that allows addressing the interrupt by name), while the internal \code{IoApic::allow} function only accepts a \code{GlobalSystemInterrupt} as argument:

\begin{codeblock}{Unmasking an IRQ Internally (IoApic.cpp)}{C++}
  \cppfile{code/ioapic_allow.cpp}
\end{codeblock}

The internal \code{IoApic::allow} function is hidden (\code{private}) from the OS, and gets only called by the exposed \code{Apic::allow} function.
This prevents accidentally setting the wrong REDTBL entry by not taking possible interrupt overrides into account.
The same principle is applied to the other operations concerning GSIs.

\section{Symmetric Multiprocessing}
\label{sec:apxsymmetric}

An overview of the complete SMP startup process can be found in \autoref{fig:smpenable}.

\subsection{Issuing Inter-Processor Interrupts}
\label{subsec:apxipis}

To issue an IPI, the ICR is written by using the \code{ICREntry} struct (see \autoref{lst:icrentry}):

\begin{codeblock}{Issuing an INIT IPI (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_initipi_example.cpp}
\end{codeblock}

\begin{codeblock}{Issuing a SIPI (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_sipi_example.cpp}
\end{codeblock}

It is important to note that the SIPI does not receive the startup address, but the page number.
For this reason, the startup routine must be relocated starting at a page boundary.
The \code{page} argument can be determined by removing the page offset from the linear address, which is done by right-shifting the address by the offset length (\SI{12}{\bit} for \SI{4}{\kilo\byte} pages).
For the physical memory location used in this implementation (\code{0x8000}) this yields page number \code{8}.

\subsection{Preparing Symmetric Multiprocessing Startup}
\label{subsec:apxpreparesmp}

Before executing the \textquote{Universal Startup Algorithm}, the boot code has to be relocated to physical lower memory.
The memory region used for this copy has to be identity-mapped to the virtual kernel address space, so the effective addresses do not change after enabling paging in protected mode.
To keep the required variables available to the startup code, these are located in the routines \textquote{TEXT} section:

\begin{codeblock}{The Boot Routine's Variables (smp\textunderscore{}boot.asm)}{nasm}
  \cppfile{code/ap_boot_variables.asm}
\end{codeblock}

The variables are initialized during runtime:

\begin{codeblock}{Preparing the Boot Routine's Variables (ApicSmp.cpp)}{C++}
  \cppfile{code/ap_boot_variables.cpp}
\end{codeblock}

This approach was taken from SerenityOS~\autocite[APIC.cpp]{serenity}:

Now, the initialized startup routine can be copied to \code{0x8000}:

\begin{codeblock}{Relocating the Boot Routine (ApicSmp.cpp)}{C++}
  \cppfile{code/ap_boot_copy.cpp}
\end{codeblock}

The \code{boot\textunderscore{}ap} function is the entry of the startup routine, it is described further in \autoref{subsec:apxapboot}.

\subsection{Universal Startup Algorithm}
\label{subsec:apxmpusa}

The \textquote{INIT-SIPI-SIPI} sequence, or \textquote{Universal Startup Algorithm} is performed by issuing IPIs as described in \autoref{subsec:apxipis} and using the PIT as time source (see \autoref{subsec:apxapictimer}):

\begin{codeblock}{The Universal Startup Algorithm (ApicSmp.cpp)}{C++}
  \cppfile{code/ap_boot_usa.cpp}
\end{codeblock}

Boot completion is signaled in the AP's entry function:

\begin{codeblock}{Signaling AP Boot Completion (smp\textunderscore{}entry.cpp)}{C++}
  \cppfile{code/smp_entry.cpp}
\end{codeblock}

\subsection{Application Processor Boot Routine}
\label{subsec:apxapboot}

Because like the BSP, every AP is initialized in real mode, it has to be switched to protected mode first:

\begin{codeblock}{Preparing the Switch to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_real_prepare.asm}
\end{codeblock}

Then, by enabling protected mode, setting the segment registers and far-jumping to the \SI{32}{\bit} code segment, the switch is performed:

\begin{codeblock}{Switching from Real Mode to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_real.asm}
\end{codeblock}

In \SI{32}{\bit} protected mode, paging is enabled and interrupt processing is prepared by reusing the control register values and the IDT from the BSP\@:

\begin{codeblock}{Reusing Values from the BSP (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_protected_bsp.asm}
\end{codeblock}

Finally, to call the AP entry function, the AP requires its own stack. If hardware context switching is to be used, the AP additionally requires its own GDT and TSS\@:

\begin{codeblock}{Calling the Entry Function (smp\textunderscore{}boot.asm)}{nasm}
  \nasmfile{code/ap_boot_protected_ap.asm}
\end{codeblock}

\subsection{Application Processor Post-Boot Routine}
\label{subsec:apxappostboot}

When the AP has booted, it initializes its own APIC, including the APIC timer and error handler:

\begin{codeblock}{Initializing the Core's Local APIC (LocalApic.cpp)}{C++}
  \cppfile{code/ap_boot_post.cpp}
\end{codeblock}

Because this added another possible recipient to internal APIC messages, the arbitration IDs are synchronized by issuing an \textquote{INIT-level-deassert IPI} using the \code{ICREntry} struct (see \autoref{lst:icrentry}):

\begin{codeblock}{Synchronizing the Arbitration IDs (LocalApic.cpp)}{C++}
  \cppfile{code/lapic_apr_example.cpp}
\end{codeblock}

Because hhuOS' paging is not designed for multiple processors, the booted AP remains in a busy loop and does not enable its interrupts.

\cleardoublepage