490 lines
24 KiB
TeX
490 lines
24 KiB
TeX
\chapter{Implementation}
|
|
\label{ch:implementation}
|
|
|
|
This chapter describes the implementation of the APIC interrupt controller for hhuOS and the source code produced during this thesis.
|
|
Code examples are simplified and do not necessarily match the actual implementation completely, to allow for self-contained examples.
|
|
|
|
\clearpage
|
|
|
|
\section{Design Decisions and Scope}
|
|
\label{sec:design}
|
|
|
|
The APIC interrupt architecture is split into multiple hardware components and tasks: The (potentially multiple) local APICs, the (usually single) I/O APIC and the APIC timer (part of each local APIC).
|
|
Furthermore, the APIC system needs to interact with its memory mapped registers and the hhuOS ACPI subsystem, to gather information about the CPU topology and interrupt overrides.
|
|
Also, the OS should be able to interact with the APIC system in a simple and easy manner, without needing to know all of its individual parts.
|
|
|
|
To keep the whole system structured and simple, the implementation is split into the following main components (see \autoref{fig:implarch}):
|
|
|
|
\begin{itemize}
|
|
\item \code{LocalApic}: Provides functionality to interact with the local APIC (masking and unmasking, register access, etc.).
|
|
\item \code{IoApic}: Provides functionality to interact with the I/O APIC (masking and unmasking, register access, etc.)
|
|
\item \code{ApicTimer}: Provides functionality to calibrate the APIC timer and handle its interrupts.
|
|
\item \code{Apic}: Condenses all the functionality above and exposes it to other parts of the OS\@.
|
|
\end{itemize}
|
|
|
|
\begin{figure}[h]
|
|
\centering
|
|
\begin{subfigure}[b]{0.5\textwidth}
|
|
\includesvg[width=1.0\linewidth]{img/Architecture.svg}
|
|
\end{subfigure}
|
|
\caption{Caller Hierarchy of the Main Components.}
|
|
\label{fig:implarch}
|
|
\end{figure}
|
|
|
|
This implementation is targeted to support systems with a single I/O APIC\footnote{
|
|
Operation with more than one I/O APIC is described further in the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.8]{mpspec}.},
|
|
\ because consumer hardware typically only uses a single one, and so does QEMU emulation.
|
|
General information on implementing multiple I/O APIC support can be found in \autoref{subsec:multiioapic}.
|
|
|
|
With the introduction of ACPI's GSIs to the OS, new types are introduced to clearly differentiate different representations of interrupts and prevent unintentional conversions:
|
|
|
|
\begin{itemize}
|
|
\item \code{GlobalSystemInterrupt}: ACPI's interrupt abstraction, detached from the hardware interrupt lines.
|
|
\item \code{InterruptRequest}: Represents an \textbf{\gls{isa}} IRQ, allowing the OS to address interrupts by the name of the device that triggers it.
|
|
When the APIC is used, interrupt overrides map IRQs to GSIs.
|
|
\item \code{InterruptVector}: Represents an interrupt's vector number, as used by the \code{InterruptDispatcher}.
|
|
The dispatcher maps interrupt vectors to interrupt handlers.
|
|
\end{itemize}
|
|
|
|
Both BIOS and UEFI are supported by hhuOS, but the hhuOS ACPI subsystem is currently only available with BIOS\footnote{
|
|
State from 11/02/2023.},
|
|
\ so when hhuOS is booted using UEFI, APIC support cannot be enabled.
|
|
Also, the APIC can handle MSIs, but they are not included in this implementation, as hhuOS currently does not utilize them.
|
|
|
|
SMP systems are partially supported: The APs are initialized, but only to a busy-looping state, as hhuOS currently is a single-core OS and lacks some required infrastructure.
|
|
All interrupts are handled using the BSP\@.
|
|
|
|
A summary of features that are outside the scope of this thesis:
|
|
|
|
\begin{itemize}
|
|
\item Operation with a discrete APIC or x2Apic.
|
|
\item Interrupts with logical destinations (flat/clustered) or custom priorities.
|
|
\item Returning from APIC operation to PIC mode\footnote{
|
|
This would be theoretically possible with single-core hardware, but probably useless.}.
|
|
\item Relocation of a local APIC's MMIO memory region\footnote{
|
|
Relocation is possible by writing a new physical APIC base address to the IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\autocite[sec.~3.11.4.5]{ia32}.}.
|
|
\item Distributing external interrupts to different APs in SMP enabled systems.
|
|
\item Usage of the system's performance counter or monitoring interrupts.
|
|
\item Meaningful treatment of APIC errors.
|
|
\item Handling of MSIs.
|
|
\end{itemize}
|
|
|
|
To be able to easily extend an APIC implementation for single-core systems to SMP systems, some things are taken into account:
|
|
|
|
\begin{itemize}
|
|
\item SMP systems need to manage multiple \code{LocalApic} and \code{ApicTimer} instances.
|
|
This is handled by the \code{Apic} class.
|
|
\item Initialization of the different components can no longer happen at the same \textquote{location}: The local APICs and APIC timers of additional APs have to be initialized by the APs themselves, because the BSP can not access an AP's registers.
|
|
\item APs are only allowed to access instances of APIC classes that belong to them.
|
|
\item Interrupt handlers that get called on multiple APs may need to take the current processor into account (for example the APIC timer interrupt handler).
|
|
\item Register access has to be synchronized, if it is performed in multiple operations on the same address space.
|
|
\end{itemize}
|
|
|
|
% \section{Code Style}
|
|
% \label{sec:codestyle}
|
|
%
|
|
% Individual state of local and I/O APICs is managed through instances of their respective classes.
|
|
% Because each CPU core can only access the local APIC contained in itself, this can create a
|
|
% misconception: It is not possible to (e.g.) allow an interrupt in a certain local APIC by calling a
|
|
% function on a certain \code{LocalApic} instance. This is communicated through code by declaring a
|
|
% range of functions as \code{static}. It is also in direct contrast to the \code{IoApic} class: I/O
|
|
% APICs can be addressed by instances, because they are not part of the CPU core: Each core can
|
|
% always access all I/O APICs (if there are multiple).
|
|
%
|
|
% Error checking is done to a small extent in this implementation: Publicly exposed functions (from
|
|
% the \code{Apic} class) do check for invalid invocations, but the internally used classes do not
|
|
% protect their invariants, because they are not used directly by other parts of the OS. These
|
|
% classes only selectively expose their interfaces (by using the \code{friend} declaration) for the
|
|
% same reason.
|
|
|
|
\section{Integration into HhuOS}
|
|
\label{sec:apxhhuos}
|
|
|
|
\subsection{Interrupt Handling in HhuOS}
|
|
\label{subsec:apxcurrenthhuos}
|
|
|
|
In hhuOS, external interrupts are handled in two stages:
|
|
|
|
\begin{enumerate}
|
|
\item After an IRQ is sent by an interrupt controller, the CPU looks up the interrupt handler address in the IDT\@.
|
|
In hhuOS, every IDT entry contains the address of the \code{dispatch} function, which is invoked with the vector number of the interrupt.
|
|
\item The \code{dispatch} function determines which interrupt handler will be called, based on the supplied vector number.
|
|
In hhuOS, interrupt handlers are implemented through the \code{InterruptHandler} interface, that provides the \code{trigger} function, which contains the interrupt handling routine.
|
|
To allow the \code{dispatch} function to find the desired interrupt handler, it has to be registered to a vector number by the OS beforehand.
|
|
This process is handled by the \code{plugin} function of the interrupt handler interface, which uses the interrupt dispatcher's \code{assign} function to register itself to the correct vector number.
|
|
HhuOS supports assigning multiple interrupt handlers to a single interrupt vector and cascading interrupts.
|
|
\end{enumerate}
|
|
|
|
To prevent the need of interacting with a specific interrupt controller implementation (e.g. \code{Pic} class) or the dispatcher, a system service (the \code{InterruptService}) is implemented to expose this functionality to other parts of the OS (it allows e.g.\ registering interrupt handlers or masking and unmasking interrupts).
|
|
|
|
Currently, hhuOS utilizes the PIT to manage the global system time, which in turn is used to trigger hhuOS' preemptive round-robin scheduler (the \code{Scheduler} class).
|
|
The PIT and other devices are initialized before the system entry point, in the \code{System::initializeSystem} function.
|
|
|
|
Devices that provide interrupts implement the \code{InterruptHandler} interface:
|
|
|
|
\begin{codeblock}{The InterruptHandler Interface (InterruptHandler.h)~\autocite{hhuos}}{C++}
|
|
\cppfile{code/interrupthandler_def.cpp}
|
|
\end{codeblock}
|
|
|
|
These interrupt handlers have to be registered to the appropriate interrupt vector:
|
|
|
|
\begin{codeblock}{Assigning an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
|
|
\cppfile{code/interruptdispatcher_assign.cpp}
|
|
\end{codeblock}
|
|
|
|
Multiple interrupt handlers can be registered to the same interrupt vector.
|
|
For every interrupt that arrives at a CPU, the first-stage interrupt handler (registered in the IDT for every vector) is called, which in turn calls the device-specific handlers:
|
|
|
|
\begin{codeblock}{Triggering an Interrupt Handler (InterruptDispatcher.cpp)~\autocite{hhuos}}{C++}
|
|
\cppfile{code/interruptdispatcher_dispatch.cpp}
|
|
\end{codeblock}
|
|
|
|
The PIT interrupt handler manages the system time and the scheduler:
|
|
|
|
\begin{codeblock}{The PIT Interrupt Handler (Pit.cpp)~\autocite{hhuos}}{C++}
|
|
\cppfile{code/pit_trigger.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Necessary Modifications to HhuOS}
|
|
\label{subsec:hhuosintegration}
|
|
|
|
To integrate the APIC implementation into hhuOS, some preexisting components of its interrupt infrastructure (see \autoref{sec:apxhhuos}) have to be modified:
|
|
|
|
\begin{enumerate}
|
|
\item The \code{InterruptService} has to forward calls to the \code{Apic} class instead of the \code{Pic} class, depending on if the APIC system is enabled -- to keep the option of running the OS on hardware that does not support the APIC (see \autoref{lst:interruptserviceafter}).
|
|
\item The \code{Pit} interrupt handler may no longer trigger the scheduler preemption if the APIC timer is enabled (see \autoref{lst:pithandlerafter}).
|
|
\item The \code{System::initializeSystem} function needs to enable the APIC system if it is available (see \autoref{lst:systemafter}).
|
|
\item The existing devices using interrupts need to use the new \code{InterruptVector} and \code{InterruptRequest} enums.
|
|
\end{enumerate}
|
|
|
|
\begin{codeblock}[label=lst:interruptserviceafter]{Using the Correct Interrupt Controller (InterruptService.cpp)}{C++}
|
|
\cppfile{code/interruptservice_after.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:pithandlerafter]{Disabling PIT Preemption (Pit.cpp)}{C++}
|
|
\cppfile{code/pit_after.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:systemafter]{Enabling the APIC System (System.cpp)}{C++}
|
|
\cppfile{code/system_after.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Public Interface of the APIC Subsystem}
|
|
\label{subsec:hhuospublicinterface}
|
|
|
|
To interact with the APIC subsystem from the \textquote{outside}, the \code{Apic} class is used, which exposes the necessary functionality:
|
|
|
|
\begin{codeblock}{The APIC Public Interface (Apic.h)}{C++}
|
|
\cppfile{code/apic_def.cpp}
|
|
\end{codeblock}
|
|
|
|
Enabling the APIC subsystem is done by the \code{enable} function (see \autoref{fig:apicenable}).
|
|
|
|
\section{Local APIC}
|
|
\label{sec:apxlocalapic}
|
|
|
|
\subsection{Disabling PIC Mode}
|
|
\label{subsec:apxdisablepic}
|
|
|
|
Setting the IMCR\footnote{
|
|
Writing the IMCR is detailed in the \textquote{MultiProcessor Specification}~\autocite[sec.~3.6.2.1]{mpspec}.}
|
|
\ using hhuOS' \code{IoPort} class:
|
|
|
|
\begin{codeblock}{Disabling PIC Mode (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_imcr.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Accessing Local APIC Registers in xApic Mode}
|
|
\label{subsec:apxxapicregacc}
|
|
|
|
The xApic register access requires a single page of strong uncachable memory, but since this requires setting attributes in the \textquote{Page Attribute Table}\footnote{
|
|
See \url{https://docs.kernel.org/x86/pat.html} (visited on 12/02/2023).},
|
|
\ this implementation only uses hhuOS' \code{mapIO} function, which maps a physical address to the kernel heap, with hhuOS' \textquote{CACHE\textunderscore{}DISABLE} flag set in the kernel page table:
|
|
|
|
\begin{codeblock}{Allocating the Local APIC's MMIO Region (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_mmio_alloc.cpp}
|
|
\end{codeblock}
|
|
|
|
A register can now be written as follows:
|
|
|
|
\begin{codeblock}[label=lst:lapicmmiowrite]{Writing a Local APIC MMIO Register (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_mmio_write.cpp}
|
|
\end{codeblock}
|
|
|
|
To reduce the usage of manual bit-shifting and -masking, this implementation provides structures for some commonly used registers, that implement conversion operators to the register format\footnote{
|
|
The usual approach of \textquote{adding} the different required flags together to obtain the desired register content was intentionally not used for the sake of expressiveness: This approach does not highlight which flags are available and which flags are \textit{not} chosen.}:
|
|
|
|
\begin{codeblock}[label=lst:msrentry]{The MSREntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_msr_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:svrentry]{The SVREntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_svr_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:lvtentry]{The LVTEntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_lvt_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:icrentry]{The ICREntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_icr_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
These can be used in combination with some convenience functions:
|
|
|
|
\begin{codeblock}{Writing the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_msr.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Writing the SVR (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_svr.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Writing the LVT (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_lvt.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Writing the ICR (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_icr.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Initializing the LVT}
|
|
\label{subsec:apxlvtinit}
|
|
|
|
The interrupt vectors are set as defined by the \code{InterruptVector} enum.
|
|
This implementation configures the LVT by using the \code{LVTEntry} struct (see \autoref{lst:lvtentry}):
|
|
|
|
\begin{codeblock}{Configuring the Local Error Interrupt (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_lvt_example.cpp}
|
|
\end{codeblock}
|
|
|
|
This process is repeated for each local interrupt, with variations for the NMI and timer interrupts.
|
|
|
|
\subsection{Handling the Spurious Interrupt}
|
|
\label{subsec:apxsvr}
|
|
|
|
This implementation sets the SVR by using the \code{SVREntry} struct (see \autoref{lst:svrentry}):
|
|
|
|
\begin{codeblock}{Setting the Spurious Interrupt Vector (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_svr_example.cpp}
|
|
\end{codeblock}
|
|
|
|
Because hhuOS uses a two-stage interrupt handling approach (described in \autoref{subsec:apxcurrenthhuos}), the spurious interrupt does not receive its own interrupt handler.
|
|
Instead, it is ignored in the \code{dispatch} function, hhuOS' \textquote{first-stage} interrupt handler:
|
|
|
|
\begin{codeblock}{Checking for Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
|
|
\cppfile{code/interruptdispatcher_check_spurious.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Ignoring Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
|
|
\cppfile{code/interruptdispatcher_ignore_spurious.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Using the APIC Timer}
|
|
\label{subsec:apxapictimer}
|
|
|
|
The timer frequency is determined by counting the ticks in one millisecond, using the PIT as calibration source:
|
|
|
|
\begin{codeblock}{Calibrating the APIC Timer (ApicTimer.cpp)}{C++}
|
|
\cppfile{code/apictimer_calibrate.cpp}
|
|
\end{codeblock}
|
|
|
|
This calibration is performed before the interrupts get enabled, so it is not possible to use hhuOS' \code{TimeService} for the delay.
|
|
Instead, the \code{PIT::earlyDelay} function is used, which configures the PIT for mode \textquote{Interrupt on Terminal Count} on channel zero and polls the channel's output status~\autocite{pit}:
|
|
|
|
\begin{codeblock}{Microsecond Delay without Interrupts (Pit.cpp)}{C++}
|
|
\cppfile{code/pit_early_delay.cpp}
|
|
\end{codeblock}
|
|
|
|
Furthermore, the calibration is only performed once, even if multiple APIC timers are used.
|
|
This has to be taken into account if multiple timers with different dividers are to be used.
|
|
|
|
To handle the APIC timer interrupt on multiple cores, \(n\) \code{ApicTimer} instances are registered to the appropriate interrupt vector, where \(n\) is the number of CPUs.
|
|
Because this means, that each APIC timer interrupt on any CPU core triggers all \(n\) interrupt handlers, the handler has to determine if it belongs to the calling CPU\@.
|
|
The handler belonging to the BSP's APIC timer also triggers the scheduler preemption (instead of the PIT):
|
|
|
|
\begin{codeblock}{Handling the APIC Timer Interrupt (ApicTimer.cpp)}{C++}
|
|
\cppfile{code/apictimer_trigger.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Handling Local APIC Errors}
|
|
\label{subsec:apxhandlingerror}
|
|
|
|
The error interrupt handler obtains the ESR's contents by writing to it first:
|
|
|
|
\begin{codeblock}{The Local APIC Error Interrupt Handler (ApicErrorHandler.cpp)}{C++}
|
|
\cppfile{code/apicerror_trigger.cpp}
|
|
\end{codeblock}
|
|
|
|
Because every CPU core can only access its own local APIC's registers, a single instance of this interrupt handler can be used for all AP's in the system.
|
|
|
|
\section{I/O APIC}
|
|
\label{sec:apxioapic}
|
|
|
|
\subsection{Accessing I/O APIC Registers}
|
|
\label{subsec:iolistings}
|
|
|
|
The I/O APIC's indirect register access requires two MMIO registers, that can be written similar to the local APIC's registers:
|
|
|
|
\begin{codeblock}{Writing an I/O APIC MMIO Register (IoApic.h)}{C++}
|
|
\cppfile{code/ioapic_write_mmio.cpp}
|
|
\end{codeblock}
|
|
|
|
Using the \textquote{Index} and \textquote{Data} registers to access the I/O APIC's indirect registers:
|
|
|
|
\begin{codeblock}{Writing an I/O APIC Indirect Register (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_write_indirect.cpp}
|
|
\end{codeblock}
|
|
|
|
To reduce the manual bit-shifting and -masking, the same approach is used as for the local APIC\@:
|
|
|
|
\begin{codeblock}[label=lst:redtblentry]{The REDTBLEntry Structure (IoApicRegisters.h)}{C++}
|
|
\cppfile{code/ioapic_redtbl_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:writeredtbl]{Writing the REDTBL (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_write_redtbl.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Interrupt Overrides}
|
|
\label{subsec:apxirqoverrides}
|
|
|
|
This implementation represents an interrupt override using the following structure:
|
|
|
|
\begin{codeblock}{The External Interrupt Override Structure (IoApic.h)}{C++}
|
|
\cppfile{code/ioapic_irqoverride.cpp}
|
|
\end{codeblock}
|
|
|
|
During initialization, the overrides are used to set the correct interrupt vectors, polarities and trigger modes for each REDTBL entry.
|
|
The \code{REDTBLEntry} struct (see \autoref{lst:redtblentry}) is used to write the REDTBL\@:
|
|
|
|
\begin{codeblock}{Initializing the REDTBL (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_redtbl_example.cpp}
|
|
\end{codeblock}
|
|
|
|
During regular operation, the overrides are used to determine the correct REDTBL entries for e.g.\ unmasking an interrupt:
|
|
|
|
\begin{codeblock}{Unmasking an IRQ (Apic.cpp)}{C++}
|
|
\cppfile{code/apic_allow.cpp}
|
|
\end{codeblock}
|
|
|
|
To convey this deviation between GSIs and PC/AT compatible IRQs very clearly, the public \code{Apic::allow} function accepts an \code{InterruptRequest} as an argument (that allows addressing the interrupt by name), while the internal \code{IoApic::allow} function only accepts a \code{GlobalSystemInterrupt} as argument:
|
|
|
|
\begin{codeblock}{Unmasking an IRQ Internally (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_allow.cpp}
|
|
\end{codeblock}
|
|
|
|
The internal \code{IoApic::allow} function is hidden (\code{private}) from the OS, and gets only called by the exposed \code{Apic::allow} function.
|
|
This prevents accidentally setting the wrong REDTBL entry by not taking possible interrupt overrides into account.
|
|
The same principle is applied to the other operations concerning GSIs.
|
|
|
|
\section{Symmetric Multiprocessing}
|
|
\label{sec:apxsymmetric}
|
|
|
|
An overview of the complete SMP startup process can be found in \autoref{fig:smpenable}.
|
|
|
|
\subsection{Issuing Inter-Processor Interrupts}
|
|
\label{subsec:apxipis}
|
|
|
|
To issue an IPI, the ICR is written by using the \code{ICREntry} struct (see \autoref{lst:icrentry}):
|
|
|
|
\begin{codeblock}{Issuing an INIT IPI (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_initipi_example.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Issuing a SIPI (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_sipi_example.cpp}
|
|
\end{codeblock}
|
|
|
|
It is important to note that the SIPI does not receive the startup address, but the page number.
|
|
For this reason, the startup routine must be relocated starting at a page boundary.
|
|
The \code{page} argument can be determined by removing the page offset from the linear address, which is done by right-shifting the address by the offset length (\SI{12}{\bit} for \SI{4}{\kilo\byte} pages).
|
|
For the physical memory location used in this implementation (\code{0x8000}) this yields page number \code{8}.
|
|
|
|
\subsection{Preparing Symmetric Multiprocessing Startup}
|
|
\label{subsec:apxpreparesmp}
|
|
|
|
Before executing the \textquote{Universal Startup Algorithm}, the boot code has to be relocated to physical lower memory.
|
|
The memory region used for this copy has to be identity-mapped to the virtual kernel address space, so the effective addresses do not change after enabling paging in protected mode.
|
|
To keep the required variables available to the startup code, these are located in the routines \textquote{TEXT} section:
|
|
|
|
\begin{codeblock}{The Boot Routine's Variables (smp\textunderscore{}boot.asm)}{nasm}
|
|
\cppfile{code/ap_boot_variables.asm}
|
|
\end{codeblock}
|
|
|
|
The variables are initialized during runtime:
|
|
|
|
\begin{codeblock}{Preparing the Boot Routine's Variables (ApicSmp.cpp)}{C++}
|
|
\cppfile{code/ap_boot_variables.cpp}
|
|
\end{codeblock}
|
|
|
|
This approach was taken from SerenityOS~\autocite[APIC.cpp]{serenity}:
|
|
|
|
Now, the initialized startup routine can be copied to \code{0x8000}:
|
|
|
|
\begin{codeblock}{Relocating the Boot Routine (ApicSmp.cpp)}{C++}
|
|
\cppfile{code/ap_boot_copy.cpp}
|
|
\end{codeblock}
|
|
|
|
The \code{boot\textunderscore{}ap} function is the entry of the startup routine, it is described further in \autoref{subsec:apxapboot}.
|
|
|
|
\subsection{Universal Startup Algorithm}
|
|
\label{subsec:apxmpusa}
|
|
|
|
The \textquote{INIT-SIPI-SIPI} sequence, or \textquote{Universal Startup Algorithm} is performed by issuing IPIs as described in \autoref{subsec:apxipis} and using the PIT as time source (see \autoref{subsec:apxapictimer}):
|
|
|
|
\begin{codeblock}{The Universal Startup Algorithm (ApicSmp.cpp)}{C++}
|
|
\cppfile{code/ap_boot_usa.cpp}
|
|
\end{codeblock}
|
|
|
|
Boot completion is signaled in the AP's entry function:
|
|
|
|
\begin{codeblock}{Signaling AP Boot Completion (smp\textunderscore{}entry.cpp)}{C++}
|
|
\cppfile{code/smp_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\subsection{Application Processor Boot Routine}
|
|
\label{subsec:apxapboot}
|
|
|
|
Because like the BSP, every AP is initialized in real mode, it has to be switched to protected mode first:
|
|
|
|
\begin{codeblock}{Preparing the Switch to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_real_prepare.asm}
|
|
\end{codeblock}
|
|
|
|
Then, by enabling protected mode, setting the segment registers and far-jumping to the \SI{32}{\bit} code segment, the switch is performed:
|
|
|
|
\begin{codeblock}{Switching from Real Mode to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_real.asm}
|
|
\end{codeblock}
|
|
|
|
In \SI{32}{\bit} protected mode, paging is enabled and interrupt processing is prepared by reusing the control register values and the IDT from the BSP\@:
|
|
|
|
\begin{codeblock}{Reusing Values from the BSP (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_protected_bsp.asm}
|
|
\end{codeblock}
|
|
|
|
Finally, to call the AP entry function, the AP requires its own stack. If hardware context switching is to be used, the AP additionally requires its own GDT and TSS\@:
|
|
|
|
\begin{codeblock}{Calling the Entry Function (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_protected_ap.asm}
|
|
\end{codeblock}
|
|
|
|
\subsection{Application Processor Post-Boot Routine}
|
|
\label{subsec:apxappostboot}
|
|
|
|
When the AP has booted, it initializes its own APIC, including the APIC timer and error handler:
|
|
|
|
\begin{codeblock}{Initializing the Core's Local APIC (LocalApic.cpp)}{C++}
|
|
\cppfile{code/ap_boot_post.cpp}
|
|
\end{codeblock}
|
|
|
|
Because this added another possible recipient to internal APIC messages, the arbitration IDs are synchronized by issuing an \textquote{INIT-level-deassert IPI} using the \code{ICREntry} struct (see \autoref{lst:icrentry}):
|
|
|
|
\begin{codeblock}{Synchronizing the Arbitration IDs (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_apr_example.cpp}
|
|
\end{codeblock}
|
|
|
|
Because hhuOS' paging is not designed for multiple processors, the booted AP remains in a busy loop and does not enable its interrupts.
|
|
|
|
\cleardoublepage |