360 lines
14 KiB
TeX
360 lines
14 KiB
TeX
\chapter{Listings}
|
|
\label{ch:listings}
|
|
|
|
This chapter contains concrete implementation examples for concepts mentioned in
|
|
\autoref{ch:implementation}, demonstrated on the example of hhuOS\@.
|
|
|
|
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Accessing Local APIC Registers in xApic Mode}
|
|
\label{sec:apxxapicregacc}
|
|
|
|
The xApic register access requires a single page of strong uncachable memory, but since this
|
|
requires setting attributes in the ``Page Attribute Table''\footnote{See
|
|
\url{https://docs.kernel.org/x86/pat.html} (visited on 12/02/2023).}, this implementation only uses
|
|
hhuOS' \code{mapIO} function\cite[MemoryService.cpp]{hhuos}, which maps a physical address to the
|
|
kernel heap, with hhuOS' ``CACHE\textunderscore{}DISABLE'' flag set in the kernel page table:
|
|
|
|
\begin{codeblock}{Allocating the Local APIC's MMIO Region (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_mmio_alloc.cpp}
|
|
\end{codeblock}
|
|
|
|
A register can now be written as follows:
|
|
|
|
\begin{codeblock}[label=lst:lapicmmiowrite]{Writing a Local APIC MMIO Register (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_mmio_write.cpp}
|
|
\end{codeblock}
|
|
|
|
To reduce the usage of manual bit-shifting and -masking, this implementation provides structures
|
|
for some commonly used registers, that implement conversion operators to the register format:
|
|
|
|
\begin{codeblock}[label=lst:msrentry]{The MSREntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_msr_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:svrentry]{The SVREntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_svr_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:lvtentry]{The LVTEntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_lvt_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:icrentry]{The ICREntry Structure (LocalApicRegisters.h)}{C++}
|
|
\cppfile{code/lapic_icr_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
These can be used in combination with some convenience functions:
|
|
|
|
\begin{codeblock}{Writing the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_msr.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Writing the SVR (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_svr.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Writing the LVT (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_lvt.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Writing the ICR (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_write_icr.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Disabling PIC-Mode}
|
|
\label{sec:apxdisablepic}
|
|
|
|
Setting the IMCR\footnote{Writing the IMCR is detailed in the MultiProcessor
|
|
specification~\cite[sec. 3.6.2.1]{mpspec}.} using hhuOS' \code{IoPort} class:
|
|
|
|
\begin{codeblock}{Disabling PIC-Mode (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_imcr.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Initializing the LVT}
|
|
\label{sec:apxlvtinit}
|
|
|
|
The interrupt vectors are set as defined by the \code{InterruptVector} enum. This implementation
|
|
configures the LVT by using the \code{LVTEntry} struct (see \autoref{lst:lvtentry}):
|
|
|
|
\begin{codeblock}{Configuring the LINT0 Local Interrupt (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_lvt_example.cpp}
|
|
\end{codeblock}
|
|
|
|
This process is repeated for each local interrupt.
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Handling the Spurious Interrupt}
|
|
\label{sec:apxsvr}
|
|
|
|
This implementation sets the SVR by using the \code{SVREntry} struct (see \autoref{lst:svrentry}):
|
|
|
|
\begin{codeblock}{Setting the Spurious Interrupt Vector (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_svr_example.cpp}
|
|
\end{codeblock}
|
|
|
|
Because hhuOS uses a two-stage interrupt handling approach (described in
|
|
\autoref{sec:currenthhuos}), the spurious interrupt does not receive its own interrupt handler.
|
|
Instead, it is ignored in the \code{dispatch} function, hhuOS' ``first-stage'' interrupt handler:
|
|
|
|
\begin{codeblock}{Checking for Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
|
|
\cppfile{code/interruptdispatcher_check_spurious.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Ignoring Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
|
|
\cppfile{code/interruptdispatcher_ignore_spurious.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Using the APIC Timer}
|
|
\label{sec:apxapictimer}
|
|
|
|
The timer frequency is determined by counting the ticks in one millisecond, using the PIT as
|
|
calibration source:
|
|
|
|
\begin{codeblock}{Calibrating the APIC Timer (ApicTimer.cpp)}{C++}
|
|
\cppfile{code/apictimer_calibrate.cpp}
|
|
\end{codeblock}
|
|
|
|
This calibration is performed before the interrupts get enabled, so it is not possible to use
|
|
hhuOS' \code{TimeService} for the delay. Instead, the \code{PIT::earlyDelay} function is used,
|
|
which configures the PIT for mode 0 on channel 0 and polls the channel's output status~\cite{pit}].
|
|
Furthermore, the calibration is only performed once, even if multiple APIC timers are used. This
|
|
removes the possibility of using multiple timers with different dividers.
|
|
|
|
To handle the APIC timer interrupt on multiple cores, \(n\) \code{ApicTimer} instances are
|
|
registered to the appropriate interrupt vector, where \(n\) is the number of CPUs. Because this
|
|
means, that each APIC timer interrupt on any CPU core triggers all \(n\) interrupt handlers, the
|
|
handler has to determine if it belongs to the calling CPU:
|
|
|
|
\begin{codeblock}{Handling the APIC Timer Interrupt (ApicTimer.cpp)}{C++}
|
|
\cppfile{code/apictimer_trigger.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Handling local APIC Errors}
|
|
\label{sec:apxhandlingerror}
|
|
|
|
The error interrupt handler obtains the ESR's contents by writing to it first:
|
|
|
|
\begin{codeblock}{The local APIC Error Interrupt Handler (ApicErrorHandler.cpp)}{C++}
|
|
\cppfile{code/apicerror_trigger.cpp}
|
|
\end{codeblock}
|
|
|
|
Because every CPU core can only access its own local APIC's registers, a single instance of this
|
|
interrupt handler can be used for each AP in the system.
|
|
|
|
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Accessing I/O APIC Registers}
|
|
\label{sec:iolistings}
|
|
|
|
The I/O APIC's indirect register access requires two MMIO registers, that can be written similar to
|
|
the local APIC's registers:
|
|
|
|
\begin{codeblock}{Writing an I/O APIC MMIO Register (IoApic.h)}{C++}
|
|
\cppfile{code/ioapic_write_mmio.cpp}
|
|
\end{codeblock}
|
|
|
|
Using the ``Index'' and ``Data'' registers to access the I/O APIC's indirect registers:
|
|
|
|
\begin{codeblock}{Writing an I/O APIC indirect Register (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_write_indirect.cpp}
|
|
\end{codeblock}
|
|
|
|
To reduce the manual bit-shifting and -masking, the same approach is used as for the local APIC:
|
|
|
|
\begin{codeblock}[label=lst:redtblentry]{The REDTBLEntry Structure (IoApicRegisters.h)}{C++}
|
|
\cppfile{code/ioapic_redtbl_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:writeredtbl]{Writing the REDTBL (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_write_redtbl.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{I/O APIC Interrupt Overrides}
|
|
\label{sec:apxirqoverrides}
|
|
|
|
This implementation represents an interrupt override using the following structure:
|
|
|
|
\begin{codeblock}{The External Interrupt Override Structure (IoApic.h)}{C++}
|
|
\cppfile{code/ioapic_irqoverride.cpp}
|
|
\end{codeblock}
|
|
|
|
During initialization, they are used to set the correct interrupt vectors, polarities and trigger
|
|
modes for each REDTBL entry. The \code{REDTBLEntry} struct (see \autoref{lst:redtblentry}) is used
|
|
to write the REDTBL:
|
|
|
|
\begin{codeblock}{Initializing the REDTBL (IoApic.cpp)}{C++}
|
|
\cppfile{code/ioapic_redtbl_example.cpp}
|
|
\end{codeblock}
|
|
|
|
During regular OS operation, they are used to determine the correct REDTBL entries for e.g.\
|
|
unmasking an interrupt:
|
|
|
|
\begin{codeblock}{Unmasking an IRQ (Apic.cpp)}{C++}
|
|
\cppfile{code/apic_allow.cpp}
|
|
\end{codeblock}
|
|
|
|
To convey this deviation between GSIs and PC/AT compatible IRQs very clearly, the internal
|
|
\code{IoApic::allow} function only accepts a \code{GlobalSystemInterrupt} as argument, while the
|
|
public \code{Apic::allow} function accepts an \code{InterruptRequest}, that allows addressing the
|
|
interrupt by name.
|
|
|
|
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Issuing Inter-Processor Interrupts}
|
|
\label{sec:apxipis}
|
|
|
|
To issue an IPI, the ICR is written by using the \code{ICREntry} struct (see
|
|
\autoref{lst:icrentry}):
|
|
|
|
\begin{codeblock}{Issuing an INIT IPI (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_initipi_example.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}{Issuing a SIPI (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_sipi_example.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Preparing Symmetric Multiprocessing Startup}
|
|
\label{sec:apxpreparesmp}
|
|
|
|
Before executing the ``Universal Startup Algorithm'', the boot code has to be relocated to physical
|
|
lower memory. The memory region used for this copy has to be identity mapped to the virtual kernel
|
|
address space, so the effective addresses don't change after enabling paging in protected mode. To
|
|
keep required variables available to the startup code, these are located in the routines ``TEXT''
|
|
section and initialized during runtime. This approach was taken from~\cite[APIC.cpp]{serenity}:
|
|
|
|
\begin{codeblock}{Preparing the Boot Routine's Variables (ApicSmp.cpp)}{C++}
|
|
\cppfile{code/ap_boot_variables.cpp}
|
|
\end{codeblock}
|
|
|
|
Now, the initialized startup routine can be copied to \code{0x8000}:
|
|
|
|
\begin{codeblock}{Relocating the Boot Routine (ApicSmp.cpp)}{C++}
|
|
\cppfile{code/ap_boot_copy.cpp}
|
|
\end{codeblock}
|
|
|
|
The \code{boot\textunderscore{}ap} function is the entry of the startup routine, it is described
|
|
further in \autoref{sec:apxapboot}.
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Universal Startup Algorithm}
|
|
\label{sec:apxmpusa}
|
|
|
|
The ``INIT-SIPI-SIPI'' sequence, or ``Universal Startup Algorithm'' is performed by issuing IPIs as
|
|
described in \autoref{sec:apxipis} and using the PIT as time source (see
|
|
\autoref{sec:apxapictimer}):
|
|
|
|
\begin{codeblock}{The Universal Startup Algorithm (ApicSmp.cpp)}{C++}
|
|
\cppfile{code/ap_boot_usa.cpp}
|
|
\end{codeblock}
|
|
|
|
Boot completion is signaled in the AP's entry function:
|
|
|
|
\begin{codeblock}{Signaling AP Boot Completion (smp\textunderscore{}entry.cpp)}{C++}
|
|
\cppfile{code/smp_entry.cpp}
|
|
\end{codeblock}
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Application Processor Boot Routine}
|
|
\label{sec:apxapboot}
|
|
|
|
Because like the BSP, every AP is initialized in real mode, it has to be switched to protected mode
|
|
first:
|
|
|
|
\begin{codeblock}{Preparing the Switch to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_real_prepare.asm}
|
|
\end{codeblock}
|
|
|
|
Then, by enabling protected mode, setting the segment registers and far-jumping to the 32-bit code
|
|
segment, the switch is performed:
|
|
|
|
\begin{codeblock}{Switching from Real Mode to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_real.asm}
|
|
\end{codeblock}
|
|
|
|
In 32-bit protected mode, paging is enabled and interrupt processing is prepared by reusing the
|
|
control register values and the IDT from the BSP:
|
|
|
|
\begin{codeblock}{Loading Registers from the BSP (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_protected_bsp.asm}
|
|
\end{codeblock}
|
|
|
|
Finally, to call the AP entry function, the AP requires its own GDT and stack:
|
|
|
|
\begin{codeblock}{Calling the Entry Function (smp\textunderscore{}boot.asm)}{nasm}
|
|
\nasmfile{code/ap_boot_protected_ap.asm}
|
|
\end{codeblock}
|
|
|
|
The rest of the AP's initialization is performed from the entry function.
|
|
|
|
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Application Processor Post-Boot Routine}
|
|
\label{sec:apxappostboot}
|
|
|
|
When the AP has booted, it initializes its own APIC, including the APIC timer and error handler:
|
|
|
|
\begin{codeblock}{Initializing the Core's Local APIC (LocalApic.cpp)}{C++}
|
|
\cppfile{code/ap_boot_post.cpp}
|
|
\end{codeblock}
|
|
|
|
Because this added another possible recipient to internal APIC messages, the arbitration IDs are
|
|
synchronized by issuing an ``INIT-level-deassert IPI'' using the \code{ICREntry} struct (see
|
|
\autoref{lst:icrentry}):
|
|
|
|
\begin{codeblock}{Synchronizing the Arbitration IDs (LocalApic.cpp)}{C++}
|
|
\cppfile{code/lapic_apr_example.cpp}
|
|
\end{codeblock}
|
|
|
|
Because hhuOS' paging is not designed for multiple processors, the booted AP remains in a busy loop
|
|
and does not enable its interrupts.
|
|
|
|
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Integration into hhuOS}
|
|
\label{sec:hhuosintegration}
|
|
|
|
To integrate the APIC implementation into hhuOS, some preexisting components of its interrupt
|
|
infrastructure (described in \autoref{sec:currenthhuos}) have to be modified:
|
|
|
|
\begin{enumerate}
|
|
\item The \code{InterruptService} has to forward calls to allow or forbid a hardware interrupt to the
|
|
\code{Apic} class instead of the \code{Pic} class, depending on if the APIC system is enabled - to
|
|
keep the option of running the OS on hardware that does not support the APIC (see
|
|
\autoref{lst:interruptserviceafter}).
|
|
\item The \code{Pit} interrupt handler may no longer trigger the scheduler preemption if the APIC timer
|
|
is enabled (see \autoref{lst:pithandlerafter}).
|
|
\item The \code{System::initializeSystem} function needs to enable the APIC system if it is available
|
|
(see \autoref{lst:systemafter}).
|
|
\end{enumerate}
|
|
|
|
\begin{codeblock}[label=lst:interruptserviceafter]{Using the correct interrupt controller (InterruptService.cpp)}{C++}
|
|
\cppfile{code/interruptservice_after.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:pithandlerafter]{Disable preemption if necessary (Pit.cpp)}{C++}
|
|
\cppfile{code/pit_after.cpp}
|
|
\end{codeblock}
|
|
|
|
\begin{codeblock}[label=lst:systemafter]{Enabling the APIC system (System.cpp)}{C++}
|
|
\cppfile{code/system_after.cpp}
|
|
\end{codeblock}
|
|
|
|
\cleardoublepage |