christoph/interrupt-handling-using-the-x86-apic
1
Fork 0
Code Actions Activity

Initial commit

Browse Source
This commit is contained in:
Christoph Urlacher
2023-02-26 21:12:51 +01:00
commit d2ec37332f
69 changed files with 58831 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
chap/appendix_figures.tex Normal file
View File

@ -0,0 +1,49 @@
\chapter{Figures}
\label{ch:figures}
\clearpage
\begin{figure}[h]
\centering
\begin{subfigure}[b]{1.0\textwidth}
\includesvg[width=1.0\linewidth]{img/mp_spec_discrete_apic_configuration.svg}
\end{subfigure}
\caption{The discrete APIC architecture~\cite[sec.~5.1]{mpspec}.}
\label{fig:discreteapic}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{1.0\textwidth}
\includesvg[width=1.0\linewidth]{img/mp_spec_integrated_apic_configuration.svg}
\end{subfigure}
\caption{The integrated APIC architecture~\cite[sec.~5.2]{mpspec}.}
\label{fig:integratedapic}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_manual_system_vs_apic_bus.svg}
\end{subfigure}
\caption{System vs APIC bus~\cite[sec.~3.11.1]{ia32}.}
\label{fig:systemvsapicbus}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{1.0\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_manual_local_apic_blockdiagram.svg}
\end{subfigure}
\caption{The local APIC block diagram~\cite[sec.~3.11.4.1]{ia32}.}
\label{fig:localapicblock}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{1.0\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_lvt.svg}
\end{subfigure}
\caption{The Local Vector Table~\cite[sec.~3.11.5.1]{ia32}.}
\label{fig:localapiclvt}
\end{figure}

360
chap/appendix_listings.tex Normal file
View File

@ -0,0 +1,360 @@
\chapter{Listings}
\label{ch:listings}
This chapter contains concrete implementation examples for concepts mentioned in
\autoref{ch:implementation}, demonstrated on the example of hhuOS\@.
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Accessing Local APIC Registers in xApic Mode}
\label{sec:apxxapicregacc}
The xApic register access requires a single page of strong uncachable memory, but since this
requires setting attributes in the ``Page Attribute Table''\footnote{See
\url{https://docs.kernel.org/x86/pat.html} (visited on 12/02/2023).}, this implementation only uses
hhuOS' \code{mapIO} function\cite[MemoryService.cpp]{hhuos}, which maps a physical address to the
kernel heap, with hhuOS' ``CACHE\textunderscore{}DISABLE'' flag set in the kernel page table:
\begin{codeblock}{Allocating the Local APIC's MMIO Region (LocalApic.cpp)}{C++}
\cppfile{code/lapic_mmio_alloc.cpp}
\end{codeblock}
A register can now be written as follows:
\begin{codeblock}[label=lst:lapicmmiowrite]{Writing a Local APIC MMIO Register (LocalApic.cpp)}{C++}
\cppfile{code/lapic_mmio_write.cpp}
\end{codeblock}
To reduce the usage of manual bit-shifting and -masking, this implementation provides structures
for some commonly used registers, that implement conversion operators to the register format:
\begin{codeblock}[label=lst:msrentry]{The MSREntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_msr_entry.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:svrentry]{The SVREntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_svr_entry.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:lvtentry]{The LVTEntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_lvt_entry.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:icrentry]{The ICREntry Structure (LocalApicRegisters.h)}{C++}
\cppfile{code/lapic_icr_entry.cpp}
\end{codeblock}
These can be used in combination with some convenience functions:
\begin{codeblock}{Writing the IA32\textunderscore{}APIC\textunderscore{}BASE MSR (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_msr.cpp}
\end{codeblock}
\begin{codeblock}{Writing the SVR (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_svr.cpp}
\end{codeblock}
\begin{codeblock}{Writing the LVT (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_lvt.cpp}
\end{codeblock}
\begin{codeblock}{Writing the ICR (LocalApic.cpp)}{C++}
\cppfile{code/lapic_write_icr.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Disabling PIC-Mode}
\label{sec:apxdisablepic}
Setting the IMCR\footnote{Writing the IMCR is detailed in the MultiProcessor
specification~\cite[sec. 3.6.2.1]{mpspec}.} using hhuOS' \code{IoPort} class:
\begin{codeblock}{Disabling PIC-Mode (LocalApic.cpp)}{C++}
\cppfile{code/lapic_imcr.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Initializing the LVT}
\label{sec:apxlvtinit}
The interrupt vectors are set as defined by the \code{InterruptVector} enum. This implementation
configures the LVT by using the \code{LVTEntry} struct (see \autoref{lst:lvtentry}):
\begin{codeblock}{Configuring the LINT0 Local Interrupt (LocalApic.cpp)}{C++}
\cppfile{code/lapic_lvt_example.cpp}
\end{codeblock}
This process is repeated for each local interrupt.
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Handling the Spurious Interrupt}
\label{sec:apxsvr}
This implementation sets the SVR by using the \code{SVREntry} struct (see \autoref{lst:svrentry}):
\begin{codeblock}{Setting the Spurious Interrupt Vector (LocalApic.cpp)}{C++}
\cppfile{code/lapic_svr_example.cpp}
\end{codeblock}
Because hhuOS uses a two-stage interrupt handling approach (described in
\autoref{sec:currenthhuos}), the spurious interrupt does not receive its own interrupt handler.
Instead, it is ignored in the \code{dispatch} function, hhuOS' ``first-stage'' interrupt handler:
\begin{codeblock}{Checking for Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
\cppfile{code/interruptdispatcher_check_spurious.cpp}
\end{codeblock}
\begin{codeblock}{Ignoring Spurious Interrupts (InterruptDispatcher.cpp)}{C++}
\cppfile{code/interruptdispatcher_ignore_spurious.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Using the APIC Timer}
\label{sec:apxapictimer}
The timer frequency is determined by counting the ticks in one millisecond, using the PIT as
calibration source:
\begin{codeblock}{Calibrating the APIC Timer (ApicTimer.cpp)}{C++}
\cppfile{code/apictimer_calibrate.cpp}
\end{codeblock}
This calibration is performed before the interrupts get enabled, so it is not possible to use
hhuOS' \code{TimeService} for the delay. Instead, the \code{PIT::earlyDelay} function is used,
which configures the PIT for mode 0 on channel 0 and polls the channel's output status~\cite{pit}].
Furthermore, the calibration is only performed once, even if multiple APIC timers are used. This
removes the possibility of using multiple timers with different dividers.
To handle the APIC timer interrupt on multiple cores, \(n\) \code{ApicTimer} instances are
registered to the appropriate interrupt vector, where \(n\) is the number of CPUs. Because this
means, that each APIC timer interrupt on any CPU core triggers all \(n\) interrupt handlers, the
handler has to determine if it belongs to the calling CPU:
\begin{codeblock}{Handling the APIC Timer Interrupt (ApicTimer.cpp)}{C++}
\cppfile{code/apictimer_trigger.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Handling local APIC Errors}
\label{sec:apxhandlingerror}
The error interrupt handler obtains the ESR's contents by writing to it first:
\begin{codeblock}{The local APIC Error Interrupt Handler (ApicErrorHandler.cpp)}{C++}
\cppfile{code/apicerror_trigger.cpp}
\end{codeblock}
Because every CPU core can only access its own local APIC's registers, a single instance of this
interrupt handler can be used for each AP in the system.
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Accessing I/O APIC Registers}
\label{sec:iolistings}
The I/O APIC's indirect register access requires two MMIO registers, that can be written similar to
the local APIC's registers:
\begin{codeblock}{Writing an I/O APIC MMIO Register (IoApic.h)}{C++}
\cppfile{code/ioapic_write_mmio.cpp}
\end{codeblock}
Using the ``Index'' and ``Data'' registers to access the I/O APIC's indirect registers:
\begin{codeblock}{Writing an I/O APIC indirect Register (IoApic.cpp)}{C++}
\cppfile{code/ioapic_write_indirect.cpp}
\end{codeblock}
To reduce the manual bit-shifting and -masking, the same approach is used as for the local APIC:
\begin{codeblock}[label=lst:redtblentry]{The REDTBLEntry Structure (IoApicRegisters.h)}{C++}
\cppfile{code/ioapic_redtbl_entry.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:writeredtbl]{Writing the REDTBL (IoApic.cpp)}{C++}
\cppfile{code/ioapic_write_redtbl.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{I/O APIC Interrupt Overrides}
\label{sec:apxirqoverrides}
This implementation represents an interrupt override using the following structure:
\begin{codeblock}{The External Interrupt Override Structure (IoApic.h)}{C++}
\cppfile{code/ioapic_irqoverride.cpp}
\end{codeblock}
During initialization, they are used to set the correct interrupt vectors, polarities and trigger
modes for each REDTBL entry. The \code{REDTBLEntry} struct (see \autoref{lst:redtblentry}) is used
to write the REDTBL:
\begin{codeblock}{Initializing the REDTBL (IoApic.cpp)}{C++}
\cppfile{code/ioapic_redtbl_example.cpp}
\end{codeblock}
During regular OS operation, they are used to determine the correct REDTBL entries for e.g.\
unmasking an interrupt:
\begin{codeblock}{Unmasking an IRQ (Apic.cpp)}{C++}
\cppfile{code/apic_allow.cpp}
\end{codeblock}
To convey this deviation between GSIs and PC/AT compatible IRQs very clearly, the internal
\code{IoApic::allow} function only accepts a \code{GlobalSystemInterrupt} as argument, while the
public \code{Apic::allow} function accepts an \code{InterruptRequest}, that allows addressing the
interrupt by name.
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Issuing Inter-Processor Interrupts}
\label{sec:apxipis}
To issue an IPI, the ICR is written by using the \code{ICREntry} struct (see
\autoref{lst:icrentry}):
\begin{codeblock}{Issuing an INIT IPI (LocalApic.cpp)}{C++}
\cppfile{code/lapic_initipi_example.cpp}
\end{codeblock}
\begin{codeblock}{Issuing a SIPI (LocalApic.cpp)}{C++}
\cppfile{code/lapic_sipi_example.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Preparing Symmetric Multiprocessing Startup}
\label{sec:apxpreparesmp}
Before executing the ``Universal Startup Algorithm'', the boot code has to be relocated to physical
lower memory. The memory region used for this copy has to be identity mapped to the virtual kernel
address space, so the effective addresses don't change after enabling paging in protected mode. To
keep required variables available to the startup code, these are located in the routines ``TEXT''
section and initialized during runtime. This approach was taken from~\cite[APIC.cpp]{serenity}:
\begin{codeblock}{Preparing the Boot Routine's Variables (ApicSmp.cpp)}{C++}
\cppfile{code/ap_boot_variables.cpp}
\end{codeblock}
Now, the initialized startup routine can be copied to \code{0x8000}:
\begin{codeblock}{Relocating the Boot Routine (ApicSmp.cpp)}{C++}
\cppfile{code/ap_boot_copy.cpp}
\end{codeblock}
The \code{boot\textunderscore{}ap} function is the entry of the startup routine, it is described
further in \autoref{sec:apxapboot}.
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Universal Startup Algorithm}
\label{sec:apxmpusa}
The ``INIT-SIPI-SIPI'' sequence, or ``Universal Startup Algorithm'' is performed by issuing IPIs as
described in \autoref{sec:apxipis} and using the PIT as time source (see
\autoref{sec:apxapictimer}):
\begin{codeblock}{The Universal Startup Algorithm (ApicSmp.cpp)}{C++}
\cppfile{code/ap_boot_usa.cpp}
\end{codeblock}
Boot completion is signaled in the AP's entry function:
\begin{codeblock}{Signaling AP Boot Completion (smp\textunderscore{}entry.cpp)}{C++}
\cppfile{code/smp_entry.cpp}
\end{codeblock}
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Application Processor Boot Routine}
\label{sec:apxapboot}
Because like the BSP, every AP is initialized in real mode, it has to be switched to protected mode
first:
\begin{codeblock}{Preparing the Switch to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_real_prepare.asm}
\end{codeblock}
Then, by enabling protected mode, setting the segment registers and far-jumping to the 32-bit code
segment, the switch is performed:
\begin{codeblock}{Switching from Real Mode to Protected Mode (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_real.asm}
\end{codeblock}
In 32-bit protected mode, paging is enabled and interrupt processing is prepared by reusing the
control register values and the IDT from the BSP:
\begin{codeblock}{Loading Registers from the BSP (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_protected_bsp.asm}
\end{codeblock}
Finally, to call the AP entry function, the AP requires its own GDT and stack:
\begin{codeblock}{Calling the Entry Function (smp\textunderscore{}boot.asm)}{nasm}
\nasmfile{code/ap_boot_protected_ap.asm}
\end{codeblock}
The rest of the AP's initialization is performed from the entry function.
% \clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Application Processor Post-Boot Routine}
\label{sec:apxappostboot}
When the AP has booted, it initializes its own APIC, including the APIC timer and error handler:
\begin{codeblock}{Initializing the Core's Local APIC (LocalApic.cpp)}{C++}
\cppfile{code/ap_boot_post.cpp}
\end{codeblock}
Because this added another possible recipient to internal APIC messages, the arbitration IDs are
synchronized by issuing an ``INIT-level-deassert IPI'' using the \code{ICREntry} struct (see
\autoref{lst:icrentry}):
\begin{codeblock}{Synchronizing the Arbitration IDs (LocalApic.cpp)}{C++}
\cppfile{code/lapic_apr_example.cpp}
\end{codeblock}
Because hhuOS' paging is not designed for multiple processors, the booted AP remains in a busy loop
and does not enable its interrupts.
\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Integration into hhuOS}
\label{sec:hhuosintegration}
To integrate the APIC implementation into hhuOS, some preexisting components of its interrupt
infrastructure (described in \autoref{sec:currenthhuos}) have to be modified:
\begin{enumerate}
\item The \code{InterruptService} has to forward calls to allow or forbid a hardware interrupt to the
\code{Apic} class instead of the \code{Pic} class, depending on if the APIC system is enabled - to
keep the option of running the OS on hardware that does not support the APIC (see
\autoref{lst:interruptserviceafter}).
\item The \code{Pit} interrupt handler may no longer trigger the scheduler preemption if the APIC timer
is enabled (see \autoref{lst:pithandlerafter}).
\item The \code{System::initializeSystem} function needs to enable the APIC system if it is available
(see \autoref{lst:systemafter}).
\end{enumerate}
\begin{codeblock}[label=lst:interruptserviceafter]{Using the correct interrupt controller (InterruptService.cpp)}{C++}
\cppfile{code/interruptservice_after.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:pithandlerafter]{Disable preemption if necessary (Pit.cpp)}{C++}
\cppfile{code/pit_after.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:systemafter]{Enabling the APIC system (System.cpp)}{C++}
\cppfile{code/system_after.cpp}
\end{codeblock}
\cleardoublepage

440
chap/appendix_tables.tex Normal file
View File

@ -0,0 +1,440 @@
\chapter{Tables}
\label{ch:tables}
This section lists all the registers and structures required to follow \autoref{ch:implementation}.
\clearpage
\renewcommand{\arraystretch}{1.2}
\section{Local APIC Registers}
\label{sec:localapicregisters}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Register Name} & \textbf{Memory Offset} \\ \hline\hline
Local APIC ID Register & 0x20 \\ \hline
Local APIC Version Register & 0x30 \\ \hline
Task Priority Register & 0x80 \\ \hline
EOI Register & 0xB0 \\ \hline
Spurious Interrupt Vector Register & 0xF0 \\ \hline
Error Status Register & 0x280 \\ \hline
Interrupt Command Register[0:31] & 0x300 \\ \hline
Interrupt Command Register[32:63] & 0x310 \\ \hline
LVT Timer Register & 0x320 \\ \hline
LVT LINT1 Register & 0x360 \\ \hline
LVT Error Register & 0x370 \\ \hline
Timer Initial Count Register & 0x380 \\ \hline
Timer Divide Configuration Register & 0x3E0 \\ \hline
\end{tabularx}
\caption{Local APIC Registers used in this Implementation~\cite[sec.~3.11.4.1]{ia32}.}
\label{tab:lapicregs}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:23 & Reserved \\ \hline
24:31 & Local APIC ID \\ \hline
\end{tabularx}
\caption{Local APIC ID Register (xApic since Pentium 4)~\cite[sec.~3.11.4.6]{ia32}.}
\label{tab:lapicregsid}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Local APIC Version \\ \hline
8:15 & Reserved \\ \hline
16:23 & Max LVT Entry \\ \hline
24 & EOI Broadcast Suppression Support \\ \hline
25:31 & Reserved \\ \hline
\end{tabularx}
\caption{Local APIC Version Register~\cite[sec.~3.11.4.8]{ia32}.}
\label{tab:lapicregsver}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:3 & Task-Priority Subclass \\ \hline
4:7 & Task-Priority Class \\ \hline
8:31 & Reserved \\ \hline
\end{tabularx}
\caption{Task Priority Register~\cite[sec.~3.11.8.3.1]{ia32}.}
\label{tab:lapicregstpr}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:31 & Send EOI Signal \\ \hline
\end{tabularx}
\caption{Local APIC EOI Register~\cite[sec.~3.11.8.5]{ia32}.}
\label{tab:lapicregseoi}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Spurious Interrupt Vector \\ \hline
8 & APIC Software Enable/Disable \\ \hline
9 & Focus Processor Checking \\ \hline
10:11 & Reserved \\ \hline
12 & EOI Broadcast Suppression \\ \hline
13:31 & Reserved \\ \hline
\end{tabularx}
\caption{Spurious Interrupt Vector Register~\cite[sec.~3.11.9]{ia32}.}
\label{tab:lapicregssvr}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:4 & Reserved \\ \hline
5 & Send Illegal Vector \\ \hline
6 & Receive Illegal Vector \\ \hline
7 & Illegal Register Access \\ \hline
8:31 & Reserved \\ \hline
\end{tabularx}
\caption{Error Status Register (Pentium 4)~\cite[sec.~3.11.5.3]{ia32}.}
\label{tab:lapicregsesr}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Interrupt Vector \\ \hline
8:10 & Delivery Mode \\ \hline
11 & Destination Mode \\ \hline
12 & Delivery Status \\ \hline
13 & Reserved \\ \hline
14 & Level \\ \hline
15 & Trigger Mode \\ \hline
16:17 & Reserved \\ \hline
18:19 & Destination Shorthand \\ \hline
20:55 & Reserved \\ \hline
56:63 & Destination Field \\ \hline
\end{tabularx}
\caption{Interrupt Command Register~\cite[sec.~3.11.6.1]{ia32}.}
\label{tab:lapicregsicr}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Interrupt Vector \\ \hline
8:11 & Reserved \\ \hline
12 & Delivery Status \\ \hline
13:15 & Reserved \\ \hline
16 & Masked \\ \hline
17:18 & Timer Mode \\ \hline
19:31 & Reserved \\ \hline
\end{tabularx}
\caption{LVT Timer Register~\cite[sec.~3.11.5.1]{ia32}.}
\label{tab:lapicregslvtt}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Interrupt Vector \\ \hline
8:11 & Reserved \\ \hline
12 & Delivery Status \\ \hline
13:15 & Reserved \\ \hline
16 & Masked \\ \hline
17:31 & Reserved \\ \hline
\end{tabularx}
\caption{LVT Error Register~\cite[sec.~3.11.5.1]{ia32}.}
\label{tab:lapicregslvterr}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Interrupt Vector \\ \hline
8:10 & Delivery Mode \\ \hline
11 & Reserved \\ \hline
12 & Delivery Status \\ \hline
13 & Pin Polarity \\ \hline
14 & Remote IRR \\ \hline
15 & Pin Polarity \\ \hline
16 & Masked \\ \hline
17:31 & Reserved \\ \hline
\end{tabularx}
\caption{LVT LINT1 Register~\cite[sec.~3.11.5.1]{ia32}.}
\label{tab:lapicregslvtlint}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:31 & Initial Count \\ \hline
\end{tabularx}
\caption{Timer Initial Count Register~\cite[sec.~3.11.5.4]{ia32}.}
\label{tab:lapicregstimerinit}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:1 & Divider \\ \hline
2 & Reserved \\ \hline
3 & Divider \\ \hline
4:31 & Reserved \\ \hline
\end{tabularx}
\caption{Timer Divide Configuration Register~\cite[sec.~3.11.5.4]{ia32}.}
\label{tab:lapicregstimerdiv}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Reserved \\ \hline
8 & BSP Flag \\ \hline
9 & Reserved \\ \hline
10 & Enable x2Apic \\ \hline
11 & Enable xApic \\ \hline
12:35 & APIC Base Address \\ \hline
36:63 & Reserved \\ \hline
\end{tabularx}
\caption{IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\cite[sec.~3.11.12.1]{ia32}.}
\label{tab:lapicregsmsr}
\end{table}
\clearpage
\section{I/O APIC Registers}
\label{sec:ioapicregs}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Register Name} & \textbf{Memory Offset} \\ \hline\hline
Index Register & 0x00 \\ \hline
Data Register & 0x10 \\ \hline\hline
\textbf{Register Name} & \textbf{Index Offset} \\ \hline\hline
I/O APIC ID Register & 0x00 \\ \hline
I/O APIC Version Register & 0x01 \\ \hline
Redirection Table & 0x10:0x3F \\ \hline
\end{tabularx}
\caption{I/O APIC Registers used in this Implementation~\cite[sec.~9.5]{ich5}.}
\label{tab:ioapicregs}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Indirect Register Index \\ \hline
\end{tabularx}
\caption{I/O APIC Index Register~\cite[sec.~9.5.2]{ich5}.}
\label{tab:ioapicregsidx}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:31 & Indirect Register Data \\ \hline
\end{tabularx}
\caption{I/O APIC Data Register~\cite[sec.~9.5.3]{ich5}.}
\label{tab:ioapicregsdat}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:14 & Reserved \\ \hline
15 & Scratchpad Bit \\ \hline
16:23 & Reserved \\ \hline
24:27 & I/O APIC ID \\ \hline
28:31 & Reserved \\ \hline
\end{tabularx}
\caption{I/O APIC ID Register~\cite[sec.~9.5.6]{ich5}.}
\label{tab:ioapicregsid}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & I/O APIC Version \\ \hline
8:14 & Reserved \\ \hline
15 & PRQ \\ \hline
16:23 & Maximum Redirection Entries \\ \hline
24:31 & Reserved \\ \hline
\end{tabularx}
\caption{I/O APIC Version Register~\cite[sec.~9.5.7]{ich5}.}
\label{tab:ioapicregsver}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:7 & Interrupt Vector \\ \hline
8:10 & Delivery Mode \\ \hline
11 & Destination Mode \\ \hline
12 & Delivery Status \\ \hline
13 & Pin Polarity \\ \hline
14 & Remote IRR \\ \hline
15 & Trigger Mode \\ \hline
16 & Masked \\ \hline
17:47 & Reserved \\ \hline
48:55 & Extended Destination ID \\ \hline
56:63 & Destination \\ \hline
\end{tabularx}
\caption{I/O APIC REDTBL Register~\cite[sec.~9.5.8]{ich5}.}
\label{tab:ioapicregsredtbl}
\end{table}
\clearpage
\section{System Description Tables}
\label{sec:sdts}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Byte Number} & \textbf{Description} \\ \hline\hline
0:35 & MADT Header \\ \hline
36:39 & Local APIC Base Address \\ \hline
40:43 & Local APIC Flags \\ \hline
44: & List of APIC Structures \\ \hline
\end{tabularx}
\caption{ACPI MADT~\cite[sec.~5.2.8]{acpi1}.}
\label{tab:madt}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Byte Number} & \textbf{Description} \\ \hline\hline
0:1 & APIC Structure Header \\ \hline
2 & ACPI Processor ID \\ \hline
3 & APIC ID \\ \hline
4:7 & Local APIC Flags (see \autoref{tab:madtlapicflags}) \\ \hline
\end{tabularx}
\caption{MADT Processor Local APIC Structure~\cite[sec.~5.2.8.1]{acpi1}.}
\label{tab:madtlapic}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0 & Enabled \\ \hline
1:31 & Reserved \\ \hline
\end{tabularx}
\caption{Local APIC Flags~\cite[sec.~5.2.8.1]{acpi1}.}
\label{tab:madtlapicflags}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Byte Number} & \textbf{Description} \\ \hline\hline
0:1 & APIC Structure Header \\ \hline
2 & I/O APIC ID \\ \hline
3 & Reserved \\ \hline
4:7 & I/O APIC Base Address \\ \hline
8:11 & I/O APIC GSI Base \\ \hline
\end{tabularx}
\caption{MADT I/O APIC Structure~\cite[sec.~5.2.8.2]{acpi1}.}
\label{tab:madtioapic}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Byte Number} & \textbf{Description} \\ \hline\hline
0:1 & APIC Structure Header \\ \hline
2 & Bus \\ \hline
3 & Source \\ \hline
4:7 & GSI \\ \hline
8:9 & Interrupt Input Flags (see \autoref{tab:madtintiflags}) \\ \hline
\end{tabularx}
\caption{MADT Interrupt Source Override Structure~\cite[sec.~5.2.8.3.1]{acpi1}.}
\label{tab:madtirqoverride}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Bit Number} & \textbf{Description} \\ \hline\hline
0:1 & Pin Polarity \\ \hline
2:3 & Trigger Mode \\ \hline
4:11 & Reserved \\ \hline
\end{tabularx}
\caption{Interrupt Input Flags~\cite[sec.~5.2.8.3.1]{acpi1}.}
\label{tab:madtintiflags}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Byte Number} & \textbf{Description} \\ \hline\hline
0:1 & APIC Structure Header \\ \hline
2:3 & Interrupt Input Flags (see \autoref{tab:madtintiflags}) \\ \hline
4:7 & GSI
\end{tabularx}
\caption{MADT I/O APIC NMI Source~\cite[sec.~5.2.8.3.2]{acpi1}.}
\label{tab:madtionmi}
\end{table}
\begin{table}[H]
\centering
\begin{tabularx}{1.0\textwidth}{| X | X |}
\hline
\textbf{Byte Number} & \textbf{Description} \\ \hline\hline
0:1 & APIC Structure Header \\ \hline
2 & ACPI Processor ID \\ \hline
3:4 & Interrupt Input Flags (see \autoref{tab:madtintiflags}) \\ \hline
5 & Local APIC Interrupt Input \\ \hline
\end{tabularx}
\caption{MADT Local APIC NMI Source~\cite[sec.~5.2.8.3.3]{acpi1}.}
\label{tab:madtlnmi}
\end{table}

427
chap/background.tex Normal file
View File

@ -0,0 +1,427 @@
\chapter{Background}
\label{ch:background}
In this section, important domain specific concepts will be explained, to create the necessary
foundation to follow \autoref{ch:implementation}. Important terms that are present in the glossary
are marked in \textbf{bold} on their first occurrence.
\clearpage
\section{Handling of External Events}
\label{sec:eventhandling}
There are two different strategies to ``listen'' for external events: ``Polling'' and
``Interrupts''.
The first strategy works by periodically \textit{polling} a device to check for changes. This could
mean reading a register of the keyboard every 50 ms to determine if a key was pressed, or reading a
sensor output even when the value remains unchanged. Because every device would have to be polled
constantly, no matter if any change actually occurred, this method is computationally inefficient
(although easy to implement without extra hardware).
The second strategy are \textbf{\glspl{interrupt}}. Instead of the CPU actively ``looking'' for
changes, the devices signal the events to the CPU themselves. Every time the CPU is notified of an
external event, it pauses the current code execution and calls a function designated to handle this
specific event.
This approach is much more efficient than the polling strategy, because the CPU does not have to
waste its processing power to check for events when none are occurring.
\section{Fundamental Concepts}
\label{sec:fundamentals}
\subsection{Interrupt}
\label{subsec:interrupt}
When a device signals an external event to the CPU, the current code execution gets
\textit{interrupted}, to handle the event. Thus, the process of signaling and handling an external
event is called ``interrupt''.
Interrupts can be caused by all sorts of events, like key presses on a keyboard or packets received
via a network card. These interrupts from external hardware devices are called
\textbf{\glspl{external interrupt}}.
Other types of interrupts mentioned in this thesis are \textbf{\glspl{ipi}}, \textbf{\glspl{msi}}
and \textbf{\glspl{local interrupt}}:
\begin{itemize}
\item IPIs: Interrupts sent between different processor cores in multiprocessor systems, for example to
initialize different cores on system startup.
\item MSIs: Interrupts sent in-band\footnote{In in-band signaling some control information (like an IRQ)
is sent over the same channel as the data. This stands in contrast to out-of-band signaling, which
uses a dedicated control line (like an interrupt line).}, for example over a PCI-bus\footnote{PCI
supports MSIs since PCI 2.2~\cite[sec.~6.8]{pci22}.}.
\item Local Interrupts: Some specific CPU-internal interrupts.
\end{itemize}
Some interrupts are essential for continuous CPU operation. These \textbf{\glspl{nmi}} always have
to be handled and cannot be ignored (usually\footnote{An example where NMIs should be disabled is
the startup of additional processors in multiprocessor systems.}). Hardware error interrupts are a
typical example for NMIs.
The existence of NMIs hints that it is possible to ignore regular interrupts. Marking an interrupt
as ``ignored'' is called \textbf{\gls{masking}} an interrupt.
\subsection{Interrupt Controller}
\label{subsec:controller}
A computer system has to process interrupts from many sources, so it is practical to have a
designated \textbf{\gls{interrupt controller}}, that receives interrupts from different devices and
forwards them to the CPU. The masking functionality is integrated into the interrupt controller, so
masked interrupts don't reach the CPU at all.
The interrupt controller receives interrupts through signals over physical connections (interrupt
lines) to different devices. These signals can be represented in multiple ways through the level on
the interrupt lines:
\textbf{\Gls{trigger mode}}:
\begin{itemize}
\item Edge-triggered signal: A signal is received when the interrupt line level changes
\item Level-triggered signal: A signal is received when the interrupt line reaches a certain level
\end{itemize}
\textbf{\Gls{pin polarity}}:
\begin{itemize}
\item High: A signal is received when the interrupt line level changes from either low to high (in
edge-triggered mode) or reaches a level above a threshold (in level-triggered mode)
\item Low: A signal is received when the interrupt line level changes from either high to low (in
edge-triggered mode) or reaches a level below a threshold (in level-triggered mode)
\end{itemize}
A signal is represented through a combination of trigger mode and pin polarity, so there are a
total of 4 combinations.
When the interrupt controller receives an interrupt signal, it requests the CPU to handle the
specific event: The interrupt controller sends an \textbf{\gls{irq}} to the CPU. Because there are
multiple devices that can signal interrupts, an IRQ is usually indexed by its hardware pin on the
interrupt controller: If a keyboard is connected to the interrupt controller on pin 1, the CPU
receives an IRQ1.
To signal that an interrupt has been handled by the software, the interrupt controller receives an
\textbf{\gls{eoi}} notice from the OS\@.
Information on specific interrupt controllers follows in \autoref{sec:intelcontrollers}.
\subsection{Interrupt Handler}
\label{subsec:handler}
When the CPU receives an IRQ, it pauses its current code execution to handle the interrupt. This is
done by executing the \textbf{\gls{interrupt handler}} function, that is registered to the specific
interrupt.
During interrupt servicing (execution of the interrupt handler), other interrupts can occur. When
the execution of an interrupt handler is paused to handle another interrupt, this is called a
\textbf{\gls{cascaded interrupt}}.
Interrupt handlers have to be registered to different IRQs, so the CPU can locate and execute the
specific code that is designated to handle a certain interrupt. The interrupt handler addresses are
stored in an \textit{interrupt vector table}: Each address corresponds to an \textbf{\gls{interrupt
vector}} number, interrupt vectors 0 to 31 are reserved for the CPU, interrupt vectors 32 to 255
are usable for IRQs. In Intel's 32 bit IA-32 CPU architecture, this table is called the
\textbf{\gls{idt}}. The \textbf{\gls{idtr}} keeps the location of the IDT, it is written by the
\code{lidt}~\cite{x86isa} instruction.
\subsection{Interrupt Trigger Mode}
\label{subsec:triggermode}
When an edge-triggered IRQ is received, its interrupt handler is called a single time, serviced,
and marked as completed in the interrupt controller (completion does not require interacting with
the device the interrupt originated from). The handler of an edge-triggered IRQ is called every
time the interrupt line changes its level. This could lead to problems if ``glitches'' occur on the
interrupt line.
An alternative is the level-triggered IRQ: When the interrupt line is above/below a certain level,
the interrupt is signaled continuously. Servicing the interrupt thus requires not only signaling
completion to the interrupt controller, but also to the device the interrupt originated from, to
de-assert the interrupt line. Otherwise, the interrupt handler would be called again after
completion.
\subsection{Spurious Interrupt}
\label{subsec:spurious}
When an interrupt disappears while the interrupt controller is issuing the IRQ to the CPU, the
interrupt controller sends a \textbf{\gls{spurious interrupt}}. The reason for this could be
electrical noise on the interrupt line, masking of an interrupt through software at the same moment
this interrupt was received, or incorrectly sent EOIs. Thus, before an interrupt handler is called,
the OS has to check the validity of the occurred interrupt and ignore it, if it is deemed spurious.
\section{Used Technologies}
\label{sec:technologies}
\subsection{Advanced Configuration and Power Interface}
\label{subsec:acpi}
\textbf{\gls{acpi}} allows the kernel to gather information about the system hardware during
runtime. It also provides interactive functionality for power management, plug and play,
hot-swapping or status monitoring. To interact with ACPI, the
\textbf{\gls{aml}}~\cite[sec.~16]{acpi1}, a small language interpreted by the kernel, has to be
used\footnote{In this thesis, information from ACPI is only read, so AML is not required.}.
ACPI defines abstractions for different types of hardware, that are organized in multiple
\textit{system description tables}. In this thesis, ACPI 1.0~\cite{acpi1} is used to read
information about the system's interrupt hardware configuration, located in the ACPI
\textbf{\gls{madt}}~\cite[sec.~5.2.8]{acpi1}. The MADT contains information on used interrupt
controllers (version, physical memory addresses to access registers, etc.), available CPU cores in
multiprocessor systems and specific interrupt configuration (trigger mode and pin polarity).
To allow compatibility to different interrupt controllers, ACPI abstracts external interrupts
through \textbf{\glspl{gsi}}~\cite[sec.~5.2.9]{acpi1}. Different interrupt controllers may have the
same devices connected to different interrupt lines, maintaining a mapping between actual hardware
interrupt lines and GSIs allows the OS to operate on different interrupt
controllers\footnote{Additional information in \autoref{subsec:ioapicpcat}.}.
\subsection{CPU Identification}
\label{subsec:cpuid}
\textbf{\gls{cpuid}} is used to gather information about a system's CPU. The x86 architecture
provides the \code{cpuid}~\cite{x86isa} instruction, which allows the software to identify present
CPU features at runtime, e.g.\ what instruction set extensions a processor implements\footnote{This
thesis uses CPUID to determine what APIC feature set is supported, see ``APIC'' and ``x2APIC''
features in the CPUID application note~\cite[sec.~5.1.2]{cpuid}.}.
\subsection{Symmetric Multiprocessing}
\label{subsec:smp}
\textbf{\gls{smp}} is a form of multiprocessing, where a system consists of multiple, homogeneous
CPU cores, that all have access to a shared main memory and the I/O devices. SMP systems are
controlled by a single OS, that treats all cores as individual, but equal processors. The shared
main memory allows every core to work on an arbitrary task, independent of its memory location.
Programming for SMP systems requires the usage of multithreading, to allow for an efficient
distribution of tasks to multiple CPU cores.
In SMP systems, a single CPU core is active initially, this core is called the \textbf{\gls{bsp}}.
Other cores, called \textbf{\glspl{ap}}, will be initialized by this core.
% TODO: I think I could remove this section
% \subsection{BIOS and UEFI}
% \label{subsec:biosuefi}
%
% The \textbf{\gls{bios}} provides low-level control over a computer system's hardware, most commonly
% used for booting an OS on system power-up. Its pre-boot environment is limited to the 16-bit
% \textbf{\gls{real mode}} and 1 MB of addressable memory. BIOS originates from the IBM PC and has
% been the default convention for computer system firmware until the \textbf{\gls{uefi}} standard.
%
% The UEFI standard is developed by the UEFI Forum, based on Intel's EFI. It improves on BIOS by
% providing modularity, support for larger partitions, 32-bit or 64-bit environments before booting
% and advanced features like networking capabilities, graphical user interfaces with mouse support or
% security features\footnote{See \url{https://uefi.org/specifications} (visited on 22/02/2023).}.
\subsection{Model Specific Register}
\label{subsec:msr}
A \textbf{\gls{msr}} is a special control register in an IA-32 CPU, that is not guaranteed to be
present in future CPU generations (it is not part of the architectural standard). MSRs that carried
over to future generations and are now guaranteed to be included in future IA-32 or Intel 64 CPUs
are called ``architectural'' MSRs. To interact with MSRs, the special instructions \code{rdmsr} and
\code{wrmsr} are used~\cite{x86isa}. These instructions allow atomic read/write operations on
MSRs\footnote{Detailed description in the IA-32 manual~\cite[sec.~4.2]{ia32}.}.
\subsection{Memory Mapped I/O}
\label{subsec:mmio}
\textbf{\gls{mmio}} is a way for the CPU to perform I/O operations with an external device.
Registers of external devices are mapped to the same address space as the main memory, thus a
memory address can either be a location in physical memory, or belong to a device's register. Read
and write operations are then performed by reading or writing the address the register is mapped
to.
\subsection{Programmable Interval Timer}
\label{subsec:pit}
The \textbf{\gls{pit}}\footnote{Specifically the Intel 8253/8254.} is a hardware timer from the
original IBM PC. It consists of multiple decrementing counters and can generate a signal (for
example an interrupt) once 0 is reached. This can be used to control a preemptive\footnote{A
scheduler is called ``preemptive'', when it is able to forcefully take away the CPU from the
currently working thread, even when the thread did not signal any form of completion.} scheduler or
the integrated PC speaker.
\section{Intel's Interrupt Controllers}
\label{sec:intelcontrollers}
Because it is logistically difficult to connect every external device directly to the CPU, this
task is delegated to an interrupt controller, usually located in the system's chipset. Notable
interrupt controllers (and the only ones considered in this thesis) are the Intel
\textbf{\gls{pic}} and Intel \textbf{\gls{apic}}.
\subsection{Programmable Interrupt Controller}
\label{subsec:intelpic}
The Intel 8259A PIC~\cite{pic} is the interrupt controller Intel used in the original Intel
8086\footnote{The 8259 was used in the Intel 8080 and 8085, the 8259A is a revised version for the
Intel 8086}, the ``father'' of the x86 processor architecture. A single PIC has 8 interrupt lines
for external devices, but multiple PICs can be cascaded for a maximum of \(8 \cdot 8 = 64\)
interrupt lines with 9 PICs. The most common PIC configuration supports 15 IRQs with two PICs: A
slave, connected to interrupt line 2 of the master, which is connected to the CPU\footnote{This
configuration is called the \textbf{\gls{pcat pic architecture}}, as it was used in the IBM
PC/AT~\cite[sec.~1.13]{pcat}.}.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includesvg[width=1.0\linewidth]{img/mp_spec_pic_configuration.svg}
\end{subfigure}
\caption{The PC/AT PIC architecture~\cite[sec.~1.13]{pcat}.}
\label{fig:pcatpic}
\end{figure}
If multiple interrupts get requested simultaneously, the PIC decides which one to service first
based on its \textbf{\gls{interrupt priority}}. The PIC IRQ priorities are determined by the used
interrupt lines, IRQ0 has the highest priority.
The PIC interrupt handling sequence can be briefly outlined as follows:
\begin{enumerate}
\item When the PIC receives an interrupt from a device (PIC interrupts are edge-triggered with high pin
polarity), it sets the corresponding bit in its \textbf{\gls{irr}}\footnote{Received IRQs that are
requesting servicing are marked in the IRR.}
\item If this interrupt is unmasked, an IRQ is sent to the CPU
\item The CPU acknowledges the IRQ
\item The PIC sets the corresponding bit in its \textbf{\gls{isr}}\footnote{IRQs that are being serviced
are marked in the ISR. To reduce confusion, interrupt service routines will be denoted as
``interrupt handler'' instead of ``ISR''.} and clears the previously set bit in the IRR. If the
same interrupt occurs again, it can now be queued a single time in the IRR\@.
\item When the interrupt has been handled by the OS, it sends an EOI to the PIC, which clears the
corresponding bit in the ISR\@.
\end{enumerate}
With the introduction of multiprocessor systems and devices with an ever-increasing need for large
amounts of interrupts (like high-performance networking hardware), the PIC reached its
architectural limits:
\begin{itemize}
\item Multiple CPU cores require sending IPIs for initialization and coordination, which is not possible
using the PIC\@.
\item The PIC is hardwired to a single CPU core. It can be inefficient to handle the entire interrupt
workload on a single core.
\item The IRQ priorities depend on how the devices are physically wired to the PIC\@.
\item The PC/AT PIC architecture only has a very limited number of interrupt lines.
\item The PIC does not support MSIs, which allow efficient handling of PCI interrupts.
\end{itemize}
\subsection{Advanced Programmable Interrupt Controller}
\label{subsec:intelapic}
The original Intel 82489DX discrete APIC was introduced with the Intel i486 processor. It consisted
of two parts: A \textbf{\gls{local apic}}, responsible for receiving special \textbf{\glspl{local
interrupt}}, and external interrupts from the second part, the \textbf{\gls{io apic}}. Unlike in
modern systems, the i486's local APIC was not integrated into the CPU core (hence ``discrete''),
see \autoref{fig:discreteapic}.
The i486 was superseded by the Intel P54C\footnote{The Intel P54C is the successor of the Intel P5,
the first Pentium processor.}, which contained an integrated local APIC for the first time, see
\autoref{fig:integratedapic}\footnote{This is now the default configuration.}.
The APIC was designed to fix the shortcomings of the PIC, specifically regarding multiprocessor
systems:
\begin{itemize}
\item Each CPU core contains its own local APIC. It has the capabilities to issue IPIs and communicate
with the chipset I/O APIC. Also, it includes the APIC timer, which can be used for per-core
scheduling.
\item The chipset I/O APIC allows configuration on a per-interrupt basis (priorities, destination,
trigger mode or pin polarity are all configurable). Also, it is able to send interrupts to
different CPU cores, allowing the efficient processing of large amounts of
interrupts\footnote{There are tools that dynamically reprogram the I/O APIC to distribute
interrupts to available CPU cores, depending on heuristics of past interrupts (IRQ-balancing).}.
\item The order in which external devices are connected to the I/O APIC is flexible.
\item The APIC architecture supports MSIs.
\end{itemize}
To allow distributing the APIC hardware between local APICs of multiple CPU cores and I/O APICs,
the APIC system sends messages over a bus. In the Intel P6 and original Pentium processors a
dedicated APIC bus is used, but since the Intel Pentium 4 and Xeon processors the chipset's system
bus is used instead (see \autoref{fig:systemvsapicbus})\footnote{The Intel Core microarchitecture
is based on the P6 architecture (replacing the Pentium 4's NetBurst architecture), but still uses
the system bus instead of a discrete bus.}. Using the system bus is considered the default in this
document.
Because the order in which external devices are connected to the interrupt controller is flexible
in the APIC architecture, but fixed in the PC/AT PIC architecture, there can be deviations in
device-to-IRQ mappings between the PIC and APIC interrupt controllers. To handle this, ACPI
provides information about how the PIC compatible interrupts (IRQ0 to IRQ15) map to ACPI's GSIs in
the MADT, which has to be taken into account by the os when operating with an APIC. The mapping
information for a single interrupt will be called \textbf{\gls{irq override}}.
There have been different revisions on the APIC architecture after the original, discrete APIC,
notably the \textbf{\gls{xapic}} and \textbf{\gls{x2apic}} architectures. This thesis is mostly
concerned about the older xApic architecture\footnote{The x2Apic architecture is backwards
compatible to the xApic architecture, also, QEMU's TCG does not support the x2Apic architecture.}.
A notable difference between xApic and x2Apic architectures is the register access: xApic uses
32-bit MMIO based register access, while x2Apic uses an MSR based register access, which allows
atomic register access to the 32- and 64-bit wide APIC control MSRs~\cite[sec.~3.11.12]{ia32}.
\section{PC/AT Compatibility}
\label{sec:pcatcompat}
For compatibility to older computer systems, two cascaded PICs are usually present in current
computer systems (see \autoref{fig:discreteapic}/\autoref{fig:integratedapic}). The local APIC can
be initialized to operate with these PICs instead of the I/O APIC, this is called
\textbf{\gls{virtual wire mode}}. It is possible to operate with both the PIC and I/O APIC as
controllers for external interrupts in ``mixed mode'', but this is very uncommon. Because the
presence of a local APIC usually guarantees the presence of an I/O APIC, this thesis only focuses
on interrupt handling with either two PICs (\textbf{\gls{pic mode}}), in case no APIC is available,
or the full APIC architecture (local APIC and I/O APIC in \textbf{\gls{symmetric io
mode}})\footnote{For more information on operating modes, consult the MultiProcessor
specification~\cite[sec.~3.6.2.1]{mpspec}.}.
To switch from PIC to symmetric I/O mode, some\footnote{References to the IMCR are only found in
the relatively old MultiProcessor specification~\cite{mpspec}.} hardware\footnote{QEMU does not
emulate the IMCR.} provides the \textbf{\gls{imcr}}, a special register that controls the physical
connection of the PIC to the CPU. In the original discrete APIC architecture, the IMCR can choose
if either the PIC or local APIC is connected to the CPU (see \autoref{fig:discreteapic}), since the
xApic architecture the IMCR only connects or disconnects the PIC to the local APIC's LINT0 pin (see
\autoref{fig:integratedapic}). When the IMCR is set to connect the PIC, and the xApic ``global
enable/disable'' flag is unset (see \autoref{subsec:lapicenable}), the system is functionally
equivalent to an IA-32 CPU without an integrated local APIC~\cite[sec.~3.11.4.3]{ia32}.
Specifics on handling PC/AT compatible external interrupts follow in \autoref{subsec:ioapicpcat}.
\section{Interrupt Handling in hhuOS}
\label{sec:currenthhuos}
In hhuOS, external interrupts are handled in two steps:
\begin{enumerate}
\item After an IRQ is sent by an interrupt controller, the CPU looks up the interrupt handler address in
the IDT. In hhuOS, every IDT entry contains the address of the \code{dispatch} function (see
\autoref{lst:irqdispatch}), which is invoked with the vector number of the interrupt.
\item The \code{dispatch} function determines which interrupt handler will be called, based on the
supplied vector number. In hhuOS, interrupt handlers are implemented through the
\code{InterruptHandler} interface (see \autoref{lst:irqhandler}), that provides the \code{trigger}
function, which contains the interrupt handling routine. To allow the \code{dispatch} function to
find the desired interrupt handler, it has to be registered to a vector number by the OS
beforehand. This process is handled by the \code{plugin} function of the interrupt handler
interface, which uses the interrupt dispatcher's \code{assign} function (see
\autoref{lst:irqassign}) to register itself to the correct vector number. HhuOS supports assigning
multiple interrupt handlers to a single interrupt vector and cascading interrupts.
\end{enumerate}
\begin{codeblock}[label=lst:irqhandler]{The InterruptHandler interface (InterruptHandler.h)~\cite{hhuos}}{C++}
\cppfile{code/interrupthandler_def.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:irqassign]{Assigning an interrupt handler (InterruptDispatcher.cpp)~\cite{hhuos}}{C++}
\cppfile{code/interruptdispatcher_assign.cpp}
\end{codeblock}
\begin{codeblock}[label=lst:irqdispatch]{Triggering an interrupt handler (InterruptDispatcher.cpp)~\cite{hhuos}}{C++}
\cppfile{code/interruptdispatcher_dispatch.cpp}
\end{codeblock}
To prevent the need of interacting with a specific interrupt controller implementation (e.g.
\code{Pic} class) or the dispatcher, a system service (the \code{InterruptService}) is implemented
to expose this functionality to other parts of the OS (it allows e.g.\ registering interrupt
handlers or masking and unmasking interrupts).
Currently, hhuOS utilizes the PIT to manage the global system time (see \autoref{lst:pithandler}),
which in turn is used to trigger hhuOS's preemptive round-robin scheduler (the \code{Scheduler}
class).
\begin{codeblock}[label=lst:pithandler]{The PIT interrupt handler (Pit.cpp)~\cite{hhuos}}{C++}
\cppfile{code/pit_trigger.cpp}
\end{codeblock}
The PIT and other devices are initialized before the system entry point, in the
\code{System::initializeSystem} function.

107
chap/conclusion.tex Normal file
View File

@ -0,0 +1,107 @@
\chapter{Conclusion}
\label{ch:conclusion}
In this thesis, support for the APIC was integrated into hhuOS. The implementation supports usage
of the local APIC in combination with the I/O APIC for regular interrupt handling without the
PIC\@. Also, the APIC's included timer is used to trigger hhuOS' scheduler, and it is demonstrated
how to initialize a multiprocessor system using the APIC's IPI capabilities. All of this is
implemented with real hardware in mind, so ACPI is used to gather system information during runtime
and adapt the initialization and usage accordingly.
\clearpage
\section{Comparing PIC and APIC Implementations}
\label{sec:comparingpicapic}
Handling interrupts using the PIC is extremely simple, as seen in hhuOS' PIC implementation.
Initialization and usage can be performed by using a total of only four different registers, two
per PIC\footnote{Omitting the infrastructure that is required for both, PIC and APIC, like the IDT
or dispatcher.}.
In comparison, the code complexity required to use the APIC is very high. The most obvious reason
is its significantly increased set of features: Local interrupts are special to the APIC, also the
APIC system is made up of multiple components that require internal communication and individual
initialization. Additionally, the APIC supports advanced processor topologies with large amounts of
cores and offers increased flexibility by allowing to configure individual interrupt's vectors,
destinations, signals and priorities, which results in additional setup.
Another source of complexity that is not present with the PIC is the APIC's variability: With the
PC/AT architecture, the entire hardware interrupt configuration is known before boot. This is not
the case when using the APIC, as the amount of interrupt controllers or even the number and order
of connected devices is only known while the system is running. Parsing this information from ACPI
tables and allowing the implementation to adapt to different hardware configurations, all while
maintaining PC/AT compatibility, increases the amount of code by a large margin.
In general, all of this effort results in a much more powerful and future-proof system: It is not
limited to specific hardware configurations, allows scaling to a large amount of interrupts or
processors, and is required for multiprocessor machines, which equals to almost all modern computer
systems.
\section{Future Improvements}
\label{sec:futureimprov}
\subsection{Dependence on ACPI}
\label{subsec:acpidependance}
The APIC system requires a supplier of system information during operation, but this must not
necessarily be ACPI. Systems following Intel's ``MultiProcessor Specification''~\cite{mpspec} can
acquire the required information by parsing configuration tables similar to ACPI's system
description tables, but provided in accordance to the MultiProcessor Specification. This would
increase the amount of systems supported by this APIC implementation, but the general compatibility
improvement is difficult to quantize, as single-core systems are still supported by the old PIC
implementation\footnote{The MultiProcessor Specification was (pre-) released in
1993~\cite{mpspecpre}, the first ACPI release was in 1996~\cite{acpipre}. Multiprocessor systems
between these years would be affected.}.
Alternatively, systems that do not support ACPI could be supported partially by utilizing the local
APIC's virtual wire mode~\cite[sec.~3.6.2.2]{mpspec}. The reliance on information provided in the
MADT stems mostly from using the I/O APIC as the external interrupt controller. By using the local
APIC for its local interrupt and multiprocessor functionality, but keeping the PC/AT compatible PIC
as the external interrupt controller, multiprocessor systems without ACPI could be supported.
\subsection{Discrete APIC and x2Apic}
\label{subsec:discretex2}
This implementation only supports the xApic architecture, as it's convenient to emulate and the
x2Apic architecture is fully backwards compatible~\cite[sec.~3.11.12]{ia32}. The original, discrete
local APIC (Intel 82489DX) is not supported explicitly, which reduces this implementation's
compatibility to older systems\footnote{Although it is possible that no or only very few
modifications are necessary, as this implementation does not utilize many of the xApic's exclusive
features~\cite[sec.~3.23.27]{ia32}.}. Supporting the x2Apic architecture does not increase
compatibility, but using the x2Apic's MSR-based atomic register access is beneficial in
multiprocessor systems. Newer ACPI versions provide separate x2Apic specific structures in the
MADT~\cite[sec.~5.2.12.12]{acpi65}, so supporting x2Apic also requires supporting modern ACPI
versions\footnote{Currently, hhuOS supports ACPI 1.0b~\cite{acpi1}.}.
\subsection{PCI Devices}
\label{subsec:pcidevices}
The APIC is capable of handling MSIs, but this is not implemented in this thesis. To support PCI
devices using the APIC instead of the PIC, either support for MSIs has to be implemented, or ACPI
has to be used to determine the corresponding I/O APIC interrupt inputs. This is rather
complicated, as it requires interacting with ACPI using AML~\cite[sec.~6.2.13]{acpi65}, which needs
an according interpreter\footnote{There exist modular solutions that can be ported to the target
OS~\cite{acpica}.}.
\subsection{Multiprocessor Systems}
\label{subsec:multiprocessor}
This implementation demonstrates AP startup in SMP systems by using the APIC's IPI mechanism, but
processors are only initialized to a spin-loop. Because the APIC is a prerequisite for more
in-depth SMP support, this implementation enables more substantial improvements in this area, like
distributing tasks to different CPU cores and using the core's local APIC timer to manage a
core-local scheduler.
Another possible area of improvement is the execution of the interrupt handlers: At the moment,
each I/O APIC redirects every received interrupt to the BSP's local APIC. Distributing interrupts
to different CPU cores could improve interrupt handling performance, especially with frequent
interrupts, like from network devices. This redirection could also happen dynamically
(``IRQ-balancing''~\cite{irqbalance}), depending on interrupt workload measurements.
\subsection{UEFI Support}
\label{subsec:uefisupport}
Currently, hhuOS' ACPI system only works with BIOS, so when booting an UEFI system, this
implementation is disabled, because it requires ACPI. By supporting ACPI on UEFI systems, this
implementation could also be used for those systems, which would improve compatibility with modern
platforms.

645
chap/implementation.tex Normal file
View File

@ -0,0 +1,645 @@
\chapter{Implementation}
\label{ch:implementation}
This section will detail how and to what extent the APIC support was implemented. The steps
performed in this implementation are described in a generally applicable way, concrete
implementation details with hhuOS as an example are found in \autoref{ch:listings}.
\autoref{sec:hhuosintegration} deals with the modifications done to the hhuOS system to integrate
the APIC implementation.
\clearpage
\section{Design Decisions and Scope}
\label{sec:design}
The APIC interrupt architecture is split into multiple hardware components and tasks: The
(potentially multiple) local APICs, the (usually single) I/O APIC and the APIC timer (part of each
local APIC). Furthermore, the APIC system needs to interact with its memory mapped registers and
the hhuOS ACPI subsystem, to gather information about the CPU topology and IRQ overrides. Also, the
OS should be able to interact with the APIC system in a simple and easy manner, without needing to
know all of its individual parts.
To keep the whole system structured and simple, the implementation is split into the following main
components (see \autoref{fig:implarch}):
\begin{itemize}
\item \code{LocalApic}: Provides functionality to interact with the local APIC
(masking and unmasking, register access, etc.).
\item \code{IoApic}: Provides functionality to interact with the I/O APIC (masking and
unmasking, register access, etc.)
\item \code{ApicTimer}: Provides functionality to calibrate the APIC timer and handle its interrupts.
\item \code{Apic}: Condenses all the functionality above and exposes it to other parts
of the OS\@.
\end{itemize}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.5\textwidth}
\includesvg[width=1.0\linewidth]{img/Architecture.svg}
\end{subfigure}
\caption{Caller hierarchy of the main components.}
\label{fig:implarch}
\end{figure}
This implementation is targeted to support systems with a single I/O APIC\footnote{Operation with
more than one I/O APIC is described further in the MultiProcessor
specification~\cite[sec.~3.6.8]{mpspec}.}, because consumer hardware typically only uses a single
one, and so does QEMU emulation. General information on implementing multiple I/O APIC support can
be found in \autoref{subsec:multiioapic}.
With the introduction of ACPI's GSIs to the OS, new types are introduced to clearly differentiate
different representations of interrupts and prevent unintentional conversions:
\begin{itemize}
\item \code{GlobalSystemInterrupt}: ACPI's interrupt abstraction, detached from the hardware
interrupt lines.
\item \code{InterruptRequest}: Represents an IRQ, allowing the OS to address interrupts by
the name of the device that triggers it. When the APIC is used, IRQ overrides map IRQs to GSIs.
\item \code{InterruptVector}: Represents an interrupt's vector number, as used by the
\code{InterruptDispatcher}. The dispatcher maps interrupt vectors to interrupt handlers.
\end{itemize}
Both BIOS and UEFI are supported by hhuOS, but the hhuOS ACPI subsystem is currently only available
with BIOS\footnote{State from 11/02/23.}, so when hhuOS is booted using UEFI, APIC support can't be
enabled. Also, the APIC can handle MSIs, but they are not included in this implementation, as hhuOS
currently does not utilize them.
SMP systems are partially supported: The APs are initialized, but only to a busy-looping state, as
hhuOS currently is a single-core OS and lacks some required infrastructure. All interrupts are
handled using the BSP\@.
Summary of features that are outside the scope of this thesis:
\begin{itemize}
\item Operation with a discrete APIC or x2Apic.
\item Interrupts with logical destinations or custom priorities.
\item Returning from APIC operation to PIC mode\footnote{This would be theoretically possible with
single-core hardware, but probably useless.}.
\item Relocation of a local APIC's MMIO memory region\footnote{Relocation is possible by writing a new
physical APIC base address to the IA32\textunderscore{}APIC\textunderscore{}BASE MSR.}.
\item Distributing external interrupts to different APs in SMP enabled systems.
\item Usage of the system's performance counter or monitoring interrupts.
\item Meaningful APIC error handling.
\item Handling of MSIs.
\end{itemize}
To be able to easily extend an APIC implementation for single-core systems to SMP systems, some
things have to be taken into account:
\begin{itemize}
\item SMP systems need to manage multiple \code{LocalApic} and \code{ApicTimer} instances. This is
handled by the \code{Apic} class.
\item Initialization of the different components can no longer happen at the same ``location'': The local
APICs and APIC timers of additional APs have to be initialized by the APs themselves, because the
BSP can not access an AP's registers.
\item APs are only allowed to access instances of APIC classes that belong to them.
\item Interrupt handlers that get called on multiple APs may need to take the current processor into
account (for example the APIC timer interrupt handler).
\item Register access has to be synchronized, if it is performed in multiple operations on the same
address space.
\end{itemize}
\section{Code Style}
\label{sec:codestyle}
Individual state of local and I/O APICs is managed through instances of their respective classes.
Because each CPU core can only access the local APIC contained in itself, this can create a
misconception: It is not possible to (e.g.) allow an interrupt in a certain local APIC by calling a
function on a certain \code{LocalApic} instance. This is communicated through code by declaring a
range of functions as \code{static}. It is also in direct contrast to the \code{IoApic} class: I/O
APICs can be addressed by instances, because they are not part of the CPU core: Each core can
always access all I/O APICs (if there are multiple).
Error checking is done to a small extent in this implementation: Publicly exposed functions (from
the \code{Apic} class) do check for invalid invocations, but the internally used classes do not
protect their invariants, because they are not used directly by other parts of the OS. These
classes only selectively expose their interfaces (by using the \code{friend} declaration) for the
same reason.
\clearpage
\section{Local APIC}
\label{sec:lapicinit}
The local APIC block diagram (see \autoref{fig:localapicblock}) shows a number of registers that
are required for initialization:
\begin{itemize}
\item \textbf{\gls{lvt}}: Used to configure how local interrupts are handled.
\item \textbf{\gls{svr}}: Contains the software-enable flag and the spurious interrupt vector number.
\item \textbf{\gls{tpr}}: Decides the order in which interrupts are handled and a possible interrupt
priority threshold, to ignore low priority interrupts.
\item \textbf{\gls{icr}}: Controls the sending of IPIs for starting up APs in SMP enabled systems.
\item \textbf{\gls{apr}}: Controls the priority in the APIC arbitration mechanism.
\end{itemize}
There are multiple registers associated with the LVT, those belong to the different local
interrupts the local APIC can handle. Local interrupts this implementation is concerned about are
listed below:
\begin{itemize}
\item LINT1: The local APIC's NMI source.
\item Timer: Periodic interrupt triggered by the APIC timer.
\item Error: Triggered by errors in the APIC system (e.g.\ invalid vector numbers or corrupted messages
in internal APIC communication).
\end{itemize}
The LINT0 interrupt is unlisted, because it is mainly important for virtual wire mode (it can be
triggered by external interrupts from the PIC). The performance and thermal monitoring interrupts
also remain unused in this implementation.
Besides the local APIC's own registers, the IMCR and \textbf{\gls{ia32 apic base msr}} also require
initialization (described in \autoref{subsec:lapicenable}).
After system power-up, the local APIC is in the following state~\cite[sec.~3.11.4.7]{ia32}:
\begin{itemize}
\item IRR, ISR and TPR are reset to \code{0x00000000}.
\item The LVT is reset to \code{0x00000000}, except for the masking bits (all local interrupts are masked
on power-up).
\item The SVR is reset to \code{0x000000FF}.
\item The APIC is in xApic mode, even if x2Apic support is present.
\item Only the BSP is enabled, other APs have to be enabled by the BSP's local APIC\@.
\end{itemize}
The initialization sequence consists of these steps:
\begin{enumerate}
\item Enable symmetric I/O mode and set the APIC operating mode.
\item Initialize the LVT and NMI\@.
\item Initialize the SVR\@.
\item Clear outstanding signals.
\item Initialize the TPR\@.
\item Initialize the APIC timer and error handler.
\item Startup the APs and initialize their respective local APICs.
\item Synchronize the APRs.
\end{enumerate}
\subsection{Accessing Local APIC Registers in xApic Mode}
\label{subsec:xapicregacc}
Accessing registers in xApic mode is done via MMIO and requires a single page (4 KB) of memory,
mapped to the ``APIC Base'' address, which can be obtained by reading the
IA32\textunderscore{}APIC\textunderscore{}BASE MSR (see
\autoref{fig:ia32apicbasemsr}/\autoref{tab:lapicregsmsr}) or parsing the MADT (see
\autoref{tab:madt}). The IA-32 manual specifies a caching strategy of ``strong
uncachable''\footnote{See IA-32 manual for information on this caching
strategy~\cite[sec.~3.12.3]{ia32}.}~\cite[sec.~3.11.4.1]{ia32} for this region (see
\autoref{sec:apxxapicregacc} for the example implementation).
The address offsets (from the base address) for the local APIC registers are listed in the IA-32
manual~\cite[sec.~3.11.4.1]{ia32} and in \autoref{tab:lapicregs}.
\subsection{Enabling the Local APIC}
\label{subsec:lapicenable}
The following steps have to be executed before any interrupt handling has been enabled by the OS\@.
Because the system boots in PIC mode, \code{0x01} should be written to the
IMCR~\cite[sec.~3.6.2.1]{mpspec} to disconnect the PIC from the local APIC's LINT0 pin (see
\autoref{fig:integratedapic}, for the example implementation see \autoref{sec:apxdisablepic}).
To set the operating mode, it is first determined which modes are supported by executing the
\code{cpuid} instruction with \code{eax=1}: If bit 9 of the \code{edx} register is set, xApic mode
is supported, if bit 21 of the \code{ecx} register is set, x2Apic mode is
supported~\cite[sec.~5.1.2]{cpuid}.
If xApic mode is supported (if the local APIC is an integrated APIC), it will be in that mode
already. The ``global enable/disable'' (``EN'') bit in the
IA32\textunderscore{}APIC\textunderscore{}BASE MSR (see
\autoref{fig:ia32apicbasemsr}/\autoref{tab:lapicregsmsr}) allows disabling xApic mode, and thus the
entire local APIC, globally\footnote{If the system uses the discrete APIC bus, xApic mode cannot be
re-enabled without a system reset~\cite[sec.~3.11.4.3]{ia32}.}.
Enabling x2Apic mode is done by setting the ``EXTD'' bit (see
\autoref{fig:ia32apicbasemsr}/\autoref{tab:lapicregsmsr}) of the
IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\cite[sec.~3.11.4.3]{ia32}.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_apic_base_msr.svg}
\end{subfigure}
\caption{The IA32\textunderscore{}APIC\textunderscore{}BASE MSR~\cite[sec.~3.11.4.4]{ia32}.}
\label{fig:ia32apicbasemsr}
\end{figure}
Because QEMU does not support x2Apic mode via its TCG\footnote{QEMU Tiny Code Generator transforms
target instructions to instructions for the host CPU.}, this implementation only uses xApic mode.
Besides the ``global enable/disable'' (``EN'') flag, the APIC can also be enabled/disabled by using
the ``software enable/disable'' flag in the SVR, see \autoref{subsec:lapicsoftenable}.
\subsection{Handling Local Interrupts}
\label{subsec:lapiclvtinit}
To configure how the local APIC handles the different local interrupts, the LVT registers are
written (see \autoref{fig:localapiclvt}).
Local interrupts can be configured to use different delivery modes
(excerpt)~\cite[sec.~3.11.5.1]{ia32}:
\begin{itemize}
\item Fixed: Simply delivers the interrupt vector stored in the LVT register.
\item NMI: Configures this interrupt as non-maskable, this ignores the stored vector number.
\item ExtINT: The interrupt is treated as an external interrupt (instead of local interrupt), used e.g.\
in virtual wire mode for LINT0.
\end{itemize}
Initially, all local interrupts are initialized to PC/AT defaults: Masked, edge-triggered,
active-high and with ``fixed'' delivery mode. Most importantly, the vector fields (bits 0:7 of an
LVT register) are set to their corresponding interrupt vector (see \autoref{sec:apxlvtinit} for an
example implementation).
After default-initializing the local interrupts, LINT1 has to be configured separately (see
\autoref{tab:lapicregslvtlint}), because it is the local APIC's NMI source\footnote{In older
specifications~\cite{mpspec}, LINT1 is defined as NMI source (see \autoref{fig:integratedapic}). It
is possible that this changed with newer architectures, so for increased compatibility this
implementation configures the local APIC NMI source as reported by ACPI. It is also possible that
ACPI reports information on the NMI source just for future-proofing.}. The delivery mode is set to
``NMI'' and the interrupt vector to \code{0x00}. This information is also provided by ACPI in the
MADT (see \autoref{tab:madtlnmi}). Other local interrupts (APIC timer and the error interrupt) will
be configured later (see \autoref{subsec:lapictimer} and \autoref{subsec:lapicerror}).
\subsection{Allowing Interrupt Processing}
\label{subsec:lapicsoftenable}
To complete a minimal local APIC initialization, the ``software enable/disable'' flag and the
spurious interrupt vector (both contained in the SVR, see
\autoref{fig:ia32apicsvr}/\autoref{tab:lapicregssvr}), are set. It makes sense to choose
\code{0xFF} for the spurious interrupt vector, as on P6 and Pentium processors, the lowest 4 bits
must be set (see \autoref{fig:ia32apicsvr}).
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_apic_svr.svg}
\end{subfigure}
\caption{The local APIC SVR register~\cite[sec.~3.11.9]{ia32}.}
\label{fig:ia32apicsvr}
\end{figure}
Because the APIC's spurious interrupt has a dedicated interrupt vector (unlike the PIC's spurious
interrupt), it can be ignored easily by registering a stub interrupt handler for the appropriate
vector (see \autoref{sec:apxsvr} for an implementation example).
The final step to initialize the BSP's local APIC is to allow the local APIC to receive interrupts
of all priorities. This is done by writing \code{0x00} to the TPR~\cite[sec.~3.11.8.3]{ia32} (see
\autoref{tab:lapicregstpr}). By configuring the TPRs of different local APICs to different
priorities or priority classes, distribution of external interrupts to CPUs can be controlled, but
this is not used in this thesis.
\subsection{Local Interrupt EOI}
\label{subsec:lapiceoi}
To notify the local APIC that a local interrupt has been handled, its EOI register (see
\autoref{tab:lapicregseoi}) has to be written. Not all local interrupts require EOIs: NMI, SMI,
INIT, ExtINT, STARTUP, or INIT-Deassert interrupts are excluded~\cite[sec.~3.11.8.5]{ia32}.
EOIs for external interrupts are also handled by the local APIC, this is described in
\autoref{subsec:ioapiceoi}.
\subsection{APIC Timer}
\label{subsec:lapictimer}
The APIC timer is integrated into the local APIC, so it requires initialization of the latter. Like
the PIT, the APIC timer can generate periodic interrupts in a specified interval by using a
counter, that is initialized with a starting value depending on the desired interval. Because the
APIC timer doesn't tick with a fixed frequency, but at bus frequency, the initial counter has to be
determined at runtime by using an external time source. In addition to the counter register, the
APIC timer interval is influenced by a divider: Instead of decrementing the counter at every bus
clock, it will be decremented every \(n\)-th bus clock, where \(n\) is the divider. This is useful
to allow for long intervals (with decreased precision), that would require a larger counter
register otherwise.
The APIC timer supports three different timer modes, that can be set in the timer's LVT register:
\begin{enumerate}
\item Oneshot: Trigger exactly one interrupt when the counter reaches zero.
\item Periodic: Trigger an interrupt each time the counter reaches zero, on zero the counter reloads its
initial value.
\item TSC-Deadline: Trigger exactly one interrupt at an absolute time.
\end{enumerate}
This implementation uses the APIC timer in periodic mode, to trigger the scheduler preemption.
Initialization requires the following steps (order recommended by OSDev~\cite{osdev}):
\begin{enumerate}
\item Measure the timer frequency with an external time source.
\item Configuration of the timer's divider register (see \autoref{tab:lapicregstimerdiv}).
\item Setting the timer mode to periodic (see \autoref{tab:lapicregslvtt}).
\item Initializing the counter register (see \autoref{tab:lapicregstimerinit}), depending on the measured
timer frequency and the desired interval.
\end{enumerate}
In this implementation, the APIC timer is calibrated by counting the amount of ticks in one
millisecond using oneshot mode (see \autoref{sec:apxapictimer} for an example implementation). The
measured amount of timer ticks can then be used to calculate the required counter for an arbitrary
millisecond interval, although very large intervals could require the use of a larger divider,
while very small intervals (in micro- or nanosecond scale) could require the opposite, to provide
the necessary precision. For this approach it is important that the timer is initialized with the
same divider that was used during calibration.
To use the timer, an interrupt handler has to be registered to its interrupt vector (see
\autoref{sec:apxapictimer} for an example implementation).
\subsection{APIC Error Interrupt}
\label{subsec:lapicerror}
Errors can occur for example when the local APIC receives an invalid vector number, or an APIC
message gets corrupted on the system bus. To handle these cases, the local APIC provides the local
error interrupt, whose interrupt handler can read the error status from the local APIC's
\textbf{\gls{esr}} (see \autoref{fig:ia32esr}/\autoref{tab:lapicregsesr}) and take appropriate
action.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_error_status_register.svg}
\end{subfigure}
\caption{Error Status Register~\cite[sec.~3.11.5.3]{ia32}.}
\label{fig:ia32esr}
\end{figure}
The ESR is a ``write/read'' register: Before reading a value from the ESR, it has to be written,
which updates the ESR's contents to the error status since the last write. Writing the ESR also
arms the local error interrupt again~\cite[sec.~3.11.5.3]{ia32}.
Enabling the local error interrupt is now as simple as enabling it in the local APIC's LVT and
registering an interrupt handler for the appropriate vector (see \autoref{sec:apxhandlingerror} for
an example implementation).
\clearpage
\section{I/O APIC}
\label{sec:ioapicinit}
% TODO: Continue moving code to the appendix from here on
To fully replace the PIC and handle external interrupts using the APIC, the I/O APIC, located in
the system chipset, has to be initialized by setting its \textbf{\gls{redtbl}}
registers~\cite[sec.~9.5.8]{ich5} (see \autoref{tab:ioapicregsredtbl}). Like the local APIC's LVT,
the REDTBL allows configuration of interrupt vectors, masking bits, interrupt delivery modes, pin
polarities and trigger modes (see \autoref{subsec:lapiclvtinit}).
Additionally, for external interrupts a destination and destination mode can be specified. This is
required because the I/O APIC is able to forward external interrupts to different local APICs over
the system bus (see \autoref{fig:integratedapic}). SMP systems use this mechanism to distribute
external interrupts to different CPU cores for performance benefits. Because this implementation's
focus is not on SMP, all external interrupts are default initialized to ``physical'' destination
mode\footnote{The alternative is "logical" destination mode, which allows addressing individual or
clusters of local APIC's in a larger volume of
processors~\cite[sec.~3.11.6.2.2]{ia32}.}~\cite[sec.~3.11.6.2.1]{ia32} and are sent to the BSP for
servicing, by using the BSP's local APIC ID as the destination. The other fields are set to
\textbf{\gls{isa}} bus defaults\footnote{Edge-triggered, active-high.}, with ``fixed'' delivery
mode, masked, and the corresponding interrupt vector, as defined by the \code{InterruptVector}
enum.
The I/O APIC does not have to be enabled explicitly, if the local APIC is enabled and the REDTBL is
initialized correctly, external interrupts will be redirected to the local APIC and handled by the
CPU\@.
Unlike the local APIC's registers, the REDTBL registers are accessed indirectly: Two registers, the
``Index'' and ``Data'' register~\cite[sec.~9.5.1]{ich5}, are mapped to the main memory and can be
used analogous to the local APIC's registers. The MMIO base address can be parsed from the MADT
(see \autoref{tab:madtioapic}). Writing an offset to the index register exposes an indirectly
accessible I/O APIC register through the data register (see \autoref{sec:iolistings} for an example
implementation). This indirect addressing scheme is useful, because the number of external
interrupts an I/O APIC supports, and in turn the number of REDTBL registers, can
vary\footnote{Intel's consumer \textbf{\glspl{ich}} always support a fixed amount of 24 external
interrupts though~\cite[sec.~9.5.7]{ich5}.}.
It is possible that one or multiple of the I/O APIC's interrupt inputs act as an NMI source. If
this is the case is reported in the MADT (see \autoref{tab:madtionmi}), so when necessary, the
corresponding REDTBL entries are initialized like the local APIC's NMI source (see
\autoref{subsec:lapiclvtinit}), and using these interrupt inputs for external interrupts is
forbidden.
\subsection{Interrupt Overrides}
\label{subsec:ioapicpcat}
In every PC/AT compatible system, external devices are hardwired to the PIC in the same order.
Because this is not the case for the I/O APIC, the interrupt line used by each PC/AT compatible
interrupt has to be determined by the OS at runtime, by using ACPI. ACPI provides ``Interrupt
Source Override'' structures~\cite[sec.~5.2.8.3.1]{acpi1} inside the MADT (see
\autoref{tab:madtirqoverride}) for each PC/AT compatible interrupt that is mapped differently to
the I/O APIC than to the PIC\@.
In addition to the interrupt input mapping, these structures also allow to customize the pin
polarity and trigger mode of PC/AT compatible interrupts.
This information does not only apply to the REDTBL initialization, but it has to be taken into
account every time an action is performed on a PC/AT compatible interrupt, like masking or
unmasking: If \code{IRQ0} (PIT) should be unmasked, it has to be determined what GSI (or in other
words, I/O APIC interrupt input) it belongs to. In many systems \code{IRQ0} is mapped to
\code{GSI2}, because the PC/AT compatible PICs are connected to \code{GSI0}. Thus, to allow the PIT
interrupt in those systems, the REDTBL entry belonging to \code{GSI2} instead of \code{GSI0} has to
be written (see \autoref{sec:apxirqoverrides} for an example implementation).
\subsection{External Interrupt EOI}
\label{subsec:ioapiceoi}
Notifying the I/O APIC that an external interrupt has been handled differs depending on the
interrupt trigger mode: Edge-triggered external interrupts are completed by writing the local
APIC's EOI register (see \autoref{subsec:lapiceoi})\footnote{Because external interrupts are
forwarded to the local APIC, the local APIC is responsible for tracking them in its IRR and ISR.}.
Level-triggered interrupts are treated separately: Upon registering a level-triggered external
interrupt, the I/O APIC sets an internal ``Remote IRR'' bit in the corresponding REDTBL
entry~\cite[sec.~9.5.8]{ich5} (see \autoref{tab:ioapicregsredtbl}).
There are three possible ways to signal completion of a level-triggered external interrupt to clear
the remote IRR bit:
\begin{enumerate}
\item Using the local APIC's EOI broadcasting feature: If EOI broadcasting is enabled, writing the local
APIC's EOI register also triggers EOIs for each I/O APIC (for the appropriate interrupt), which
clears the remote IRR bit.
\item Sending a directed EOI to an I/O APIC: I/O APICs with versions greater than \code{0x20} include an
I/O EOI register. Writing the vector number of the handled interrupt to this register clears the
remote IRR bit.
\item Simulating a directed EOI for I/O APICs with versions smaller than \code{0x20}: Temporarily masking
and setting a completed interrupt as edge-triggered clears the remote IRR
bit~\cite[io\textunderscore{}apic.c]{linux}.
\end{enumerate}
Because the first option is the only one supported by all APIC versions, it is used in this
implementation\footnote{Disabling EOI broadcasting is not supported by all local
APICs~\cite[sec.~3.11.8.5]{ia32}.}.
At this point, after initializing the local and I/O APIC for the BSP, the APIC system is fully
usable. External interrupts now have to be enabled/disabled by writing the ``masked'' bit in these
interrupts' REDTBL entries, interrupt handler completion is signaled by writing the local APIC's
EOI register, and spurious interrupts are detected by using the local APIC's spurious interrupt
vector.
\subsection{Multiple I/O APICs}
\label{subsec:multiioapic}
Most consumer hardware, for example all IA processors~\cite{ia32} and ICH hubs~\cite{ich5}, only
provide a single I/O APIC, although technically multiple I/O APICs are supported by the
MultiProcessor specification~\cite[sec.~3.6.8]{mpspec}.
If ACPI reports multiple I/O APICs (by supplying multiple MADT I/O APIC structures, see
\autoref{tab:madtioapic}), the previously described initialization has to be performed for each I/O
APIC individually. Additionally, the I/O APIC's ID, also reported by ACPI, has to be written to the
corresponding I/O APIC's ID register (see \autoref{tab:ioapicregsid}), because this register is
always initialized to zero~\cite[sec.~9.5.6]{ich5}.
Using a variable number of I/O APICs requires determining the target I/O APIC for each operation
that concerns a GSI, like masking or unmasking. For this reason, ACPI provides the ``GSI
Base''~\cite[sec.~5.2.8.2]{acpi1} for each available I/O APIC, the number of GSIs a single I/O APIC
can handle can be determined by reading the I/O APIC's version register~\cite[sec.~9.5.7]{ich5}
(see \autoref{tab:ioapicregsver})\footnote{This approach was previously used in this
implementation, but removed for simplicity.}.
\clearpage
\section{Symmetric Multiprocessing}
\label{sec:smpinit}
Like single-core systems, SMP systems boot using only a single core, the BSP. By using the APIC's
capabilities to send IPIs between cores, additional APs can be put into startup state and booted
for system use.
To determine the amount of usable processors, the MADT is parsed (see \autoref{tab:madtlapic}).
Note, that some processors may be reported as disabled, those may not be used by the OS (see
\autoref{tab:madtlapicflags}).
\subsection{Inter-Processor Interrupts}
\label{subsec:ipis}
Issuing IPIs works by writing the local APIC's ICR (see
\autoref{fig:ia32icr}/\autoref{tab:lapicregsicr}). It allows specifying IPI type, destination
(analogous to REDTBL destinations, see \autoref{sec:ioapicinit}) and vector (see
\autoref{sec:apxipis} for an example implementation).
Depending on the APIC architecture, two different IPIs are required: The INIT IPI for systems using
a discrete APIC, and the \textbf{\gls{sipi}} for systems using the xApic or x2Apic architectures:
\begin{itemize}
\item The INIT IPI causes an AP to reset its state and start executing at the address specified at its
system reset vector. If paired with a system warm-reset, the AP can be instructed to start
executing the AP boot sequence by writing the appropriate address to the warm-reset
vector~\cite[sec.~B.4.1]{mpspec}.
\item Since the xApic architecture, the SIPI is used for AP startup: It causes the AP to start executing
code in real mode, at a page specified in the IPIs interrupt vector~\cite[sec.~B.4.2]{mpspec}. By
copying the AP boot routine to a page in lower physical memory, and sending the SIPI with the
correct page number, an AP can be booted.
\end{itemize}
To wait until the IPI is sent, the ICR's delivery status bit can be polled.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.7\textwidth}
\includesvg[width=1.0\linewidth]{img/ia32_interrupt_command_register.svg}
\end{subfigure}
\caption{Interrupt Command Register~\cite[sec.~3.11.6.1]{ia32}.}
\label{fig:ia32icr}
\end{figure}
\subsection{Universal Startup Algorithm}
\label{subsec:apstartup}
SMP initialization is performed differently on various processors. Intel's MultiProcessor
specification defines a ``universal startup algorithm'' for multiprocessor
systems~\cite[sec.~B.4]{mpspec}, which can be used to boot SMP systems with either discrete APIC,
xApic or x2Apic, as it issues both, INIT IPI and SIPI\footnote{Technically, it always issues the
INIT IPI, and the SIPI only for xApic or x2Apic, but since the SIPI is ignored by discrete APICs,
it can be sent either way. This ``INIT-SIPI-SIPI'' sequence is also stated in the IA-32
manual~\cite[sec.~3.9.4]{ia32}.}.
This algorithm has some prerequisites: It is required to copy the AP boot routine (detailed in
\autoref{subsec:apboot}) to lower memory, where the APs will start their execution. Also, the APs
need allocated stack memory to call the entry function, and in case of a discrete APIC that uses
the INIT IPI, the system needs to be configured for a warm-reset (by writing \code{0xAH} to the
CMOS shutdown status byte, located at \code{0xF}~\cite[sec.~B.4]{mpspec}), because the INIT IPI
does not support supplying the address where AP execution should begin, unlike the SIPI. The
warm-reset vector (a 32-bit field, located at physical address
\code{40:67}~\cite[sec.~B.4]{mpspec}) needs to be set to the physical address the AP startup
routine was copied to. Additionally, the entire AP startup procedure has to be performed with all
sources of interrupts disabled, which offers a small challenge, since some timings need to be taken
into account\footnote{This implementation uses the PIT's mode 0 on channel 0 for timekeeping.}.
The usage of delays in the algorithm is quite specific, but the specification provides no further
information on the importance of these timings or required precision. The algorithm allowed for
successful startup of additional APs when tested in QEMU (with and without KVM) and on certain real
hardware, although for different processors or emulators (like Bochs), different timings might be
required~\cite[lapic.c]{xv6}.
After preparation, the universal startup algorithm is now performed as follows, for each AP
sequentially (see \autoref{sec:apxmpusa} for an example implementation):
\begin{enumerate}
\item Assert and de-assert the level-triggered INIT IPI\@.
\item Delay for 10 milliseconds.
\item Send the SIPI\@.
\item Delay for 200 microseconds.
\item Send the SIPI again.
\item Delay for 200 microseconds again.
\item Wait until the AP has signaled boot completion, then continue to the next.
\end{enumerate}
If the system uses a discrete APIC, the APs will reach the boot routine by starting execution at
the location specified in the warm-reset vector, if the system uses the xApic or x2Apic
architecture, the APs will reach the boot routine because its location was specified in the SIPI\@.
Signaling boot completion from the APs entry function can be done by using a global bitmap
variable, where the \(n\)-th bit indicates the running state of the \(n\)-th processor. This
variable does not have to be synchronized across APs, because the startup is performed
sequentially.
\subsection{Application Processor Boot Routine}
\label{subsec:apboot}
After executing the ``INIT-SIPI-SIPI'' sequence, the targeted AP will start executing its boot
routine in real mode. The general steps required are similar to those required when booting a
single-core system, but since the BSP in SMP systems is already fully operational at this point,
much can be recycled. The AP boot routine this implementation uses can be roughly described as
follows (see \autoref{sec:apxapboot} for an example implementation):
\begin{enumerate}
\item Load a temporary \textbf{\gls{gdt}}, used for switching to protected mode.
\item Enable protected mode by writing \code{cr0}.
\item Far jump to switch to protected mode and reload the code-segment register, set up the other
segments manually.
\item Load the \code{cr3}, \code{cr0} and \code{cr4} values used by the BSP to enable paging (in that
order).
\item Load the IDT used by the BSP\@.
\item Determine the AP's APIC ID by using CPUID\@.
\item Load the GDT and \textbf{\gls{tss}} prepared for this AP\@.
\item Load the stack prepared for this AP\@.
\item Call the (C++) AP entry function.
\end{enumerate}
The APIC ID is used to determine which GDT and stack were prepared for a certain AP\@. It is
necessary for each AP to have its own GDT, because each processor needs its own TSS for context
switching, for example when interrupt-based system calls are used on all CPUs.
Because it is relocated into lower physical memory (in this implementation to \code{0x8000}), this
code has to be position independent. For this reason, absolute physical addresses have to be used
when jumping, loading the IDTR and GDTR, or referencing variables. Also, any variables required
during boot have to be available after relocation, this can be achieved by locating them inside the
``TEXT'' section of the routine, so they stay with the rest of the instructions when copying. These
variables have to be initialized during runtime, before the routine is copied (see
\autoref{sec:apxpreparesmp} for an example implementation).
\subsection{Application Processor Post-Boot Routine}
\label{subsec:apsystementry}
In the entry function, called at the end of the boot routine, the AP signals boot completion as
described in \autoref{subsec:apstartup} and initializes its local APIC by repeating the necessary
steps from \autoref{subsec:lapiclvtinit}, \autoref{subsec:lapicsoftenable},
\autoref{subsec:lapictimer} and \autoref{subsec:lapicerror}\footnote{MMIO memory does not have to
be allocated again, as all local APICs use the same memory region in this implementation. Also, the
initial value for the APIC timer's counter can be reused, if already calibrated.}.
Because multiple local APICs are present and active in the system now, the possibility arises that
a certain local APIC receives multiple messages from different local APICs at a similar time. To
decide the order of handling these messages, an arbitration mechanism based on the local APIC's ID
is used~\cite[sec.~3.11.7]{ia32}. To make sure the arbitration priority matches the local APIC's
ID, the ARPs can be synchronized by issuing an INIT-level-deassert IPI\footnote{This is not
supported on Pentium 4 and Xeon processors.} (see \autoref{sec:apxappostboot} for an example
implementation).
\clearpage

26
chap/introduction.tex Normal file
View File

@ -0,0 +1,26 @@
\chapter{Introduction}
\label{ch:introduction}
Computer systems are very useful, because they are able to interact with the ``outside world'', for
instance by reading values from sensors, controlling external appliances or interacting with a user
through human interface devices. In each of these scenarios, the system's CPU has to react to
``external changes'', like a key press or sensor reading. An efficient hardware solution to this
problem are ``interrupts''.
In this thesis, support for the ``APIC'', a modern and widely used interrupt controller
architecture, introduced by Intel for the Pentium 4 processor, will be implemented into hhuOS, ``A
small operating system for learning purposes''~\cite{hhuos}. This support will cover a complete
replacement of the older ``PIC'' interrupt controller, introduction of an alternative timer - a
part of the APIC architecture - for scheduling, and utilizing the APIC to boot multiprocessor
systems.
The following chapter explains important background concepts, in \autoref{ch:implementation} the
required steps to use the APIC and their implementation are explained in general,
\autoref{ch:verification} deals with the verification process of the developed software on emulated
and real hardware, and \autoref{ch:conclusion} draws conclusions regarding the previous
implementation and future improvements.
Specific details on the code created during this thesis are given in \autoref{ch:listings},
separated from the main body.
\clearpage

82
chap/verification.tex Normal file
View File

@ -0,0 +1,82 @@
\chapter{Verification}
\label{ch:verification}
Common techniques for testing software include component-based tests, like ``unit tests'', and
``end-to-end tests'', where the complete functionality of a software system with all its parts is
tested. As unit testing is mostly suitable for independent slices of a system's application logic,
it is not very useful for testing low-level software designed to run directly on hardware devices.
This chapter deals with the process and results of testing hhuOS with the APIC implementation
developed during this thesis.
\clearpage
\section{Methods of Verification}
\label{sec:verificationmethods}
This application can be tested by running the entire hhuOS operating system on both emulated and
real hardware, and monitoring its state of operation\footnote{There is almost no logic that can be
tested isolated or hardware independently.}.
This can be done in multiple ways:
\begin{itemize}
\item When running hhuOS in an emulated environment, the current machine state can be inspected
directly\footnote{QEMU offers the ``QEMU monitor'' to query information about specific hardware
components, like the local or I/O APIC (\code{info lapic} and \code{info pic}).}.
\item To test the functionality of an interrupt controller specifically, stub interrupt handlers can be
used to verify that a specific interrupt (like an IPI) has been registered.
\item By attaching a debugger (very simple for emulated hardware), it can be observed that e.g.\ the
timer interrupt is correctly increasing the timestamp for each running CPU\@.
\item By exposing some of the APIC's internal data through hhuOS' virtual file system, end-to-end
verification can be performed from the system itself, as debugging and monitoring is significantly
more difficult when running on real hardware.
\item Because the interrupt controller is a vital part for many components of an operating system,
observing a successful boot is already a significant indicator for a correct implementation.
\end{itemize}
\section{Results}
\label{sec:verificationresults}
QEMU was used as the main development platform, the implemented features were tested by using the
QEMU monitor. Specifically, QEMU provides the current register state of all local and I/O APICs, by
attaching a debugger to the running operating system correct behavior was observed on the emulated
hardware level:
\begin{itemize}
\item Working local interrupts - verified by observing the local APIC's IRR\@.
\item Propagation of interrupts to the CPU - verified by observing the local APIC's ISR\@.
\item A working local APIC timer - verified by observing the APIC timer's registers.
\end{itemize}
Although hhuOS is developed mainly for learning purposes, every OS' core task remains to be the
management of computer hardware. For this reason this implementation was additionally tested on a
``ThinkPad T60s'' with an Intel ``Core 2 Duo'' processor. By providing internal status data through
the virtual file system, implemented features (including booting additional APs) were verified on
this very specific set of hardware.
More specifically, information about detected and enabled local APICs and the I/O APIC, register
values from the BSP's LVT and the REDTBL, the contents of the PIC's \textbf{\gls{imr}}, and amounts
of occurred interrupts by core (similar to \code{/proc/interrupts} in Linux~\cite{linux}]) are
exposed on the path \code{/device/apic/}. This way, the following things could be verified on real
hardware:
\begin{itemize}
\item The APIC is indeed used instead of the PIC - verified by observing the PIC's IMR\@.
\item Successful MMIO for local and I/O APIC - verified by observing the LVT and REDTBL\@.
\item Devices ``plugging in'' to the APIC instead of the PIC - verified by observing the LVT and
REDTBL\@.
\item Interrupt handlers are called correctly - verified by using the keyboard.
\item Working startup of additional APs - verified by checking the state of the AP's local APICs and
observing the system log.
\item Handling interrupts on another AP - verified by redirecting the keyboard interrupt to another
processor and observing the number of occurred keyboard interrupts on this
processor\footnote{Interestingly, this was easier to verify on real hardware than in QEMU, because
QEMU usually instantly crashes when interrupts are enabled on an AP beside the BSP. Also,
redirecting the keyboard interrupt in QEMU significantly changes the behaviour of any key press and
renders the keyboard unusable. This is expected, as hhuOS' paging is not designed for multiple
processors. On the ThinkPad T60s though, it was possible (surprisingly) to use the redirected
keyboard interrupt to \code{cat} the \code{/device/apic/irqs} file to observe the interrupt
statistics, which showed the keyboard interrupts arriving on a different core than the BSP. After a
certain time, the system crashes on the ThinkPad aswell.}.
\end{itemize}