diff --git a/chapters/02_methodology/02_00_methodology.tex b/chapters/02_methodology/02_00_methodology.tex deleted file mode 100644 index e024f7a..0000000 --- a/chapters/02_methodology/02_00_methodology.tex +++ /dev/null @@ -1,11 +0,0 @@ -%! TeX program = lualatex -%! TeX root = ../../thesis.tex -\documentclass[../../thesis.tex]{subfiles} - -\begin{document} - -\chapter{Methodology}\label{ch:methodology} - -Specific FARM-model variant for this research questions, fault model. - -\end{document} diff --git a/chapters/04_methodology/04_00_methodology.tex b/chapters/04_methodology/04_00_methodology.tex new file mode 100644 index 0000000..728be25 --- /dev/null +++ b/chapters/04_methodology/04_00_methodology.tex @@ -0,0 +1,120 @@ +%! TeX program = lualatex +%! TeX root = ../../thesis.tex +\documentclass[../../thesis.tex]{subfiles} + +\begin{document} + +\chapter{Methodology}\label{ch:methodology} + +\begin{itemize} + \item General: How do the experiments answer the research questions? + \item Specific FARM-model variant for this research questions, fault model. +\end{itemize} + +\section{Experimental Design}\label{sec:experimentaldesign} + +This thesis evaluates the effect of execution mode and software hardening on program behavior under injected faults. +Each selected benchmark is evaluated under native execution, \Gls{aot} compiled \Gls{wasm} execution, and interpreted \Gls{wasm} execution. +Comparisons are performed between corresponding benchmark variants using the same inputs and expected results. + +The execution mode is the factor that influences the experiment the most. +For benchmarks with hardened implementations, the hardening techniques are additional influences.\todo{Open if I get to that, but preferably at least CoRed} +The experiment outcomes describe if execution completes successfully with the expected result, detects an error, or terminates for other reasons, like timeouts. + +The experiments are designed to determine what changes when adding \Gls{wamr} to an unhardened or hardened benchmark, and specifically, what changes between \Gls{wamr}'s \Gls{aot} compiled and interpreted running modes. + +\section{Applying the FARM-Model}\label{sec:applyingthefarmmodel} + +With \Gls{farm}~\autocite{arlatFaultInjectionDependability1990}, Arlat et al.\ introduced a model for applications of \Gls{fi} to clearly specify the faults introduced, the conditions under which the system works, and the interpretation of the observed outcomes. +\Gls{farm} structures its specification into four sets: faults (\(F\)) describe the disturbances introduced into the system, activations (\(A\)) specify the execution conditions of the system, readouts (\(R\)) are the outcomes collected from experiment execution, and measures (\(M\)) finally derive quantities from the observations to evaluate the system's behavior. + +In this thesis, \Gls{farm} provides the framework to specify and compare the native, \Gls{aot}, and interpreted experiment variants and interpret the experiment results. + +\subsection{Faults}\label{ssec:farmfaults} + +The \Gls{fi} campaigns inject single-bit faults into the machine state exposed by the \Gls{bochs} simulator. +Fault locations are separated into memory, register, and instruction-pointer categories that are evaluated separately. +The injection locations and times are determined from a fault-free execution (\gls{goldenrun}) of the selected benchmark. +The evaluated fault space is thus specific to the executed benchmark, its inputs, the execution mode and the chosen traced region. + +\todo[inline]{Write about comparability of results, equivalence classes etc.} + +The fault model excludes multiple simultaneous faults or multi-bit faults. +After an injection, the program execution continues without intervention until an experiment outcome is reached. + +\subsection{Activations}\label{ssec:farmactivations} + +\todo[inline]{This reads more like a list of notes} + +Each benchmark defines a traced region through explicit start and stop markers. +The \gls{goldenrun} records a fault-free execution trajectory of that region and provides the simulator starting state for each injection. +Additionally, the \gls{goldenrun} determines the injection locations. +The trace boundary alone determines which program activity contributes to the experiment. +All benchmarks use equivalent boundaries on the source-level: state initialization and \Gls{wamr} runtime initialization lie outside of this boundary. +In contrast to natively executed benchmark variants, the \Gls{aot} or interpreted variants may call \Gls{wamr} runtime code within the boundary. + +The effects of injected faults are later evaluated through the readouts described in \autoref{ssec:farmreadouts}. + +\subsection{Pruning}\label{ssec:farmpruning} + +Instead of injecting a single-bit fault at every possible location derived from the \gls{goldenrun}, the fault space is pruned before the campaign is run. +Equivalence classes of faults leading to the same outcome are identified and represented by a single \gls{pilot} experiment, so the outcomes of all experiments that belong to an equivalence class can be determined by performing injections on only its \gls{pilot}. +Because injections into different bits can cause different outcomes, only injections into the same bit can belong to a single equivalence class, additionally, there must not be intervening accesses into a memory location within a single equivalence class. + +For example, if a single byte at the memory location \(m\) is written by the program under test at time \(t_{0}\) and read back (without \(m\) being accessed in between) at time \(t_{1}\) with \(t_{1} > t_{0}\), each injection into bit \(b\) of \(m\) during the time interval \([t_{0}, t_{1}]\) would cause the same outcome (multiple simultaneous faults are outside the scope of this thesis). +Instead of performing \(t_{1} - t_{0} + 1\) individual injections into bit \(b\) of \(m\), the pruner marks \([t_{0}, t_{1}]\) as an equivalence class, represented by a single injection into bit \(b\) of \(m\) at time \(t_{1}\) (or any other \(t\in [t_{0}, t_{1}]\)). + +This example is a case of \textquote{def/use pruning}, as introduced by Smith et al.~\autocite{smithMethodDetermineEquivalent1995} and Güthoff and Sieh~\autocite{guthoffCombiningSoftwareimplementedSimulationbased1995}. + +\subsection{Readouts}\label{ssec:farmreadouts} + +Outcomes are classified by using explicit markers on the source level and listening for simulator events (possible outcomes are listed in \autoref{tab:experimentoutcomes}). +The positive and negative markers report the result of the benchmark's success condition outside the trace boundary, the detected marker records a fault that manifested to an error but was detected by the program. +In \Gls{wamr} benchmark variants, \Gls{wamr}'s exception handler reports an explicit outcome as well. +Traps, timeouts and invalid memory accesses are reported by the simulator without the need for source-level instrumentation. + +\todo[inline]{Table is duplicated with background chapter on FAIL} + +\begin{table}[H] + \centering + \begin{tabularx}{\textwidth}{lX} + Marker Type & Description \\ + \midrule + \code{OK\_MARKER} & The experiment's success condition passed \\ + \code{FAIL\_MARKER} & The experiment's success condition did not pass \\ + \code{DETECTED\_MARKER} & The experiment signalled an error through some detection mechanism \\ + \code{TRAP} & The experiment did not complete because the CPU threw an exception \\ + \code{TIMEOUT} & The experiment did not complete in the time limit \\ + \code{ACCESS\_OUTERSPACE} & The program accessed an address outside the program's address space \\ + \code{WRITE\_TEXTSEGMENT} & The program wrote to an address located inside the program's text segment \\ + \code{GROUP1\_MARKER} & An \Gls{aot} or interpreted variant reached \Gls{wamr}'s internal exception handler \\ + \end{tabularx} + \caption{Outcomes detected by \Gls{fail}'s experiment runner.}\label{tab:experimentoutcomes} +\end{table} + +\subsection{Measures}\label{ssec:farmmeasures} + +To quantify the experiment outcomes, the evaluation must reconstruct weighted outcome totals using the \gls{pilot} outcomes with the corresponding sizes of the equivalence classes' fault spaces. +The intuitive reason for this is, as Schirmeier et al.\ state in~\autocite{schirmeierAvoidingPitfallsFaultInjection2015}, that \textquote{the longer data lives in a memory cell, the more probable a soft error will affect it}. + +Continuing the example from \autoref{ssec:farmpruning}, an outcome observed for an injection into a particular bit of a memory location \(m\) represents \(t_{1} - t_{0} + 1\) possible injection times within the equivalence class. +Its contribution to the weighted total for that outcome is therefore +\[ + w = t_{1} - t_{0} + 1. +\] +The weighting is applied separately to each injected bit because injections into different bits of \(m\) could produce different outcomes. +For example, if \(m\) is one byte wide and injections into three of its eight bits produce \code{FAIL\_MARKER}, this interval contributes \(3w\) to the weighted \code{FAIL\_MARKER} total (and \(8w\) to the total weight across all outcomes). + +Generally, if \(C\) is the set of equivalence classes for a benchmark variant and outcome, \(s_{c}\) and \(e_{c}\) are the start and end times of each class \(c\in C\), \(B_{c}\) is the set of injected bits, and \(o_{c,b}\) is the outcome corresponding to the injection into bit \(b\in B_{c}\), then the weighted total for an outcome \(o\) is +\[ + W_o = \sum\limits_{c \in C} \sum\limits_{b \in B_c} (e_c - s_c + 1)\,\mathbf{1}\{o_{c,b} = o\}, +\] +where \(\mathbf{1}\{o_{c,b} = o\}\) is the indicator function, which equals one if the recorded outcome matches \(o\), and zero otherwise.\footnote{% + In terms of \Gls{fail}'s database, this corresponds to summing \code{t.time2 - t.time1 + 1} after joining the trace intervals to their \gls{pilot} results and grouping them by outcome. + The join accounts for the individual injected bits, so no additional factor for the bit width is required in the sum. +} + +Weighted outcome totals are reported separately for each fault category, execution mode and injection location (memory, registers or instruction-pointer). +Outcome shares describe the composition of each variant's fault space, while absolute totals show differences in vulnerability caused by executaion duration and memory footprint. + +\end{document}