shutdown cleanups revisited

This change became necessary as we observed weird fail-client SIGSEGV
crashes with both Bochs and Gem5 backends and different experiments.

Some Fail* components are instantiated statically: the
SimulatorController instance "simulator", containing the
ListenerManager and the CoroutineManager, and the active
ExperimentFlow subclass(es)
(experiments/instantiate-experiment*.ah.in).  The experiment(s) is
registered as an active flow in the CoroutineManager at startup.

As plugins (which are ExperimentFlows themselves) are often created on
an experiment's stack, ExperimentFlows deregister themselves on
destruction (e.g., when leaving the plugin variable's scope).  The
core problem is, that the creation and destruction order of statically
instantiated objects depends on the link order; if the experiment is
destroyed after the CoroutineManager, its automatic self-deregistering
feature talks to the smoking ruins of the latter.

This change removes all static instantiations of ExperimentFlow and
replaces them with constructions on the heap.  Additionally it makes
sure that the CoroutineManager recognizes that a shutdown is in
progress, and refrains from touching potentially already destroyed
data structures when a (mistakenly globally instantiated)
ExperimentFlow deregisters in this case.

Change-Id: I8a7d42fb141222cd2cce6040ab1a01f9de61be24

src/experiments/ecos_kernel_test/instantiateExperiment.cc

@@ -1,8 +1,7 @@
 #include "experiment.hpp"
 #include "sal/SALInst.hpp"
 
-static EcosKernelTestExperiment experiment;
 void instantiateEcosKernelTestExperiment()
 {
-	fail::simulator.addFlow(&experiment);
+	fail::simulator.addFlow(new EcosKernelTestExperiment);
 }

src/experiments/instantiate-experiment-indirect.ah.in

@@ -12,6 +12,10 @@
 // You need to provide the implementation of this function in your experiment
 // directory:
 void instantiate@EXPERIMENT_TYPE@();
+// The experiment needs to be instantiated dynamically (on the stack, or the
+// heap), as the ExperimentFlow destructor deregisters from the
+// CoroutineManager which may not exist anymore if the global
+// construction/destruction order is inappropriate.
 
 aspect @EXPERIMENT_TYPE@ExperimentHook {
 	advice execution ("void fail::SimulatorController::initExperiments()") : after () {

src/experiments/instantiate-experiment.ah.in

@@ -9,10 +9,14 @@
 #include "../experiments/@EXPERIMENT_NAME@/experiment.hpp"
 #include "sal/SALInst.hpp"
 
+// The experiment needs to be instantiated dynamically (on the stack, or the
+// heap), as the ExperimentFlow destructor deregisters from the
+// CoroutineManager which may not exist anymore if the global
+// construction/destruction order is inappropriate.
+
 aspect @EXPERIMENT_TYPE@ExperimentHook {
-	@EXPERIMENT_TYPE@ experiment;
 	advice execution ("void fail::SimulatorController::initExperiments()") : after () {
-		fail::simulator.addFlow(&experiment);
+		fail::simulator.addFlow(new @EXPERIMENT_TYPE@);
 	}
 };
 

src/experiments/nanojpeg/instantiateExperiment.cc

@@ -1,8 +1,7 @@
 #include "experiment.hpp"
 #include "sal/SALInst.hpp"
 
-static NanoJPEGExperiment experiment;
 void instantiateNanoJPEGExperiment()
 {
-	fail::simulator.addFlow(&experiment);
+	fail::simulator.addFlow(new NanoJPEGExperiment);
 }

src/experiments/rampage/instantiateExperiment.cc

@@ -1,8 +1,7 @@
 #include "experiment.hpp"
 #include "sal/SALInst.hpp"
 
-static RAMpageExperiment experiment;
 void instantiateRAMpageExperiment()
 {
-	fail::simulator.addFlow(&experiment);
+	fail::simulator.addFlow(new RAMpageExperiment);
 }