The SumTree implements an efficient tree data structure for
"roulette-wheel" sampling, or "sampling with fault expansion", i.e.,
sampling of trace entries / pilots without replacement and with a
picking probability proportional to the entries' sizes.
For every sample, the naive approach picks a random number between 0
and the sum of all entry sizes minus one. It then iterates over all
entries and sums their sizes until the sum exceeds the random number.
The current entry gets picked. The main disadvantage is the linear
complexity, which gets unpleasant for millions of entries.
The core idea behind the SumTree implementation is to maintain the
size sum of groups of entries, kept in "buckets". Thereby, a bucket
can be quickly jumped over. To keep bucket sizes (and thereby linear
search times) bounded, more bucket hierarchy levels are introduced
when a defined bucket size limit is reached.
Note that the current implementation is built for a pure growth phase
(when the tree gets filled with pilots from the database), followed by
a sampling phase when the tree gets emptied. It does not handle a
mixed add/remove case very smartly, although it should remain
functional.
Change-Id: If05e9700bc84761b5bc31006402641e7112b3a72
The compiler should be able to completely optimize away side-effect
free usage of this logger. Can be used as a drop-in replacement for
Loggers to silence logging output for known-good code without having
to remove the corresponding "LOG << ..." code.
Change-Id: Ifb276223f61686773dd6108aafd567e99c88b223
gethostbyname() doesn't set errno but h_errno. Thus, we need to call
herror() instead of perror() to print an appropriate error message.
(Thanks, Björn.)
Change-Id: I8fd4bdd4af41774dd290151c5ad37090d006f423
This change makes the RegisterImporter continue importing if it
encounters an instruction pointer that is not part of the disassembled
ELF binary (and, thus, cannot be disassembled). This is OK if we
don't want to inject into registers used by these instructions.
Change-Id: Ia9b5e7f789367f8386d63f235451dae5d399610d
The disassembled memory region's end (variable "End") is exclusive
now. Up to now, the two branches defining this variable disagreed on
inclusiveness, leading to an infinite loop in one case.
Change-Id: I055fc240f6ec2f4a1d1937e48617c86612cff5c5
Since commit c142818, DatabaseCampaignMessage.proto is generated into
the binary directory, which must be announced to the protobuf
compiler.
Change-Id: I0c3c4c6525df687012330573979dc897dfa8d701
Enabling both CONFIG_FAST_BREAKPOINTS / CONFIG_FAST_WATCHPOINTS and
CONFIG_EVENT_BREAKPOINTS / CONFIG_EVENT_BREAKPOINTS_RANGE /
CONFIG_EVENT_MEMREAD / CONFIG_EVENT_MEMWRITE led to an (unnecessary)
include cycle.
Change-Id: I8144e3e72da69b98e21a844a4bfded1b77bdce07
This fixes a gcc 4.7 warning: comparison between signed and unsigned
integer expressions [-Wsign-compare].
Change-Id: If440567c2767494f456e8e68df29e2f8d1582955
BochsController::getTimerTicksPerSecond() only works reliably when the
simulation is already running (e.g., after a restore()). This broke
timeout conditions for the very first experiment in a FailBochs instance.
Change-Id: Ice5f0aa0c6759f2d9341ad4f21d5c346307b4c12
This change fixes another Bochs crash in the floppy controller,
triggered by one out of 670 million experiments in my current
campaign.
Change-Id: I8a2ff78c9d2c8fca12eefb97f508bc213373bbfa
This change makes the DatabaseCampaign load all pilot_ids from the result
table in memory instead of LEFT JOINing them for each variant. This vastly
improves campaign speed (possibly making commit 5567c59 superfluous) at the
cost of slightly increased startup time for half-completed (large)
campaigns.
By exploiting the generally continuous nature of pilot IDs and using a
boost::icl::interval_map, the additional memory requirements are
insignificant.
Change-Id: I1e744fb9ca33efea77a2a785cea3c94106f360df
When no variants matching the command line parameters were found, the
campaign printed an uninitialized sent_pilots count.
Change-Id: Ib1d70ae86f02059daeb9a62567d6c83802e4986e
The fix for tools/convert-trace/Gem5Converter.cc repairs a real issue
which caused a runtime warning to never be displayed.
Change-Id: I5d94acdbffc99d5f2a78f047062c011c6f8ab4e3
This crippled version of the weather-monitor experiment is not needed
anymore, as gem5's restore() works now.
Change-Id: Icc3bced6b04241d55f2acc46f341cf553b6dbd08
As the eCos kernel tests do not sufficiently check for failures, a
(compared to the golden run) measurably *reduced* simulation time can be
used as an indication for SDC.
Change-Id: I303f6c3f7ef56cb691c1f226eff74d113dd16629
With this change, x86 special-purpose registers (e.g., EFLAGS) get the
same register ID in both 32 bit and 64 bit configurations.
Change-Id: I69db9397481414f99ca05ecb0ea9dc8ab7d989c9
This bug mapped EFLAGS to the same Fail* register ID as R9 on x86_64. This
probably has not had consequences yet, as most FailBochs users use 32-bit
code.
Change-Id: I00a680675bb9e73c2781276f3ef651162c8e4445
This change limits fault injection to general-purpose registers, instead of
relying on the LLVM/Fail* bridge to only recognize the status register
(EFLAGS on x86) and general-purpose registers. Since this bridge just
learned to translate x86's control and segment registers, and these
registers need special handling for fault injection (def/use pruning does
not work here), only import register accesses from the RT_GP subset.
Status register and instruction pointer injection remain functional, and
import-trace now should work architecture independently.
Change-Id: Id8ad2f0a9dab1861bf16ea9443c3bdfe7213d3fa
When socket(2), setsockopt(2), bind(2), listen(2), or accept(2) return an
unexpected error status, it is usually not a good idea to let the campaign
continue. This is especially a problem as the perror(3) message gets lost
in normal campaign output and may be missed by the user.
Change-Id: I92747174e0706a613bedd8c6664cc8d888e07533
Classes deriving from ExperimentData usually contain the
experiment-specific Protobuf message, which needs to be properly
destroyed. This is particularly a problem in the generic
DatabaseCampaign, as it never downcasts ExperimentData objects
retrieved from JobServer::getDone(). As the embedded
DatabaseCampaignMessage (usually named "fsppilot") is allocated on the
heap (this happens in the campaign's cb_send_pilot() function, asking
for a mutable_fsppilot()), the lack of a virtual destructor in
ExperimentData led to a memory leak, rendering the campaign server
inoperable after handling ~1E7 messages (with a 4GiB / 32-bit process
memory limit).
Change-Id: I4cb8a26d5a702e03189c4aae340051ce62a9c9ce
Due to the previous DatabaseCampaign fix, this may not be necessary
anymore, but it's nevertheless a good idea to handle thread creation
failures properly.
Change-Id: I8317a77dd5338509727e737040944320e7755ae3
It is necessary to copy pilot IDs of existing results to a temporary table
before fetching undone jobs from the DB: Otherwise, due to MyISAMs
table-level locking, collect_result_thread() will block in INSERT (SHOW
PROCESSLIST state "Waiting for table level lock") until the (streamed)
pilot query finishes. As one pilot query follows after the other,
collect_result_thread() may even starve until the memory for the
JobServer's "done" queue runs out, resulting in a crash and the loss of all
queued results.
Change-Id: Ib0ec5fa84db466844b1e9aa0e94142b4d336b022
The patched eCos variant we analyze intentionally overflows the 16550
UART FIFOs, flooding the terminal with Bochs error messages. Enabling
CONFIG_BOCHS_NON_VERBOSE now also enforces ignoring error messages,
regardless of log verbosity settings in the bochsrc.
Change-Id: If14e2532234e61bf60720a45150ef4973e8d508b
Before this change, running prune-trace with, e.g.
"prune-trace -d fsp_mibench -v bitmap% --benchmark-exclude clockcnv"
resulted in an implied "--benchmark none", rendering --benchmark-exclude
ineffective and resulting in nothing being pruned. Now, the "none" default
only applies when neither --benchmark nor --benchmark-exclude (analogously
for --variant / --variant-exclude) is provided.
Change-Id: Ic7c88919d7cfde1261749a745dc6a679472ff348
Using Database::insert_multiple() instead of prepared statements
speeds up trace import by a factor of 3-4. While being there, we now
properly deal with nonexistent extended trace values (i.e., put NULLs
into the DB).
Side note: The ElfImporter should switch to insert_multiple(), too.
Change-Id: I96785e9775e3ef4f242fd50720d5c34adb4e88a1
At least for the Bochs backend there might be side effects when saving
the simulator state while tracing, which therefore should be avoided.
As there is no known use-case for using a --save-symbol different to
--start-symbol, this change disables the semantics behind
--save-symbol completely and only keeps the command-line switch for
backward compatibility reasons (existing automatic test scripts etc.).
The generic-tracing experiment now complains and aborts if a
--save-symbol different to --start-symbol is given.
Change-Id: I6072d846be96e016534cc83db375a400cfc25303
With m_tracetype=TRACE_MEM, bool first was never reset to false in the
tracing plugin's main loop. This bug was most probably never
triggered, though, as nobody only traces memory accesses.
This change also slightly simplifies the internal logic in the tracing
plugin.
Change-Id: I65d7df6a3781ec552cfb892bbf3394b421e227f1
As we copy a 32-bit word from the dereferenced address, we also need to
check whether address+3 is also mapped. (Yes, I've seen this in the
wild.)
Change-Id: I43f891c56e077333670c9cb48c0ee8e9342fa41d
Up to now, BochsMemory::isMapped() always returned true in 32-bit protected
mode with a 4GB linear address space (as used by, e.g., eCos), even for
addresses greater than the configured memory size. This led to lots of
bogus memory dereferences in the (extended) tracing plugin.
This change (a follow-up to commit 5171645) additionally checks the return
value of getHostMemAddr(), and announces BX_RW (read/write access) instead
of BX_READ as the intended type of memory access. In the aforementioned
scenario, memory addresses greater than the memory size are now correctly
detected as "not mapped".
Change-Id: Ic2fa7554c869cb90191164535a601bae4dbb49b6
Memory accesses that don't belong to the preceding IP event in the
trace *do* have a use case: a hardware interrupt causes the CPU to
push its state onto the (kernel) stack. At the moment we cannot
distinguish this case from a malformed trace (as we don't record the
occurrence of interrupts), hence this warning needs to be disabled for
now.
This reverts commit 84edd02b6f.
We now use boost::icl::interval_set internally, consuming extremely
lower amounts of memory. boost::icl was introduced with Boost 1.46;
Debian 7.0 comes with 1.49, so this dependency should be no problem
anymore.
Both the class interface and the memory-map file format stay the same.
Change-Id: I38e8148384c90aa493984d0f6280817df00f1702
The new CLIENT_JOB_INITIAL configuration option allows to configure
the client to request more than one job in the first request round.
If a reasonable initial value is chosen, this removes the job ramp-up
after each fail-client restart, and slightly improves overall
throughput.
Change-Id: Idac2721264ec264c520d341fac64a8311a974708
The JobClient currently waits a LONG time until it really shuts down
after not having reached the server in sendResultsToServer() (which is
unfortunately the by far most probable point in the code to determine
this):
- A different bug (fixed in the previous commit) provoked the
situation that a (way) too large amount of jobs was fetched
before.
- sendResult() (called after each experiment iteration) realized
that CLIENT_JOB_REQUEST_SEC seconds are over, and tried to
prematurely call home to send first results (without planning to
get new jobs yet).
- If the server was gone (done, or aborted), connect in
sendResultsToServer() failed after several retries and timeouts.
- All subsequent calls to sendResult() retried connecting to the
server (again, with retries and timeouts), once for each remaining
job.
- When all jobs were done, getParam() tries to connect a last time,
finally telling the experiment that nobody's home.
This resulted in client shutdown times of up to four hours (for the
default CLIENT_JOB_LIMIT of 1000) after the campaign server
terminated. This change solves the issue by not handing out new
(cached) jobs after the connect failed once, making the experiment
terminate quickly.
Change-Id: I0d8cb2e084d783aca74c51a503fa72eb2b2eb0b7
If we don't properly initialize the job timing statistics, the number
of jobs to be requested in the second request to the server is based
on the wrong timings. In our test case, CLIENT_JOB_LIMIT jobs were
requested at once.
Change-Id: I7e9d8ab6fe14e4488b3a74baf061d9a07f3a77c4
Delay insertion of to-be-sent jobs into m_runningJobs until they are
really sent, as getMessage() won't work anymore (as in: segfault) if
this job is concurrently re-sent (due to campaign end), its result is
received, and deleted in the campaign. This becomes non-hypothetical
with larger values for CLIENT_JOB_LIMIT and CLIENT_JOB_REQUEST_SEC.
Additionally, reinsert the remaining jobs into the input queue if
communication fails, instead of inefficiently delaying redistribution
until the campaign end.
Change-Id: If85e3c8261deda86beb8d4d93343429223753f22
Bounding the outgoing queue is always a good idea: If the campaign has
separate threads for outgoing and incoming jobs (true for the
DatabaseCampaign), this keeps memory requirements reasonable. If the
campaign works in a single thread, this is not disadvantageous either.
Change-Id: Ic75272daa8266f051adf7b23e2ffe87f5c965b86
To allow the JobServer to shutdown properly, the accept() loop in
JobServer::run() needs to regularly check whether we're done. This
change introduces a timed, non-blocking variant of accept() into
SocketComm to achieve this.
Change-Id: Id411096be816c4ed6c7b0b37674410e22152eb22
To avoid accessing destroyed resources in CommThreads talking to clients,
we need to properly join them on shutdown. The m_CommMutex becomes a
JobServer member to make sure it isn't destroyed before the JobServer
itself.
Change-Id: I35b9fb93ace08a7a9476650f8f5e93597a3a8aa0
This change cleans up in/out queue synchronization in the job server.
End-of-jobs conditions are now properly signaled through the
SynchronizedQueue, allowing to resume and abort blocked readers when
no more input is expected.
Change-Id: I3eaf37115ccf8c5b5afe3d971c7109cd62b68906
The Fail* tools expect trace events to be ordered in a specific way:
memory-access events are supposed to come *after* the instruction
event for the instruction that caused them. Using a different order
may cause subtle problems with both fault-space pruning and fast
forwarding. This change introduces a warning message when such a
malformed trace is detected (i.e., when the instruction pointer of a
memory-access event does not match the preceding instruction event).
Change-Id: I8ae7420fd8ff26e2574590748bdcc5a63db76490
According to
<http://dev.mysql.com/doc/refman/5.5/en/c-api-threaded-clients.html>,
(potentially) threaded clients should use the reentrant
libmysqlclient_r. This is just a precaution, I haven't seen any
issues with the normal libmysqlclient.
Change-Id: Icb29df6dd54eb666e3b43b73fbda406acccd11cb
According to
<http://dev.mysql.com/doc/refman/5.5/en/c-api-threaded-clients.html>,
a MySQL connection handle must not be used concurrently with an open
result set and mysql_use_result() in one thread
(DatabaseCampaign::run()), and mysql_query() in another
(DatabaseCampaign::collect_result_thread()). This indeed leads to
crashes when bounding the outgoing job queue (SERVER_OUT_QUEUE_SIZE),
and maybe even more insidous effects in other cases. The solution is
to create separate connections for both threads.
Additionally, call mysql_library_init() before spawning any threads.
Change-Id: I2981f2fdc67c9a2cbe8781f1a21654418f621aeb
Although we know that a known_outcome=1 pilot does not exhibit
behavior different from the golden run, the database schema does not
yet know what this behavior looks like (in terms of result-table
column values). In order to be able to JOIN valid results for all
memory writes in the trace table (fspgroup maps them all onto *one*
pilot per variant), we need to run these experiments, too.
Additionally, don't join the fspgroup table; we only need this one for
result calculations afterwards.
Change-Id: Idcd2991274fede84526b1eee68a231774625d11a
With the recent updates to record one additional instruction at the trace
start, I broke memory-map handling (restrictMemoryAddresses() and
restrictInstructionAddresses()). This change repairs this functionality.
Change-Id: I0daf9f474d0efe3f8e30a168c0ccc1e993e7ddc6
The internal m_iponly / m_memonly bools are a bit hackish; especially it's
unclear what should happen if both are set. The m_tracetype enum now
encompasses all possible configurations, while the plugin's user interface
remains unchanged.
Change-Id: Ibdd872b5cc5781836428b27bfb2db3825700e671
This change implements what the source-code comment already promised but
didn't keep: As we only record time deltas instead of absolute time values,
prevtime must not be overwritten unless the current delta was really added
to the trace. This has caused timing information to be stored incorrectly
if certain events were skipped (e.g., because they didn't match the memory
map configured by the user).
Change-Id: Id40271d117dd91b1122136c62329d64174f304b0
Richard noticed that instr2 values are off by one when done with the
MemoryImporter vs. with his own importer. The core problem is that
the dynamic instruction counter in the Importer base class
(Importer::copy_to_database, instruction_count_t instr) gets increased
*after* reporting an IP event to the importer implementation; this has
the side-effect that memory access events have a +1 dynamic
instruction count offset with regard to the IP event of the
instruction they belong to.
Bottom line: IP events and all memory events belonging to that
instruction should have the same dynamic instruction number.
Christian argued for the numbers starting with 0, which, as a side
effect, relativizes the repercussions of the change introduced in the
previous commit, as the new "first" event gets the sequence number 0
now.
- All experiments and importers only dealing with memory accesses
(MemoryImporter) are affected by this change: The dynamic
instruction count now starts with 0 instead of 1. Together with
the previous commit, the only change is one additional dynamic
instruction at position 0. Note that existing trace files do not
have this additional instruction, which shifts all trace positions
by 1.
- All importers that process *only* IP events (InstructionImporter,
RandomJumpImporter, RegisterImporter) won't see any difference.
Commit 036e340, though, introduced a +1 offset.
- Experiments that use these instruction counts for navigating to
the target instruction must be checked to properly deal with the
dynamic instruction #0 (no forwarding necessary). All dynamic
instruction offsetting should now work uniformly for both memory
accesses and all other fault models. To be sure everything works
in order, sanity-check the current absolute instruction pointer
right before fault injection.
Change-Id: I3f509f1b47836fa78fd029a7bb7c36c878912d97
When starting the tracing plugin (simulator.addFlow()), at the moment
the *current* dynamic instruction (e.g., the one the start symbol
points to) is skipped, and tracing commences with the second
instruction. This change records an additional instruction event at
the trace begin.
Note that this change affects all tracing-plugin users. The first
event gets recorded when starting the plugin (simulator.addFlow()).
This avoids compatibility/off-by-one issues when recording traces with
the generic-tracing experiment vs. with custom experiments.
Change-Id: Ic24e17a68b8a44edad3be994e9edd6d6712bfda1
This reverts commit 036e340bd9.
Problems with this one were:
- Broken event timings. m_prevtime wasn't reset to m_curtime in
TracingPlugin::handleSingleIP(), resulting in a large deltatime
being recorded for the second event, too. This effectively
doubled the experiment's start time.
- Code repetition (copy/pasted for special handling of first event),
making planned changes (advanced tracing for IP events) more
difficult.
- Unnecessary additional tracing-plugin interface method.
Change-Id: I4b74d1a3f4563aabe6626399f9b30a2171b4c285
Without this change, import-trace won't recognize, e.g., the -e and -t
parameters if they come after a parameter that was added by the Importer:
import-trace -i objdump --objdump arm-none-eabi-objdump -e B.elf -t C.tc
[...]
[import-trace 14:37:32] couldn't open trace.pb
Change-Id: I9532b01e432055479c79d801b1ca2736a8fd21cc
There's one fspgroup entry for every trace entry, the pilot_id is
therefore *not* part of the (unique) primary key. If this had been
right in the first place, it would have revealed an equivalence-based
fault-space pruning bug early ... :-/
Change-Id: I449d4985645c6631c0a8db0c64510364677b1354
If you only need raw failure counts, no-effect write equivalence classes
are redundant and only slow down access to the trace and fspgroup tables.
This switch prevents any accesstype='W' entries from finding their way into
the trace table.
Change-Id: Ifb415994063a2107769bc80ebd2fd780de5a4dda
Initially this was implemented by directly passing through trace
events to the MemoryImporter, keeping a record of conditional jumps
and opcodes, and UPDATEing all inserted rows in a second pass when the
MemoryImporter is finished.
Unfortunately, UPDATE is very slow, and keeping all information in
memory till the end doesn't scale indefinitely. Therefore the
implementation now delays passing memory access events upwards to the
MemoryImporter only until enough branch history is aggregated, and
taps into Importer's database operations with a set of new virtual
functions that are called downwards.
Change-Id: I159b2533932087087fb3049f4ff07a5f17a25a00
A MemoryImporter that additionally imports Relyzer-style conditional
branch history, instruction opcodes, and a virtual
duration=time2-time1+1 column (MariaDB 5.2+ only) for fault-space
pruning purposes.
Change-Id: I6764a26fa8aae21655be44134b88fdee85e67ff6
This change touches several subsystems, tools and experiments
(sal, util, cmake, import-trace, generic-tracing, nanojpeg), and
changes details not worth separate commits.
Change-Id: Icd1d664d1be5cfc2212dbf77801c271183214d08
This tool can now import extended trace information with the
--extended-trace command-line parameter. The existing importers cease
using artificial access_info_t objects in favor of passing through the
original Trace_Event wherever possible. This allows us to import
extended trace information for all importers.
Change-Id: I3613e9d05d5e69ad49e96f4dc5ba0b1c4ef95a11
It's OK if we cannot map every register LLVM knows to a Fail register
ID, but we need to explicitly skip these cases in the
RegisterImporter.
Change-Id: I2152f819fb94aa4de5720c5798b229b66988d382
Horst Schirmeier