Classes deriving from ExperimentData usually contain the
experiment-specific Protobuf message, which needs to be properly
destroyed. This is particularly a problem in the generic
DatabaseCampaign, as it never downcasts ExperimentData objects
retrieved from JobServer::getDone(). As the embedded
DatabaseCampaignMessage (usually named "fsppilot") is allocated on the
heap (this happens in the campaign's cb_send_pilot() function, asking
for a mutable_fsppilot()), the lack of a virtual destructor in
ExperimentData led to a memory leak, rendering the campaign server
inoperable after handling ~1E7 messages (with a 4GiB / 32-bit process
memory limit).
Change-Id: I4cb8a26d5a702e03189c4aae340051ce62a9c9ce
Due to the previous DatabaseCampaign fix, this may not be necessary
anymore, but it's nevertheless a good idea to handle thread creation
failures properly.
Change-Id: I8317a77dd5338509727e737040944320e7755ae3
It is necessary to copy pilot IDs of existing results to a temporary table
before fetching undone jobs from the DB: Otherwise, due to MyISAMs
table-level locking, collect_result_thread() will block in INSERT (SHOW
PROCESSLIST state "Waiting for table level lock") until the (streamed)
pilot query finishes. As one pilot query follows after the other,
collect_result_thread() may even starve until the memory for the
JobServer's "done" queue runs out, resulting in a crash and the loss of all
queued results.
Change-Id: Ib0ec5fa84db466844b1e9aa0e94142b4d336b022
The patched eCos variant we analyze intentionally overflows the 16550
UART FIFOs, flooding the terminal with Bochs error messages. Enabling
CONFIG_BOCHS_NON_VERBOSE now also enforces ignoring error messages,
regardless of log verbosity settings in the bochsrc.
Change-Id: If14e2532234e61bf60720a45150ef4973e8d508b
Before this change, running prune-trace with, e.g.
"prune-trace -d fsp_mibench -v bitmap% --benchmark-exclude clockcnv"
resulted in an implied "--benchmark none", rendering --benchmark-exclude
ineffective and resulting in nothing being pruned. Now, the "none" default
only applies when neither --benchmark nor --benchmark-exclude (analogously
for --variant / --variant-exclude) is provided.
Change-Id: Ic7c88919d7cfde1261749a745dc6a679472ff348
Using Database::insert_multiple() instead of prepared statements
speeds up trace import by a factor of 3-4. While being there, we now
properly deal with nonexistent extended trace values (i.e., put NULLs
into the DB).
Side note: The ElfImporter should switch to insert_multiple(), too.
Change-Id: I96785e9775e3ef4f242fd50720d5c34adb4e88a1
At least for the Bochs backend there might be side effects when saving
the simulator state while tracing, which therefore should be avoided.
As there is no known use-case for using a --save-symbol different to
--start-symbol, this change disables the semantics behind
--save-symbol completely and only keeps the command-line switch for
backward compatibility reasons (existing automatic test scripts etc.).
The generic-tracing experiment now complains and aborts if a
--save-symbol different to --start-symbol is given.
Change-Id: I6072d846be96e016534cc83db375a400cfc25303
With m_tracetype=TRACE_MEM, bool first was never reset to false in the
tracing plugin's main loop. This bug was most probably never
triggered, though, as nobody only traces memory accesses.
This change also slightly simplifies the internal logic in the tracing
plugin.
Change-Id: I65d7df6a3781ec552cfb892bbf3394b421e227f1
As we copy a 32-bit word from the dereferenced address, we also need to
check whether address+3 is also mapped. (Yes, I've seen this in the
wild.)
Change-Id: I43f891c56e077333670c9cb48c0ee8e9342fa41d
Up to now, BochsMemory::isMapped() always returned true in 32-bit protected
mode with a 4GB linear address space (as used by, e.g., eCos), even for
addresses greater than the configured memory size. This led to lots of
bogus memory dereferences in the (extended) tracing plugin.
This change (a follow-up to commit 5171645) additionally checks the return
value of getHostMemAddr(), and announces BX_RW (read/write access) instead
of BX_READ as the intended type of memory access. In the aforementioned
scenario, memory addresses greater than the memory size are now correctly
detected as "not mapped".
Change-Id: Ic2fa7554c869cb90191164535a601bae4dbb49b6
Memory accesses that don't belong to the preceding IP event in the
trace *do* have a use case: a hardware interrupt causes the CPU to
push its state onto the (kernel) stack. At the moment we cannot
distinguish this case from a malformed trace (as we don't record the
occurrence of interrupts), hence this warning needs to be disabled for
now.
This reverts commit 84edd02b6f.
We now use boost::icl::interval_set internally, consuming extremely
lower amounts of memory. boost::icl was introduced with Boost 1.46;
Debian 7.0 comes with 1.49, so this dependency should be no problem
anymore.
Both the class interface and the memory-map file format stay the same.
Change-Id: I38e8148384c90aa493984d0f6280817df00f1702
The new CLIENT_JOB_INITIAL configuration option allows to configure
the client to request more than one job in the first request round.
If a reasonable initial value is chosen, this removes the job ramp-up
after each fail-client restart, and slightly improves overall
throughput.
Change-Id: Idac2721264ec264c520d341fac64a8311a974708
The JobClient currently waits a LONG time until it really shuts down
after not having reached the server in sendResultsToServer() (which is
unfortunately the by far most probable point in the code to determine
this):
- A different bug (fixed in the previous commit) provoked the
situation that a (way) too large amount of jobs was fetched
before.
- sendResult() (called after each experiment iteration) realized
that CLIENT_JOB_REQUEST_SEC seconds are over, and tried to
prematurely call home to send first results (without planning to
get new jobs yet).
- If the server was gone (done, or aborted), connect in
sendResultsToServer() failed after several retries and timeouts.
- All subsequent calls to sendResult() retried connecting to the
server (again, with retries and timeouts), once for each remaining
job.
- When all jobs were done, getParam() tries to connect a last time,
finally telling the experiment that nobody's home.
This resulted in client shutdown times of up to four hours (for the
default CLIENT_JOB_LIMIT of 1000) after the campaign server
terminated. This change solves the issue by not handing out new
(cached) jobs after the connect failed once, making the experiment
terminate quickly.
Change-Id: I0d8cb2e084d783aca74c51a503fa72eb2b2eb0b7
If we don't properly initialize the job timing statistics, the number
of jobs to be requested in the second request to the server is based
on the wrong timings. In our test case, CLIENT_JOB_LIMIT jobs were
requested at once.
Change-Id: I7e9d8ab6fe14e4488b3a74baf061d9a07f3a77c4
Delay insertion of to-be-sent jobs into m_runningJobs until they are
really sent, as getMessage() won't work anymore (as in: segfault) if
this job is concurrently re-sent (due to campaign end), its result is
received, and deleted in the campaign. This becomes non-hypothetical
with larger values for CLIENT_JOB_LIMIT and CLIENT_JOB_REQUEST_SEC.
Additionally, reinsert the remaining jobs into the input queue if
communication fails, instead of inefficiently delaying redistribution
until the campaign end.
Change-Id: If85e3c8261deda86beb8d4d93343429223753f22
Bounding the outgoing queue is always a good idea: If the campaign has
separate threads for outgoing and incoming jobs (true for the
DatabaseCampaign), this keeps memory requirements reasonable. If the
campaign works in a single thread, this is not disadvantageous either.
Change-Id: Ic75272daa8266f051adf7b23e2ffe87f5c965b86
To allow the JobServer to shutdown properly, the accept() loop in
JobServer::run() needs to regularly check whether we're done. This
change introduces a timed, non-blocking variant of accept() into
SocketComm to achieve this.
Change-Id: Id411096be816c4ed6c7b0b37674410e22152eb22
To avoid accessing destroyed resources in CommThreads talking to clients,
we need to properly join them on shutdown. The m_CommMutex becomes a
JobServer member to make sure it isn't destroyed before the JobServer
itself.
Change-Id: I35b9fb93ace08a7a9476650f8f5e93597a3a8aa0
This change cleans up in/out queue synchronization in the job server.
End-of-jobs conditions are now properly signaled through the
SynchronizedQueue, allowing to resume and abort blocked readers when
no more input is expected.
Change-Id: I3eaf37115ccf8c5b5afe3d971c7109cd62b68906
The Fail* tools expect trace events to be ordered in a specific way:
memory-access events are supposed to come *after* the instruction
event for the instruction that caused them. Using a different order
may cause subtle problems with both fault-space pruning and fast
forwarding. This change introduces a warning message when such a
malformed trace is detected (i.e., when the instruction pointer of a
memory-access event does not match the preceding instruction event).
Change-Id: I8ae7420fd8ff26e2574590748bdcc5a63db76490
According to
<http://dev.mysql.com/doc/refman/5.5/en/c-api-threaded-clients.html>,
(potentially) threaded clients should use the reentrant
libmysqlclient_r. This is just a precaution, I haven't seen any
issues with the normal libmysqlclient.
Change-Id: Icb29df6dd54eb666e3b43b73fbda406acccd11cb
According to
<http://dev.mysql.com/doc/refman/5.5/en/c-api-threaded-clients.html>,
a MySQL connection handle must not be used concurrently with an open
result set and mysql_use_result() in one thread
(DatabaseCampaign::run()), and mysql_query() in another
(DatabaseCampaign::collect_result_thread()). This indeed leads to
crashes when bounding the outgoing job queue (SERVER_OUT_QUEUE_SIZE),
and maybe even more insidous effects in other cases. The solution is
to create separate connections for both threads.
Additionally, call mysql_library_init() before spawning any threads.
Change-Id: I2981f2fdc67c9a2cbe8781f1a21654418f621aeb
Although we know that a known_outcome=1 pilot does not exhibit
behavior different from the golden run, the database schema does not
yet know what this behavior looks like (in terms of result-table
column values). In order to be able to JOIN valid results for all
memory writes in the trace table (fspgroup maps them all onto *one*
pilot per variant), we need to run these experiments, too.
Additionally, don't join the fspgroup table; we only need this one for
result calculations afterwards.
Change-Id: Idcd2991274fede84526b1eee68a231774625d11a
With the recent updates to record one additional instruction at the trace
start, I broke memory-map handling (restrictMemoryAddresses() and
restrictInstructionAddresses()). This change repairs this functionality.
Change-Id: I0daf9f474d0efe3f8e30a168c0ccc1e993e7ddc6
The internal m_iponly / m_memonly bools are a bit hackish; especially it's
unclear what should happen if both are set. The m_tracetype enum now
encompasses all possible configurations, while the plugin's user interface
remains unchanged.
Change-Id: Ibdd872b5cc5781836428b27bfb2db3825700e671
This change implements what the source-code comment already promised but
didn't keep: As we only record time deltas instead of absolute time values,
prevtime must not be overwritten unless the current delta was really added
to the trace. This has caused timing information to be stored incorrectly
if certain events were skipped (e.g., because they didn't match the memory
map configured by the user).
Change-Id: Id40271d117dd91b1122136c62329d64174f304b0
Richard noticed that instr2 values are off by one when done with the
MemoryImporter vs. with his own importer. The core problem is that
the dynamic instruction counter in the Importer base class
(Importer::copy_to_database, instruction_count_t instr) gets increased
*after* reporting an IP event to the importer implementation; this has
the side-effect that memory access events have a +1 dynamic
instruction count offset with regard to the IP event of the
instruction they belong to.
Bottom line: IP events and all memory events belonging to that
instruction should have the same dynamic instruction number.
Christian argued for the numbers starting with 0, which, as a side
effect, relativizes the repercussions of the change introduced in the
previous commit, as the new "first" event gets the sequence number 0
now.
- All experiments and importers only dealing with memory accesses
(MemoryImporter) are affected by this change: The dynamic
instruction count now starts with 0 instead of 1. Together with
the previous commit, the only change is one additional dynamic
instruction at position 0. Note that existing trace files do not
have this additional instruction, which shifts all trace positions
by 1.
- All importers that process *only* IP events (InstructionImporter,
RandomJumpImporter, RegisterImporter) won't see any difference.
Commit 036e340, though, introduced a +1 offset.
- Experiments that use these instruction counts for navigating to
the target instruction must be checked to properly deal with the
dynamic instruction #0 (no forwarding necessary). All dynamic
instruction offsetting should now work uniformly for both memory
accesses and all other fault models. To be sure everything works
in order, sanity-check the current absolute instruction pointer
right before fault injection.
Change-Id: I3f509f1b47836fa78fd029a7bb7c36c878912d97
When starting the tracing plugin (simulator.addFlow()), at the moment
the *current* dynamic instruction (e.g., the one the start symbol
points to) is skipped, and tracing commences with the second
instruction. This change records an additional instruction event at
the trace begin.
Note that this change affects all tracing-plugin users. The first
event gets recorded when starting the plugin (simulator.addFlow()).
This avoids compatibility/off-by-one issues when recording traces with
the generic-tracing experiment vs. with custom experiments.
Change-Id: Ic24e17a68b8a44edad3be994e9edd6d6712bfda1
This reverts commit 036e340bd9.
Problems with this one were:
- Broken event timings. m_prevtime wasn't reset to m_curtime in
TracingPlugin::handleSingleIP(), resulting in a large deltatime
being recorded for the second event, too. This effectively
doubled the experiment's start time.
- Code repetition (copy/pasted for special handling of first event),
making planned changes (advanced tracing for IP events) more
difficult.
- Unnecessary additional tracing-plugin interface method.
Change-Id: I4b74d1a3f4563aabe6626399f9b30a2171b4c285
Without this change, import-trace won't recognize, e.g., the -e and -t
parameters if they come after a parameter that was added by the Importer:
import-trace -i objdump --objdump arm-none-eabi-objdump -e B.elf -t C.tc
[...]
[import-trace 14:37:32] couldn't open trace.pb
Change-Id: I9532b01e432055479c79d801b1ca2736a8fd21cc
There's one fspgroup entry for every trace entry, the pilot_id is
therefore *not* part of the (unique) primary key. If this had been
right in the first place, it would have revealed an equivalence-based
fault-space pruning bug early ... :-/
Change-Id: I449d4985645c6631c0a8db0c64510364677b1354
If you only need raw failure counts, no-effect write equivalence classes
are redundant and only slow down access to the trace and fspgroup tables.
This switch prevents any accesstype='W' entries from finding their way into
the trace table.
Change-Id: Ifb415994063a2107769bc80ebd2fd780de5a4dda
Initially this was implemented by directly passing through trace
events to the MemoryImporter, keeping a record of conditional jumps
and opcodes, and UPDATEing all inserted rows in a second pass when the
MemoryImporter is finished.
Unfortunately, UPDATE is very slow, and keeping all information in
memory till the end doesn't scale indefinitely. Therefore the
implementation now delays passing memory access events upwards to the
MemoryImporter only until enough branch history is aggregated, and
taps into Importer's database operations with a set of new virtual
functions that are called downwards.
Change-Id: I159b2533932087087fb3049f4ff07a5f17a25a00
A MemoryImporter that additionally imports Relyzer-style conditional
branch history, instruction opcodes, and a virtual
duration=time2-time1+1 column (MariaDB 5.2+ only) for fault-space
pruning purposes.
Change-Id: I6764a26fa8aae21655be44134b88fdee85e67ff6
This change touches several subsystems, tools and experiments
(sal, util, cmake, import-trace, generic-tracing, nanojpeg), and
changes details not worth separate commits.
Change-Id: Icd1d664d1be5cfc2212dbf77801c271183214d08
This tool can now import extended trace information with the
--extended-trace command-line parameter. The existing importers cease
using artificial access_info_t objects in favor of passing through the
original Trace_Event wherever possible. This allows us to import
extended trace information for all importers.
Change-Id: I3613e9d05d5e69ad49e96f4dc5ba0b1c4ef95a11
It's OK if we cannot map every register LLVM knows to a Fail register
ID, but we need to explicitly skip these cases in the
RegisterImporter.
Change-Id: I2152f819fb94aa4de5720c5798b229b66988d382
This change became necessary as we observed weird fail-client SIGSEGV
crashes with both Bochs and Gem5 backends and different experiments.
Some Fail* components are instantiated statically: the
SimulatorController instance "simulator", containing the
ListenerManager and the CoroutineManager, and the active
ExperimentFlow subclass(es)
(experiments/instantiate-experiment*.ah.in). The experiment(s) is
registered as an active flow in the CoroutineManager at startup.
As plugins (which are ExperimentFlows themselves) are often created on
an experiment's stack, ExperimentFlows deregister themselves on
destruction (e.g., when leaving the plugin variable's scope). The
core problem is, that the creation and destruction order of statically
instantiated objects depends on the link order; if the experiment is
destroyed after the CoroutineManager, its automatic self-deregistering
feature talks to the smoking ruins of the latter.
This change removes all static instantiations of ExperimentFlow and
replaces them with constructions on the heap. Additionally it makes
sure that the CoroutineManager recognizes that a shutdown is in
progress, and refrains from touching potentially already destroyed
data structures when a (mistakenly globally instantiated)
ExperimentFlow deregisters in this case.
Change-Id: I8a7d42fb141222cd2cce6040ab1a01f9de61be24
Contemporary AspectC++ versions can deal with the LLVM headers very
well, and #ifdef __puma stuff in Fail* headers results in
unmaintainable #ifdef __puma blocks in other parts of Fail* (e.g., the
trace importer).
Make sure you're using a 64-bit ac++ when living in a 64-bit userland
(the 32-bit version doesn't know about __int128), and be aware that
AspectC++ r325 introduced a regression that has not been fixed yet.
Change-Id: I5bb759b08995a74b020d44a2b40e9d7a6e18111c
For def/use pruning, the linear address mapping of the x86
sub-registers (e.g., AX represents the lower 16 bits of EAX) must
overlap. If it doesn't, e.g., AX and EAX are considered separate
registers by def/use pruning, resulting in a failure to correlate an
EAX def with a subsequent AX use.
The only user of this mapping up to now, RegisterImporter, forced all
register widths to 8 bits, thereby fortunately canceling out this
problem. Nevertheless it makes no sense to continue encoding a
constant width in these virtual addresses.
Existing trace, fspgroup, fsppilot and result tables may be converted
to the new encoding by using this query:
UPDATE <tablename> SET data_address = ((data_address >> 4) & ~0xf) | data_address & 0xf;
Change-Id: I7a942b78c34f6140803a86af639eeedef3550f34
This change adapts the gem5 backend to the Register class interface change
in commit 52723a8. The necessary modifications suggested adding the "misc"
registers from gem5, too.
Change-Id: I32561c3fc905b9cd396e32ce80c791c01d5682fb
This change introduces a CMake-style FindMySQL.cmake properly looking for
libmysqlclient_r with mysql_config. This also fixes linking on some
machines.
Change-Id: Ifdbfdc3c7440dead37a8b63aaa86732d636aa0e2
The extended trace register list is needed in multiple locations; the CPU
class is the logical module to contain this information.
Increased number of x86 registers to be traced; we can remove those
that prove unusable for fault-space pruning later on.
Change-Id: Ic46ecdbc55167a6d92872c190317fc0d1a3ad92d
This allows a register to be a member of multiple UniformRegisterSets.
Needed to architecture-specifically store, e.g., a list of registers to
record in traces.
Change-Id: Ia5d20768540efe252a84df967d43e569f107da3a
Several database implementations, including, e.g., MySQL and PostgreSQL,
allow multiple rows to be INSERTed at once. At least for MySQL this
improves INSERT speed significantly.
Change-Id: Ic223f7915053f30b0cb00c567310f3e780c9fa59
Database::insert_id() retrieves the auto-generated ID from the last INSERT
or UPDATE on a table containing an AUTO_INCREMENT column.
Change-Id: I6ec94d1820bb7e77bd37df33145caa93016fd328
This was necessary because LLVM 3.1's
ELFObjectFile<ELFT>::sectionContainsSymbolsectionContainsSymbol() (needed
for proper functioning of our LLVM disassembler, unless you're using
-ffunction-sections) is "unimplemented".
Change-Id: I81112627ebd1c92b718ac6f4ed58d7f188aedf0c
- Don't enforce the join order, MariaDB usually gets this right.
- Update DB statistics before terminating.
Change-Id: If7bbbe146321430d199811062d05b3c179c5732f
data_address is definitely part of the unique key to trace entries, but
instr2 is arbitrary (could be instr1, time1 or time2 as well). Moving
data_address up the hierarchy to speed up certain FSP experiments.
Change-Id: I37a1f6c1e5b3957ba2f5bf46e0cd1a9c4aa7bfef
When recording a "full" trace (register values, plus dereferenced
values), we use a fixed dereference size of 4 bytes.
Change-Id: Ifbf6730377e7df2e7f1fc1e6b91bc114ccf10fbc
The new parameter -s/--stats shows trace statistics such as number of
instructions, number of memory reads/writes, and duration in simulator
time.
Change-Id: I39a48ff62d9c308c420fe52e7ed17ed57ae9c139
Quoting connect(3posix): "If connect() fails, the state of the socket is
unspecified. Conforming applications should close the file descriptor and
create a new socket before attempting to reconnect."
Change-Id: Ibcdcc0f546560a41009832894659a37947243f2f
Horst Schirmeier