# Test Setup
This repository contains a tool suite that can be used for automation of test of
different tools (e.g. rowhammer PoCs) on test systems. In general, it is a
combination of the older **HammerTest** and **HammerISO** components. In
contrast to them, this setup provides an additional **local** mode that can be
used to automatically map and evaluate experiment results.

## General approach of the local mode
In contrast to **HammerTest** and **HammerISO**, this project provides a
**local** mode. In that, the systems are automatically matched: There might be
experiments that require testing of many DDR4 DIMMs. The DIMMs are labeled with
a decimal number. The basic idea is that if the insertion order of the DIMMs is
known, it is not required to explicitly declare the DIMM that is tested at a
certain point in time as long as each DIMM is only tested once and there is a
shared storage that stores the last tested DIMM.

Because the live system is unable to store information about the last time it
was running on the same host, and the hosts might be identical, the MAC
addresses of the first interfaces of the systems are used to match the
information. In contrast to many other information they stay the same as long as
not manually changed. Thereby, the systems can be identified during reboots.

## Add a Experiment
**It is strongly advised to read the Architecture section in order to get a
better understanding of the underlying procedures.**

In order to add a new Experiment, the first step is to decide if a new PoC is
required or if the experiment uses an already existing PoC (take a look in the
`PoC` directory). If the PoC required for the experiment does not exist already,
create a new directory `PoC/<your new PoC>` and put your PoC (binary or source
code if dynamic compilation is required) there.

Next, the experiment has to be specified. Create a file
`Scripts/PoCModules/<ExperimentName>.sh`. See the following listing as example:
```bash
#!/bin/sh

setvar_MyExperiment() {
    POCNAME="My new cool experiment"
    POCITERATIONS=1
}

init_MyExperiment() {
    if [ "$BUILT_MyPoC" != "Y" ]; then
        cd PoC/mypoc && make; cd ../..
        export $BUILT_MyPoC="Y"
    fi
    if [ "$BUILT_HugePage" != "Y" ]; then
        sh ./PoC/trrespass/hugepage.sh
        export BUILT_HugePage="Y"
    fi
}

measure_MyExperiment() {
    (time timeout -s SIGINT $TIMEOUT ./PoC/mypoc/mypoc) 2>&1 | tee $LOGDIR/MyExperiment.log
}

eval_MyExperiment() {
    cat $LOGDIR/MyExperiment.log | grep flip | wc -l > $EVALDIR/MyExperiment.txt
}

print_MyExperiment() {
    echo "MyExperiment:                               $(cat $EVALDIR/MyExperiment.txt)"
}
```

Building the PoC might require additional tools that are not part of the
installation. In order to add tools, edit the file `ISO/packages.x86_64` and
write the names of the packages required to that file.

Afterwards, the experiment has to be added to the configuration in order to be
executed. Edit the files `Scripts/localConfig.sh` and/or
`Scripts/remoteConfig.sh` and add your experiment. If it should be executed, the
value of the according variables should be set to `Y`. If it should not be
executed, there should be an entry with a value of `N` anyway for documentation.

Afterwards, the ISO images can be built again using the following commands:
```
make local
make remote
```

## Architecture
This section contains a brief description of the architecture of the testing
setup.

### run.sh
In `run.sh`, the basic testing procedure is implemented: It initializes the
`TMUX` console, creates directories for the data and fetches the module
identifiers as described in the section about the local mode. Afterwards, it
executes the different steps implemented in the scripts within the `Scripts`
directory in order to perform the tests.

In the end, it increases the counters in case of local mode and sends the
results to a server when in remote mode and the `noauto` flag was not specified.

### PoC
The `PoC` directory contains the single PoC binaries (or source code if dynamic
compilation is required) that are executed within the tests. At the time of
writing, the following PoCs are included:
* `blacksmith`: Binary PoC from the paper [BLACKSMITH: Scalable Rowhammering in the Frequency Domain](https://comsec.ethz.ch/wp-content/files/blacksmith_sp22.pdf)
* `hammertool`: Binary PoC developed during my Bachelor Thesis
* `rowhammerjs`: Source Code PoC from the paper [Rowhammer.js: A Remote Software-Induced Fault Attack in JavaScript](https://arxiv.org/pdf/1507.06955.pdf)
* `trrespass`: Source Code PoC from the paper [TRRespass: Exploiting the Many Sides of Target Row Refresh](https://download.vusec.net/papers/trrespass_sp20.pdf)

### Scripts
The execution of the experiments runs in 4 stages which are each implemented in
a single file:
* `collectInformation.sh`: Collect information about the system which executes
the test (e.g. CPU information, DIMM information, DMI table information)
* `reverseEngineerAddressFunctions.sh`: Automatic time-based reverse engineering
of the address function using the `hammertool` PoC. The calculated address
functions are stored in the file system and can be used for the single
experiments afterwards.
* `experiments.sh`: Initializes several counters and performs the experiments
according to the configuration (see below). The single experiments are thereby
implemented in single files within the `Scripts/PoCModules` directory.
* `evaluate.sh`: Evaluation of the results (e.g. counting bit flips) and brief
output of the results. Again, the selection of the experiments is based on the
configuration (the same experiments executed before) and the implementation for
the single experiments is done in the files within the `Scripts/PoCModules`
directory.

In addition to the single steps of the overall procedure, there are
configuration files that specify which experiments are executed:
* `localConfig.sh`: Configuration for **local** mode (shared storage, automatic
tracking of the single modules used, performance of the experiments, upload to
the shared storage)
* `remoteConfig.sh`: Configuration for the **remote** mode (performance of the
experiments and upload to a server).

Note that there are two ISO images built (select them by using the according
`make` target:
```
make local
make remote
```

### Scripts/PoCModules
This directory contains the modular specifications of the single experiments.
For each experiment, the following functions have to be defined:
* `setvar_<ExperimentName>`: Should set the variable `POCNAME` to the human
readable name of the experiment and the variable `POCITERATIONS` to the number
of iterations (of duration `TIMEOUT_SEC`) the experiment takes
* `init_<ExperimentName>`: Should initialize everything that is required for the
execution of the experiment (e.g. build binaries from source code after the
required modifications, remember that the address functions are stored in
`$CONFIGDIR/addrFunctions.cnf`).

Because the combination of the experiments can be selected by the user, it is
required to implement the initialization steps for the tools within the files
for **every** experiment that uses them. It is not sufficient to implement them
only once and require the user to select the experiments in groups.

In order to avoid duplicated initialization (e.g. when there are two experiments
using the `TRRespass` PoC, it is advised to use global variables and set them to
`Y` once one initialization step is done. The next initialization call for the
same PoC will check the variable and skip the initialization.
* `measure_<ExperimentName>`: Perform the measurement. It is ensured that the
`init_<ExperimentName>` function was called before, this function should not
contain any initialization code (except it is required to run that code each
time the experiment is started). The output of the PoC should be written to
`$LOGDIR/<ExperimentName>.log`.
* `eval_<ExperimentName>`: This function should parse the output of the PoC
stored by the `measure_<ExperimentName>` function before and generate a file
`$EVALDIR/RowhammerJS.txt` that contains the number of bit flips found by the
PoC. If the evaluation is more complex, that should be implemented there as
well.
* `print_<ExperimentName>`: This function should provide a brief
user-understandable output of the experiments results that is shown in the
console after all experiments were performed and evaluated.

### ISO
The ISO image is based on ArchLinux and built with the [Archiso](https://wiki.archlinux.org/title/Archiso)
tool. See the documentation for that tool for more information about the general
usage, etc. This section will highlight some important files but will not
replace the documentation.
* `ISO/packages.x86_64`: List of packages that are installed within the live
system on the created ISO image
* `ISO/localroot`: Root directory for the **local** ISO image
* `ISO/remoteroot`: Root directory for the **remote** ISO image
* `ISO/localroot/root/.ssh/authorized_keys`: List of authorized SSH keys (add
your key to that list if you want to SSH to the host afterwards)
* `ISO/<root>/root/autostart.sh`: Script performing the steps of initialization.
Afterwards, it executes the `run.sh` script.
* `ISO/<root>/etc/profile`: Configuration that initiates the execution of the
`autostart.sh` script within each new TMUX session.
