Week 6 — Runtime Application Self-Protection¶

Click inside the slides and use the arrow keys; use the button at the bottom right of the slides or the link above for full screen.

cd code
.\build.ps1
cd week-06\01-butunluk-hmac
.\demo.ps1        # or double-click demo.cmd

cd code
./build.sh
cd week-06/01-butunluk-hmac
sh demo.sh

0. Before we start¶

This section prepares you for the week. It first briefly recalls the earlier topics this week builds on; it then defines each of this week's concepts in one sentence and links it to the section where it is explained in full.

What we bring from earlier weeks¶

• The white-box attacker model — the user who owns the device is also a potential attacker; they can read memory, attach a debugger, and modify code (Week 1, §3; see also the "white-box attacker's six paths" table: Week 1, §5). This week we turn this model into RASP's single attacker model under the name MATE (§1.1).
• Digest, MAC, and HMAC — a digest is a fixed-size, one-way fingerprint computed from data; a MAC/HMAC proves a message hasn't changed using a shared secret key (Week 3, §2.4). This week we use HMAC-SHA-256 to verify the application's own code (§3) and to build a control-flow chain (§9).
• Deriving keys from a master secret (HKDF) — RFC 5869 HKDF is a function that derives new keys from a master secret through Extract and Expand steps (Week 3, §7). This week we trace the same two steps by hand, deriving a key from a device fingerprint instead of a master secret (§10.1).
• Device binding — a secret being meaningful only on a specific device is the innermost layer of the security shell (Week 3, §13.1). This week we make that layer part of a RASP response policy (§10).

This week's concept map¶

How do the concepts connect to each other?¶

The concepts above are not a random list; each one builds on the previous one. If you forget a concept further along, go back to the "prerequisite" column of the table below and refresh it first.

1. What is RASP? Detect → defend → deter¶

Over the first five weeks we hardened the application statically: we validated input (Weeks 1, 4), encrypted data (Week 3), obscured the code (Weeks 4, 5). But all of that rested on one assumption: the program will run as written. This week we drop that assumption and look at runtime — not the moment the program is compiled and sitting on disk, but the moment it is actually executing. While it runs, the application may be in a hostile environment: the user owns the device, may have rooted it, may have attached a debugger, may have patched pieces of code, may have hooked functions.

RASP (Runtime Application Self-Protection) is a set of protections embedded inside the application that watch themselves at runtime and respond to attacks. The application is its own guard.

RASP does three jobs; this triad is the backbone of the week:

• Detection: understanding that something is wrong — the binary has been patched, a debugger is attached, the environment is an emulator, the app is on a rooted device, a component has been repackaged.
• Defence: giving a meaningful response to detection — erase the secret, reject the operation (fail-closed), bind the key to the device, tie a critical operation to the control flow.
• Deterrence: making the attacker's job expensive — return a fake (decoy) result instead of crashing, delay the response, report the event to the server. The goal is not unbreakability; it is slowing the attack down enough (the RASP-flavoured version of Week 4's "obscurity delays, it does not block" principle).

1.1 The attacker model: MATE ("Man-At-The-End")¶

In Week 1 we saw three attacker types. RASP's world is entirely MATE: the attacker owns the endpoint where the application runs. Against a server, the attacker is outside (and TLS, authentication protect it); but inside the client application, the attacker sees everything: they can read memory, modify code, step through it with a debugger. This is the same assumption as the white-box model in Week 11.

What sets the MATE attacker apart from the other attacker types we saw in Week 1 is position, not capability:

Why does this table matter? Because against a MATE attacker, network-attacker defences (TLS, server-side input validation) do nothing at all — TLS protects the connection's content, but the attacker is the application that opens the connection; the moment it decrypts, it sees the data in the clear in its own memory. That is why RASP is added on top of, not instead of, the earlier weeks' defences.

1.2 This week's example architecture and its central idea¶

Recall the example architecture we've used all term: a mobile payment app and a security library (native C/C++) that does its critical work. The phone is an untrusted environment. This week we add a RASP layer to that library that protects itself while it runs. All example values are synthetic (made up); no real key, card number, device id, or product name is used.

The central idea in one sentence: don't trust a single control. Every detection signal can be bypassed on its own; the strength comes from multiple independent signals, a response policy, and device binding working together. This is the runtime counterpart of Week 3's security shell.

1.3 Worked example: handling the same event three different ways¶

Let's make concrete why detection, defence, and deterrence are separate concepts. Scenario: right before unwrapping the payment key, the crypto library notices a debugger is attached (the Section 4 TracerPid signal is

0). Three developers handle this in three different ways:

1. Detection only, no response: the code increments the supheli (suspicious) variable but never checks it; the function continues its normal flow. Result: an attacker with a debugger attached reads the key directly. The detection was wasted — because it was never wired to a defence.
2. Detection + a crude defence: the code writes if (supheli) exit(1);. Result: the attacker can't read the key, but they see exactly which line the program stopped on; within a few tries they patch that if (the same weakness as the "the checker itself gets patched too" lesson in Section 3).
3. Detection + defence + deterrence: the code mixes the signal, not into exit() directly, but as data into the key-derivation operation (a small-scale version of the control-flow counter from Section 9), and if triggered, produces a decoy result (Section 10). Result: the attacker sees no error message; they get a result that looks "successful" but is useless, and they can't tell which check fired.

All three options share the same detection code; they differ only in the defence and deterrence layers. As this week's demos progress we'll keep moving toward the third approach (see the full comparison in Section 10.2).

2. RASP architecture: when, where, and how do checks work?¶

In the sections ahead we'll see the individual checks (integrity, debugger, environment, hook, root) one by one. But in the field, a RASP solution's strength comes less from how clever each individual check is, and more from how these checks are combined. A check that runs in one place, once, and whose result is evaluated with a single if, no matter how well it's written, is the easiest check to find and bypass. This section covers the design principles for placing checks.

When does it run? Three trigger moments¶

A single trigger moment isn't enough: a debugger check done only at startup will never see a debugger that attaches after the app has started.

Where does it run? Distributed checking¶

• Multiplicity: the same check isn't done in one function; it's done, each time slightly differently, in many places in the code. When the attacker patches one point, the others still run.
• Mutual checking: the managed layer (Java) and the native layer check each other's integrity; if one is modified, the other notices ("mutual integrity checking" in the split fingerprint of Week 5 and the interface table of Week 1).
• Inside the protected function itself: a valuable function (e.g., string decryption or key derivation) does several checks itself, at its own entry point. In the field, a single such function is often protected with close to fifteen checks of its own (code integrity, debugger, environment, root, control-flow counter), in addition to the checks already done at load time. For performance reasons, only the internal-use builds push some of these checks down to the entry point.
• Decoy parameters: at the entry points of sensitive functions there can be extra parameters, hidden behind a macro for the programmer's readability, that trigger an integrity check ("decoy parameters" in Week 4).

How does it decide? Not centralized, data-dependent¶

The weakest design is this:

if (hata_ayiklayici_var() || root_var() || butunluk_bozuk())
    return HATA;                     /* changing this single line bypasses everything */
anahtari_coz();

An attacker who flips a single comparison (or makes the hata_ayiklayici_var function always return "no") gets past the whole protection. In a strong design, the checks' results don't feed an if; they feed the data of the next computation:

• Check results participate in key derivation: if the environment is clean, the correct key comes out; if not, a different but plausible-looking key comes out (the control-flow counter and the decoy output in the response policy are two applications of this idea).
• Check results are carried as opaque values (Week 4): they can't be flipped to "passed" by changing a single byte.
• The response is given far away in time and code from the trigger (the response-policy section).

What does "data dependency" mean, concretely? In a weak design you write if (kontrol_gecti) anahtari_kullan(); — the anahtar (key) is already sitting ready in memory, the if only decides whether to use it; if the attacker patches that if, the key is already there and gets used. In a strong design, the key itself doesn't exist yet; it is computed from the check result: anahtar = HMAC(kontrol_sonucu, sabit_tuz). If the check fails, kontrol_sonucu is a different value, and anahtar comes out different (and useless) — there is no if for the attacker to skip, because there's no decision to skip, only a value to compute. The acc chain in Section 9.1 and the device key in Section 10.1 are two concrete applications of this idea.

Layers protect each other¶

Each layer covers the previous one's weakness: obfuscation hides the checks, integrity checking notices the hidden checks being changed, the control-flow counter notices the integrity check being skipped, device binding stops the data being used somewhere else even if all of that is bypassed; server-side validation has the final word even if the client is completely compromised (the telemetry and server-side risk engine in the response policy).

2.1 Worked example: protecting one function at three points¶

Let's not leave "multiplicity" abstract. Take the anahtar_coz() function (the native function that unwraps the payment key) in the example architecture and apply the three trigger moments in order:

1. At load time: when the library is loaded into memory (in the native library's init function), an integrity check (Section 3) runs once. If the environment is already broken from the start, anahtar_coz is made uncallable (a global flag is cleared).
2. Right before the sensitive operation: at anahtar_coz()'s own entry, an anti-debug check (Section 4) and a hook check (Section 6) also run — because the attacker may have attached the debugger after load time.
3. In the background, at irregular intervals: a separate thread reruns the environment check (Section 5) at random intervals (e.g., 2–7 seconds); the next anahtar_coz() call reads that thread's latest result.

Result: an attacker who wants to bypass anahtar_coz() has to defeat three separate checks at three separate points in time at once. Patching only point 2 (the function's own entry) doesn't affect the checks at points 1 and 3 — the attacker thinks they've finished the job, but the background thread catches the inconsistency within a few seconds and the response policy kicks in (Section 10).

3. Integrity checking: the application verifies itself (self-hashing)¶

RASP's first and most basic step: "Have I been modified?" An attacker tries to bypass a license check, an if (odendi) (if paid) branch, or a crypto call by patching it (the je → jmp patch we saw in Week 4). The defence: the application computes a digest of its own binary (or of a protected code/data region) at runtime and compares it against a known-in-advance golden value. If even a single byte has changed, the patch is caught.

Demo 1 — Runtime integrity checking and patch detection¶

The core of the code (reads the file and computes the HMAC; kripto_hmac_sha256 from cen429_kripto.h):

/* Find your own path: Linux /proc/self/exe, Windows GetModuleFileNameA */
oz_yol(yol, sizeof(yol));
dosya_oku(yol, &veri, &boy);
kripto_hmac_sha256(RASP_ANAHTAR, 32, veri, boy, ozet);   /* HMAC over the file */

/* verify: constant-time comparison (prevents a timing leak) */
unsigned char fark = 0;
for (int i = 0; i < 32; i++) fark |= beklenen[i] ^ ozet[i];
if (fark == 0) { /* BUTUNLUK TAMAM */ } else { /* YAMA ALGILANDI */ }

Actual output captured on WSL (shortened):

ADIM 2 - Normal calisma: butunluk dogrulanir (yama yok)
Beklenen   : bae498a6a982279a833bf7e33d2db9cf...ee4d01e6
Hesaplanan : bae498a6a982279a833bf7e33d2db9cf...ee4d01e6
SONUC: BUTUNLUK TAMAM - hedef degismemis.

ADIM 3 - Saldiri: ikili dosyanin bir KOPYASI alinip 1 bayti degistiriliyor
> ofset 21268 baytini XOR 0xFF ile degistir
ADIM 4 - Yamali kopyanin HMAC'i altin degerle karsilastiriliyor
Beklenen   : bae498a6a982279a833bf7e33d2db9cf...ee4d01e6
Hesaplanan : 05fb42748cb5b57305f3ffacd61ad757...131c6705
SONUC: YAMA ALGILANDI - hedef degistirilmis! (tamper)   (cikis kodu 3)

While unpatched, the digest was exactly the same as the golden value. Changing a single byte completely changed the digest (the avalanche effect), and the patch was caught. The comparison is done in constant time (not memcmp), because as we saw in Week 3, a plain memcmp stops at the first differing byte and leaks timing.

3.1 Worked example: which bytes does the HMAC digest, and how does a patch break it?¶

Which bytes are digested? Looking at the hedef_hmac() function, the answer is clear: dosya_oku() reads the entire target (its own binary) into memory (with fread, from offset 0 to the end of the file), and kripto_hmac_sha256 digests this whole byte sequence in one pass — not a specific function or section, but every byte of the file. In demo.sh's actual output, ofset 21268 was changed; this lands roughly at the middle of the file (BOY / 2) — meaning the binary's total size in that run was roughly 21268 × 2 ≈ 42536 bytes. No matter which byte of the file the attacker changes (beginning, middle, or end), that byte is included in the computation and the digest changes.

How the patch breaks it — step by step on a small example. Let's see the same avalanche effect, not on a huge binary, but in a single character of text; the logic is identical. Let's HMAC-SHA-256 the string "MERHABA" ("HELLO") with a fixed key, then flip the last character from A to B and hash again:

printf 'MERHABA' | openssl dgst -sha256 -hmac "ornek-anahtar"
printf 'MERHABB' | openssl dgst -sha256 -hmac "ornek-anahtar"

SHA2-256(stdin)= 4d45a6382deee7f75c75b3cf5a6a81a2776d317e1db9ec9b0125b69e09492615
SHA2-256(stdin)= 4fbdf8768822083c3fc3797c5f1d278dd1ea58ab0bfde31a8ebf85e260fbcbd7

Only 1 of the input's 7 bytes changed (the last character, A=0x41 → B=0x42; these two values differ in only two bits); but all 32 bytes of the output are different. This is the avalanche effect of cryptographic digest functions: if a single bit of the input changes, on average half of the output changes. The bae498a6...ee4d01e6 → 05fb4274...131c6705 transformation you saw in Demo 1's actual output is this same property on a huge binary: XOR 0xFF flips every bit of a single byte, and this spreads to all 256 bits of output through HMAC-SHA-256's internals (each round of SHA-256's compression function).

How often should "periodic" be? A cost-latency trade-off. Let's put a number to Section 2's "in the background, at irregular intervals" principle. The shorter the check interval, the faster a patch gets noticed, but CPU/battery cost also rises; the longer the interval, the more "unnoticed operation" window is left to the attacker:

In the field, the preferred approach is to use not just one of these four options alone, but all of them together: once at startup (cheap, broad coverage), every time right before a critical operation (cheap, narrow coverage but the most critical moment), and at random intervals in the background (not cheap, but it stops the attacker from predicting "when does the check run"). A fixed interval (e.g., exactly every 30 seconds) gives the attacker predictability; a random interval prevents that — which is why the third row of the table is marked "random."

4. Debugger detection¶

The attacker's most powerful tool is a debugger (gdb, lldb, x64dbg, WinDbg, Visual Studio): it stops the program, steps through it, reads memory and keys, and changes branches. RASP tries to figure out whether a debugger is attached.

Demo 2 — Debugger detection (Linux ptrace/TracerPid, Windows PEB)¶

/* Read TracerPid from /proc/self/status; if >0 we're being watched */
long tp = tracer_pid();
if (tp > 0) supheli++;      /* a process is watching us via ptrace */

ADIM 1 - Normal calisma (hata ayiklayici YOK): temiz beklenir
1) /proc/self/status TracerPid           : 0 (izleyen yok)
2) Ana surec (parent) adi                : sh
SONUC: Temiz - izleyen bir arac gorunmuyor.

ADIM 2 - gdb altinda calistir (TracerPid > 0 beklenir)
1) /proc/self/status TracerPid           : 2909 (IZLENIYOR)
SONUC: Hata ayiklayici/izleme ARACI algilandi (2 sinyal).

Without a debugger TracerPid was 0, under gdb it was >0; the program told the two situations apart. On Windows, if you run under Visual Studio with F5 (under the debugger), IsDebuggerPresent returns YES and the result is "detected."

4.1 Worked example: tracing TracerPid's move from 0 to non-zero step by step¶

Demo 2's output showed TracerPid moving from 0 to 2909; let's now unpack why step by step. /proc/self/status is a text file the kernel keeps for every process, readable by the process itself too (in Week 1 we described /proc as "a window the kernel opens onto the process"). The TracerPid: line inside it holds the PID (Process ID) of the process watching that process with the ptrace() system call; if nobody is watching, the value is 0.

Step 1 — without a debugger. The antidebug program is run directly from the shell and reads its own /proc/self/status file (representative, shortened content):

Name:      antidebug
Pid:       4821
PPid:      4820          <- the shell that launched it (sh)
TracerPid: 0             <- nobody is tracing with ptrace

The program reads this line, finds the numeric value 0; since tp > 0 is false, it doesn't increment the supheli (suspicious) variable. The "SONUC: Temiz" (result: clean) line in the demo output comes from here.

Step 2 — when run under gdb. On its second pass, demo.sh starts the program from inside gdb. When gdb starts or attaches to the target process, it tells the kernel "I'm watching this process"; this happens via a ptrace(PTRACE_TRACEME, ...) or ptrace(PTRACE_ATTACH, ...) call. The kernel records this and fills the watched process's /proc/self/status TracerPid field with the watching gdb process's own PID:

Name:      antidebug
Pid:       4821
PPid:      2909          <- the parent process is now gdb
TracerPid: 2909          <- gdb is tracing us with ptrace

This is exactly the 2909 value seen in the demo output. The program itself did nothing — it only read a field the kernel already keeps; that's why the technique is classified as "read-only" and has no side effects (exactly why the safety promise in the demo's README can be kept).

Step 3 — the decision. Since tp > 0, the supheli counter goes up by 1; since the parent process's name is also gdb, a second independent signal (the ana_surec_adi (parent-process-name) check) gets added too; the "algilandi (2 sinyal)" (detected, 2 signals) phrase in the demo output is the sum of these two. Reading a single field on its own may not be enough, but this kernel-level information is a signal that is hard to fake: zeroing out TracerPid directly is impossible, because it's read from a kernel data structure, not from the process's own memory — the only way for an attacker to hide it is to hook the library function that reads /proc/self/status, which is exactly the topic of Section 6.

4.2 The Windows counterpart: reading the bits in NtGlobalFlag¶

Windows has no direct /proc file, but the same idea (reading a piece of information the kernel already keeps) has a counterpart: the PEB (Process Environment Block — a memory structure kept by the kernel for every process, which the process itself can also read). The PEB's NtGlobalFlag field carries three special flag bits if the process was started under a debugger (not only attached later):

These three bits together make up 0x70 (0x10 | 0x20 | 0x40 = 0x70); when the Windows loader starts a process under a debugger, it automatically switches the heap allocator into a stricter-checking mode and sets these three bits. The line (ntglobal & 0x70) != 0 in Demo 2's Windows code reads exactly this: whether the NtGlobalFlag value carries at least one of these three bits. Just like TracerPid, this too is a read-only operation; the program changes nothing, it reads a field the kernel/loader has already written.

5. Environment detection: virtual machine and emulator¶

An attacker most often runs the application in a controlled analysis environment — a virtual machine, an emulator, or a sandbox. An emulator is a software environment that imitates a device (an Android emulator, QEMU); a virtual machine (VM) abstracts real hardware to run another operating system on top of it. The attacker does their analysis here instead of on a real device, because it's easier. RASP tries to sense this from environmental clues.

Demo 3 — VM/emulator and timing detection (CPUID)¶

(A) CPUID.1:ECX[31] hipervizor biti : VAR
    Hipervizor satici imzasi        : (satici imzasi gizli/bos)
(B) 2.000.000 islem suresi          : 0.806 ms
SONUC: Hipervizor GORULDU. Ama WSL2/Hyper-V/VBS de boyle gorunur;
bu TEK basina 'analiz ortami' demek degildir - zayif sinyal.

5.1 Worked example: how do you build a risk score and a threshold?¶

Let's put numbers to Section 1.2's "don't trust a single control" principle here. Every environment-detection signal is given a weight (a contribution score between 0 and 100); all the weights are summed and a risk score comes out. The decision is not a single if (hipervizor_var), but whether this score is above a threshold.

Example 1 — a developer working on WSL2 (a legitimate user). As in Demo 3's actual output, the hypervisor bit is PRESENT but the vendor signature is hidden/empty, timing is normal, there's no emulator trace, no debugger attached. Total score: only 15 (the hypervisor bit alone). The score is low; the user isn't blocked.

Example 2 — an attacker working in an analysis environment. Hypervisor bit PRESENT (15) + vendor signature visible (25) + timing anomaly (20) + debugger attached (35) = total 95. The score is high; the operation is rejected or a decoy is returned (Section 10).

The difference is not the presence of a single bit, but how many independent signals appear together. Where should the threshold be set? That's a risk appetite decision that varies from product to product — a trade-off of "how many legitimate users am I willing to block, in exchange for how many attacks I prevent":

6. Hook and instrumentation detection (LD_PRELOAD, Frida)¶

A sneakier threat than a debugger: function hooking and dynamic instrumentation. A hook is intercepting and changing a function's call; instrumentation is injecting code into a running program to observe or change its behaviour (the most common tool for this is Frida). Without stopping the program, the attacker replaces the functions it calls — including our own anti-debug/anti-root checks — with their own version. This lets them make Demo 2's checks lie and always return "clean."

Detection ideas (Catalogue K7/K8/K18):

• Is the LD_PRELOAD environment variable empty? (getenv — weak, because getenv itself can also be hooked)
• Which shared object does a function actually resolve from? dlsym(RTLD_DEFAULT, ...) + dladdr are used to look up which .so file a function actually comes from; if a foreign .so shows up instead of the expected libc, there's a hook. (This doesn't rely on getenv faking.)
• Unexpected executable regions in /proc/self/maps, known framework indicators.

Demo 4 — Hooking with LD_PRELOAD and detection with dladdr¶

void *p = dlsym(RTLD_DEFAULT, "time");   /* returns a preloaded hook if one exists */
Dl_info info;
dladdr(p, &info);                        /* the .so that provides the function */
int kanca = !mesru_mi(info.dli_fname);   /* anything other than libc/vdso/ld = HOOK */

ADIM 1 - Normal calisma (LD_PRELOAD yok): temiz beklenir
   time     -> linux-vdso.so.1
   getenv   -> /lib/x86_64-linux-gnu/libc.so.6
   time(NULL) dondurdu : 1789852509
SONUC: Temiz - preload/kanca gorunmuyor.

ADIM 2 - Saldiri: sahte kanca LD_PRELOAD ile YALNIZ bu surece yukleniyor
   time     -> bin/linux/libsahtekanca.so (KANCA!)
   time(NULL) dondurdu : 1234567890
SONUC: Fonksiyon kancasi / preload ALGILANDI (2 sinyal).

In the clean run, time resolved from the kernel's vDSO (this is legitimate, not a hook). Once the hook was loaded, the same time resolved from libsahtekanca.so and returned a fixed fake value (1234567890); dladdr caught the hook by pointing at the function's location. Note: to avoid counting linux-vdso as a false positive in the clean run, detection treats libc, vDSO, and the dynamic loader as legitimate sources — this is the lesson that "writing a detector is easier than writing a correct detector."

6.1 Worked example: finding a function's source step by step with dlsym + dladdr¶

The heart of Demo 4 is two function calls: dlsym and dladdr. Let's unpack what the two do together, step by step.

Step 1 — what does dlsym(RTLD_DEFAULT, "time") do? dlsym searches the running process's symbol table for a symbol with the given name ("time") and returns its memory address. RTLD_DEFAULT is a special handle meaning "use the program's normal symbol search order (its own binary, then loaded libraries, including LD_PRELOADed ones)." This step's output is just a number (a memory address); it carries no information about which file it came from.

Step 2 — who provided this address? This is dladdr's job: it takes a given address, looks it up in the running process's map of loaded shared objects (.so files), and finds which .so's memory range the address falls into. It writes the result into a Dl_info structure; the field we use, dli_fname, is that .so's file path.

Step 3 — in the clean run. With LD_PRELOAD empty, the time symbol resolves through the normal search order and comes from the kernel-provided vDSO (virtual Dynamic Shared Object — a special virtual .so the kernel automatically adds to a process's memory to provide fast reads without a system call). Demo output: time -> linux-vdso.so.1. The mesru_mi() (is-it-legitimate) function recognises this name (it looks for the linux-vdso substring) and says not a hook.

Step 4 — once the hook is loaded via LD_PRELOAD. demo.sh starts the program with the environment variable LD_PRELOAD=./libsahtekanca.so. The dynamic linker (ld-linux.so) puts this library at the very front of the normal search order; since the library defines its own time() function, dlsym(RTLD_DEFAULT, "time") now finds not the real time in vDSO, but the fake time in libsahtekanca.so. dladdr sees this address falls within libsahtekanca.so's memory range; dli_fname returns "bin/linux/libsahtekanca.so". Since mesru_mi() doesn't recognise this name (not libc/vDSO/ld), it says HOOK — exactly the time -> bin/linux/libsahtekanca.so (KANCA!) line in the demo output.

Step 5 — why is this stronger than a getenv(LD_PRELOAD) check? The getenv check looks at the environment variable itself — but the environment variable is only used to load the library at the program's start-up moment; once it's loaded, the attacker can delete it (unsetenv) or hook getenv itself in their own fake library to return empty (exactly what Activity 1 tries). The dladdr check never looks at the environment variable at all; it directly asks "where is this function actually running from right now?" — since the hook is already loaded, the answer to that question comes, not from the environment variable, but from the running code itself, which is why it doesn't rely on getenv faking.

7. Dynamic memory protection and memory-monitoring detection¶

One of the items the syllabus puts in this week is dynamic memory protection: measures taken, while the program runs, against sensitive data in memory being read and modified. In Week 1 we saw erasing secrets from memory and preventing them from being swapped out; in Week 3, the "data in use" layers. This section asks the same question again, from the runtime attacker's angle: what can we do if the attacker is watching or modifying memory?

What does an attacker do with memory?¶

For all of these attacks the attacker needs to access process memory: through a debugger, through the OS's process-memory interfaces, or through a library loaded into the process. That's why memory protection is thought of together with the debugger and hook detection from earlier sections.

Countermeasure 1: keep little in memory, keep it short, keep it scattered¶

The most effective memory protection is leaving nothing to read in memory:

1. Short lifetime: the secret is only ever in the clear inside the function that uses it, and is erased before the function returns (Week 3: the window of exposure).
2. Masking and splitting: while unused, the secret sits XORed with a random mask or split into pieces; the secret itself never sits whole in one place in memory.
3. Erase with a random value: once done, memory is filled, not with zeros, but with random values, so it doesn't even leave the trace "a secret was just here." In the field, on the native side, this is the one place a random generator is used.
4. Never unwrap it at all: in whitebox cryptography, the key is never in the clear in memory at any point (Week 11). In the field's shell matrix, this is the only shell left on the native side during a payment.

Countermeasure 2: make memory harder for the OS to read¶

These measures don't stop an attacker with administrator (root) privileges; but they force the attacker to move to a more powerful position, and that's where root detection comes in.

A concrete example: what does PR_SET_DUMPABLE change? On Linux, every process is "dumpable" by default — the kernel can produce a core dump on a crash and another process owned by the same user can attach to it with ptrace(PTRACE_ATTACH, pid, ...). The prctl(PR_SET_DUMPABLE, 0) call turns this flag off:

This works complementary to the TracerPid check in Section 4: TracerPid detects "is someone watching right now?"; PR_SET_DUMPABLE = 0 prevents "an ordinary tool owned by the same user shouldn't be able to watch." An attacker with root privilege can still get past this flag (depending on the ptrace_scope setting and their privilege) — which is why it should be considered together with the root indicator in Section 8.

Countermeasure 3: notice when data has been changed¶

The way to notice a critical structure in memory (a counter, a permission flag, a configuration table) being silently modified is to never leave that data alone:

typedef struct {
    uint32_t deger;        /* the actual value                      */
    uint32_t golge;        /* deger XOR the runtime mask             */
    uint32_t ozet;         /* a short MAC over deger and golge       */
} KorunanSayac;

/* On read, the consistency of all three is checked; if it doesn't hold, the value has been changed in memory. */
int sayac_oku(const KorunanSayac *s, uint32_t *cikti)
{
    if ((s->golge ^ calisma_maskesi) != s->deger) return -1;
    if (kisa_mac(s->deger, s->golge) != s->ozet)   return -1;
    *cikti = s->deger;
    return 0;
}

If the attacker changes only the deger (value) field in memory, the shadow copy and the digest no longer match. To change all three consistently, they'd also need to find the mask and the digest key; this raises the cost of the attack. The same idea appears in Week 5's device fingerprint being stored with two different digests in two different places, and in Week 2's tamper-resistant log.

7.1 Worked example: how does KorunanSayac catch an attack?¶

Let's trace the structure above with numbers. Let calisma_maskesi (the runtime mask) be a fixed value randomly generated at startup: 0xA5A5A5A5. When a payment-attempt counter deger = 3 (third attempt), the structure is filled like this:

sayac_oku() checks on every read whether golge XOR calisma_maskesi equals deger: 0xA5A5A5A6 XOR 0xA5A5A5A5 = 0x00000003 — equal to deger, consistent; the function returns 0.

Now suppose the attacker uses a memory editor to change only the deger field to 0x00000063 (99) (trying to reset/increase an attempt counter or a score). The golge and ozet fields are unchanged. The next sayac_oku() call: golge XOR calisma_maskesi = 0xA5A5A5A6 XOR 0xA5A5A5A5 = 0x00000003 — but deger is now 0x00000063. They don't match; the function returns -1, and the inconsistency is caught. For the attacker to get past this, they'd also have to recompute golge with the correct mask and produce a consistent ozet — which, as long as they don't know kisa_mac's key, is exactly as hard as recreating the HMAC golden value from Section 3.1.

Countermeasure 4: notice when you're being watched¶

Most memory-watching tools work by attaching to the process as a debugger or by loading a library into the process. That's why the foundation of watch detection is the methods we saw in the debugger and hook detection sections. In addition, some indirect signals are used:

• Timing: code that is single-stepped or paused at breakpoints runs far slower than normal; an unexpectedly long time between two points is a signal (the same idea as the timing measurement in environment detection).
• Unexpected modules: libraries present in process memory that the application itself never loaded.
• Broken consistency: protected values like in Countermeasure 3 becoming inconsistent.

None of these signals is conclusive on its own; all of them are used as contributions to the risk score in the response policy.

8. Root/privileged environment and component signature verification¶

8.1 Root / privileged environment indicator¶

Root is full privilege on a device, normally restricted; jailbreak is the name for forcing that same privilege open on a closed system like iOS. On a rooted (or jailbroken) device, an application's security assumptions collapse: any process can read memory, file-system protections are bypassed. RASP looks at root indicators (Catalogue K4/K5): the presence of the su binary, dangerous packages, writable system paths, root-hiding frameworks. The desktop counterpart: is the application running with elevated privilege (Linux geteuid()==0, an elevated token on Windows)?

The difference between mobile "root" and desktop "elevated privilege" is just a name; the idea is the same. On Android, an application finding the su binary shows that root access is installed; on Linux/desktop, a process's geteuid() returning 0 shows the process is actually running as the root user. A small but important difference between the two: on mobile, root may be installed but not in use at that moment (the indicator is just an indicator); on desktop, geteuid()==0 reports the actual current state (the kernel itself keeps this information, just as hard to fake as TracerPid in Section 4.1 was). That's why the desktop/ server-side "privileged process" check is a slightly more reliable signal than the mobile "root indicator" scan — still, it should be added to the risk score in Section 8.1.1 rather than deciding on its own.

Demo 7 — Root/privileged environment indicator (read-only)¶

ADIM 2 - Isaretli durum: sahte bir 'su' isaret dosyasi olusturuluyor
1) Ayricalik seviyesi : normal kullanici
2) Tehlikeli gosterge taramasi:
   [BULUNDU] cikti/sahte_su (ek isaret)
SONUC: Ayricalikli/riskli ortam GOSTERGESI var (1 sinyal).

8.1.1 Tying the root indicator to a risk score too¶

The risk-score idea from Section 5.1 applies to the root/privilege indicator as well; in fact, the field prefers a list rather than a single indicator, because every single indicator can be both missed (an attacker hides it) and produce a false positive (a legitimate user has genuinely rooted their device).

Demo 7's actual output — a single signal (cikti/sahte_su found → 1 signal) — corresponds to a low score; on its own, it shouldn't be enough to stop a banking application. For the score to come out high, several indicators are expected to appear together — just as in the environment example in Section 5.1.

8.2 Component/package signature and digest verification (the general form of APK signature verification)¶

On Android, an application verifies the APK signature (APK Signature Scheme v1/v2/v3) against repackaging: even if an attacker opens the app, modifies it, and repackages it, they can't carry the signature produced by the original developer's private key. This is the general form of "is the component I loaded/called the expected, unmodified version?" (Catalogue K2/K3/K6).

Demo 6 — Component signature verification and repackaging detection¶

ADIM 3 - Yukleme oncesi dogrulama (modul degismedi): TUTAR
Modul imzasi TUTTU -> guvenle yuklenebilir.
ADIM 4 - Saldiri: modul YENIDEN PAKETLENIR (tek bayt eklenir)
Modul imzasi TUTMADI -> YENIDEN PAKETLENMIS/degistirilmis.
Yukleme REDDEDILDI.   (cikis kodu 3)

8.2.1 Worked example: how is repackaging caught by adding a single byte?¶

Let's combine Demo 6's actual "modül YENIDEN PAKETLENIR (tek bayt eklenir)" (module repackaged, one byte added) step with Section 3.1's logic. The module's verification is also an HMAC-SHA-256 computed over the whole file, just like Demo 1; the difference is that this time the target isn't the program itself but a separate file (the plugin module).

Step 1 — the original module. The module file is N bytes; the loader reads it, computes HMAC(anahtar, modul[0..N)), and compares it against the previously stored signature. It matches → "TUTAR" (holds) → loaded with confidence.

Step 2 — the attacker opens the module, adds a feature, repackages it. This operation appends one byte to the end of the file (it's now N+1 bytes) — even if it never touches the module's intended functionality. This time the loader computes HMAC(anahtar, modul[0..N+1)). Because the input's length itself has changed (a more fundamental difference than the single byte value change in Section 3.1), HMAC's internal block-processing sees different blocks from the very start, and the result comes out completely different — just as we saw in Demo 1.

Step 3 — the comparison fails. The new HMAC doesn't match the stored signature → "TUTMADI" (doesn't hold) → "YENIDEN PAKETLENMIS/degistirilmis" (repackaged/modified) → loading is refused (exit code 3). For the attacker to reproduce the signature even after adding a single byte, they'd need the signing key; in this teaching HMAC, that key lives inside the loader, but in a real APK signature (asymmetric), it is the private key held only by the original developer — which is exactly the crux of the distinction in Section 8.2's "An important difference" note: anyone who can verify an HMAC can also produce one; with an asymmetric signature, the verifier can only verify, never produce.

9. Control-flow integrity: why a single if isn't enough¶

Every check so far (integrity, anti-debug, environment, root) makes one decision at one point: if (tehlike) reddet; (if danger, reject). The attacker's job is simple: patch that one jump (je → jmp) and skip the check (the patch from Section 3). So what's the way to make checks unskippable?

The answer: tie the security checks to a control-flow counter / key chain (Catalogue K17). A critical operation can only produce the correct result if all the checkpoints have been passed in order. We build this as a data dependency: each checkpoint advances a key chain (acc = HMAC(acc, "asama-i")); the critical operation unwraps a secret with the key derived from this chain.

If a check is skipped, or its order is broken, acc comes out different, the key is wrong, and GCM decryption is rejected — the real result can't be produced. So a single jmp patch is useless.

Demo 5 — Control-flow counter and a skip attack¶

SENARYO 1 - Normal: butun kontrol noktalari sirayla calisir
   [gecildi] kontrol noktasi 0, 1, 2
SONUC: ODEME ONAYLANDI -> "ODEME-ONAYI-TOKEN-4242"

SENARYO 2 - Saldiri: kontroller tamamen ATLANIR
   [ATLANDI] hicbir kontrol noktasi calismadi
   (uyari: sayac=0 iz=0x0 beklenen=3/0x7)
SONUC: ODEME REDDEDILDI (zincir anahtari yanlis). ... decoy doner

SENARYO 4 - Saldiri: kontroller YANLIS SIRADA calisir
SONUC: ODEME REDDEDILDI (zincir anahtari yanlis). ... decoy doner

In the normal flow, all three checkpoints were passed in order, the chain came out correct, and the payment was approved. When the checks were skipped, only partly passed, or passed in the wrong order, the chain came out different; the critical operation couldn't produce the real result and returned a fake (decoy) one. The program also keeps a double counter (a sayac counter plus an iz trace bit) (K17); together with the key chain, this gives a triple consistency check.

Why is the "double counter" needed — isn't the acc chain enough? The acc chain proves, indirectly, which stages ran in which order (wrong stage/order → wrong key), but as we see in the demo's warning line (uyari: sayac=0 iz=0x0 beklenen=3/0x7), the program also keeps two more simple counters: sayac (a plain integer counting how many checkpoints ran) and iz (a bit set where each checkpoint sets its own bit position, e.g., if checkpoints 0, 1, and 2 passed, iz = 0b111 = 0x7). Their purpose is different from acc's: acc is cryptographically strong but on its own gives no readable diagnostic information like "how many checks passed, which ones" (you cannot infer "all three checks passed" from an HMAC output). sayac and iz give the developer and the log a readable summary, while the actual authorization decision still rests only on the key derived from acc — even if the sayac and iz fields themselves are patched to 3 and 0x7, the key still comes out wrong because the acc chain is wrong. The three values (acc, sayac, iz) thus cross-check each other: if one is inconsistent, it's noticed in the log, but none of them alone can change the authorization decision.

9.1 Worked example: computing the key chain by hand¶

Let's not leave acc = HMAC(acc, "asama-i") abstract; let's compute it by hand with real openssl commands. Let the starting value acc0 be produced by HMACing empty data with a fixed key of 32 zero bytes (in a real system this is randomly generated per session):

A0=$(printf '' | openssl dgst -sha256 -mac HMAC \
     -macopt hexkey:0000000000000000000000000000000000000000000000000000000000000000 \
     -binary | xxd -p -c 256)
echo "acc0 = $A0"

acc0 = b613679a0814d9ec772f95d778c35fc5ff1697c493715653c6c712144292c5ad

Correct order — checkpoint 0, then 1, then 2, in order. At each stage, the previous acc is used as the key, and the stage label is HMACed as the data; the result becomes the new acc:

A1=$(printf 'asama-0' | openssl dgst -sha256 -mac HMAC -macopt hexkey:$A0 -binary | xxd -p -c 256)
A2=$(printf 'asama-1' | openssl dgst -sha256 -mac HMAC -macopt hexkey:$A1 -binary | xxd -p -c 256)
A3=$(printf 'asama-2' | openssl dgst -sha256 -mac HMAC -macopt hexkey:$A2 -binary | xxd -p -c 256)
echo "acc3 (DOGRU zincir) = $A3"

acc3 (DOGRU zincir) = fc5c2a86e948b190dbe61619375a77bee2aa79911feec3d37f5d7d20615cada8

The critical operation (e.g., payment approval) only derives the correct key and produces the real result if acc3 is exactly equal to this value — Demo 5's SCENARIO 1.

Attack — checkpoints are skipped, only the first stage runs. The attacker patches around the calls to checkpoint functions 0 and 2 (the same idea as the je → jmp patch in Section 3); only the asama-1 label is processed:

B1=$(printf 'asama-1' | openssl dgst -sha256 -mac HMAC -macopt hexkey:$A0 -binary | xxd -p -c 256)
echo "acc (ATLANMIS zincir) = $B1"

acc (ATLANMIS zincir) = bffc11620959254cef65b0e97fc8b54a1fe42cd6afe161b66aca5a758c1e17ff

fc5c2a86... and bffc1162... are completely different values. The critical operation can't open the real secret with the key derived from this wrong acc; this is exactly the "ODEME REDDEDILDI ... decoy doner" (payment rejected ... a decoy is returned) result we saw in Demo 5's SCENARIO 2 and 4. The attacker's jmp patch doesn't even change which check gets skipped — every skipped combination produces a different but always wrong acc, because the HMAC chain spreads every change in its input through the avalanche effect (the same property from Section 3.1, here in its chained form).

10. Response policy, device binding, and deterrence¶

Detection alone is useless; what really matters is the response. A bad response (e.g., exit(1) the instant detection fires) tells the attacker exactly which check fired and points them straight at that check's patch. A good response policy slows the attacker down and misleads them:

Device binding (Week 3, §13.1, the innermost shell of Demo 9) becomes part of RASP here: the valuable secret is wrapped with a key derived, via HKDF, from the device fingerprint and the version string. Even if the phone is cloned and the files copied to another device, the key comes out different and the secret can't be opened.

10.1 Worked example: deriving a key from a device fingerprint (HKDF step by step)¶

Let's trace Demo 8's "device/version binding" step by hand with real values. kripto_hkdf_sha256 implements the two steps (Extract, Expand) of the RFC 5869 HKDF we saw in Week 3; here ikm (input key material) is the device fingerprint, tuz (salt) is the version string, and info is a fixed context label:

Step 1 — Extract. This step compresses ikm (a device fingerprint is plain text; it may not be cryptographically "well distributed"), whose length and randomness are uncertain, into a fixed-length, secure intermediate key; this intermediate key's standard name is PRK (Pseudo-Random Key): PRK = HMAC-SHA256(anahtar = tuz, veri = ikm). Let's do the same computation with openssl:

printf '%s' 'cihaz=SIM-MODEL-A;seri=SN-0001;uretici=DEMO' \
  | openssl dgst -sha256 -mac HMAC -macopt hexkey:$(printf '%s' 'surum=1.0.0' | xxd -p -c 256) -binary \
  | xxd -p -c 256

PRK (real device)    = a0aaf6995343abfb24d1d44d6aac18b20326fc5b9e84bdd82e89bc7ca99562ce

Step 2 — Expand. Since we need 32 bytes (a single block), one HMAC is enough: T1 = HMAC-SHA256(key = PRK, data = info || 0x01) (0x01 is RFC 5869's block counter). This T1 directly becomes the 32-byte data key:

key (real device)      = a19f06fd7b93894705615ac8933a2fa9e0b7cd290c7247a39e09dbc5be175d10

Step 3 — repeating the same computation with "another device's" fingerprint. Let's just change the serial number from SN-0001 to SN-9999 (exactly like an attacker copying the data file to another phone) and repeat the same two steps:

PRK (another device)   = b6b88f3dce3ea3099c4ff0673d356c040689ebd72be4cc98fb158bfa4c0ac039
key (another device)   = b87e4dee6c5579a1456b977ce8cbf8d14f3363aa4e1c7c1a7198342c7689e220

Result. The two keys (a19f06fd... and b87e4dee...) differ so much they share no common bytes — a single-character serial-number difference produces a completely different key from start to finish, because of HKDF's HMAC core (the avalanche effect from Section 3.1). This is exactly the "cihaz/surum baglama tutmadi" (device/version binding failed) result we saw in Demo 8's SCENARIO 3 ("another device"): because the key the packet was encrypted with (a19f06fd...) and the key being used to try to decrypt it (b87e4dee...) are not the same, AES-GCM's authentication tag doesn't hold and kripto_gcm_coz fails — the secret is never exposed, even if the files are copied to another device.

Demo 8 — The RASP engine: detection + response + device/version binding (capstone)¶

SENARYO 1 - Normal: kontroller gecer, cihaz dogru -> sir acilir
SONUC: TEMIZ. Sir acildi ve islem yapiliyor -> "ODEME-ANAHTARI-7C4A"
(sir kullanildiktan sonra bellekten guvenle silindi)

SENARYO 2 - Tamper: bir kontrol basarisiz -> sil + decoy + bayrak
SONUC: TAMPER ALGILANDI -> algilama kontrolu basarisiz (tamper)
   Politika: sir silindi, tamper bayragi kaldirildi, olay kaydedildi.
   Cokmek yerine SAHTE (decoy) sonuc dondu: 60797327...

SENARYO 3 - Baska cihaz: kontroller gecer ama cihaz anahtari tutmaz
SONUC: TAMPER ALGILANDI -> cihaz/surum baglama tutmadi (klonlama?)
   ... decoy doner

In the normal flow, the checks passed, the device was correct, the secret was opened, used, and erased right away. When a check failed (tamper), the engine erased the secret and returned a decoy. When the package was moved to another device, even though the checks passed, the device-bound key didn't hold and the secret couldn't be opened. This is the innermost layer of Week 3's security shell (device binding), merged with RASP.

10.2 Worked example: three different responses to the same event, three different outcomes¶

In Section 1.3 we briefly compared three different responses to the same detection signal; now let's compare it outcome by outcome using Demo 8's tamper scenario. Event: an anti-debug check failed (Demo 8 SCENARIO 2).

All three rest on the same detection code (Section 1.3); they differ only in when and what they return. In Demo 8's actual output, option © is applied: "Cokmek yerine SAHTE (decoy) sonuc dondu: 60797327...". This number is different every run (Exercise 7); if it were a fixed "ERROR-0x1234" value, the attacker would recognise this constant and could tell it apart from the real result — the decoy being random prevents that distinction too.

11. RASP's limits and ethics¶

12. Term project: this week¶

This week you will write the RASP (self-protection) sections of your security guide and add your first RASP check to your project. This is the direct subject of the midterm project demo (Week 7) and Quiz-1 (Week 8, weeks 1–6); it is graded as A2.4 RASP Techniques (15p → LO.3) in the midterm rubric and F2.4 Binary Application Protections (15p → LO.3) in the final rubric.

• S10 — RASP detection inventory: for every critical operation in your project, list in a table which RASP checks you applied (integrity, anti-debug, environment, hook, root, signature), what each one detects, and how it could be bypassed.
• S10 — Response policy: write down what happens when tampering is detected: which secret is erased, whether it's fail-closed, whether it's a decoy, whether there is device/version binding, whether there is a server notification.

| Control            | What it detects      | How it's bypassed     | Hardening           |
| ------------------ | --------------------- | ---------------------- | -------------------- |
| Integrity (HMAC)   | Binary patch          | Checker gets patched  | Overlapping checker|
| Anti-debug         | Debugger               | ptrace no-op / patch  | Multiple signals+counter |
| Hook detection     | LD_PRELOAD/Frida       | Hiding from maps      | dladdr + maps      |
| Device binding     | Cloning                | (bound to the device) | HKDF + AES-GCM     |

| Trigger             | Response                         | Notification     |
| ------------------- | --------------------------------- | ----------------- |
| integrity failed    | erase secret + decoy + fail-close | event to server   |
| device mismatch     | don't open, decoy                 | event to server   |

13. Class activities¶

14. Code-reading exercises¶

if (IsDebuggerPresent()) {
    exit(1);
}
do_payment();

uint32_t crc = crc32(kod, kod_boy);
if (crc != BEKLENEN_CRC) tamper();

if (getenv("LD_PRELOAD") == NULL) printf("temiz\n");

unsigned int ecx;
__cpuid_ecx(1, &ecx);
if (ecx & (1u << 31)) {
    /* hipervizor biti set */
    exit(1);           /* "analiz ortami, calismayi reddet" */
}

15. On your own¶

These exercises aren't graded; they're for reinforcement. All of them are done on your own computer, on the code/week-06 demos.

16. Self-check¶

17. Quiz-1-style sample questions¶

18. Resources and further reading¶

Textbook — J. Viega, M. Messier, Secure Programming Cookbook for C and C++, O'Reilly, 2003:

• Recipe 12.2 (Detecting Modification — updated from CRC to HMAC/signature)
• Recipes 12.12–12.15 (Detecting Debuggers — Unix/Windows; SoftICE → modern tools)
• Chapter 12 (Anti-Tampering) overall is the core of this week and of Week 9.

Open standards and resources

• OWASP MASVS-RESILIENCE and MASTG (mobile resilience: anti-debug, anti-tamper, root/emulator detection, signature verification).
• Android APK Signature Scheme v2/v3 (signature against repackaging); Play Integrity / hardware-backed attestation.
• MITRE CWE: CWE-693 (Protection Mechanism Failure), CWE-919 (mobile weaknesses), CWE-925 (improper verification).
• SEI CERT C: ENV and FIO rules (safe use of the environment and files).