Week 12 — Security Certifications and Penetration Test Planning¶

Click inside the slides and use the arrow keys; use the button at the bottom right of the slides or the link above for full screen.

# Windows (PowerShell)
.\build.ps1
cd week-12\01-birim-test
.\bin\windows\birim_test.exe
cd ..\02-saldiri-potansiyeli
.\bin\windows\saldiri_potansiyeli.exe

# WSL / Linux
./build.sh
cd week-12/01-birim-test && ./bin/linux/birim_test
cd ../02-saldiri-potansiyeli && ./bin/linux/saldiri_potansiyeli

0. Before we start¶

This section prepares you for the week. It first briefly recalls the earlier topics this week builds on; it then defines each of this week's concepts in one sentence and links it to the section where it is explained in full.

What we bring from earlier weeks¶

• Vulnerability — a weak point in a system that can be abused, e.g. an unbounded buffer (Week 1, §1). This week we separate this concept from finding and risk one by one, and rebuild all three in the language of an independent evaluation (end of §0, §6).
• Kerckhoffs's principle — the principle that a system's security must rest not on the design staying secret, but only on the key staying secret (Week 3, §2). This week this principle returns as the reason behind the evaluator's assumption that "the attacker fully knows the system" and behind working white-box (end of §0).
• Static and dynamic analysis — finding bugs without running the code (static) or while running it with sanitizers (dynamic) (Week 4, §9; for the dynamic side, §10). This week we use these two under evaluation's official names, SAST and DAST, and place them in the vulnerability-assessment order (§3).
• Fuzzing — feeding a program unexpected/random inputs to look for crashes or corruption (Week 4, §11). This week we treat fuzzing as the fourth step of the vulnerability-assessment order, a method that must be reported together with its duration and the code coverage it reached (§3).
• CVSS — the Common Vulnerability Scoring System, which expresses a vulnerability's impact as a standard score (0–10) (Week 2, §15). This week we compare CVSS with attack potential and see why the two answer different questions (§5).

This week's concept map¶

Let's open up the idea of risk a little further: risk is the joint assessment of a vulnerability's probability of being exploited and the impact that would result if it were — roughly "how likely × how bad." Not every vulnerability carries the same risk: if exploiting it is nearly impossible (e.g., a server in a locked room, never connected to a network), the risk is low; the same vulnerability on a server exposed to the internet carries high risk — same vulnerability, different risk. The risk that still cannot be closed after all mitigations is called residual risk; a good report writes this explicitly, it does not hide it (we'll see an example in §6). In short: a vulnerability is a technical weakness, a finding is that weakness documented with evidence during an evaluation, and risk is the contextual answer to the question "how dangerous is this weakness, really?"

Let's now ground these three terms and the white-box/black-box distinction in a single example and a comparison table; these two pieces are the last connecting step before we carry this week's concepts into the body.

Telling the terms apart: a small end-to-end example¶

The definitions above make sense one by one, but the differences between them stay abstract until you see them in a single example. Consider the following synthetic function:

int oturum_ac(const char *kullanici, const char *sifre)
{
    char sorgu[256];
    /* HATA: kullanici ve sifre dogrudan sorguya ekleniyor */
    sprintf(sorgu, "SELECT * FROM kullanicilar WHERE ad='%s' AND sifre='%s'",
            kullanici, sifre);
    return veritabani_calistir(sorgu);   /* 1 = giris basarili */
}

In this function, anyone who types ' OR '1'='1 into the kullanici field can log in without knowing the password (a classic SQL injection, CWE-89). Now let's track, line by line, how (and how well) each method catches this single bug — this is the first concrete example of section 3's "which method finds what" question:

Now let's place the terms using this example:

• Vulnerability: building an unparameterized SQL query with sprintf — a technical weak point.
• Finding: the evaluator's written documentation of it: "T-14: the oturum_ac function is vulnerable to SQL injection, evidence: screenshot, CWE-89."
• Risk: if this application is exposed to the internet and holds real user data behind it, the risk is high (high probability × high impact); if it's a prototype running only on a closed test network, the same vulnerability's risk can be assessed as lower.
• Recommendation: use a parameterised query (prepared statement).

Standard or certificate? A frequently confused pair¶

• A standard is a rule: it says "key derivation should be done this way." The standard itself approves no product; it only defines the criterion.
• A certificate is the document proving that a product or organisation meets that rule. There is no certificate without a standard; but reading a standard (or "designing to" it) does not by itself mean you have obtained a certificate — an independent verification (evaluation) is required.

Why is white-box preferred? A short comparison¶

In the coming sections we'll often see the sentence "the evaluator assumes the attacker fully knows the system" — this assumption comes from Kerckhoffs's principle, which we met in Week 3: a system's security must rest not on the design staying secret, but only on the key staying secret; the defence "nobody knows our algorithm, so we're secure" is invalid. The evaluator's working white-box is exactly a natural consequence of this principle: since a real attacker can sooner or later learn the design, the evaluator should see everything from the start too.

Let's put all the terms together in a single sentence¶

Let's see the terms we've defined one by one so far in a single flow of a real process — this is the one-sentence summary of the rest of the week:

A standard states which requirements must be met; an evaluation laboratory, with white-box access (per Kerckhoffs's principle, assuming the attacker knows everything), examines the product; it finds vulnerabilities through code review, SAST, DAST, fuzzing, and penetration testing; it turns each vulnerability, with evidence, into a finding; it rates each finding with attack potential and CVSS; once the developer fixes it, the laboratory re-examines only the changed part through delta assessment; in the end, a certification authority issues the certificate for a specific TOE identity.

If you can read this single paragraph and recognise every term, you've already built the skeleton of everything we'll see from section 1 through 9; the remaining sections will fill in this skeleton's flesh. Now: why independent evaluation?

1. Why independent evaluation?¶

The team that writes a piece of software is not its best tester: it shares the design's assumptions, and it never tries the paths it thinks "surely won't happen." That is why, in domains such as payments, identity, health, and the public sector, products go through evaluation by an independent laboratory. At the end of the evaluation a certificate (or approval) is issued; the certificate does not say "this product cannot be broken," it says "this product met such-and-such requirements of such-and-such standard, against such-and-such attacker model, as of such-and-such date."

We've designed this course's project from the start "as if it will go through a certification." This week we work through that process itself, step by step, through an evaluator's eyes.

The standards and certification landscape¶

What does certification say, and not say?¶

The most common mistake a reader of a certificate makes is equating the word "certified" with "unbreakable." A certificate actually says four things together; if any one of the four is missing, the sentence becomes meaningless:

1. Which product — the TOE's full identity: version + binary file + source code + hash value (we'll see this in section 2).
2. Against which standard — e.g., Common Criteria, EMVCo, PCI.
3. Against which attacker model — e.g., "a moderately skilled attacker with no physical access, attacking over the network."
4. As of which date — once the software is updated, the certificate remains valid for that old version; the new version must either go through delta assessment or be considered uncertified.

Why can't the developer approve its own product?¶

This is not a matter of skill, but of point of view. The team that writes a product carries three biases at once:

• It shares the design's assumptions — a developer who says "a user will never type 300 characters into this field" also writes their test to that assumption; they never try an out-of-range input.
• It never looks at the paths it thinks "nobody touches this" — because in its own mental model that path is already closed.
• It is reluctant to criticize its own code — under time pressure, "it's probably secure" is not evidence, but human nature leans toward it.

An independent laboratory shares none of these three biases: it reads the developer's documentation but questions its boundaries, makes the path marked "nobody touches this" the first thing it checks, and — having no stake in the product's success — does not accept the sentence "probably secure." It demands evidence.

Same product, different tests under different schemes: a short example¶

The same mobile payment application may need to go through different evaluations depending on its target market:

• If the application will be sold to a bank: the bank's information security management system likely falls under ISO/IEC 27001 — but that certifies the bank's processes, not the application itself.
• If the application processes card data: it falls under PCI DSS (organisation) and/or PCI MPoC/SSF (software) — how the software processes, stores, and transmits card data is tested.
• If the application will carry a payment scheme's (e.g., a card network's) brand: it must go through an EMVCo evaluation — the security of the chip/software-based payment flows is tested by a laboratory penetration test.
• If the application makes a general mobile security claim: it can be tested against the OWASP MASVS level — not a certificate, but a recognised test framework.

The same codebase is tested through four different lenses with four different questions. In your project, the answer to "which standard are we closest to?" determines which columns of the requirement template in S17 you fill in.

The cost of certification: why isn't every product certified?¶

Independent evaluation is not free — it requires laboratory time, expert engineer-hours, sometimes special equipment; the process can take weeks to months. That's why not every piece of software is certified against every standard; a team asks the question: "is this certificate's cost worth the trust it provides?"

• For a payment library that will be sold to a bank, the answer is usually yes: without the certificate the bank may not buy the library at all; the certificate's cost is much smaller than the sale it would otherwise lose.
• For an internal company tool (used only by its own employees, never sold externally), a full-scope Common Criteria evaluation is generally disproportionate; a lighter internal security review (code review + SAST + basic penetration test) is preferred instead.

The historical driving force behind certification: why did it emerge?¶

Let's read the short history at the start of this section again, this time with the "why" question — each step is actually a response to a real trust crisis:

• Before TCSEC (1985), the US government could not find a common criterion to evaluate security claims; every vendor used its own definition of "secure" — comparison was impossible.
• Common Criteria (1999) was created to solve a problem caused by the US/European/Canadian criteria (TCSEC/ITSEC/CTCPEC) being separately developed and incomparable with each other — so that the same product would not need to be re-evaluated in every country for international trade.
• CVE/CWE/CVSS (as we saw in Week 2, §15), between 1999 and 2006, arose as a response to the chaos created by vulnerabilities being reported in unnamed and incomparable ways.

Frequently asked question: is a certificate indefinite?¶

A point weaker students often skip but which matters: a certificate does not say "this product is secure forever." A certificate can lose validity over time, or need to be re-reviewed, for three separate reasons:

1. The product changes — a new version is released; the delta assessment from section 2 kicks in, and the certificate must move to the new TOE identity.
2. Attack techniques advance — an attack that requires "multiple experts, months" today can, a few years later, drop to "an ordinary user, one day" with cheap tools (the scoring in section 5 can change over time).
3. The standard is updated — a new version of a standard can decide that a control once deemed sufficient is no longer enough.

2. The evaluation process: 13 steps¶

The flow below is a generalized version of a software component (e.g., a mobile payment library) being evaluated at a third-party laboratory. Product, organisation, and scheme names have been removed; the steps are similar across many schemes.

TOE identity: why doesn't "version 3.2.0" suffice?¶

The first of the 13 steps we're about to see demands a far more precise definition than a version number. Let's see why with an example: a team says "version 3.2.0," but three different binaries can exist under that same version number — one built with debug symbols (debug build), one an optimised release build, and one a copy a developer produced locally with different compiler flags. Their behaviour (debug output, timing, even the presence of some security checks) can differ. That is why a TOE identity consists of four parts (the figure above):

1. Version label — for human readers (e.g., "3.2.0").
2. Binary — which build (release/debug, which architecture: ARM64/x86_64).
3. Source code — which commit/tag (git tag/commit hash).
4. Hash value — the SHA-256 of the binary; proves whether even a single byte has changed.

Let's name the three phases in this figure: Preparation (steps 1–4: TOE, documents, template, workshop), Evaluation (steps 5–9: code review → vulnerability analysis → penetration test → functional conformance → findings), Continuity (steps 10–13: impact analysis → delta assessment → trade-off/residual risk → change management). The certificate is issued at the end of the Evaluation phase; but the Continuity phase never ends for the life of the product.

What does the "secure channel" in step 2 mean?¶

The table's phrase "delivered over a secure channel" isn't there for nothing: the source code and the security guide are the most sensitive pieces of information for a product that isn't certified yet. Sending them as an email attachment or a generic file-sharing link damages the product's confidentiality before the evaluation even begins.

• Encrypted transfer: files are encrypted with AEAD (Week 3, §2) and the key is shared over a separate channel; or an end-to-end encrypted enterprise file-sharing system is used.
• Integrity check: a hash value of the delivered package is also communicated separately; when the laboratory receives the file it verifies the hash — to detect any corruption or tampering in transit (the enterprise counterpart of the AEAD tag verification from week 3).
• Access log: a record is kept of who downloaded which file and when; this makes it possible to later detect unauthorized copying of the source code.

End-to-end example: "GüvenPay Wallet SDK v3.2.0" goes through a laboratory¶

The table above summarizes the 13 steps; but reading a table and living through a process are not the same thing. Below we walk a completely fictional product ("GüvenPay Wallet SDK," version 3.2.0 — a mobile wallet application's card-storage and payment library) through these 13 steps without skipping. The goal is to see concretely, at each step, who does what, and which document passes from hand to hand.

Step 1 — Target of evaluation (TOE). The GüvenPay team writes to the laboratory: "Product to be evaluated: GüvenPay Wallet SDK, version 3.2.0, Android ARM64 build, SHA-256 hash a1b2c3... (the version-identity discipline from week 1 returns here). Out of scope: the mobile operating system hosting the application, the user's device." As soon as the evaluator reads this line, they ask their first question: "so, is the platform trusted?" The answer is no — that is the assumption (we'll see this below).

Step 2 — Delivery of documents. GüvenPay sends the source code, the API documentation, and — most importantly — the security guide to the laboratory over an encrypted channel. In the guide, sentences such as "the card number is stored encrypted only with AES-256-GCM" are each written with a requirement reference (see week 3 AEAD, week 10 version signing).

Step 3 — Requirement template. The laboratory pours the numbered requirements of the chosen standard (here, assumed: an EMVCo + PCI MPoC mix) into a template: "Requirement 4.2 — Card data must be encrypted at rest. Status: GüvenPay says 'met,' evidence: guide section 3.1, source code kart_deposu.c lines 40–85." This is a small example of the S17 compliance matrix in your project.

Step 4 — Workshop. The laboratory and the GüvenPay team review the template together in a meeting (in person or online): "There's no evidence yet for Requirement 6.1 (key rotation) — how will you close this?" GüvenPay presents a document-update plan.

Step 5 — Source code review. Two of the laboratory's engineers read kart_deposu.c, anahtar_yonetimi.c, and the network layer line by line (white-box). One engineer notices a fixed salt used in key derivation — this is noted as the first finding candidate before moving on to the vulnerability-analysis step.

Step 6 — Vulnerability analysis. The laboratory extracts GüvenPay's asset list (card number, CVV, session key; we defined this asset concept in Week 1, §6) and asks, for each one, "where, in what state (in memory/on disk/on the network), can it be exposed?" Result: a penetration test plan (section 7) is prepared; the fixed-salt finding is also added to the plan as a test item.

Step 7 — Penetration test. Every item in the plan is carried out with the eight-field test card (section 7). For the fixed-salt item: the same key-derivation function is run on two different devices, and the same key is observed to come out (evidence). This finding is rated with attack potential (section 5) — knowledge of target is public (a constant in the source code), equipment is standard, time is short → low attack potential → serious finding.

Step 8 — Functional conformance. The laboratory also tests GüvenPay's expected functions with the standard test suite: does the payment flow send the correct amount, does it correctly reject a wrong PIN entry? This is a step separate from, but complementary to, security testing — a product that is secure but doesn't work cannot be certified either.

Step 9 — Findings and fixes. The laboratory report: "Finding F-03: key derivation uses a fixed salt, evidence: the same key from two devices. Recommendation: a random salt per device (the CSPRNG rule from Week 3, §3) + KDF." GüvenPay responds: "Accepted, will be fixed in v3.2.1." Some minor findings (e.g., "log messages aren't in English") are closed as "not a security issue, a good-practice suggestion."

Step 10 — Security impact analysis. While preparing v3.2.1, GüvenPay writes a document: "Change type: bug fix (security). Affected file: anahtar_yonetimi.c. Security impact: key derivation now uses a random salt per device; the weakness in the previous version is closed, no other asset is affected."

Step 11 — Delta assessment. The laboratory does not review the whole product from scratch; it reviews only the anahtar_yonetimi.c file, the impact analysis, and the new TOE identity (v3.2.1, new hash value). The test from step 7 is repeated: this time it is verified that the two devices produce different keys → the finding is closed.

Step 12 — Trade-off and residual risk. The report writes one remaining risk explicitly: "Key rotation frequency is once a year; more frequent rotation is recommended, but GüvenPay has documented, with its reasons, that it accepts this risk because of the performance cost." This is the certification counterpart of the trade-off discipline from Week 1, §19.

Step 13 — Change management. GüvenPay's release of v3.2.1 is classified, approved, tested, released, and verified by the laboratory according to its own baseline; this cycle repeats with every version, for the life of the product — a certificate is not a document obtained once and forgotten, it is a maintained state.

The evaluator's basic stance¶

The evaluator works with two assumptions: (1) it has access to all source code and documentation (a white-box context) and (2) the platform the product runs on is untrusted. That's why a control merely existing is not enough; how easily it is bypassed is measured. It doesn't report on a single control but on how long it takes to bypass controls together; in a well-designed product, the attacker is forced to chain several controls.

Roles in the process: who is who?¶

In the GüvenPay example we said "laboratory" and "GüvenPay team," but a real evaluation has at least four separate parties; mixing them up, especially on "who decides what," causes confusion:

Why in this order? (Preparation → Evaluation → Continuity)¶

We split the 13 steps into three phases; let's think about why the order is this way — each phase builds on the one before it and cannot be skipped:

• Preparation comes first, because if what is to be tested (the TOE) is undefined, no test can be repeatable. A laboratory first wants a precise answer to "what exactly am I examining?"; starting a code review without this answer means working without even knowing which file is in scope.
• Evaluation comes in the middle, because without the requirement template from preparation (step 3), the question "is this a finding, or is it normal?" cannot be answered — a finding is defined against a requirement, not in a vacuum.
• Continuity comes last, because the concept of a "change" can only be discussed once a certificate/decision exists; for a not-yet-certified product, "delta assessment" is meaningless — there is no baseline to change from.

3. Vulnerability-assessment methods¶

Steps 5 and 6 of the evaluation process (code review and vulnerability analysis) require using a set of methods in the right order. We approach these defensively, with the question "how do I test my own product before I deliver it?" Order matters: you start with cheap, broad-coverage methods and move toward expensive, deep ones.

Code review: where the evaluator looks¶

Code review finds logic errors that automated tools miss. The places the evaluator prioritizes (all from previous weeks):

• Crypto usage: is the right algorithm/mode/padding used? Where does the key come from, where is it written, when is it erased? (weeks 3, 10, 11)
• Input validation and memory: bounds checking, integer overflow, format string, UAF (week 4 CERT).
• Error and secret leakage: sensitive data in logs, internal state in error messages (weeks 4, 6).
• Control-bypass paths: a security check that can be bypassed with a single branch or a single return value (Week 9, §6 opaque boolean, §5 random exit).

Why is each item on this list? Because they're all bugs that automated tools miss: a SAST tool can say "here AES_encrypt is being called," but it generally can't answer "was the right mode used" (the AES-GCM/CBC/CTR distinction from week 3, which is the week for section 3's material) — that's a job for a human who understands context.

Static and dynamic analysis, fuzzing¶

We saw these three methods in week 4 as part of the secure build pipeline. In evaluation, the same tools are run this time by an independent party. What matters is that the developer has already run these tools itself and presented its results (S16): the evaluator sees a big difference between "no one has ever tested this product" and "the developer ran these tools, closed these findings."

Tracking a memory bug with five methods¶

In section 0 we compared five methods on a logic bug (SQL injection); now let's do the opposite and track a memory bug — because the methods are strong in different areas.

void tampon_kopyala(char *hedef, const char *kaynak)
{
    strcpy(hedef, kaynak);   /* HATA: hedefin boyutu kontrol edilmiyor */
}

void isle(const char *disaridan_gelen)
{
    char arabellek[64];
    tampon_kopyala(arabellek, disaridan_gelen);   /* disaridan_gelen 64 bayttan uzunsa tasma */
}

Comparison: logic bugs like SQLi are strong for code review and SAST, weak for fuzzing; memory bugs are very strong for fuzzing and sanitizers. That's why section 3's order (code review → SAST → DAST/sanitizer → fuzzing → penetration test) passes every bug class through at least one point where a method is strong.

False positives and false negatives: two risks of automated tools¶

Let's unpack the section-opening sentence "SAST is fast but produces false positives" — these two terms come up constantly in evaluation:

• False positive: a case where the tool says "there's a problem here" but there actually isn't a problem. Example: SAST flags a strcpy call the moment it sees it; but if the target buffer is already allocated safely large, this is a false positive. A large number of false positives leads the developer to disable the tool (alert fatigue).
• False negative: a real problem the tool misses. Example: if a SAST tool only checks against a known list of dangerous functions, it misses the same bug made by a custom wrapper function not on that list. A false negative is more dangerous than a false positive: the trust of "the tool said it's clean" turns out to be false.

Why "cheap to expensive" order is a matter of cost¶

Let's also read the section-opening order (code review → SAST → DAST/sanitizer → fuzzing → penetration test) from the angle of relative cost. The table below is not a real measurement, only an illustrative ranking given to make the idea concrete:

How do the methods verify each other? Cross-checking¶

Verifying something one method finds with another method increases confidence in an evaluation. Example: SAST flags a candidate integer overflow (it might be a false positive); if DAST/sanitizer runs the same path with a real input and shows the overflow actually occurs, it is no longer a false positive — it's a verified finding. A penetration test usually combines (chains) these verified findings into a real attack scenario — two findings that look low-impact on their own can form a serious chain when used together.

4. The tests the standards demand¶

Different standards emphasize different types of testing. Knowing which requirement family your project is closest to determines which tests you'll prioritise.

A concrete test example for every standard¶

Let's make the table's "test it primarily expects" row concrete — one example test question for every standard:

• ISO/IEC 27001: "Is the list of employees who have access to card data current, is access reviewed every quarter?" — this is not a code question, it's a process question; the evidence is an access log, not a test card.
• Common Criteria: "In how many hours, at what level of expertise, can the key-derivation function be broken by reverse engineering?" — directly section 5's attack-potential question.
• FIPS 140-3: "Does the module self-test at startup (power-up self-test), does it run with an algorithm-validated implementation?" — an algorithm validation question, not an attack scenario.
• ETSI EN 303 645: "Does the device have a default/guessable password?" — a baseline, broad-coverage control question; it doesn't expect an in-depth penetration test.
• EMVCo / PCI: "How long does card data stay exposed in memory, is it erased when the transaction ends?" — requires both code review and a laboratory penetration test.
• OWASP MASVS: "Does the application protect sensitive data while running on a rooted/jailbroken device?" — requires both static and dynamic testing.

What does it look like if GüvenPay's six questions are answered one by one?¶

Let's apply the six example questions above to GüvenPay and see how the same product looks through different lenses — this is the concrete form of the "the same product can pass four different exams" idea (section 1):

Two separate scopes in the payment domain: software, or organisation?¶

In the payment-security domain, "PCI" is not a single standard, it's a family of standards; which one applies depends on "what" is being evaluated:

• PCI DSS (Data Security Standard): audits the compliance of an organisation/environment that processes card data — servers, network segmentation, access logs. It's not the software itself that's in scope, but the environment the software runs in. Like ISO/IEC 27001 (section 1), it is process and environment oriented.
• PCI software standards (e.g., SSF — Software Security Framework, MPoC — Mobile Payments on COTS): evaluate whether the software itself (how it processes, stores, deletes card data) is securely designed — this is closer to the scope of an SDK like GüvenPay.
• EMVCo: evaluates, through a laboratory penetration test, whether chip- and software-based payment flows (transaction verification, cryptographic protocol steps) are correctly implemented.

Test depth isn't the same across every standard¶

The same phrase "penetration test" can mean work at very different depths from standard to standard:

The same requirement, six different words: a comparison¶

A single security requirement like "card data must be encrypted" shows up in different standards with different wording, but the same underlying idea. Seeing this similarity will help a lot when filling in the compliance matrix in week 13:

Who writes and updates the standards?¶

A question weaker students often ask but rarely get an answer to: "who's making up these standards?" Short answer: none of them is the product of a single person or company; each is created through the joint work of many organisations in that field and is regularly updated (usually every few years) — much like CVSS, which we saw in week 2, evolving from v2 to v3.1 to v4.0. The reason for these updates is usually either (a) a new attack technique emerging, or (b) clarifying an item in the previous version that was found ambiguous or inconsistent.

5. Attack potential and finding rating¶

We said a control's mere "existing" isn't enough, that "how easily it's bypassed" must be measured. Common Criteria and payment schemes make this measurement with attack potential scoring. A finding's severity is determined by the answer to the question "what did it take to break it?" This is the formal, evaluation-side version of Week 9, §9's "measuring obfuscation" (strength, resilience) framework.

Attack potential factors¶

We score the effort required to carry out an attack with five to six factors; each factor's points are summed, and the total maps to a resistance level (e.g., "basic / moderate / high / very high").

Interpretation: if the total score is high, the attack is hard; the control is good. If the score is low (e.g., an ordinary user, in an hour, with free tools), this is a serious finding. The logic here is the same as Week 9, §9's four criteria: the strength of a defence is measured by the cost of overcoming it.

Why does each factor exist? The reasoning, one by one¶

The five factors are not chosen at random; each represents a different type of cost a real attacker faces. This is the point weaker students confuse the most — the answer to "why these five, and not others?":

• Elapsed time: an attacker's time is not infinite; even a well-resourced attacker usually budgets days, not months, to find a flaw. As duration increases, the number of people who would actually try this attack in the real world shrinks.
• Expertise: not everyone can do reverse engineering or cryptanalysis. As the required expertise level rises, the pool of people who could carry out the attack narrows — "an ordinary user" covers millions of people, while "multiple experts" covers maybe a few hundred.
• Knowledge of target: an attack that can be carried out with publicly available information (e.g., a general API document) is far more accessible than one that requires internal/confidential documents — because the attacker doesn't first need to leak or steal that information.
• Opportunity (window of access): is an attack carried out with unlimited attempts (e.g., on your own device, offline), or within a restricted window (e.g., three minutes of access to an ATM)? If the number of attempts is limited, the attacker must succeed on the first try — which is far harder.
• Equipment: an attack that can be carried out with a standard, free tool (e.g., a debugger) is in a completely different category of accessibility than one requiring custom-adapted, expensive laboratory equipment (e.g., a focused ion beam).

Score table (the simplified scale we'll use in class)¶

The table above says "low/high"; scoring requires numbers. In class (and in the demo) we use the following simplified Common Criteria/JIL scale:

What the total score maps to:

End-to-end example: scoring a finding from start to finish¶

Scenario. "A license check in a release build depended on a branch that a proficient (not expert) user could find and bypass with a single-byte patch, in one hour, using a free disassembler."

Let's read and score the five factors one by one:

Toplam = 0 + 3 + 0 + 0 + 0 = 3        →  3 ≤ 9  →  TEMEL

Result: BASIC level = low resistance = serious finding. In the report, this is written as "close this first."

Now let's add a control and re-score. We put week 9's opaque boolean + random exit on top, plus a server-side check. The same attack now: takes a week (4), requires an expert (6), learning the internal behaviour needs restricted knowledge (3), the server-side check makes attempts restricted (4), equipment is still free (0):

Toplam = 4 + 6 + 3 + 4 + 0 = 17       →  14 ≤ 17 ≤ 19  →  YÜKSEK

So the control moved the finding from level BASIC (3) to level HIGH (17). Instead of saying "I added a control," we spoke with a measurement — this is exactly the sentence to write into S16.

./bin/linux/saldiri_potansiyeli 0 1 0 0 0     # korumasız hâl  -> puan 3  (TEMEL)
./bin/linux/saldiri_potansiyeli 1 2 1 1 0     # korumalı hâl   -> puan 17 (YÜKSEK)

Second end-to-end example: a scenario at the opposite end of the scale ("secure-element cracking")¶

In the first example (the license check) the attack was very easy. Now let's look at the other end of the scale, at the demo's third built-in example ("Secure element (SE) cracking") — this time to see how hard an attack can be.

Scenario B. "Extracting the symmetric master key from a payment card's secure element, via a hardware-level side-channel/fault-injection attack. The attacker needs a team with expertise in multiple domains (hardware + cryptanalysis), access to the chip manufacturer's internal documents, and specially adapted laboratory equipment (e.g., a precision measurement probe, a focused ion beam); the attack takes months and needs very restricted physical access to the device."

Let's score the five factors one by one with the same simplified scale (the score table above):

Toplam = 19 + 8 + 11 + 10 + 7 = 55        →  55 >= 25  →  OTESI (yalniz cok yetenekli saldirgan)

Result: BEYOND level = very high resistance = low-priority finding (still not zero risk — a highly resourced attacker may have access; the report writes this as "residual risk," as we'll see in sections 6 and 12).

Comparing the three results:

Extra exercise: let's hand-verify the demo's third built-in example¶

We saw the two extreme examples (3 and 55); let's also hand-calculate the demo's second built-in finding ("WBC key extraction") and compare it with the tool's output.

"Extracting a key from a white-box cryptography implementation; the attacker applies a differential analysis technique (DCA — differential computation analysis) that requires multiple-expert-level knowledge, needs sensitive (not available to the general public) documentation about the product's internal structure, the attack takes about a month, access to the device is restricted, and it uses specialized (a commercially available analysis tool) equipment."

Toplam = 10 + 8 + 7 + 4 + 4 = 33        ->  33 >= 25  ->  OTESI

In the demo, the command ./bin/linux/saldiri_potansiyeli 2 3 2 1 1 (order: time expertise knowledge opportunity equipment) verifies the same result (puan=33 -> OTESI) — matching exactly the "WBC anahtar cikarma (DCA)" example embedded in the code. This shows that even a scenario that at first glance seems "much harder than the license patch but not as extreme as SE cracking" can quickly climb into the BEYOND band once the five factors are summed together; the numbers can come out higher than the intuitive "moderately hard" feeling — which is why this needs calculation, not guessing.

Rating with CVSS¶

Where attack potential asks "how hard is it to break?", CVSS (Common Vulnerability Scoring System) gives a standard score to the question "how big is this vulnerability's impact?" The two measures complement each other: one weighs likelihood (the attack's difficulty), the other weighs impact (loss of confidentiality/integrity/ availability). A finding report usually gives both; prioritization is done accordingly.

The same finding, two different scores: why does CVSS come out different from attack potential?¶

Let's now score the same finding (Scenario B — extracting a key from a secure element) both with attack potential (above: 55, BEYOND) and with CVSS, and see step by step why the two don't give the same number.

The CVSS v3.1 base score is computed from eight metrics (introduced in Week 2, §15); here we use the relevant four:

• Attack vector (AV): where does the attacker access from? Network (N) · Adjacent (A) · Local (L) · Physical (P). Scenario B requires physical access → AV:P.
• Attack complexity (AC): do conditions outside the attacker's control make it harder? Low (L) · High (H). A hardware attack requiring special equipment + an expert team → AC:H.
• Privileges required (PR) / user interaction (UI): does the attacker need a pre-existing account or a user action? Neither is needed here → PR:N, UI:N.
• Impact (C/I/A): how bad is the outcome if it succeeds? If the master key is captured, the confidentiality, integrity, and the trustworthiness of transaction approval collapse for all cards → C:H, I:H, A:H (total loss).

Vector: AV:P/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H

The CVSS v3.1 formula combines two sub-scores: the Impact sub-score is computed only from C/I/A, the Exploitability sub-score from AV/AC/PR/UI. Let's add up the numbers ourselves:

Etki alt puani (Scope: Degismedi, C=H, I=H, A=H -> her biri 0.56):
  ISS    = 1 - [(1-0.56) x (1-0.56) x (1-0.56)]
         = 1 - (0.44 x 0.44 x 0.44)
         = 1 - 0.085184
         = 0.914816
  Impact = 6.42 x ISS = 6.42 x 0.914816 = 5.873   (ara deger)

Somurulebilirlik alt puani (AV:P=0.20, AC:H=0.44, PR:N=0.85, UI:N=0.85):
  Exploitability = 8.22 x 0.20 x 0.44 x 0.85 x 0.85 = 0.523   (ara deger)

Taban puan = yukari_yuvarla( min(5.873 + 0.523, 10) )
           = yukari_yuvarla(6.396)
           = 6.4   ->  "Orta" bandi (NVD: 4.0-6.9)

Now let's compare the same impact (C:H/I:H/A:H) but with a different access form — this is only a comparison for teaching purposes, no such flaw is claimed in the GüvenPay product. If it were a hypothetical flaw achieving the same result over the network, with low complexity (AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H):

Exploitability = 8.22 x 0.85 x 0.77 x 0.85 x 0.85 = 3.887
Taban puan     = yukari_yuvarla( min(5.873 + 3.887, 10) ) = yukari_yuvarla(9.760) = 9.8  -> "Kritik"

Why does it come out different? Because the two measures answer different questions at different resolutions:

• CVSS's AV/AC metrics are coarse, categorical options (N/A/L/P; L/H) — saying "physical access and high complexity" does not distinguish whether the attack takes a week or six months, or whether it needs one person or a team spanning five specialties.
• Attack potential, on the other hand, sums time, expertise, knowledge, opportunity, and equipment separately, at a fine grain (from 0 up to 19) — it is designed precisely to capture the difference between "six months + five experts + a special ion beam" and "one day + a single user + a free tool."
• CVSS measures impact (what happens if this flaw is realized); attack potential measures cost (what it costs to carry out this flaw). Both can come out "high" (as in Scenario B: catastrophic impact but very high cost), or both can come out "low" (as in the license-patch example).

Chained attacks: combining multiple vulnerabilities¶

At the end of section 2 we said "in a well-designed product, the attacker is forced to chain several controls"; let's unpack this in the language of attack potential. Even a control with very high resistance on its own (e.g., BEYOND) can see its total resistance drop once it's combined with another vulnerability. Example: Scenario B's secure element (BEYOND, score 55) is nearly unexploitable on its own; but if the device's operating system is also an old version with a known root exploit, the attacker can first gain full access to the device through that easier flaw, then turn the "very restricted" opportunity needed for the hardware attack into "unlimited" — which drops the opportunity factor from 10 to 0 and brings the total score down from 55 to 45 (still BEYOND, but an easier BEYOND).

6. Finding → recommendation → action, and impact analysis¶

Evaluation is not a "pass/fail" stamp, it's an improvement cycle. For every finding:

1. Finding: what the laboratory found (with evidence, in a repeatable form).
2. Recommendation: the fix direction the laboratory suggests.
3. Action: what the developer did (or why they didn't — a justified acceptance is also an action).
4. Closure: some findings can also close as "not a security vulnerability, a good-practice suggestion."

This cycle is the source of the finding–action list (week 7) expected in your post-midterm project.

Security impact analysis and delta assessment¶

When fixes produce a new version, re-evaluating everything from scratch is expensive. That's why two documents come into play (steps 10 and 11 of the process):

• Security impact analysis: classifies the changes in the new version (new feature / improvement / bug fix), lists the affected files, and writes the security impact of every change.
• Delta assessment: based on this document, the laboratory re-evaluates only the changed part; it asks for the marked-up documents, the change log, the file list, and the new TOE identity.

End-to-end example: from a finding's birth to its closure¶

Let's trace the "F-03: key derivation with a fixed salt" finding from the GüvenPay example in section 2 concretely through each of the four steps — this is the full example of how your project's finding–action table should be filled in.

1. Finding (laboratory, November 12): "In anahtar_yonetimi.c line 22, the key-derivation function uses a fixed string (\"guvenpay_tuz_2024\"). Evidence: running with the same input on two different test devices produced the same 64-byte key (screen output attached). Attack potential: time < 1 day (visible in the source code), expertise ordinary, knowledge public, opportunity unlimited, equipment free → total 0, BASIC (using section 5's score table)."
2. Recommendation (laboratory, same report): "Use a unique salt generated from a CSPRNG (week 3) for every device; store the salt in the device's secure storage; derive the key with HKDF."
3. Action (GüvenPay, November 15): "Accepted. In anahtar_yonetimi.c, a 32-byte random salt is now generated with the operating system's CSPRNG (week 3) and written to the device's secure storage. The change went into v3.2.1."
4. Closure (laboratory, after delta assessment, November 22): "Verified: the two devices now produce different keys. Finding closed."

For comparison, let's also see an example closed as "not a security vulnerability, a good-practice suggestion": "Finding F-07: log messages are only in English, no Turkish error message." GüvenPay's response: "This is not a security vulnerability, the log messages contain no sensitive data (evidence: the log output was also checked during the F-03 review) — noted as a usability suggestion, removed from the security finding list." The laboratory accepts this justification and closes the finding as "not affected."

End-to-end example: a security impact analysis document¶

Let's write step 10's impact analysis exactly for GüvenPay's v3.2.0 → v3.2.1 transition (this is a scaled-down version of the document you'll fill in when you make a fix after the midterm in your own project):

Residual risk: not everything closes; what doesn't close is written explicitly¶

Let's make the residual risk concept we defined in section 0 concrete here. Consider GüvenPay's F-08 finding (key rotation once a year): the laboratory recommends doing it more often, but GüvenPay accepts the risk, citing performance and user-experience cost as its justification. The report writes this as: "F-08: key rotation frequency is once a year. Attack potential: HIGH (17) — not easy, but not impossible either. GüvenPay has accepted this risk, citing performance cost; the residual risk has been recorded by the laboratory, not closed."

F-03's full life cycle: on a timeline¶

We've seen the F-03 finding from different angles across many sections this week — how it was found in section 2, the finding-action cycle in this section, the test cards in section 7 (PT-03/PT-07). Let's gather all of it into a single timeline; this is the complete picture of how a finding is traced from start to finish:

Scope creep in delta assessment¶

A common trap when making a fix: the developer, thinking "I was already in there," fixes F-03 while also making other unrelated small changes (e.g., updating an interface string, deleting an unused function). If the impact analysis document only describes F-03, the laboratory sees an unexpected file change during delta assessment.

When does a fix introduce a new vulnerability?¶

Step 10's impact analysis document is always explained with "it improved" examples; but in reality a fix can also introduce a new problem. A fictional example: while closing F-03, GüvenPay writes the salt to secure storage but accidentally leaves a line in a debug build that also writes the salt to the log. During delta assessment the laboratory doesn't just ask "is the salt unique now?" but also looks at the whole anahtar_yonetimi.c file and notices this new log line — recording it as a new finding (F-11).

7. Penetration test plan¶

A penetration test is a planned and permitted activity that combines the methods above from an attacker's viewpoint. "Planned" and "permitted" are not optional words: a test whose scope and rules aren't written down is not legitimate security work. A plan contains at least the following four headings.

1. Scope¶

What will be tested, and what won't, is written down explicitly: which binary/version, which components, which environment (test environment, or production), which data (only synthetic). What's excluded is also written down (e.g., third-party servers, real user data).

2. Rules of engagement¶

• Written permission and an authorized signature; the test window (date/time).
• Allowed and forbidden methods (e.g., excluding tests that would cause a service outage).
• Data-handling rule: no real personal data is used; findings are kept confidential.
• Kill switch: if a critical impact is observed, the test stops and is reported immediately.
• Communication and escalation points.

3. Methodology¶

A recognised methodology is chosen and followed: OWASP MASTG/MASVS for mobile, OWASP WSTG for web, PTES or NIST SP 800-115 for the general process. Choosing a methodology makes the result repeatable and comparable.

4. Test template (eight fields)¶

Every test in the evaluation is written with the same template; this is also the skeleton of your project's S16 section:

None of the eight fields is chosen at random; each answers a question necessary for the test to be repeatable and debatable:

• Test ID: you need to know which finding (section 6) or which report line (section 8) a given test is tied to — a test without an ID is a claim without evidence.
• Related requirement: if a test isn't tied to any requirement, it can't answer "why are we doing this test?" — this field is the bridge to the S17 compliance matrix.
• Purpose: without the sentence "what is being verified," a reader can't understand what the test is trying to prove.
• Preconditions: if which version, which environment, which data the test was run with isn't written down, nobody else can repeat the same test (this is vital in delta assessment).
• Method/steps: a test not written step by step carries the risk of "I don't remember how I did it"; repeatability is the keyword here too.
• Expected result: if it isn't written beforehand, the person running the test may be tempted to interpret the result however they like (cognitive bias, the small-scale version of section 1's "developer can't approve its own product" logic).
• Observed result: an unevidenced "it worked" sentence isn't even a finding in the sense we defined in section 0.
• Attack potential/decision: turns the test's result into a priority (the priority matrix in section 8).

An example of a filled-in test card¶

Below is a truly filled-in version of the eight-field template — a test directly tied to Week 3, §2's AES-GCM/AEAD demo:

Let's also see a FAIL (failed) example¶

PT-07 passed; but the section is incomplete without also seeing a test card that fails — because one of the most common student mistakes is showing only passing tests. In GüvenPay's first version (v3.2.0, before F-03 was fixed), the same test would have resulted like this:

8. Reporting¶

A good security report is not a "you failed" list, it's a decision-ready document:

• Executive summary: the overall status and the top three most critical findings, for a non-technical reader. The reader is usually a manager or a client; it answers not "how many findings are there" but "are we ready for production" directly.
• Methodology and scope: what was tested, what wasn't, with which methodology. Why anything excluded is out of scope is also written — otherwise the reader can't tell "wasn't tested" from "was forgotten."
• Findings: each with evidence, attack potential and CVSS, and a recommendation. Findings are listed in order of severity (most serious first); each should be understandable even read on its own.
• Finding–action table: the developer's response and closure status. This is the report's counterpart of section 6's four-step cycle (finding → recommendation → action → closure).
• Residual risk: risks that weren't closed or were transferred, with justification. This section cannot be skipped: the part of a report that earns the most trust is actually not "we solved everything" but the sentence "we couldn't solve this, here's why."

A report serves multiple readers¶

To understand why all five of the sections above are needed, let's ask who will read the same report — each section is actually written for a different reader:

Example: an executive-summary paragraph¶

Example: a finding-row report format¶

Every row in the findings table is written with the same discipline — evidence, attack potential/CVSS, and recommendation together:

(*The CVSS values in this example are for illustration; real scoring is done with the AV/AC/PR/UI/C/I/A choices from section 5 and a standard CVSS calculator.)

Prioritising findings: a simple priority matrix¶

We saw in section 5 that CVSS measures impact and attack potential measures cost. If a report has dozens of findings, which one to look at first comes from the combination of these two. The matrix below is not a formal standard, it's a simple teaching tool we use for this course — real schemes have more detailed priority rules, but the idea is the same:

Let's apply it to the GüvenPay example: F-03 (CVSS Medium, attack potential BASIC) falls into the "high priority" cell in the table — exactly why it was the first finding closed. F-08 (CVSS Medium, attack potential HIGH) falls into the "medium priority" cell — which is why it's reasonable for it to be accepted as a residual risk.

How long should a report be?¶

A weaker student often falls into one of two extremes: either writing a one-page, evidence-free "everything's secure" report, or dumping every single detail (every SAST warning, every unverified candidate finding) one after another. Both are wrong:

• Too short a report: the reader can't see what a decision is based on; the word "secure" is not evidence (the "claim is not evidence" rule from section 1).
• Too long, unfiltered a report: dumping dozens of unverified SAST warnings as-is causes the reader to lose the real three-to-five critical findings in the noise (the reporting-level counterpart of the false-positive discussion in section 3).

9. Term project: this week (S16 — test plan and results)¶

1. Test plan: write eight-field test cards for the two–three most critical assets/requirements (linked to S17).
2. Methodology: state which methodology (MASTG/WSTG/PTES/NIST SP 800-115) you're basing it on.
3. Results: run the tests and write the observed results — at the final, a plan is not expected, a result is.
4. Finding–action: pour your midterm feedback into a finding–action table.
5. Residual risk: write down the risks you haven't closed and why.

Why item 4 matters especially: your midterm feedback is already a set of "findings"¶

Every piece of feedback the instructor gave in your midterm presentation is, in this week's language, actually a finding: a weakness, an observation marked with evidence (the observation made during the presentation). Apply this week's four-step cycle (section 6) directly to your own project:

1. Finding: the concrete sentence the instructor said at the midterm (e.g., "input validation is missing").
2. Recommendation: the concrete fix direction that follows from that feedback.
3. Action: what you did by the final (code change, commit reference).
4. Closure: a test that proves the fix (a new or updated test card).

10. Self-check¶

This week's terms: side-by-side comparison¶

Let's put the five most commonly confused term pairs from this week side by side one last time before the exam:

11. Resources and further reading¶

• The secure-programming technical guide (course resource) — the structure of the delivered security guide and the change/impact-analysis documents; the requirement block and status ("met / transferred / not met").
• OWASP MASVS and MASTG — mobile application security requirements and testing guide.
• OWASP WSTG — web application security testing guide.
• PTES (Penetration Testing Execution Standard) and NIST SP 800-115 — penetration-testing process methodologies.
• Common Criteria (ISO/IEC 15408) and related attack-potential guides — a bridge to week 13.
• CVSS — vulnerability severity scoring; the detailed calculation steps of the base-score formula are published in the CVSS v3.1 Specification (FIRST.org).

This week in one paragraph¶

A product is examined at an independent laboratory with white-box access, in the order code review → SAST → DAST → fuzzing → penetration test; every vulnerability becomes, with evidence, a finding; every finding turns into a priority via attack potential (five factors: time, expertise, knowledge, opportunity, equipment) and CVSS (impact); once the developer makes a fix, the laboratory re-examines only the changed part through delta assessment; every risk that isn't closed is written down explicitly; and this whole chain — finding, test card, impact analysis, action — stays traceable by cross-referencing itself. The F-03 chain we traced end to end in the GüvenPay example (Step 5 → PT-03 → finding → fix → PT-07 → closure) is a single real-time example of this paragraph.