Merge branch '5-mutation-test-spec' into 'master'

Create the exercise specs on mutation testing

Closes #5

See merge request !7

docs/2021/exercise3.md

@@ -161,6 +161,10 @@ the system (e.g. invoking service methods).
 At the end of the exercise, do not forget to schedule an one-on-one meeting
 with a teaching assistant to demonstrate your work.
 
+The due date of this exercise is: **1 December 2021, 21:00 UTC+7**. Please
+ensure any updates to the fork repository related to this exercise were made
+and pushed before the due date.
+
 ### Checklist
 
 1. [ ] Choose one **complex function** from your group project.

docs/2021/exercise4.md

+# Exercise 4: Mutation Testing
+
+## Mutation Testing
+
+Mutation testing is a test that involves modifying a program in small ways,
+usually by changing a variable or an operator. The goal here is to "kill" the
+modified program with test cases, that is, by rejecting the mutant using the
+original program's test cases.
+
+### Creating Mutants
+
+The mutant is created by applying one of these operation on a program line.
+
+1. Absolute Value Insertion (**ABS**): Each arithmetic expression (and
+   subexpression) is modified by the functions `abs()`, `negAbs()`, and
+   `failOnZero()`.
+
+    ```java
+        a = m * (o + p);
+    ∆1  a = abs (m * (o + p));
+    ∆2  a = m * abs ((o + p));
+    ∆3  a = failOnZero (m * (o + p));
+    ```
+
+2. Arithmetic Operator Replacement (**AOR**): Each occurrence of one of the
+   arithmetic operators is replaced by each of the other operators. It can also
+   be replaced by the special mutation operators such as `leftOp` and `rightOp`.
+
+    ```java
+        a = m * (o + p);
+    ∆1  a = m + (o + p);
+    ∆2  a = m * (o * p);
+    ∆3  a = m leftOp (o + p);
+    ```
+
+3. Relational Operator Replacement (**ROR**): Each occurrence of one of the
+   relational operators (`<`, `≤`, `>`, `≥`, `=`, `≠`) is replaced by each of
+   the other operators and by `falseOp` and `trueOp`.
+
+    ```java
+        if (X <= Y)
+    ∆1  if (X > Y)
+    ∆2  if (X < Y)
+    ∆3  if (X falseOp Y)  // always returns false
+    ```
+
+4. Conditional Operator Replacement (**COR**): Each occurrence of one of the
+   logical operators (`&&`, `||`, etc.) is replaced by each of the other
+   operators. In addition, each is replaced by `falseOp`, `trueOp`, `leftOp`,
+   and `rightOp`.
+
+    ```java
+        if (X <= Y && a > 0)
+    ∆1  if (X <= Y || a > 0)
+    ∆2  if (X <= Y leftOp a > 0) // returns result of left clause
+    ```
+
+5. Shift Operator Replacement (**SOR**): Each occurrence of one of the shift
+   operators `<<`, `>>`, and `>>>` is replaced by each of the other operators.
+   In addition, each is replaced by the special mutation operator `leftOp`.
+
+    ```java
+        byte b = (byte) 16;
+        b = b >> 2;
+    ∆1  b = b << 2;
+    ∆2  b = b leftOp 2; // result is b
+    ```
+
+6. Logical Operator Replacement (**LOR**): Each occurrence of one of the logical
+   operators (bitwise and - `&`, bitwise or - `|`, exclusive or - `^`) is
+   replaced by each of the other operators. In addition, each is replaced by
+   `leftOp` and `rightOp`.
+
+    ```java
+        int a = 60;    int b = 13;
+        int c = a & b;
+    ∆1  int c = a | b;
+    ∆2  int c = a rightOp b; // result is b
+    ```
+
+7. Assignment Operator Replacement (**ASR**): Each occurrence of one of the
+   assignment operators (`=`, `+=`, `-=`, etc.) is replaced by each of the other
+   operators.
+
+    ```java
+        a = m * (o + p);
+    ∆1  a += m * (o + p);
+    ∆2  a *= m * (o + p);
+    ```
+
+8. Unary Operator Insertion (**UOI**): Each unary operator (arithmetic `+`,
+   arithmetic `-`, conditional `!`, logical `~`) is inserted in front of each
+   expression of the correct type.
+
+    ```java
+        a = m * (o + p);
+    ∆1  a = m * -(o + p);
+    ∆2  a = -(m * (o + p));
+    ```
+
+9. Unary Operator Deletion (**UOD**): Each unary operator (arithmetic `+`,
+   arithmetic `-`, conditional `!`, logical `~`) is deleted.
+
+    ```java
+        if !(X <= Y && !Z)
+    ∆1  if (X > Y && !Z)
+    ∆2  if !(X < Y && Z)
+    ```
+
+10. Scalar Variable Replacement (**SVR**): Each variable reference is replaced
+    by every other variable of the appropriate type that is declared in the
+    current scope.
+
+    ```java
+        a = m * (o + p);
+    ∆1  a = o * (o + p);
+    ∆2  a = m * (m + p);
+    ∆3  a = m * (o + o);
+    ∆4  p = m * (o + p);
+    ```
+
+11. Bomb Statement Replacement (**BSR**): Each statement is replaced by a
+    special `Bomb()` function.
+
+    ```java
+        a = m * (o + p);
+    ∆1  Bomb() // Raises exception when reached
+    ```
+
+## Killing Mutants
+
+To kill a mutant, you need to recall the RIPR model introduced on
+**Introduction to Software Testing by Paul Amann and Jeff Offutt. Chapter 2**:
+
+1. Reachability: The test causes the faulty statement to be reached
+   (in mutation – the mutated statement).
+2. Infection: The test causes the faulty statement to result in an
+   **incorrect state**.
+3. Propagation: The incorrect state propagates to **incorrect output**.
+4. Revealability : The tester must observe part of the incorrect output.
+
+The RIPR model leads to two variants of mutant killing:
+
+- Weakly killing mutants: create a test that kill mutants with incorrect state. To achieve this, you need to
+  create a test that satisfies **reachability and infection, but not propagation**.
+- Strongly killing mutants: create a test that kill mutants with incorrect output. To achieve this, you need to
+  create a test that satisfies **reachability, infection, and propagation**.
+
+To demonstrate how we kill a mutant, take a look at the snippet below:
+
+```java
+boolean isEven (int X) {
+    if (X < 0)
+        X = 0 - X;
+∆       X = 0;
+    if (double) (X/2) == ((double) X) / 2.0
+        return (true);
+    else
+        return (false);
+    }
+```
+
+Ignoring the revealability, we can summarise our RIPR model as follows:
+
+```markdown
+Reachability (R): X < 0
+Infection (I): X != 0
+Propagation (P): odd(0-X)
+```
+
+Now that we have found the reachability, infection, and propagation of the
+mutant, we can finally create a **full test specification** to kill the mutant.
+
+To weakly kill the mutant, we need to satisfy these conditions: `R && I && !P`.
+
+```markdown
+Weak kill = R && I && !P
+          = (X < 0) && (X != 0) && !odd(0-X)
+          = (X < 0) && even(0-X)
+```
+
+One of the `X` value that satifies this is `X=-6`.
+
+To strongly kill the mutant, we need to satisfy these conditions: `R && I && P`.
+
+```markdown
+Weak kill = R && I && P
+          = (X < 0) && (X != 0) && odd(0-X)
+          = (X < 0) && odd(0-X)
+```
+
+One of the `X` value that satifies this is `X=-5`.
+
+For more information regarding mutant killings, please read the
+**Introduction to Software Testing by Paul Amann and Jeff Offutt on Chapter 9.2.2**
+
+## Tasks
+
+You are asked to create a mutation test on your group
+project **individually**. Implement this exercise using a **new branch** of
+your **previously forked group project codebase**.
+
+This exercise consist of 2 phases, where each phase requires you to
+**commit and push your work** to [GitLab CS](https://gitlab.cs.ui.ac.id).
+
+At the end of the exercise, do not forget to schedule an one-on-one meeting
+with a teaching assistant to demonstrate your work.
+
+### Task 1: Creating Mutants
+
+You have learnt about how mutation testing works. Now it is your turn to
+conduct it in your forked repository. Please do these tasks below.
+
+1. [ ] Pick one complex method that is different from what you use on previous exercise,
+   and then try to create at least 2 mutants from it with 2 different kinds of mutations.
+   **You can write the mutants on code or in a written document**.
+2. [ ] Create the **full test specification to strongly kill  your mutants** in a written
+   document by analyzing the reachability, infection, and propagation of those mutants.
+3. [ ] Compare the **full test specification** with the current method's test suite?
+   Are they current test suite enough to kill all mutants?
+
+You can relate question 2 and 3 with your **full test specification** for each mutants.
+
+### Task 2: Mutation Testing Tools
+
+There are several tools that can support mutation testing.
+Try to use one of them by doing these tasks below.
+
+1. [ ] Choose one mutation testing tools and apply it on your forked repository.
+2. [ ] Use that tool to conduct mutation testing on your selected method (not the mutants
+   you implement).
+3. [ ] Improve your test based on the conducted mutation testing.
+4. [ ] Modify your `.gitlab-ci.yml` to include mutation testing using the selected tool
+   if necessary.
+5. [ ] Commit and push your improvement.
+
+After conducting all the tasks above, please explain these in a written document:
+
+## Deliverables
+
+At the end of this exercise, you are required to prepare the following
+artifacts:
+
+- [ ] A written document that describes the process of your work in completing
+  this exercise. You can write the document as a Markdown-formatted text file
+  or a PDF file. Give the document a descriptive name, e.g. `exercise4.md`, and
+  put it into a folder called `docs` in your fork.
+- [ ] The mutants you created.
+- [ ] One or more changed test suites, if any, that had been updated from the
+  result of mutation testing tools.
+  > If there are no changes in the test suites, explain the reasons in the
+  > documentation.
+
+## Work Demonstration
+
+Arrange an one-on-one meeting with a teaching assistant to demonstrate your work.
+You are expected to be able to:
+
+- Describe how you conduct your **full test specification** and how it can kill your mutants.
+- Explain about the mutation testing tools you're using.
+- Explain the test improvement using mutation testing tools.
+- Explain the benefit of mutatin testing.
+
+The due date of this exercise is: **8 December 2021, 21:00 UTC+7**. Please
+ensure any updates to the fork repository related to this exercise were made
+and pushed before the due date.
+
+## References
+
+- Introduction to Software Testing by Paul Amann and Jeff Offutt.
+- [Mutation Testing Tools for Java](https://pitest.org/java_mutation_testing_systems/)
+
+*[RIPR]: Reachability Infection Propagation Revealability

mkdocs.yml

@@ -55,6 +55,7 @@ nav:
     - Exercise 1: 2021/exercise1.md
     - Exercise 2: 2021/exercise2.md
     - Exercise 3: 2021/exercise3.md
+    - Exercise 4: 2021/exercise4.md
   - Year 2020:
     - 2020/index.md
     - Midterm Exam 1: 2020/midexam1.md