Fakultas Ilmu Komputer UI

Commit 65cf37bc authored by SAMUEL TUPA FEBRIAN's avatar SAMUEL TUPA FEBRIAN
Browse files

Merge branch '5-mutation-test-spec' into 'master'

Create the exercise specs on mutation testing

Closes #5

See merge request !7
parents f7823126 b5aca7b1
Pipeline #88824 passed with stage
in 51 seconds
......@@ -161,6 +161,10 @@ the system (e.g. invoking service methods).
At the end of the exercise, do not forget to schedule an one-on-one meeting
with a teaching assistant to demonstrate your work.
The due date of this exercise is: **1 December 2021, 21:00 UTC+7**. Please
ensure any updates to the fork repository related to this exercise were made
and pushed before the due date.
### Checklist
1. [ ] Choose one **complex function** from your group project.
......
# Exercise 4: Mutation Testing
## Mutation Testing
Mutation testing is a test that involves modifying a program in small ways,
usually by changing a variable or an operator. The goal here is to "kill" the
modified program with test cases, that is, by rejecting the mutant using the
original program's test cases.
### Creating Mutants
The mutant is created by applying one of these operation on a program line.
1. Absolute Value Insertion (**ABS**): Each arithmetic expression (and
subexpression) is modified by the functions `abs()`, `negAbs()`, and
`failOnZero()`.
```java
a = m * (o + p);
∆1 a = abs (m * (o + p));
∆2 a = m * abs ((o + p));
∆3 a = failOnZero (m * (o + p));
```
2. Arithmetic Operator Replacement (**AOR**): Each occurrence of one of the
arithmetic operators is replaced by each of the other operators. It can also
be replaced by the special mutation operators such as `leftOp` and `rightOp`.
```java
a = m * (o + p);
∆1 a = m + (o + p);
∆2 a = m * (o * p);
∆3 a = m leftOp (o + p);
```
3. Relational Operator Replacement (**ROR**): Each occurrence of one of the
relational operators (`<`, `≤`, `>`, `≥`, `=`, `≠`) is replaced by each of
the other operators and by `falseOp` and `trueOp`.
```java
if (X <= Y)
∆1 if (X > Y)
∆2 if (X < Y)
∆3 if (X falseOp Y) // always returns false
```
4. Conditional Operator Replacement (**COR**): Each occurrence of one of the
logical operators (`&&`, `||`, etc.) is replaced by each of the other
operators. In addition, each is replaced by `falseOp`, `trueOp`, `leftOp`,
and `rightOp`.
```java
if (X <= Y && a > 0)
∆1 if (X <= Y || a > 0)
∆2 if (X <= Y leftOp a > 0) // returns result of left clause
```
5. Shift Operator Replacement (**SOR**): Each occurrence of one of the shift
operators `<<`, `>>`, and `>>>` is replaced by each of the other operators.
In addition, each is replaced by the special mutation operator `leftOp`.
```java
byte b = (byte) 16;
b = b >> 2;
∆1 b = b << 2;
∆2 b = b leftOp 2; // result is b
```
6. Logical Operator Replacement (**LOR**): Each occurrence of one of the logical
operators (bitwise and - `&`, bitwise or - `|`, exclusive or - `^`) is
replaced by each of the other operators. In addition, each is replaced by
`leftOp` and `rightOp`.
```java
int a = 60; int b = 13;
int c = a & b;
∆1 int c = a | b;
∆2 int c = a rightOp b; // result is b
```
7. Assignment Operator Replacement (**ASR**): Each occurrence of one of the
assignment operators (`=`, `+=`, `-=`, etc.) is replaced by each of the other
operators.
```java
a = m * (o + p);
∆1 a += m * (o + p);
∆2 a *= m * (o + p);
```
8. Unary Operator Insertion (**UOI**): Each unary operator (arithmetic `+`,
arithmetic `-`, conditional `!`, logical `~`) is inserted in front of each
expression of the correct type.
```java
a = m * (o + p);
∆1 a = m * -(o + p);
∆2 a = -(m * (o + p));
```
9. Unary Operator Deletion (**UOD**): Each unary operator (arithmetic `+`,
arithmetic `-`, conditional `!`, logical `~`) is deleted.
```java
if !(X <= Y && !Z)
∆1 if (X > Y && !Z)
∆2 if !(X < Y && Z)
```
10. Scalar Variable Replacement (**SVR**): Each variable reference is replaced
by every other variable of the appropriate type that is declared in the
current scope.
```java
a = m * (o + p);
∆1 a = o * (o + p);
∆2 a = m * (m + p);
∆3 a = m * (o + o);
∆4 p = m * (o + p);
```
11. Bomb Statement Replacement (**BSR**): Each statement is replaced by a
special `Bomb()` function.
```java
a = m * (o + p);
∆1 Bomb() // Raises exception when reached
```
## Killing Mutants
To kill a mutant, you need to recall the RIPR model introduced on
**Introduction to Software Testing by Paul Amann and Jeff Offutt. Chapter 2**:
1. Reachability: The test causes the faulty statement to be reached
(in mutation – the mutated statement).
2. Infection: The test causes the faulty statement to result in an
**incorrect state**.
3. Propagation: The incorrect state propagates to **incorrect output**.
4. Revealability : The tester must observe part of the incorrect output.
The RIPR model leads to two variants of mutant killing:
- Weakly killing mutants: create a test that kill mutants with incorrect state. To achieve this, you need to
create a test that satisfies **reachability and infection, but not propagation**.
- Strongly killing mutants: create a test that kill mutants with incorrect output. To achieve this, you need to
create a test that satisfies **reachability, infection, and propagation**.
To demonstrate how we kill a mutant, take a look at the snippet below:
```java
boolean isEven (int X) {
if (X < 0)
X = 0 - X;
X = 0;
if (double) (X/2) == ((double) X) / 2.0
return (true);
else
return (false);
}
```
Ignoring the revealability, we can summarise our RIPR model as follows:
```markdown
Reachability (R): X < 0
Infection (I): X != 0
Propagation (P): odd(0-X)
```
Now that we have found the reachability, infection, and propagation of the
mutant, we can finally create a **full test specification** to kill the mutant.
To weakly kill the mutant, we need to satisfy these conditions: `R && I && !P`.
```markdown
Weak kill = R && I && !P
= (X < 0) && (X != 0) && !odd(0-X)
= (X < 0) && even(0-X)
```
One of the `X` value that satifies this is `X=-6`.
To strongly kill the mutant, we need to satisfy these conditions: `R && I && P`.
```markdown
Weak kill = R && I && P
= (X < 0) && (X != 0) && odd(0-X)
= (X < 0) && odd(0-X)
```
One of the `X` value that satifies this is `X=-5`.
For more information regarding mutant killings, please read the
**Introduction to Software Testing by Paul Amann and Jeff Offutt on Chapter 9.2.2**
## Tasks
You are asked to create a mutation test on your group
project **individually**. Implement this exercise using a **new branch** of
your **previously forked group project codebase**.
This exercise consist of 2 phases, where each phase requires you to
**commit and push your work** to [GitLab CS](https://gitlab.cs.ui.ac.id).
At the end of the exercise, do not forget to schedule an one-on-one meeting
with a teaching assistant to demonstrate your work.
### Task 1: Creating Mutants
You have learnt about how mutation testing works. Now it is your turn to
conduct it in your forked repository. Please do these tasks below.
1. [ ] Pick one complex method that is different from what you use on previous exercise,
and then try to create at least 2 mutants from it with 2 different kinds of mutations.
**You can write the mutants on code or in a written document**.
2. [ ] Create the **full test specification to strongly kill your mutants** in a written
document by analyzing the reachability, infection, and propagation of those mutants.
3. [ ] Compare the **full test specification** with the current method's test suite?
Are they current test suite enough to kill all mutants?
You can relate question 2 and 3 with your **full test specification** for each mutants.
### Task 2: Mutation Testing Tools
There are several tools that can support mutation testing.
Try to use one of them by doing these tasks below.
1. [ ] Choose one mutation testing tools and apply it on your forked repository.
2. [ ] Use that tool to conduct mutation testing on your selected method (not the mutants
you implement).
3. [ ] Improve your test based on the conducted mutation testing.
4. [ ] Modify your `.gitlab-ci.yml` to include mutation testing using the selected tool
if necessary.
5. [ ] Commit and push your improvement.
After conducting all the tasks above, please explain these in a written document:
## Deliverables
At the end of this exercise, you are required to prepare the following
artifacts:
- [ ] A written document that describes the process of your work in completing
this exercise. You can write the document as a Markdown-formatted text file
or a PDF file. Give the document a descriptive name, e.g. `exercise4.md`, and
put it into a folder called `docs` in your fork.
- [ ] The mutants you created.
- [ ] One or more changed test suites, if any, that had been updated from the
result of mutation testing tools.
> If there are no changes in the test suites, explain the reasons in the
> documentation.
## Work Demonstration
Arrange an one-on-one meeting with a teaching assistant to demonstrate your work.
You are expected to be able to:
- Describe how you conduct your **full test specification** and how it can kill your mutants.
- Explain about the mutation testing tools you're using.
- Explain the test improvement using mutation testing tools.
- Explain the benefit of mutatin testing.
The due date of this exercise is: **8 December 2021, 21:00 UTC+7**. Please
ensure any updates to the fork repository related to this exercise were made
and pushed before the due date.
## References
- Introduction to Software Testing by Paul Amann and Jeff Offutt.
- [Mutation Testing Tools for Java](https://pitest.org/java_mutation_testing_systems/)
*[RIPR]: Reachability Infection Propagation Revealability
......@@ -55,6 +55,7 @@ nav:
- Exercise 1: 2021/exercise1.md
- Exercise 2: 2021/exercise2.md
- Exercise 3: 2021/exercise3.md
- Exercise 4: 2021/exercise4.md
- Year 2020:
- 2020/index.md
- Midterm Exam 1: 2020/midexam1.md
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment