Fakultas Ilmu Komputer UI

Commit 117e08a6 authored by SAMUEL TUPA FEBRIAN's avatar SAMUEL TUPA FEBRIAN
Browse files

Merge branch '5-mutation-test-spec' into 'master'

Create the exercise specs on mutation testing

Closes #5

See merge request !7
parents f7823126 d3c747ca
Pipeline #88820 passed with stage
in 57 seconds
# Exercise 4: Mutation Testing
## Mutation Testing
Mutation testing is a test that involves modifying a program in small ways, usually by changing a variable or an operator.
The goal here is to "kill" the modified program with testcases by reject the mutant using the original program's test cases.
### Creating Mutants
The mutant is created by applying one of these operation on a program line.
1. Absolute Value Insertion (ABS): Each arithmetic expression (and subexpression) is
modified by the functions abs(), negAbs(), and failOnZero().
```java
a = m * (o + p);
1 a = abs (m * (o + p));
2 a = m * abs ((o + p));
3 a = failOnZero (m * (o + p));
```
2. Arithmetic Operator Replacement (AOR): Each occurrence of one of the arithmetic operators is replaced
by each of the other operators. Iit can also be replaced by the special mutation operators such as leftOp,
and rightOp.
```java
a = m * (o + p);
1 a = m + (o + p);
2 a = m * (o * p);
3 a = m leftOp (o + p);
```
3. Relational Operator Replacement (ROR): Each occurrence of one of the relational operators
(<, ≤, >, ≥, =, ≠) is replaced by each of the other operators and by falseOp and trueOp.
```java
if (X <= Y)
1 if (X > Y)
2 if (X < Y)
3 if (X falseOp Y) // always returns false
```
4. Conditional Operator Replacement (COR): Each occurrence of one of the logical operators (&&, ||, etc)
is replaced by each of the other operators; in addition, each is replaced by falseOp, trueOp, leftOp,
and rightOp.
```java
if (X <= Y && a > 0)
1 if (X <= Y || a > 0)
2 if (X <= Y leftOp a > 0) // returns result of left clause
```
5. Shift Operator Replacement (SOR): Each occurrence of one of the shift operators <<, >>, and >>>
is replaced by each of the other operators. In addition, each is replaced by the special mutation
operator leftOp.
```java
byte b = (byte) 16;
b = b >> 2;
1 b = b << 2;
2 b = b leftOp 2; // result is b
```
6. Logical Operator Replacement (LOR): Each occurrence of one of the logical operators (bitwise and -
&, bitwise or - |, exclusive or - ^) is replaced by each of the other operators; in addition, each is
replaced by leftOp and rightOp.
```java
int a = 60; int b = 13;
int c = a & b;
1 int c = a | b;
2 int c = a rightOp b; // result is b
```
7. Assignment Operator Replacement (ASR): Each occurrence of one of the assignment operators (=, +=, -=,
etc) is replaced by each of the other operators.
```java
a = m * (o + p);
1 a += m * (o + p);
2 a *= m * (o + p);
```
8. Unary Operator Insertion (UOI): Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~)
is inserted in front of each expression of the correct type.
```java
a = m * (o + p);
1 a = m * -(o + p);
2 a = -(m * (o + p));
```
9. Unary Operator Deletion (UOD): Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~)
is deleted.
```java
if !(X <= Y && !Z)
1 if (X > Y && !Z)
2 if !(X < Y && Z)
```
10. Scalar Variable Replacement (SVR): Each variable reference is replaced by every other variable of the
appropriate type that is declared in the current scope.
```java
a = m * (o + p);
1 a = o * (o + p);
2 a = m * (m + p);
3 a = m * (o + o);
4 p = m * (o + p);
```
11. Bomb Statement Replacement (BSR): Each statement is replaced by a special Bomb() function.
```java
a = m * (o + p);
1 Bomb() // Raises exception when reached
```
## Killing Mutants
To kill a mutant, you need to recall the RIPR model introduced on
**Introduction to Software Testing by Paul Amann and Jeff Offutt. Chapter 2**.
1. Reachability: The test causes the faulty statement to be reached (in mutation – the mutated statement)
2. Infection: The test causes the faulty statement to result in an **incorrect state**.
3. Propagation: The incorrect state propagates to **incorrect output**.
4. Revealability : The tester must observe part of the incorrect output.
The RIPR model leads to two variants of mutant killing:
- Weakly killing mutants: create a test that kill mutants with incorrect state. To achieve this, you need to
create a test that satisfies **reachability and infection, but not propagation**.
- Strongly killing mutants: create a test that kill mutants with incorrect output. To achieve this, you need to
create a test that satisfies **reachability, infection, and propagation**.
To demonstrate how we kill a mutant, take a look at the snippet below.
```java
boolean isEven (int X) {
if (X < 0)
X = 0 - X;
X = 0;
if (double) (X/2) == ((double) X) / 2.0
return (true);
else
return (false);
}
```
Ignoring the revealibility, we can summarise our RIPR model as follows:
```
Reachability (R): X < 0
Infection (I): X != 0
Propagation (P): odd(0-X)
```
Now that we have founf the reachability, infection, and propagation of this mutant, we can finally
create a **full test specification** to kill this mutant.
To weakly kill the mutant, we need to satisfy these conditions: ``R && I && !P``.
```
Weak kill = R && I && !P
= (X < 0) && (X != 0) && !odd(0-X)
= (X < 0) && even(0-X)
```
One of the ``X`` value that satifies this is ``X=-6``.
To strongly kill the mutant, we need to satisfy these conditions: ``R && I && P``.
```
Weak kill = R && I && P
= (X < 0) && (X != 0) && odd(0-X)
= (X < 0) && odd(0-X)
```
One of the ``X`` value that satifies this is ``X=-5``.
For more information regarding mutant killings, please read the
**Introduction to Software Testing by Paul Amann and Jeff Offutt on Chapter 9.2.2**
## Task
You are asked to create a mutation test on your group
project **individually**. Implement this exercise using a **new branch** of
your **previously forked group project codebase**.
This exercise consist of 2 phases, where each phase requires you to
**commit and push your work** to the `gitlab.cs`.
At the end of the exercise, do not forget to schedule an one-on-one meeting
with a teaching assistant to demonstrate your work.
### Task 1: Creating Mutants
You've learnt about how mutation testing works. Now it's your turn to
conduct it in your forked repository. Please do these tasks below.
1. [ ] Pick one complex method that is different from what you use on previous exercise,
and then try to create at least 2 mutants from it with 2 different kinds of mutations.
**You can write the mutants on code or in a written document**.
2. [ ] Create the **full test specification to strongly kill your mutants** in a written
document by analyzing the reachability, infection, and propagation of those mutants.
3. [ ] Compare the **full test specification** with the current method's test suite?
Are they current test suite enough to kill all mutants?
You can relate question 2 and 3 with your **full test specification** for each mutants.
### Task 2: Mutation Testing Tools
There are several tools that can support mutation testing.
Try to use one of them by doing these tasks below.
1. [ ] Choose one mutation testing tools and apply it on your forked repository.
2. [ ] Use that tool to conduct mutation testing on your selected method (not the mutants
you implement).
3. [ ] Improve your test based on the conducted mutation testing.
4. [ ] Modify your `.gitlab-ci.yml` to include mutation testing using the selected tool
if necessary.
5. [ ] Commit and push your improvement.
After conducting all the tasks above, please explain these in a written document:
## Deliverables
At the end of this exercise, you are required to prepare the following
artifacts:
- [ ] A written document that describes the process of your work in completing
this exercise. You can write the document as a Markdown-formatted text file
or a PDF file. Give the document a descriptive name, e.g. `exercise4.md`, and
put it into a folder called `docs` in your fork.
- [ ] The mutants you created.
- [ ] One or more changed test suites, if any, that had been updated from the
result of mutation testing tools.
> If there are no changes in the test suites, explain the reasons in the
> documentation.
## Work Demonstration
Arrange an one-on-one meeting with a teaching assistant to demonstrate your work.
You are expected to be able to:
- Describe how you conduct your **full test specification** and how it can kill your mutants.
- Explain about the mutation testing tools you're using.
- Explain the test improvement using mutation testing tools.
- Explain the benefit of mutatin testing.
The due date of this exercise is: **8 December 2021, 21:00 UTC+7**. Please
ensure any updates to the fork repository related to this exercise were made
and pushed before the due date.
## References
- Introduction to Software Testing by Paul Amann and Jeff Offutt.
- [Mutation Testing Tools for Java](https://pitest.org/java_mutation_testing_systems/)
......@@ -55,6 +55,7 @@ nav:
- Exercise 1: 2021/exercise1.md
- Exercise 2: 2021/exercise2.md
- Exercise 3: 2021/exercise3.md
- Exercise 4: 2021/exercise4.md
- Year 2020:
- 2020/index.md
- Midterm Exam 1: 2020/midexam1.md
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment