Fakultas Ilmu Komputer UI
Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
PMPL
Course Site
Commits
25ef25d8
Commit
25ef25d8
authored
Dec 01, 2021
by
Daya Adianto
Browse files
Merge branch 5-mutation-test-spec with refs/heads/master into refs/merge-requests/7/train
parents
f7823126
b5aca7b1
Pipeline
#88825
passed with stage
in 46 seconds
Changes
3
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
docs/2021/exercise3.md
View file @
25ef25d8
...
...
@@ -161,6 +161,10 @@ the system (e.g. invoking service methods).
At the end of the exercise, do not forget to schedule an one-on-one meeting
with a teaching assistant to demonstrate your work.
The due date of this exercise is:
**1 December 2021, 21:00 UTC+7**
. Please
ensure any updates to the fork repository related to this exercise were made
and pushed before the due date.
### Checklist
1.
[ ] Choose one
**complex function**
from your group project.
...
...
docs/2021/exercise4.md
0 → 100644
View file @
25ef25d8
# Exercise 4: Mutation Testing
## Mutation Testing
Mutation testing is a test that involves modifying a program in small ways,
usually by changing a variable or an operator. The goal here is to "kill" the
modified program with test cases, that is, by rejecting the mutant using the
original program's test cases.
### Creating Mutants
The mutant is created by applying one of these operation on a program line.
1.
Absolute Value Insertion (
**ABS**
): Each arithmetic expression (and
subexpression) is modified by the functions
`abs()`
,
`negAbs()`
, and
`failOnZero()`
.
```java
a = m * (o + p);
∆1 a = abs (m * (o + p));
∆2 a = m * abs ((o + p));
∆3 a = failOnZero (m * (o + p));
```
2.
Arithmetic Operator Replacement (
**AOR**
): Each occurrence of one of the
arithmetic operators is replaced by each of the other operators. It can also
be replaced by the special mutation operators such as
`leftOp`
and
`rightOp`
.
```java
a = m * (o + p);
∆1 a = m + (o + p);
∆2 a = m * (o * p);
∆3 a = m leftOp (o + p);
```
3.
Relational Operator Replacement (
**ROR**
): Each occurrence of one of the
relational operators (
`<`
,
`≤`
,
`>`
,
`≥`
,
`=`
,
`≠`
) is replaced by each of
the other operators and by
`falseOp`
and
`trueOp`
.
```java
if (X <= Y)
∆1 if (X > Y)
∆2 if (X < Y)
∆3 if (X falseOp Y) // always returns false
```
4.
Conditional Operator Replacement (
**COR**
): Each occurrence of one of the
logical operators (
`&&`
,
`||`
, etc.) is replaced by each of the other
operators. In addition, each is replaced by
`falseOp`
,
`trueOp`
,
`leftOp`
,
and
`rightOp`
.
```java
if (X <= Y && a > 0)
∆1 if (X <= Y || a > 0)
∆2 if (X <= Y leftOp a > 0) // returns result of left clause
```
5.
Shift Operator Replacement (
**SOR**
): Each occurrence of one of the shift
operators
`<<`
,
`>>`
, and
`>>>`
is replaced by each of the other operators.
In addition, each is replaced by the special mutation operator
`leftOp`
.
```java
byte b = (byte) 16;
b = b >> 2;
∆1 b = b << 2;
∆2 b = b leftOp 2; // result is b
```
6.
Logical Operator Replacement (
**LOR**
): Each occurrence of one of the logical
operators (bitwise and -
`&`
, bitwise or -
`|`
, exclusive or -
`^`
) is
replaced by each of the other operators. In addition, each is replaced by
`leftOp`
and
`rightOp`
.
```java
int a = 60; int b = 13;
int c = a & b;
∆1 int c = a | b;
∆2 int c = a rightOp b; // result is b
```
7.
Assignment Operator Replacement (
**ASR**
): Each occurrence of one of the
assignment operators (
`=`
,
`+=`
,
`-=`
, etc.) is replaced by each of the other
operators.
```java
a = m * (o + p);
∆1 a += m * (o + p);
∆2 a *= m * (o + p);
```
8.
Unary Operator Insertion (
**UOI**
): Each unary operator (arithmetic
`+`
,
arithmetic
`-`
, conditional
`!`
, logical
`~`
) is inserted in front of each
expression of the correct type.
```java
a = m * (o + p);
∆1 a = m * -(o + p);
∆2 a = -(m * (o + p));
```
9.
Unary Operator Deletion (
**UOD**
): Each unary operator (arithmetic
`+`
,
arithmetic
`-`
, conditional
`!`
, logical
`~`
) is deleted.
```java
if !(X <= Y && !Z)
∆1 if (X > Y && !Z)
∆2 if !(X < Y && Z)
```
10.
Scalar Variable Replacement (
**SVR**
): Each variable reference is replaced
by every other variable of the appropriate type that is declared in the
current scope.
```java
a = m * (o + p);
∆1 a = o * (o + p);
∆2 a = m * (m + p);
∆3 a = m * (o + o);
∆4 p = m * (o + p);
```
11.
Bomb Statement Replacement (
**BSR**
): Each statement is replaced by a
special
`Bomb()`
function.
```java
a = m * (o + p);
∆1 Bomb() // Raises exception when reached
```
## Killing Mutants
To kill a mutant, you need to recall the RIPR model introduced on
**Introduction to Software Testing by Paul Amann and Jeff Offutt. Chapter 2**
:
1.
Reachability: The test causes the faulty statement to be reached
(in mutation – the mutated statement).
2.
Infection: The test causes the faulty statement to result in an
**incorrect state**
.
3.
Propagation: The incorrect state propagates to
**incorrect output**
.
4.
Revealability : The tester must observe part of the incorrect output.
The RIPR model leads to two variants of mutant killing:
-
Weakly killing mutants: create a test that kill mutants with incorrect state. To achieve this, you need to
create a test that satisfies
**reachability and infection, but not propagation**
.
-
Strongly killing mutants: create a test that kill mutants with incorrect output. To achieve this, you need to
create a test that satisfies
**reachability, infection, and propagation**
.
To demonstrate how we kill a mutant, take a look at the snippet below:
```
java
boolean
isEven
(
int
X
)
{
if
(
X
<
0
)
X
=
0
-
X
;
∆
X
=
0
;
if
(
double
)
(
X
/
2
)
==
((
double
)
X
)
/
2.0
return
(
true
);
else
return
(
false
);
}
```
Ignoring the revealability, we can summarise our RIPR model as follows:
```
markdown
Reachability (R): X < 0
Infection (I): X != 0
Propagation (P): odd(0-X)
```
Now that we have found the reachability, infection, and propagation of the
mutant, we can finally create a
**full test specification**
to kill the mutant.
To weakly kill the mutant, we need to satisfy these conditions:
`R && I && !P`
.
```
markdown
Weak kill = R && I && !P
= (X < 0) && (X != 0) && !odd(0-X)
= (X < 0) && even(0-X)
```
One of the
`X`
value that satifies this is
`X=-6`
.
To strongly kill the mutant, we need to satisfy these conditions:
`R && I && P`
.
```
markdown
Weak kill = R && I && P
= (X < 0) && (X != 0) && odd(0-X)
= (X < 0) && odd(0-X)
```
One of the
`X`
value that satifies this is
`X=-5`
.
For more information regarding mutant killings, please read the
**Introduction to Software Testing by Paul Amann and Jeff Offutt on Chapter 9.2.2**
## Tasks
You are asked to create a mutation test on your group
project
**individually**
. Implement this exercise using a
**new branch**
of
your
**previously forked group project codebase**
.
This exercise consist of 2 phases, where each phase requires you to
**commit and push your work**
to
[
GitLab CS
](
https://gitlab.cs.ui.ac.id
)
.
At the end of the exercise, do not forget to schedule an one-on-one meeting
with a teaching assistant to demonstrate your work.
### Task 1: Creating Mutants
You have learnt about how mutation testing works. Now it is your turn to
conduct it in your forked repository. Please do these tasks below.
1.
[ ] Pick one complex method that is different from what you use on previous exercise,
and then try to create at least 2 mutants from it with 2 different kinds of mutations.
**You can write the mutants on code or in a written document**
.
2.
[ ] Create the
**full test specification to strongly kill your mutants**
in a written
document by analyzing the reachability, infection, and propagation of those mutants.
3.
[ ] Compare the
**full test specification**
with the current method's test suite?
Are they current test suite enough to kill all mutants?
You can relate question 2 and 3 with your
**full test specification**
for each mutants.
### Task 2: Mutation Testing Tools
There are several tools that can support mutation testing.
Try to use one of them by doing these tasks below.
1.
[ ] Choose one mutation testing tools and apply it on your forked repository.
2.
[ ] Use that tool to conduct mutation testing on your selected method (not the mutants
you implement).
3.
[ ] Improve your test based on the conducted mutation testing.
4.
[ ] Modify your
`.gitlab-ci.yml`
to include mutation testing using the selected tool
if necessary.
5.
[ ] Commit and push your improvement.
After conducting all the tasks above, please explain these in a written document:
## Deliverables
At the end of this exercise, you are required to prepare the following
artifacts:
-
[ ] A written document that describes the process of your work in completing
this exercise. You can write the document as a Markdown-formatted text file
or a PDF file. Give the document a descriptive name, e.g.
`exercise4.md`
, and
put it into a folder called
`docs`
in your fork.
-
[ ] The mutants you created.
-
[ ] One or more changed test suites, if any, that had been updated from the
result of mutation testing tools.
> If there are no changes in the test suites, explain the reasons in the
> documentation.
## Work Demonstration
Arrange an one-on-one meeting with a teaching assistant to demonstrate your work.
You are expected to be able to:
-
Describe how you conduct your
**full test specification**
and how it can kill your mutants.
-
Explain about the mutation testing tools you're using.
-
Explain the test improvement using mutation testing tools.
-
Explain the benefit of mutatin testing.
The due date of this exercise is:
**8 December 2021, 21:00 UTC+7**
. Please
ensure any updates to the fork repository related to this exercise were made
and pushed before the due date.
## References
-
Introduction to Software Testing by Paul Amann and Jeff Offutt.
-
[
Mutation Testing Tools for Java
](
https://pitest.org/java_mutation_testing_systems/
)
*
[RIPR]: Reachability Infection Propagation Revealability
mkdocs.yml
View file @
25ef25d8
...
...
@@ -55,6 +55,7 @@ nav:
-
Exercise 1
:
2021/exercise1.md
-
Exercise 2
:
2021/exercise2.md
-
Exercise 3
:
2021/exercise3.md
-
Exercise 4
:
2021/exercise4.md
-
Year 2020
:
-
2020/index.md
-
Midterm Exam 1
:
2020/midexam1.md
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment