Skip to content

Commit 9d85736

Browse files
caaladorcaalador
committed
feat: add documentation for setting thresholds
Add documentation for default and customized thresholds in load tests.
1 parent 7c8e9e0 commit 9d85736

1 file changed

Lines changed: 275 additions & 0 deletions

File tree

‎articles/flow/testing/load-testing/thresholds.adoc‎

Lines changed: 275 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,275 @@
1+
---
2+
title: Load Testing Thresholds
3+
page-title: Setting Thresholds for Load Tests
4+
description: How to set and control failure thresholds for the load tests
5+
meta-description: Learn how to set and control failure thresholds for the load tests
6+
order: 60
7+
version: since:com.vaadin:vaadin@V25.2
8+
---
9+
10+
= [since:com.vaadin:vaadin@V25.2]#Load Testing Thresholds#
11+
12+
== Overview
13+
14+
k6 thresholds define pass/fail criteria for your load tests.
15+
16+
When a threshold is breached, k6 marks the test as failed, and the Maven build fails if `k6.failOnThreshold=true` (the default).
17+
18+
The TestBench k6 converter plugin generates some default thresholds automatically in every k6 script produced.
19+
You can configure the default thresholds, disable individual ones, or add custom thresholds for any k6 metric.
20+
21+
== Default Thresholds
22+
23+
Every generated k6 script includes these thresholds out of the box:
24+
25+
[cols="2,3,1", options="header"]
26+
|===
27+
| Metric | Expression | Default
28+
29+
| `http_req_duration`
30+
| 95th percentile response time
31+
| `p(95)<2000` (2 seconds)
32+
33+
| `http_req_duration`
34+
| 99th percentile response time
35+
| `p(99)<5000` (5 seconds)
36+
37+
| `checks`
38+
| All checks must pass
39+
| `rate==1` with `abortOnFail: true`
40+
|===
41+
42+
This generates the following block in the k6 script:
43+
44+
[source,javascript]
45+
----
46+
export const options = {
47+
thresholds: {
48+
checks: [{ threshold: 'rate==1', abortOnFail: true, delayAbortEval: '5s' }],
49+
http_req_duration: ['p(95)<2000', 'p(99)<5000'],
50+
},
51+
}
52+
----
53+
54+
== Editing Default Thresholds
55+
56+
You can change the default threshold values with Maven properties or through configuration in the POM.
57+
58+
=== Via command line
59+
60+
[source,bash]
61+
----
62+
# Tighter response time requirements
63+
mvn k6:convert -Dk6.harFile=recording.har \
64+
-Dk6.threshold.httpReqDurationP95=1000 \
65+
-Dk6.threshold.httpReqDurationP99=3000
66+
67+
# Allow checks to fail without aborting the test
68+
mvn k6:convert -Dk6.harFile=recording.har \
69+
-Dk6.threshold.checksAbortOnFail=false
70+
71+
# Disable the p95 threshold entirely (set to 0)
72+
mvn k6:convert -Dk6.harFile=recording.har \
73+
-Dk6.threshold.httpReqDurationP95=0
74+
----
75+
76+
=== Via POM configuration
77+
78+
[source,xml]
79+
----
80+
<plugin>
81+
<groupId>com.vaadin</groupId>
82+
<artifactId>testbench-converter-plugin</artifactId>
83+
<configuration>
84+
<!-- 95th percentile response time in ms (0 to disable) -->
85+
<httpReqDurationP95>1000</httpReqDurationP95>
86+
<!-- 99th percentile response time in ms (0 to disable) -->
87+
<httpReqDurationP99>3000</httpReqDurationP99>
88+
<!-- Set to false to continue the test even when checks fail -->
89+
<checksAbortOnFail>false</checksAbortOnFail>
90+
</configuration>
91+
</plugin>
92+
----
93+
94+
=== Default threshold parameters
95+
96+
[cols="3,1,4", options="header"]
97+
|===
98+
| Parameter | Default | Description
99+
100+
| `k6.threshold.httpReqDurationP95`
101+
| `2000`
102+
| 95th percentile response time threshold in ms. Set to `0` to disable.
103+
104+
| `k6.threshold.httpReqDurationP99`
105+
| `5000`
106+
| 99th percentile response time threshold in ms. Set to `0` to disable.
107+
108+
| `k6.threshold.checksAbortOnFail`
109+
| `true`
110+
| When `true`, the test aborts immediately on the first check failure. When `false`, failures are recorded but the test continues.
111+
|===
112+
113+
== Adding Custom Thresholds
114+
115+
You can add pass/fail thresholds for any https://grafana.com/docs/k6/latest/using-k6/metrics/[k6 built-in metric] using the `k6.threshold.custom` property.
116+
117+
The format is a comma-separated list of `metric:expression` pairs:
118+
119+
[source]
120+
----
121+
metric1:expression1,metric2:expression2,...
122+
----
123+
124+
=== Via command line
125+
126+
[source,bash]
127+
----
128+
# Fail if more than 1% of requests fail
129+
mvn k6:run -Dk6.testFile=test.js \
130+
-Dk6.threshold.custom="http_req_failed:rate<0.01"
131+
132+
# Multiple custom thresholds
133+
mvn k6:run -Dk6.testFile=test.js \
134+
-Dk6.threshold.custom="http_req_failed:rate<0.01,http_req_waiting:p(95)<500,http_reqs:count>100"
135+
----
136+
137+
=== Via POM configuration
138+
139+
[source,xml]
140+
----
141+
<plugin>
142+
<groupId>com.vaadin</groupId>
143+
<artifactId>testbench-converter-plugin</artifactId>
144+
<configuration>
145+
<!-- Custom thresholds in metric:expression format -->
146+
<customThresholds>http_req_failed:rate&lt;0.01,http_req_waiting:p(95)&lt;500</customThresholds>
147+
</configuration>
148+
</plugin>
149+
----
150+
151+
NOTE: In XML, use `&lt;` instead of `<` inside threshold expressions.
152+
153+
=== Combining defaults and custom thresholds
154+
155+
Custom thresholds are added alongside the defaults.
156+
If you add a custom `http_req_duration` expression, it is merged with the default p95/p99 thresholds:
157+
158+
[source,bash]
159+
----
160+
# Adds a median threshold alongside the existing p95 and p99
161+
mvn k6:convert -Dk6.harFile=recording.har \
162+
-Dk6.threshold.custom="http_req_duration:p(50)<1000"
163+
----
164+
165+
This produces:
166+
167+
[source,javascript]
168+
----
169+
export const options = {
170+
thresholds: {
171+
checks: [{ threshold: 'rate==1', abortOnFail: true, delayAbortEval: '5s' }],
172+
http_req_duration: ['p(95)<2000', 'p(99)<5000', 'p(50)<1000'],
173+
},
174+
}
175+
----
176+
177+
== Common Threshold Examples
178+
179+
Below are commonly used threshold configurations for Vaadin load tests.
180+
181+
=== Strict performance requirements
182+
183+
[source,bash]
184+
----
185+
mvn k6:run -Dk6.testFile=test.js \
186+
-Dk6.threshold.httpReqDurationP95=500 \
187+
-Dk6.threshold.httpReqDurationP99=1000 \
188+
-Dk6.threshold.custom="http_req_failed:rate<0.001,http_req_waiting:p(95)<300"
189+
----
190+
191+
Generated output:
192+
[source,javascript]
193+
----
194+
thresholds: {
195+
checks: [{ threshold: 'rate==1', abortOnFail: true, delayAbortEval: '5s' }],
196+
http_req_duration: ['p(95)<500', 'p(99)<1000'],
197+
http_req_failed: ['rate<0.001'],
198+
http_req_waiting: ['p(95)<300'],
199+
},
200+
----
201+
202+
=== Relaxed thresholds for stress testing
203+
204+
[source,bash]
205+
----
206+
mvn k6:run -Dk6.testFile=test.js \
207+
-Dk6.threshold.httpReqDurationP95=10000 \
208+
-Dk6.threshold.httpReqDurationP99=0 \
209+
-Dk6.threshold.checksAbortOnFail=false \
210+
-Dk6.threshold.custom="http_req_failed:rate<0.05"
211+
----
212+
213+
Generated output:
214+
[source,javascript]
215+
----
216+
thresholds: {
217+
checks: ['rate==1'],
218+
http_req_duration: ['p(95)<10000'],
219+
http_req_failed: ['rate<0.05'],
220+
},
221+
----
222+
223+
=== Throughput minimum
224+
225+
[source,bash]
226+
----
227+
mvn k6:run -Dk6.testFile=test.js \
228+
-Dk6.threshold.custom="http_reqs:count>1000"
229+
----
230+
231+
== k6 Metric Reference
232+
233+
The following k6 built-in metrics can be used with custom thresholds.
234+
For the full reference, see the https://grafana.com/docs/k6/latest/using-k6/metrics/reference/[k6 metrics documentation].
235+
236+
[cols="2,2,3", options="header"]
237+
|===
238+
| Metric | Type | Description
239+
240+
| `http_req_duration`
241+
| Trend
242+
| Total time for the request (send + wait + receive)
243+
244+
| `http_req_waiting`
245+
| Trend
246+
| Time spent waiting for response (TTFB)
247+
248+
| `http_req_connecting`
249+
| Trend
250+
| Time spent establishing TCP connection
251+
252+
| `http_req_sending`
253+
| Trend
254+
| Time spent sending data
255+
256+
| `http_req_receiving`
257+
| Trend
258+
| Time spent receiving data
259+
260+
| `http_req_failed`
261+
| Rate
262+
| Rate of failed requests
263+
264+
| `http_reqs`
265+
| Counter
266+
| Total number of requests
267+
268+
| `checks`
269+
| Rate
270+
| Rate of successful checks
271+
|===
272+
273+
*Trend* metrics support: `p(N)<value`, `avg<value`, `min<value`, `max<value`, `med<value` +
274+
*Rate* metrics support: `rate<value`, `rate>value` +
275+
*Counter* metrics support: `count<value`, `count>value`

0 commit comments

Comments
 (0)