criterion performance measurements
overview
want to understand this report?
KB 0/#req 4096/#threads 1/http-client-local-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6945447807956394 | 0.6958670445819131 | 0.696443534191764 |
| Standard deviation | 0.0 | 1.1729300854895685e-3 | 1.2917187645368968e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 4096/#threads 1/http-client-local-manager-no-timeout
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6003688477891095 | 0.6130005049885668 | 0.6202969779162886 |
| Standard deviation | 0.0 | 1.13280202324006e-2 | 1.263786182686494e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 4096/#threads 1/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6399668437564033 | 0.6465790182401586 | 0.6498700256997139 |
| Standard deviation | 0.0 | 5.7264897757030525e-3 | 5.752430026336491e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 4096/#threads 1/http-streams-ioref-no-catchany
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5704915041423265 | 0.570886766685908 | 0.5711227560022452 |
| Standard deviation | 0.0 | 3.6113212956155534e-4 | 4.0874548593929555e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-client-local-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.48078434008489607 | 0.4824543109336044 | 0.4838987492661299 |
| Standard deviation | 6.798699777552591e-17 | 2.3312008927069494e-3 | 2.5018405803341696e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-client-local-manager-no-timeout
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4059948627739432 | 0.4133531771746779 | 0.4180653161116474 |
| Standard deviation | 0.0 | 7.0860464667137895e-3 | 8.161664051154769e-3 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.46511857610306717 | 0.466823384352453 | 0.4680375411965149 |
| Standard deviation | 0.0 | 1.8323655871898626e-3 | 2.1029813422725613e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-streams-ioref-no-catchany
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3688056551472146 | 0.38098780766120227 | 0.3876993618719092 |
| Standard deviation | 6.798699777552591e-17 | 1.0713009190929388e-2 | 1.1624752890697353e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-client-local-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3107998484810478 | 0.3141481097994447 | 0.31749637111784157 |
| Standard deviation | 2.8613077061659456e-3 | 3.96986736801207e-3 | 4.756846012448269e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-client-local-manager-no-timeout
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2660641422109244 | 0.2668085120781516 | 0.2684955611329388 |
| Standard deviation | 1.6154186185492848e-4 | 1.159421947632865e-3 | 1.4845513000652439e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3141007918252806 | 0.31854096797985615 | 0.32175417110114324 |
| Standard deviation | 1.8197670476842454e-3 | 4.7623089073243864e-3 | 6.5920027767347265e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-streams-ioref-no-catchany
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2388713571653348 | 0.24310553481319191 | 0.2476229396542681 |
| Standard deviation | 2.5012780861497774e-3 | 4.782056331322019e-3 | 6.736971341094461e-3 |
Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-client-local-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.45067336857751256 | 0.4528660051574159 | 0.4544718620082034 |
| Standard deviation | 0.0 | 2.43770533154772e-3 | 2.7814256552466214e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-client-local-manager-no-timeout
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.18205588867873862 | 0.18308040475594076 | 0.18483978607630608 |
| Standard deviation | 6.223612850806512e-4 | 1.7265915159426412e-3 | 2.4597240027326296e-3 |
Outlying measurements have moderate (0.1388888888888887%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.22822302941026015 | 0.2324300266992941 | 0.23619514810671283 |
| Standard deviation | 3.013075473335044e-3 | 4.99244653242614e-3 | 7.267206023084981e-3 |
Outlying measurements have moderate (0.1388888888888888%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-streams-ioref-no-catchany
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1641972665833877 | 0.16495018836427688 | 0.16578670053862912 |
| Standard deviation | 8.01987764059204e-4 | 1.1272710875031484e-3 | 1.5009546974905094e-3 |
Outlying measurements have moderate (0.12244897959183675%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-client-local-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4411739388232982 | 0.4460839398080177 | 0.4486804589266249 |
| Standard deviation | 6.798699777552591e-17 | 4.273327736862047e-3 | 4.497303036251693e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-client-local-manager-no-timeout
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1514758553240861 | 0.152424454650673 | 0.15378640923659026 |
| Standard deviation | 1.0061379230180835e-3 | 1.5656422073784954e-3 | 2.137765994317149e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1701893100511124 | 0.17335940088903934 | 0.17726367737303414 |
| Standard deviation | 3.0135908523595692e-3 | 4.735149566777617e-3 | 7.161798953214419e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-streams-ioref-no-catchany
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.14901515947159225 | 0.15037634195874408 | 0.15284955102667588 |
| Standard deviation | 1.6476039667725454e-3 | 2.364748827040983e-3 | 3.040608029280516e-3 |
Outlying measurements have moderate (0.12244897959183672%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-client-local-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4228685793277273 | 0.42489491319261635 | 0.42806366511886645 |
| Standard deviation | 0.0 | 2.7795806094655473e-3 | 2.999384179977384e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-client-local-manager-no-timeout
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.14590952573864402 | 0.14736837479506665 | 0.14915428594662936 |
| Standard deviation | 1.1272884245153005e-3 | 2.2538296916626904e-3 | 3.3747182402558105e-3 |
Outlying measurements have moderate (0.12244897959183675%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.15874909669698914 | 0.16415553382335415 | 0.17105501418518518 |
| Standard deviation | 5.36069433701367e-3 | 8.590003243564827e-3 | 1.1197163736210774e-2 |
Outlying measurements have moderate (0.12622108945334687%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-streams-ioref-no-catchany
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.18103666915166214 | 0.1881811242050511 | 0.20082180187014975 |
| Standard deviation | 1.7456661012178556e-3 | 1.2918851747345916e-2 | 1.753634208988458e-2 |
Outlying measurements have moderate (0.1513180032174908%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.