criterion performance measurements
overview
want to understand this report?
KB 0/#req 4096/#threads 1/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7278861041635064 | 0.7349605701291111 | 0.7418426721905705 |
| Standard deviation | 1.3597399555105182e-16 | 1.175709972331994e-2 | 1.1920150433322532e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 4096/#threads 1/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7249702998530628 | 0.7268695733312418 | 0.7300524192303758 |
| Standard deviation | 1.3597399555105182e-16 | 2.773563224660993e-3 | 2.9341629905454475e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 4096/#threads 1/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6892508928292008 | 0.689901797291342 | 0.6904024326403211 |
| Standard deviation | 0.0 | 7.706877365883793e-4 | 8.67125860496736e-4 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 4096/#threads 1/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.64599017291798 | 0.6481612870990044 | 0.6493933427334283 |
| Standard deviation | 0.0 | 1.9309218007465552e-3 | 2.13398295657369e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.49245917830621316 | 0.4950251183501398 | 0.4963610676402095 |
| Standard deviation | 0.0 | 2.2278459231758856e-3 | 2.313932046736249e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.509022977829496 | 0.5130217350213897 | 0.5154783472975274 |
| Standard deviation | 0.0 | 3.7247963775888376e-3 | 4.254977276767909e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4688083025689265 | 0.4811350664269132 | 0.4874942523388926 |
| Standard deviation | 0.0 | 1.0691439716226438e-2 | 1.1014433094324497e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 2048/#threads 2/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.46778164317251864 | 0.4691583169620696 | 0.46998320184962994 |
| Standard deviation | 0.0 | 1.2606471477497772e-3 | 1.4287425356502622e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.33170050789139827 | 0.3351992948733948 | 0.3411942984024317 |
| Standard deviation | 0.0 | 5.215969454869923e-3 | 5.481242507175072e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3396187698530993 | 0.3436774306299739 | 0.3456674702372741 |
| Standard deviation | 0.0 | 3.5168792082820362e-3 | 3.58295696731542e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3440524057811634 | 0.3482573733745613 | 0.3505432298451248 |
| Standard deviation | 0.0 | 3.682926035617218e-3 | 3.9592195458260674e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 1024/#threads 4/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.31465231563055107 | 0.31827060287982617 | 0.3211912518952623 |
| Standard deviation | 1.8195493555074604e-3 | 3.85728208467394e-3 | 5.228555544954764e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.44931976214170205 | 0.45545674273838443 | 0.4602117834297508 |
| Standard deviation | 0.0 | 7.338049344816491e-3 | 8.235972069503944e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.44660077141483995 | 0.46207869952483566 | 0.46992499400270504 |
| Standard deviation | 0.0 | 1.3408145744684852e-2 | 1.359018068681686e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3021286488203603 | 0.303368965841746 | 0.3049923863931644 |
| Standard deviation | 7.662070600654713e-4 | 1.5810516202486604e-3 | 2.0436212348732613e-3 |
Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.
KB 0/#req 512/#threads 8/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.23115200865020027 | 0.23374693782464445 | 0.23972049245782942 |
| Standard deviation | 9.315102606164631e-4 | 4.794662106737427e-3 | 6.374667950790616e-3 |
Outlying measurements have moderate (0.1388888888888888%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4518893785526694 | 0.46040724603643995 | 0.46485789580818154 |
| Standard deviation | 0.0 | 7.3990773193852895e-3 | 7.708751531351161e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4777597183932339 | 0.4993615693525623 | 0.5126133521832922 |
| Standard deviation | 0.0 | 2.0100753454764534e-2 | 2.2952761153692915e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.30261689286670185 | 0.3060603326955787 | 0.30814255288398446 |
| Standard deviation | 1.4621986558347455e-3 | 3.374675064953174e-3 | 4.2536550645712766e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
KB 0/#req 256/#threads 16/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.20521781042672074 | 0.20755966304692594 | 0.2099015156671311 |
| Standard deviation | 1.9915456359186645e-3 | 3.1539533371525456e-3 | 4.421881896276148e-3 |
Outlying measurements have moderate (0.13888888888888876%) effect on estimated standard deviation.
KB 0/#req 128/#threads 32/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4640902614267797 | 0.47081001647777593 | 0.47671194127558386 |
| Standard deviation | 0.0 | 9.592933335062876e-3 | 1.022243361225395e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 128/#threads 32/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4854867282999256 | 0.4862660362491648 | 0.487296871892202 |
| Standard deviation | 0.0 | 9.309159960626634e-4 | 1.0450868902726446e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 128/#threads 32/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2993873821074808 | 0.30405323132901063 | 0.30975850870748667 |
| Standard deviation | 2.8935545244324623e-3 | 6.9515668330322215e-3 | 9.08283443924827e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
KB 0/#req 128/#threads 32/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.21784566980523437 | 0.22140473075141043 | 0.22713367915986474 |
| Standard deviation | 2.1387584376880465e-3 | 6.1562130963470985e-3 | 8.489568037839509e-3 |
Outlying measurements have moderate (0.1388888888888888%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-client
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5083060258673843 | 0.5252610370319238 | 0.5407067561735978 |
| Standard deviation | 0.0 | 2.554619941551834e-2 | 2.6752770312818506e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-client-global-manager
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.49938722140391106 | 0.507851847308052 | 0.5130621815742286 |
| Standard deviation | 0.0 | 7.89595782602737e-3 | 9.024563673434944e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-streams-mvar
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2858758023381678 | 0.2890920053796858 | 0.29454256346537 |
| Standard deviation | 5.690294289269574e-4 | 5.165910875819158e-3 | 6.578277070470348e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
KB 0/#req 64/#threads 64/http-streams-ioref
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2009714310309274 | 0.20445987888580652 | 0.20722290137052152 |
| Standard deviation | 2.966298410899109e-3 | 4.06436818175516e-3 | 4.789994637516271e-3 |
Outlying measurements have moderate (0.1388888888888888%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.