criterion performance measurements
overview
want to understand this report?
plain/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1810663624031855e-3 | 1.2337914114476207e-3 | 1.2956923122505542e-3 |
| Standard deviation | 1.682696707726267e-4 | 1.969590653671359e-4 | 2.3270487260890007e-4 |
Outlying measurements have severe (0.871613641405977%) effect on estimated standard deviation.
plain/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.068001720842705e-3 | 2.131342006815145e-3 | 2.1986958256651015e-3 |
| Standard deviation | 1.4890804652307e-4 | 1.9447456787983084e-4 | 2.4107802251515596e-4 |
Outlying measurements have severe (0.6460844296966931%) effect on estimated standard deviation.
plain/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.251281087948135e-3 | 9.429086682313724e-3 | 9.713301538732446e-3 |
| Standard deviation | 5.183473194187339e-4 | 7.165594587393573e-4 | 1.0449508893730474e-3 |
Outlying measurements have moderate (0.40400667441981086%) effect on estimated standard deviation.
plain/100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8641721733997408e-2 | 1.9072227990235353e-2 | 1.9677007091786446e-2 |
| Standard deviation | 9.904545984829292e-4 | 1.3098545409427178e-3 | 1.8372579997599022e-3 |
Outlying measurements have moderate (0.30293761309616535%) effect on estimated standard deviation.
plain/500
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.637213026409434e-2 | 9.90604415042705e-2 | 0.10241781682915062 |
| Standard deviation | 2.7303109334533516e-3 | 4.8070506492148566e-3 | 7.020269483605975e-3 |
Outlying measurements have moderate (0.10200913649314063%) effect on estimated standard deviation.
plain/1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.19608386268275224 | 0.20182253314267462 | 0.20393912337792575 |
| Standard deviation | 1.1554854110739602e-3 | 4.200878947223891e-3 | 5.79652108421113e-3 |
Outlying measurements have moderate (0.13888888888888884%) effect on estimated standard deviation.
with-timeout/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.29097020934175244 | 0.2965350189339975 | 0.29993697761764637 |
| Standard deviation | 2.1924522606565603e-3 | 5.560933391904584e-3 | 7.654187486033753e-3 |
Outlying measurements have moderate (0.16000000000000003%) effect on estimated standard deviation.
with-timeout/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2979558291790374 | 0.30345747228582076 | 0.31237847742171243 |
| Standard deviation | 3.0256349944544746e-3 | 8.740981912373642e-3 | 1.186130808201388e-2 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
with-timeout/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3193134690760344 | 0.33002136421240713 | 0.33849338499999726 |
| Standard deviation | 6.524974713587814e-3 | 1.0812420251543208e-2 | 1.434900616504881e-2 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
with-timeout/100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3537824657546719 | 0.3685867756738876 | 0.37680404912243187 |
| Standard deviation | 0.0 | 1.305202886516863e-2 | 1.4232735112565483e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
with-timeout/500
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5212065476160109 | 0.5311539791826293 | 0.5388547565829208 |
| Standard deviation | 0.0 | 1.188046007715095e-2 | 1.3338137715082948e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
with-timeout/1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.6533221018002174 | 0.6608261650173852 | 0.6678019621254988 |
| Standard deviation | 0.0 | 1.1651918277222251e-2 | 1.2082435014544956e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
with-mvar/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.144799680748481e-2 | 8.332385266790374e-2 | 8.816369443465824e-2 |
| Standard deviation | 1.763718923785259e-3 | 4.629153637951555e-3 | 7.51888294263787e-3 |
Outlying measurements have moderate (0.1797504436713848%) effect on estimated standard deviation.
with-mvar/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.178303078369492e-2 | 8.265900296939263e-2 | 8.348275719627907e-2 |
| Standard deviation | 1.029323755411523e-3 | 1.4057404751913582e-3 | 2.1692149731033714e-3 |
Outlying measurements have slight (9.000000000000001e-2%) effect on estimated standard deviation.
with-mvar/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.242703580874626e-2 | 9.317448007025055e-2 | 9.404053858908537e-2 |
| Standard deviation | 7.861932593138894e-4 | 1.2097443148161983e-3 | 1.7081683606092744e-3 |
Outlying measurements have slight (9.876543209876543e-2%) effect on estimated standard deviation.
with-mvar/100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10014327850984645 | 0.10089088518021397 | 0.10208626599701245 |
| Standard deviation | 1.0688721840209432e-3 | 1.607835212993144e-3 | 1.9767827973354835e-3 |
Outlying measurements have slight (9.87654320987653e-2%) effect on estimated standard deviation.
with-mvar/500
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.2172014090527147 | 0.22164389541069904 | 0.22700266259205404 |
| Standard deviation | 2.5595655508233207e-3 | 6.308601307034099e-3 | 8.947279425539153e-3 |
Outlying measurements have moderate (0.13888888888888887%) effect on estimated standard deviation.
with-mvar/1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3562058715549841 | 0.3588149491391486 | 0.3608058496441766 |
| Standard deviation | 0.0 | 3.0570076854863895e-3 | 3.448340827523033e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
with-ioref/5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3305753703710294e-2 | 4.8808707195991535e-2 | 5.8057767422919904e-2 |
| Standard deviation | 8.1120268811389e-3 | 1.3140011479441494e-2 | 2.1075212248205896e-2 |
Outlying measurements have severe (0.8458292386190603%) effect on estimated standard deviation.
with-ioref/10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.105488709101674e-2 | 4.575850411723976e-2 | 5.46818985013609e-2 |
| Standard deviation | 7.591655501546925e-3 | 1.2539164198913209e-2 | 1.956105706962526e-2 |
Outlying measurements have severe (0.8346939205659872%) effect on estimated standard deviation.
with-ioref/50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.033035582621978e-2 | 7.441571373856064e-2 | 8.729010871280589e-2 |
| Standard deviation | 1.4804038821787257e-2 | 2.1172330635811094e-2 | 2.9652613888080392e-2 |
Outlying measurements have severe (0.7876464445745001%) effect on estimated standard deviation.
with-ioref/100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.994956251154288e-2 | 9.247986647221845e-2 | 0.1030045004757758 |
| Standard deviation | 7.739519940583623e-3 | 1.7848301766732046e-2 | 2.7636254489220344e-2 |
Outlying measurements have severe (0.6459511587387258%) effect on estimated standard deviation.
with-ioref/500
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.16761814047569362 | 0.18734743598312997 | 0.20653754167809696 |
| Standard deviation | 1.5702319524931035e-2 | 2.5461321203359262e-2 | 3.346082278539362e-2 |
Outlying measurements have moderate (0.31766726912087234%) effect on estimated standard deviation.
with-ioref/1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.24084144628747797 | 0.2784425915205913 | 0.3197525444375917 |
| Standard deviation | 2.287182092046817e-2 | 4.429284681132456e-2 | 6.2194415238403404e-2 |
Outlying measurements have moderate (0.38024058193855176%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.