# `nupp.bench` Measuring Nupp code from inside an ordinary program. A benchmark here is a program, not a case a runner discovered. It links this module, runs under `nupp run`, and reports itself: ```nupp:fragment local bench = nupp.bench local function sized(b: nupp.bench.Case): nil for _ = 1, b.n do bench.keep(makePoint()) end end bench.case("presize.sized", sized) bench.report() ``` The library and `nupp bench` have different jobs. An application's hot loop lives in the application, so only something the loop can call can measure it. The command discovers those programs and gives each named case an isolated process. `report` raises rather than returning a status, because a chunk's return value is discarded and a run that did not raise exits zero. It writes the record first, so a case that trips the gate is still a case whose record the runner can merge. ## Types ### `Case` _record_ ```nupp record bench.Case ... ``` One case handed to a measured body. #### Members | Name | Kind | Description | | --- | --- | --- | | [`name`](#nupp.bench.Case.name) | field | The case's name, as case was given it. | | [`n`](#nupp.bench.Case.n) | field | Iterations the body is to run. | #### `name` _field_ ```nupp name: string ``` The case's name, as `case` was given it. #### `n` _field_ ```nupp n: integer ``` Iterations the body is to run. Fixed before the measured rounds begin, and carried in the record, so a comparison can re-run the same work. ### `FormatOptions` _type_ ```nupp type bench.FormatOptions = { --- Compare suite variants by their geometric-mean ratio. geometricMean: boolean?, --- Forks behind the measurements, when the runner merged them. --- --- Decides which columns the table carries. One fork reports a within-process --- interquartile range, which is a spread and not an interval; more than one --- reports the interval and the coverage it attained. forks: integer?, --- Lines appended below the table, each already wrapped. notes: {string}? } ``` Optional sections in the human result report. ### `FrameSession` _record_ ```nupp record bench.FrameSession ... ``` A running frame measurement, as `frames` returns it. The application owns its loop, so this offers a condition and a report rather than taking the loop over. There is no exit-time fallback: nothing hands this module a callback when the chunk returns, collection before shutdown is not guaranteed, and the compiler's entry point ends in `os.exit`. A session that is never reported produces no record, and the runner says so. #### Members | Name | Kind | Description | | --- | --- | --- | | [`name`](#nupp.bench.FrameSession.name) | field | | | [`budgetMs`](#nupp.bench.FrameSession.budgetMs) | field | | | [`wanted`](#nupp.bench.FrameSession.wanted) | field | | | [`samples`](#nupp.bench.FrameSession.samples) | field | | | [`started`](#nupp.bench.FrameSession.started) | field | | | [`session`](#nupp.bench.FrameSession.session) | field | | | [`sampling`](#nupp.bench.FrameSession.sampling) | field | | | [`done`](#nupp.bench.FrameSession.done) | field | | | [`more`](#nupp.bench.FrameSession.more) | method | Whether more frames are wanted. | | [`begin`](#nupp.bench.FrameSession.begin) | method | Marks the start of a frame. | | [`finish`](#nupp.bench.FrameSession.finish) | method | Marks the end of a frame and records it. | | [`report`](#nupp.bench.FrameSession.report) | method | Ends the session, records it, and reports. | #### `name` _field_ ```nupp name: string ``` #### `budgetMs` _field_ ```nupp budgetMs: number ``` #### `wanted` _field_ ```nupp wanted: integer ``` #### `samples` _field_ ```nupp samples: {number} ``` #### `started` _field_ ```nupp started: number ``` #### `session` _field_ ```nupp session: any ``` #### `sampling` _field_ ```nupp sampling: any ``` #### `done` _field_ ```nupp done: boolean ``` #### `more` _method_ ```nupp more: function bench.FrameSession:more(): boolean ``` Whether more frames are wanted. False once `count` frames have been recorded, and always true when the session was opened without one. ##### Returns | Type | Description | | --- | --- | | `boolean` | | #### `begin` _method_ ```nupp begin: function bench.FrameSession:begin(): nil ``` Marks the start of a frame. ##### Returns | Type | Description | | --- | --- | | `nil` | | #### `finish` _method_ ```nupp finish: function bench.FrameSession:finish(): nil ``` Marks the end of a frame and records it. Elapsed on the monotonic clock, not `os.clock`, which counts processor time. A frame that waited on presentation, on I/O, or on a sleep spends very little of either and blows its budget anyway, and missing the budget is the whole measurement. ##### Returns | Type | Description | | --- | --- | | `nil` | | #### `report` _method_ ```nupp report: function bench.FrameSession:report(): nil ``` Ends the session, records it, and reports. A frame report is a distribution rather than a median: a budget missed one frame in a hundred is a visible stutter and an unmoved mean. This writes the record and applies the gate, which is why an application's loop needs no second call. `bench.report` is idempotent, so a program with both cases and frames may still call it itself. ##### Returns | Type | Description | | --- | --- | | `nil` | | ##### Raises | Type | Condition | | --- | --- | | `string` | when a gated counter moved, or the record cannot be written | ### `Invocation` _record_ ```nupp record bench.Invocation ... ``` One expanded suite case handed to setup, run and teardown callbacks. #### Members | Name | Kind | Description | | --- | --- | --- | | [`name`](#nupp.bench.Invocation.name) | field | The unqualified case name from the suite declaration. | | [`data`](#nupp.bench.Invocation.data) | field | Case-specific data supplied by the declaration. | | [`parameters`](#nupp.bench.Invocation.parameters) | field | One value for each parameter in the expanded case. | #### `name` _field_ ```nupp name: string ``` The unqualified case name from the suite declaration. #### `data` _field_ ```nupp data: any ``` Case-specific data supplied by the declaration. #### `parameters` _field_ ```nupp parameters: {[string]: any} ``` One value for each parameter in the expanded case. ### `Measurement` _record_ ```nupp record bench.Measurement ... ``` What one measured case contributes to a record. #### Members | Name | Kind | Description | | --- | --- | --- | | [`name`](#nupp.bench.Measurement.name) | field | The case's name. | | [`kind`](#nupp.bench.Measurement.kind) | field | case, suite or frames. | | [`n`](#nupp.bench.Measurement.n) | field | Iterations per round, for a case. | | [`rounds`](#nupp.bench.Measurement.rounds) | field | Rounds timed, for a case. | | [`medianMs`](#nupp.bench.Measurement.medianMs) | field | The median round, in milliseconds. | | [`minMs`](#nupp.bench.Measurement.minMs) | field | The fastest round, in milliseconds. | | [`samplesMs`](#nupp.bench.Measurement.samplesMs) | field | Suite samples in milliseconds, normalized to one represented operation. | | [`meanMs`](#nupp.bench.Measurement.meanMs) | field | | | [`stdevMs`](#nupp.bench.Measurement.stdevMs) | field | | | [`p90Ms`](#nupp.bench.Measurement.p90Ms) | field | | | [`p99Ms`](#nupp.bench.Measurement.p99Ms) | field | The 99th percentile of this measurement's samples, in milliseconds: rounds for a case, samples for a suite, frames... | | [`sampleIterations`](#nupp.bench.Measurement.sampleIterations) | field | Calls to a suite variant inside one timed sample, and the operations one call represents. | | [`operationsPerInvocation`](#nupp.bench.Measurement.operationsPerInvocation) | field | | | [`warmupIterations`](#nupp.bench.Measurement.warmupIterations) | field | | | [`minSamples`](#nupp.bench.Measurement.minSamples) | field | | | [`minDurationMs`](#nupp.bench.Measurement.minDurationMs) | field | | | [`maxSamples`](#nupp.bench.Measurement.maxSamples) | field | | | [`maxDurationMs`](#nupp.bench.Measurement.maxDurationMs) | field | | | [`suite`](#nupp.bench.Measurement.suite) | field | Comparative-suite identity. | | [`variant`](#nupp.bench.Measurement.variant) | field | | | [`caseName`](#nupp.bench.Measurement.caseName) | field | | | [`parameters`](#nupp.bench.Measurement.parameters) | field | | | [`baselineVariant`](#nupp.bench.Measurement.baselineVariant) | field | | | [`allocatedKb`](#nupp.bench.Measurement.allocatedKb) | field | Kilobytes allocated, read with the collector stopped. | | [`retainedKb`](#nupp.bench.Measurement.retainedKb) | field | Kilobytes still held once the collector has run again, which is what "retained" has to mean. | | [`p50Ms`](#nupp.bench.Measurement.p50Ms) | field | Frame durations in milliseconds at the named quantiles. | | [`p999Ms`](#nupp.bench.Measurement.p999Ms) | field | | | [`frames`](#nupp.bench.Measurement.frames) | field | Frames recorded, and how many exceeded the budget. | | [`overBudget`](#nupp.bench.Measurement.overBudget) | field | | | [`abortSites`](#nupp.bench.Measurement.abortSites) | field | Abort site identities seen while measuring, each severity\|reason\|location\|zone. | | [`totalAborts`](#nupp.bench.Measurement.totalAborts) | field | Abort events and blacklistings behind those sites. | | [`blacklisted`](#nupp.bench.Measurement.blacklisted) | field | | | [`profilePath`](#nupp.bench.Measurement.profilePath) | field | Collapsed-stack profile written for this measured window, when requested. | | [`p25Ms`](#nupp.bench.Measurement.p25Ms) | field | Quantiles of this measurement's own samples, in milliseconds. | | [`p75Ms`](#nupp.bench.Measurement.p75Ms) | field | | | [`iqrMs`](#nupp.bench.Measurement.iqrMs) | field | The interquartile range of this measurement's own samples, in milliseconds. | | [`trend`](#nupp.bench.Measurement.trend) | field | What a monotone-trend test made of this process's samples in execution order: trend, no-trend-detected, or unknown. | | [`trendTau`](#nupp.bench.Measurement.trendTau) | field | Kendall's tau and the two-sided p-value behind trend, and how far the level actually moved between the ends of the... | | [`trendPValue`](#nupp.bench.Measurement.trendPValue) | field | | | [`trendDrift`](#nupp.bench.Measurement.trendDrift) | field | | | [`concentration`](#nupp.bench.Measurement.concentration) | field | The fraction of this measurement's samples lying within ten percent of its median. | | [`outlierCount`](#nupp.bench.Measurement.outlierCount) | field | Samples beyond three interquartile ranges of the nearer quartile, and the largest of them over the median. | | [`outlierMaxRatio`](#nupp.bench.Measurement.outlierMaxRatio) | field | | | [`forkCount`](#nupp.bench.Measurement.forkCount) | field | Processes this measurement summarizes, when the runner merged forks into it. | | [`forkSummariesMs`](#nupp.bench.Measurement.forkSummariesMs) | field | One normalized summary per fork, in execution order. | | [`intervalLowMs`](#nupp.bench.Measurement.intervalLowMs) | field | The interval for this benchmark's population median, when one is available. | | [`intervalHighMs`](#nupp.bench.Measurement.intervalHighMs) | field | | | [`intervalCoverage`](#nupp.bench.Measurement.intervalCoverage) | field | | | [`intervalWithheld`](#nupp.bench.Measurement.intervalWithheld) | field | Why there is no interval: below-minimum-forks or trend-warning. | #### `name` _field_ ```nupp name: string ``` The case's name. #### `kind` _field_ ```nupp kind: string ``` `case`, `suite` or `frames`. #### `n` _field_ ```nupp n: integer? ``` Iterations per round, for a case. #### `rounds` _field_ ```nupp rounds: integer? ``` Rounds timed, for a case. #### `medianMs` _field_ ```nupp medianMs: number? ``` The median round, in milliseconds. Recorded, never gated. #### `minMs` _field_ ```nupp minMs: number? ``` The fastest round, in milliseconds. #### `samplesMs` _field_ ```nupp samplesMs: {number}? ``` Suite samples in milliseconds, normalized to one represented operation. #### `meanMs` _field_ ```nupp meanMs: number? ``` #### `stdevMs` _field_ ```nupp stdevMs: number? ``` #### `p90Ms` _field_ ```nupp p90Ms: number? ``` #### `p99Ms` _field_ ```nupp p99Ms: number? ``` The 99th percentile of this measurement's samples, in milliseconds: rounds for a case, samples for a suite, frames for a frame session. #### `sampleIterations` _field_ ```nupp sampleIterations: integer? ``` Calls to a suite variant inside one timed sample, and the operations one call represents. #### `operationsPerInvocation` _field_ ```nupp operationsPerInvocation: integer? ``` #### `warmupIterations` _field_ ```nupp warmupIterations: integer? ``` #### `minSamples` _field_ ```nupp minSamples: integer? ``` #### `minDurationMs` _field_ ```nupp minDurationMs: number? ``` #### `maxSamples` _field_ ```nupp maxSamples: integer? ``` #### `maxDurationMs` _field_ ```nupp maxDurationMs: number? ``` #### `suite` _field_ ```nupp suite: string? ``` Comparative-suite identity. Kept apart so a reader need not parse `name`. #### `variant` _field_ ```nupp variant: string? ``` #### `caseName` _field_ ```nupp caseName: string? ``` #### `parameters` _field_ ```nupp parameters: {[string]: any}? ``` #### `baselineVariant` _field_ ```nupp baselineVariant: string? ``` #### `allocatedKb` _field_ ```nupp allocatedKb: number? ``` Kilobytes allocated, read with the collector stopped. A case stores one round; a suite stores one represented operation. A size rather than a count of allocations, and recorded only. #### `retainedKb` _field_ ```nupp retainedKb: number? ``` Kilobytes still held once the collector has run again, which is what "retained" has to mean. An earlier implementation reported the number above and called it this one: with collection stopped for the rounds, the heap delta is what they allocated, not what survived them. #### `p50Ms` _field_ ```nupp p50Ms: number? ``` Frame durations in milliseconds at the named quantiles. `p99Ms` is above. #### `p999Ms` _field_ ```nupp p999Ms: number? ``` #### `frames` _field_ ```nupp frames: integer? ``` Frames recorded, and how many exceeded the budget. #### `overBudget` _field_ ```nupp overBudget: integer? ``` #### `abortSites` _field_ ```nupp abortSites: {string}? ``` Abort site identities seen while measuring, each `severity|reason|location|zone`. Identities rather than counts: a count is partly a function of how much work ran, and a loop that started aborting is the finding at any count. Nil when no session could be opened, which is not the same as an empty set: one says nothing aborted and the other says nobody looked. #### `totalAborts` _field_ ```nupp totalAborts: integer? ``` Abort events and blacklistings behind those sites. Recorded, never gated. #### `blacklisted` _field_ ```nupp blacklisted: integer? ``` #### `profilePath` _field_ ```nupp profilePath: string? ``` Collapsed-stack profile written for this measured window, when requested. #### `p25Ms` _field_ ```nupp p25Ms: number? ``` Quantiles of this measurement's own samples, in milliseconds. The table shows `[p25, p99]` beside the score, which is a range rather than a percentage either side of the median on purpose: a benchmark whose samples are bimodal puts its median in the gap between the two clusters, and a symmetric "+-x%" around that median describes a distribution the run never produced. The upper end is the tail rather than the third quartile because the tail is what a reader is looking for. A slow mode holding a tenth of the samples moves p99 and leaves p75 where it was, so a box would have hidden exactly the case worth seeing. `p75Ms` stays in the record for anyone who wants the box. #### `p75Ms` _field_ ```nupp p75Ms: number? ``` #### `iqrMs` _field_ ```nupp iqrMs: number? ``` The interquartile range of this measurement's own samples, in milliseconds. Within-process spread. It says how much the samples inside one process varied and is not a confidence interval: those samples share a heap, a set of compiled traces and a thermal state, so they are not independent draws and no interval may be derived from them. An interval comes from fork summaries, which the runner assembles across processes. #### `trend` _field_ ```nupp trend: string? ``` What a monotone-trend test made of this process's samples in execution order: `trend`, `no-trend-detected`, or `unknown`. There is no value asserting a steady state. A test that found no trend has not established one, and saying otherwise is the mistake this field exists to avoid making. #### `trendTau` _field_ ```nupp trendTau: number? ``` Kendall's tau and the two-sided p-value behind `trend`, and how far the level actually moved between the ends of the series. `trend` needs both a significant test and a drift worth acting on, so a series that moved half a percent with a p-value of 0.005 is not reported as trending. #### `trendPValue` _field_ ```nupp trendPValue: number? ``` #### `trendDrift` _field_ ```nupp trendDrift: number? ``` #### `concentration` _field_ ```nupp concentration: number? ``` The fraction of this measurement's samples lying within ten percent of its median. Near one when the samples describe a single rate. Near zero when they split into clusters and the median falls in the gap, which is what a collector running on alternate samples produces and which makes the reported score describe a rate the benchmark never ran at. #### `outlierCount` _field_ ```nupp outlierCount: integer? ``` Samples beyond three interquartile ranges of the nearer quartile, and the largest of them over the median. Classified, never removed. A real warmup or deoptimization phase falls exactly where these fences do, so excluding what they catch would delete the behavior `trend` is looking for. Every sample stays in `samplesMs` and in the median. #### `outlierMaxRatio` _field_ ```nupp outlierMaxRatio: number? ``` #### `forkCount` _field_ ```nupp forkCount: integer? ``` Processes this measurement summarizes, when the runner merged forks into it. Absent on a measurement a child wrote: a child is one process and knows nothing about the others. #### `forkSummariesMs` _field_ ```nupp forkSummariesMs: {number}? ``` One normalized summary per fork, in execution order. The input the interval below was computed from, kept so a reader can recompute it. #### `intervalLowMs` _field_ ```nupp intervalLowMs: number? ``` The interval for this benchmark's population median, when one is available. `intervalCoverage` is the probability this interval actually attains, not one that was requested: it falls out of the fork count through the sign test, and a run with too few forks gets no interval rather than a relabelled one. #### `intervalHighMs` _field_ ```nupp intervalHighMs: number? ``` #### `intervalCoverage` _field_ ```nupp intervalCoverage: number? ``` #### `intervalWithheld` _field_ ```nupp intervalWithheld: string? ``` Why there is no interval: `below-minimum-forks` or `trend-warning`. ### `SuiteCase` _type_ ```nupp type bench.SuiteCase = { name: string, data: any?, parameters: {[string]: {any}}?, --- Operations represented by one call to a variant's `run` callback. operations: integer? } ``` One workload in a suite. Parameter lists are expanded as a Cartesian product. ### `SuiteOptions` _type_ ```nupp type bench.SuiteOptions = { name: string, variants: {bench.Variant}, cases: {bench.SuiteCase}, --- Variant used as the denominator in the human result table. baselineVariant: string?, --- Calls made before timing begins. warmupIterations: integer?, --- Calls to `run` inside one timed sample. State is set up once per sample. sampleIterations: integer?, --- Stop once both this many samples and this many milliseconds of measured time --- exist. minSamples: integer?, minDurationMs: number?, --- Safety bounds when a sample is much faster or slower than expected. maxSamples: integer?, maxDurationMs: number? } ``` A comparative benchmark suite. ### `Variant` _type_ ```nupp type bench.Variant = { name: string, setup: (function(bench.Invocation): any)?, run: function(any, bench.Invocation): any, teardown: (function(any, bench.Invocation): nil)? } ``` One implementation of every case in a suite. ## Functions ### `bench.case` _function_ ```nupp function bench.case(name: string, body: function(bench.Case): nil, fixedN: integer?): nil ``` Measures one case and adds it to the record this program will report. The body is handed the case and runs its own loop. Calling a one-iteration closure `n` times instead would put a call boundary inside the measurement and change what the recorder sees, so the loop belongs to the body. #### Arguments | Name | Type | Description | | --- | --- | --- | | `name` | `string` | what to record the case as | | `body` | `function(bench.Case): nil` | the work, which iterates `case.n` times | | `fixedN` | `integer?` | an explicit iteration count, skipping calibration | #### Returns | Type | Description | | --- | --- | | `nil` | | #### Raises | Type | Condition | | --- | --- | | `string` | when the name is empty or already declared | ### `bench.compare` _function_ ```nupp function bench.compare(record: any, baseline: any): {string}, {string} ``` Compares one record against one baseline record and returns what moved. Exported because the runner owns the baseline for a whole set and has to apply the same rules a standalone case applies to its own. Two copies of these rules would disagree, and the one that mattered would be whichever the reader was not looking at. A comparison with no comparable baseline is reported as exactly that. It is a result, and a different one from a regression: the runner has to say which, or a lost baseline reads like a pass. #### Arguments | Name | Type | Description | | --- | --- | --- | | `record` | `any` | what this run measured | | `baseline` | `any` | the record to compare it against | #### Returns | Type | Description | | --- | --- | | `{string}` | the gated differences, which are failures | | `{string}` | the differences that were only reported | ### `bench.decodeBaseline` _function_ ```nupp function bench.decodeBaseline(text: string, path: string, oldest: integer?): any, string? ``` Decodes a baseline record, refusing one this version cannot read. Anything that decodes as JSON used to read as a baseline: a record from a newer bench, or a file that is not a record at all, matched no case, so every case read as having no baseline, and that looks like a pass. Exported so the runner applies the same rule to its own documents, from the oldest schema it reads. #### Arguments | Name | Type | Description | | --- | --- | --- | | `text` | `string` | what the baseline file holds | | `path` | `string` | where it was read from, for the message | | `oldest` | `integer?` | the oldest schema the reader accepts, `OLDEST_BASELINE_SCHEMA` by default | #### Returns | Type | Description | | --- | --- | | `any` | the decoded baseline, or nil where it is refused | | `string?` | why it was refused, naming its schema and the ones this version reads | ### `bench.format` _function_ ```nupp function bench.format(record: any, options: bench.FormatOptions?): string ``` Formats a record as the compact result table used by both a standalone program and the set runner. The shape follows the useful part of JMH's final report, but names the statistic `p50`: these scores are medians, not averages with confidence intervals. Case scores are normalized to one body iteration. The calibrated batch size stays in the JSON record, where the runner also finds it for the next comparison. #### Arguments | Name | Type | Description | | --- | --- | --- | | `record` | `any` | | | `options` | `bench.FormatOptions?` | | #### Returns | Type | Description | | --- | --- | | `string` | | ### `bench.frames` _function_ ```nupp function bench.frames(name: string, budgetMs: number?, count: integer?): bench.FrameSession ``` Opens a frame measurement. Plain parameters rather than an options record, because a record would make the ordinary call site write `new bench.FrameOptions(...)` -- a table literal is not one, which is what the first version of this API got wrong in its documented example. #### Arguments | Name | Type | Description | | --- | --- | --- | | `name` | `string` | what to record the session as | | `budgetMs` | `number?` | milliseconds a frame is allowed; frames over it are counted | | `count` | `integer?` | frames to record before `more` turns false, or nil to let the application decide when to stop | #### Returns | Type | Description | | --- | --- | | `bench.FrameSession` | | #### Raises | Type | Condition | | --- | --- | | `string` | when the name is empty or already declared | ### `bench.keep` _function_ ```nupp function bench.keep(value: any): nil ``` Keeps a value the measured body produced, so the work that produced it survives. This is the one rule writing a case requires. LuaJIT removes work whose result does not escape its trace, which is correct and is also the most common way a benchmark comes out impossibly fast: the loop under test is deleted and the measurement is of nothing. The store is to a field of this module, which is visible outside any trace and so cannot be sunk. Nothing checks that this stays true of LuaJIT; a collapse in the reported durations is the only signal, which is why it is worth reading them. #### Arguments | Name | Type | Description | | --- | --- | --- | | `value` | `any` | whatever the measured body produced | #### Returns | Type | Description | | --- | --- | | `nil` | | ### `bench.report` _function_ ```nupp function bench.report(): nil ``` Writes the record, compares it against a baseline when one was named, and raises when a gated counter moved. The write happens first on purpose. A case that trips the gate is a case whose record the runner has to merge, or the report says only that something failed. Raising is how a status reaches the shell: a chunk's return value is discarded and a run that did not raise exits zero. `os.exit` would also produce one and would discard the run's own profile and trace-abort reports, which are written after the chunk returns. #### Returns | Type | Description | | --- | --- | | `nil` | | #### Raises | Type | Condition | | --- | --- | | `string` | when a gated counter moved, or the record cannot be written | ### `bench.suite` _function_ ```nupp function bench.suite(options: bench.SuiteOptions): nil ``` Declares a comparative suite. Every case, parameter set and variant is a named benchmark and therefore gets its own process under the set runner. #### Arguments | Name | Type | Description | | --- | --- | --- | | `options` | `bench.SuiteOptions` | | #### Returns | Type | Description | | --- | --- | | `nil` | | #### Raises | Type | Condition | | --- | --- | | `string` | when the declaration or its sampling bounds are invalid |