`nupp.bench`

Measuring Nupp code from inside an ordinary program.

A benchmark here is a program, not a case a runner discovered. It links this module, runs under nupp run, and reports itself:

local bench = nupp.bench

local function sized(b: nupp.bench.Case): nil
    for _ = 1, b.n do
        bench.keep(makePoint())
    end
end

bench.case("presize.sized", sized)
bench.report()

The library and nupp bench have different jobs. An application's hot loop lives in the application, so only something the loop can call can measure it. The command discovers those programs and gives each named case an isolated process.

report raises rather than returning a status, because a chunk's return value is discarded and a run that did not raise exits zero. It writes the record first, so a case that trips the gate is still a case whose record the runner can merge.

Module contents

Types

Type	Kind	Description
`Case`	record	One case handed to a measured body.
`FormatOptions`	type	Optional sections in the human result report.
`FrameSession`	record	A running frame measurement, as frames returns it.
`Invocation`	record	One expanded suite case handed to setup, run and teardown callbacks.
`Measurement`	record	What one measured case contributes to a record.
`SuiteCase`	type	One workload in a suite.
`SuiteOptions`	type	A comparative benchmark suite.
`Variant`	type	One implementation of every case in a suite.

Functions

Function	Kind	Description
`case`	function	Measures one case and adds it to the record this program will report.
`compare`	function	Compares one record against one baseline record and returns what moved.
`decodeBaseline`	function	Decodes a baseline record, refusing one this version cannot read.
`format`	function	Formats a record as the compact result table used by both a standalone program and the set runner.
`frames`	function	Opens a frame measurement.
`keep`	function	Keeps a value the measured body produced, so the work that produced it survives.
`report`	function	Writes the record, compares it against a baseline when one was named, and raises when a gated counter moved.
`suite`	function	Declares a comparative suite.

Types#

`Case`record#

record bench.Case ...

One case handed to a measured body.

Members

Name	Kind	Description
`name`	field	The case's name, as case was given it.
`n`	field	Iterations the body is to run.

`name`field#

name: string

The case's name, as case was given it.

`n`field#

n: integer

Iterations the body is to run. Fixed before the measured rounds begin, and carried in the record, so a comparison can re-run the same work.

`FormatOptions`type#

type bench.FormatOptions = {
    --- Compare suite variants by their geometric-mean ratio.
    geometricMean: boolean?,

    --- Forks behind the measurements, when the runner merged them.
    ---
    --- Decides which columns the table carries. One fork reports a within-process
    --- interquartile range, which is a spread and not an interval; more than one
    --- reports the interval and the coverage it attained.
    forks: integer?,

    --- Lines appended below the table, each already wrapped.
    notes: {string}?
}

Optional sections in the human result report.

`FrameSession`record#

record bench.FrameSession ...

A running frame measurement, as frames returns it.

The application owns its loop, so this offers a condition and a report rather than taking the loop over. There is no exit-time fallback: nothing hands this module a callback when the chunk returns, collection before shutdown is not guaranteed, and the compiler's entry point ends in os.exit. A session that is never reported produces no record, and the runner says so.

Members

Name	Kind	Description
`name`	field
`budgetMs`	field
`wanted`	field
`samples`	field
`started`	field
`session`	field
`sampling`	field
`done`	field
`more`	method	Whether more frames are wanted.
`begin`	method	Marks the start of a frame.
`finish`	method	Marks the end of a frame and records it.
`report`	method	Ends the session, records it, and reports.

`name`field#

name: string

`budgetMs`field#

budgetMs: number

`wanted`field#

wanted: integer

`samples`field#

samples: {number}

`started`field#

started: number

`session`field#

session: any

`sampling`field#

sampling: any

`done`field#

done: boolean

`more`method#

more: function bench.FrameSession:more(): boolean

Whether more frames are wanted. False once count frames have been recorded, and always true when the session was opened without one.

Returns

Type	Description
`boolean`

`begin`method#

begin: function bench.FrameSession:begin(): nil

Marks the start of a frame.

Returns

Type	Description
`nil`

`finish`method#

finish: function bench.FrameSession:finish(): nil

Marks the end of a frame and records it.

Elapsed on the monotonic clock, not os.clock, which counts processor time. A frame that waited on presentation, on I/O, or on a sleep spends very little of either and blows its budget anyway, and missing the budget is the whole measurement.

Returns

Type	Description
`nil`

`report`method#

report: function bench.FrameSession:report(): nil

Ends the session, records it, and reports.

A frame report is a distribution rather than a median: a budget missed one frame in a hundred is a visible stutter and an unmoved mean.

This writes the record and applies the gate, which is why an application's loop needs no second call. bench.report is idempotent, so a program with both cases and frames may still call it itself.

Returns

Type	Description
`nil`

Raises

Type	Condition
`string`	when a gated counter moved, or the record cannot be written

`Invocation`record#

record bench.Invocation ...

One expanded suite case handed to setup, run and teardown callbacks.

Members

Name	Kind	Description
`name`	field	The unqualified case name from the suite declaration.
`data`	field	Case-specific data supplied by the declaration.
`parameters`	field	One value for each parameter in the expanded case.

`name`field#

name: string

The unqualified case name from the suite declaration.

`data`field#

data: any

Case-specific data supplied by the declaration.

`parameters`field#

parameters: {[string]: any}

One value for each parameter in the expanded case.

`Measurement`record#

record bench.Measurement ...

What one measured case contributes to a record.

Members

Name	Kind	Description
`name`	field	The case's name.
`kind`	field	case, suite or frames.
`n`	field	Iterations per round, for a case.
`rounds`	field	Rounds timed, for a case.
`medianMs`	field	The median round, in milliseconds.
`minMs`	field	The fastest round, in milliseconds.
`samplesMs`	field	Suite samples in milliseconds, normalized to one represented operation.
`meanMs`	field
`stdevMs`	field
`p90Ms`	field
`p99Ms`	field	The 99th percentile of this measurement's samples, in milliseconds: rounds for a case, samples for a suite, frames...
`sampleIterations`	field	Calls to a suite variant inside one timed sample, and the operations one call represents.
`operationsPerInvocation`	field
`warmupIterations`	field
`minSamples`	field
`minDurationMs`	field
`maxSamples`	field
`maxDurationMs`	field
`suite`	field	Comparative-suite identity.
`variant`	field
`caseName`	field
`parameters`	field
`baselineVariant`	field
`allocatedKb`	field	Kilobytes allocated, read with the collector stopped.
`retainedKb`	field	Kilobytes still held once the collector has run again, which is what "retained" has to mean.
`p50Ms`	field	Frame durations in milliseconds at the named quantiles.
`p999Ms`	field
`frames`	field	Frames recorded, and how many exceeded the budget.
`overBudget`	field
`abortSites`	field	Abort site identities seen while measuring, each severity\|reason\|location\|zone.
`totalAborts`	field	Abort events and blacklistings behind those sites.
`blacklisted`	field
`profilePath`	field	Collapsed-stack profile written for this measured window, when requested.
`p25Ms`	field	Quantiles of this measurement's own samples, in milliseconds.
`p75Ms`	field
`iqrMs`	field	The interquartile range of this measurement's own samples, in milliseconds.
`trend`	field	What a monotone-trend test made of this process's samples in execution order: trend, no-trend-detected, or unknown.
`trendTau`	field	Kendall's tau and the two-sided p-value behind trend, and how far the level actually moved between the ends of the...
`trendPValue`	field
`trendDrift`	field
`concentration`	field	The fraction of this measurement's samples lying within ten percent of its median.
`outlierCount`	field	Samples beyond three interquartile ranges of the nearer quartile, and the largest of them over the median.
`outlierMaxRatio`	field
`forkCount`	field	Processes this measurement summarizes, when the runner merged forks into it.
`forkSummariesMs`	field	One normalized summary per fork, in execution order.
`intervalLowMs`	field	The interval for this benchmark's population median, when one is available.
`intervalHighMs`	field
`intervalCoverage`	field
`intervalWithheld`	field	Why there is no interval: below-minimum-forks or trend-warning.

`name`field#

name: string

The case's name.

`kind`field#

kind: string

case, suite or frames.

`n`field#

n: integer?

Iterations per round, for a case.

`rounds`field#

rounds: integer?

Rounds timed, for a case.

`medianMs`field#

medianMs: number?

The median round, in milliseconds. Recorded, never gated.

`minMs`field#

minMs: number?

The fastest round, in milliseconds.

`samplesMs`field#

samplesMs: {number}?

Suite samples in milliseconds, normalized to one represented operation.

`meanMs`field#

meanMs: number?

`stdevMs`field#

stdevMs: number?

`p90Ms`field#

p90Ms: number?

`p99Ms`field#

p99Ms: number?

The 99th percentile of this measurement's samples, in milliseconds: rounds for a case, samples for a suite, frames for a frame session.

`sampleIterations`field#

sampleIterations: integer?

Calls to a suite variant inside one timed sample, and the operations one call represents.

`operationsPerInvocation`field#

operationsPerInvocation: integer?

`warmupIterations`field#

warmupIterations: integer?

`minSamples`field#

minSamples: integer?

`minDurationMs`field#

minDurationMs: number?

`maxSamples`field#

maxSamples: integer?

`maxDurationMs`field#

maxDurationMs: number?

`suite`field#

suite: string?

Comparative-suite identity. Kept apart so a reader need not parse name.

`variant`field#

variant: string?

`caseName`field#

caseName: string?

`parameters`field#

parameters: {[string]: any}?

`baselineVariant`field#

baselineVariant: string?

`allocatedKb`field#

allocatedKb: number?

Kilobytes allocated, read with the collector stopped. A case stores one round; a suite stores one represented operation. A size rather than a count of allocations, and recorded only.

`retainedKb`field#

retainedKb: number?

Kilobytes still held once the collector has run again, which is what "retained" has to mean. An earlier implementation reported the number above and called it this one: with collection stopped for the rounds, the heap delta is what they allocated, not what survived them.

`p50Ms`field#

p50Ms: number?

Frame durations in milliseconds at the named quantiles. p99Ms is above.

`p999Ms`field#

p999Ms: number?

`frames`field#

frames: integer?

Frames recorded, and how many exceeded the budget.

`overBudget`field#

overBudget: integer?

`abortSites`field#

abortSites: {string}?

Abort site identities seen while measuring, each severity|reason|location|zone. Identities rather than counts: a count is partly a function of how much work ran, and a loop that started aborting is the finding at any count.

Nil when no session could be opened, which is not the same as an empty set: one says nothing aborted and the other says nobody looked.

`totalAborts`field#

totalAborts: integer?

Abort events and blacklistings behind those sites. Recorded, never gated.

`blacklisted`field#

blacklisted: integer?

`profilePath`field#

profilePath: string?

Collapsed-stack profile written for this measured window, when requested.

`p25Ms`field#

p25Ms: number?

Quantiles of this measurement's own samples, in milliseconds.

The table shows [p25, p99] beside the score, which is a range rather than a percentage either side of the median on purpose: a benchmark whose samples are bimodal puts its median in the gap between the two clusters, and a symmetric "+-x%" around that median describes a distribution the run never produced.

The upper end is the tail rather than the third quartile because the tail is what a reader is looking for. A slow mode holding a tenth of the samples moves p99 and leaves p75 where it was, so a box would have hidden exactly the case worth seeing. p75Ms stays in the record for anyone who wants the box.

`p75Ms`field#

p75Ms: number?

`iqrMs`field#

iqrMs: number?

The interquartile range of this measurement's own samples, in milliseconds.

Within-process spread. It says how much the samples inside one process varied and is not a confidence interval: those samples share a heap, a set of compiled traces and a thermal state, so they are not independent draws and no interval may be derived from them. An interval comes from fork summaries, which the runner assembles across processes.

`trend`field#

trend: string?

What a monotone-trend test made of this process's samples in execution order: trend, no-trend-detected, or unknown.

There is no value asserting a steady state. A test that found no trend has not established one, and saying otherwise is the mistake this field exists to avoid making.

`trendTau`field#

trendTau: number?

Kendall's tau and the two-sided p-value behind trend, and how far the level actually moved between the ends of the series.

trend needs both a significant test and a drift worth acting on, so a series that moved half a percent with a p-value of 0.005 is not reported as trending.

`trendPValue`field#

trendPValue: number?

`trendDrift`field#

trendDrift: number?

`concentration`field#

concentration: number?

The fraction of this measurement's samples lying within ten percent of its median.

Near one when the samples describe a single rate. Near zero when they split into clusters and the median falls in the gap, which is what a collector running on alternate samples produces and which makes the reported score describe a rate the benchmark never ran at.

`outlierCount`field#

outlierCount: integer?

Samples beyond three interquartile ranges of the nearer quartile, and the largest of them over the median.

Classified, never removed. A real warmup or deoptimization phase falls exactly where these fences do, so excluding what they catch would delete the behavior trend is looking for. Every sample stays in samplesMs and in the median.

`outlierMaxRatio`field#

outlierMaxRatio: number?

`forkCount`field#

forkCount: integer?

Processes this measurement summarizes, when the runner merged forks into it.

Absent on a measurement a child wrote: a child is one process and knows nothing about the others.

`forkSummariesMs`field#

forkSummariesMs: {number}?

One normalized summary per fork, in execution order. The input the interval below was computed from, kept so a reader can recompute it.

`intervalLowMs`field#

intervalLowMs: number?

The interval for this benchmark's population median, when one is available.

intervalCoverage is the probability this interval actually attains, not one that was requested: it falls out of the fork count through the sign test, and a run with too few forks gets no interval rather than a relabelled one.

`intervalHighMs`field#

intervalHighMs: number?

`intervalCoverage`field#

intervalCoverage: number?

`intervalWithheld`field#

intervalWithheld: string?

Why there is no interval: below-minimum-forks or trend-warning.

`SuiteCase`type#

type bench.SuiteCase = {
    name: string,
    data: any?,
    parameters: {[string]: {any}}?,

    --- Operations represented by one call to a variant's `run` callback.
    operations: integer?
}

One workload in a suite. Parameter lists are expanded as a Cartesian product.

`SuiteOptions`type#

type bench.SuiteOptions = {
    name: string,
    variants: {bench.Variant},
    cases: {bench.SuiteCase},

    --- Variant used as the denominator in the human result table.
    baselineVariant: string?,

    --- Calls made before timing begins.
    warmupIterations: integer?,

    --- Calls to `run` inside one timed sample. State is set up once per sample.
    sampleIterations: integer?,

    --- Stop once both this many samples and this many milliseconds of measured time
    --- exist.
    minSamples: integer?,
    minDurationMs: number?,

    --- Safety bounds when a sample is much faster or slower than expected.
    maxSamples: integer?,
    maxDurationMs: number?
}

A comparative benchmark suite.

`Variant`type#

type bench.Variant = {
    name: string,
    setup: (function(bench.Invocation): any)?,
    run: function(any, bench.Invocation): any,
    teardown: (function(any, bench.Invocation): nil)?
}

One implementation of every case in a suite.

Functions#

`bench.case`function#

function bench.case(name: string, body: function(bench.Case): nil, fixedN: integer?): nil

Measures one case and adds it to the record this program will report.

The body is handed the case and runs its own loop. Calling a one-iteration closure n times instead would put a call boundary inside the measurement and change what the recorder sees, so the loop belongs to the body.

Arguments

Name	Type	Description
`name`	`string`	what to record the case as
`body`	`function(bench.Case): nil`	the work, which iterates `case.n` times
`fixedN`	`integer?`	an explicit iteration count, skipping calibration

Returns

Type	Description
`nil`

Raises

Type	Condition
`string`	when the name is empty or already declared

`bench.compare`function#

function bench.compare(record: any, baseline: any): {string}, {string}

Compares one record against one baseline record and returns what moved.

Exported because the runner owns the baseline for a whole set and has to apply the same rules a standalone case applies to its own. Two copies of these rules would disagree, and the one that mattered would be whichever the reader was not looking at.

A comparison with no comparable baseline is reported as exactly that. It is a result, and a different one from a regression: the runner has to say which, or a lost baseline reads like a pass.

Arguments

Name	Type	Description
`record`	`any`	what this run measured
`baseline`	`any`	the record to compare it against

Returns

Type	Description
`{string}`	the gated differences, which are failures
`{string}`	the differences that were only reported

`bench.decodeBaseline`function#

function bench.decodeBaseline(text: string, path: string, oldest: integer?): any, string?

Decodes a baseline record, refusing one this version cannot read.

Anything that decodes as JSON used to read as a baseline: a record from a newer bench, or a file that is not a record at all, matched no case, so every case read as having no baseline, and that looks like a pass. Exported so the runner applies the same rule to its own documents, from the oldest schema it reads.

Arguments

Name	Type	Description
`text`	`string`	what the baseline file holds
`path`	`string`	where it was read from, for the message
`oldest`	`integer?`	the oldest schema the reader accepts, `OLDEST_BASELINE_SCHEMA` by default

Returns

Type	Description
`any`	the decoded baseline, or nil where it is refused
`string?`	why it was refused, naming its schema and the ones this version reads

`bench.format`function#

function bench.format(record: any, options: bench.FormatOptions?): string

Formats a record as the compact result table used by both a standalone program and the set runner. The shape follows the useful part of JMH's final report, but names the statistic p50: these scores are medians, not averages with confidence intervals.

Case scores are normalized to one body iteration. The calibrated batch size stays in the JSON record, where the runner also finds it for the next comparison.

Arguments

Name	Type	Description
`record`	`any`
`options`	`bench.FormatOptions?`

Returns

Type	Description
`string`

`bench.frames`function#

function bench.frames(name: string, budgetMs: number?, count: integer?): bench.FrameSession

Opens a frame measurement.

Plain parameters rather than an options record, because a record would make the ordinary call site write new bench.FrameOptions(...) -- a table literal is not one, which is what the first version of this API got wrong in its documented example.

Arguments

Name	Type	Description
`name`	`string`	what to record the session as
`budgetMs`	`number?`	milliseconds a frame is allowed; frames over it are counted
`count`	`integer?`	frames to record before `more` turns false, or nil to let the application decide when to stop

Returns

Type	Description
`bench.FrameSession`

Raises

Type	Condition
`string`	when the name is empty or already declared

`bench.keep`function#

function bench.keep(value: any): nil

Keeps a value the measured body produced, so the work that produced it survives.

This is the one rule writing a case requires. LuaJIT removes work whose result does not escape its trace, which is correct and is also the most common way a benchmark comes out impossibly fast: the loop under test is deleted and the measurement is of nothing.

The store is to a field of this module, which is visible outside any trace and so cannot be sunk. Nothing checks that this stays true of LuaJIT; a collapse in the reported durations is the only signal, which is why it is worth reading them.

Arguments

Name	Type	Description
`value`	`any`	whatever the measured body produced

Returns

Type	Description
`nil`

`bench.report`function#

function bench.report(): nil

Writes the record, compares it against a baseline when one was named, and raises when a gated counter moved.

The write happens first on purpose. A case that trips the gate is a case whose record the runner has to merge, or the report says only that something failed.

Raising is how a status reaches the shell: a chunk's return value is discarded and a run that did not raise exits zero. os.exit would also produce one and would discard the run's own profile and trace-abort reports, which are written after the chunk returns.

Returns

Type	Description
`nil`

Raises

Type	Condition
`string`	when a gated counter moved, or the record cannot be written

`bench.suite`function#

function bench.suite(options: bench.SuiteOptions): nil

Declares a comparative suite. Every case, parameter set and variant is a named benchmark and therefore gets its own process under the set runner.

Arguments

Name	Type	Description
`options`	`bench.SuiteOptions`

Returns

Type	Description
`nil`

Raises

Type	Condition
`string`	when the declaration or its sampling bounds are invalid

nupp.bench

Module contents

Types

Functions

Types#

Caserecord#

Members

namefield#

nfield#

FormatOptionstype#

FrameSessionrecord#

Members

namefield#

budgetMsfield#

wantedfield#

samplesfield#

startedfield#

sessionfield#

samplingfield#

donefield#

moremethod#

Returns

beginmethod#

Returns

finishmethod#

Returns

reportmethod#

Returns

Raises

Invocationrecord#

Members

namefield#

datafield#

parametersfield#

Measurementrecord#

Members

namefield#

kindfield#

nfield#

roundsfield#

medianMsfield#

minMsfield#

samplesMsfield#

meanMsfield#

stdevMsfield#

p90Msfield#

p99Msfield#

sampleIterationsfield#

operationsPerInvocationfield#

warmupIterationsfield#

minSamplesfield#

minDurationMsfield#

maxSamplesfield#

maxDurationMsfield#

suitefield#

variantfield#

caseNamefield#

parametersfield#

baselineVariantfield#

allocatedKbfield#

retainedKbfield#

p50Msfield#

p999Msfield#

framesfield#

overBudgetfield#

abortSitesfield#

totalAbortsfield#

blacklistedfield#

profilePathfield#

p25Msfield#

p75Msfield#

iqrMsfield#

trendfield#

trendTaufield#

trendPValuefield#

trendDriftfield#

concentrationfield#

outlierCountfield#

outlierMaxRatiofield#

forkCountfield#

`nupp.bench`

`Case`record#

`name`field#

`n`field#

`FormatOptions`type#

`FrameSession`record#

`name`field#

`budgetMs`field#

`wanted`field#

`samples`field#

`started`field#

`session`field#

`sampling`field#

`done`field#

`more`method#

`begin`method#

`finish`method#

`report`method#

`Invocation`record#

`name`field#

`data`field#

`parameters`field#

`Measurement`record#

`name`field#

`kind`field#

`n`field#

`rounds`field#

`medianMs`field#

`minMs`field#

`samplesMs`field#

`meanMs`field#

`stdevMs`field#

`p90Ms`field#

`p99Ms`field#

`sampleIterations`field#

`operationsPerInvocation`field#

`warmupIterations`field#

`minSamples`field#

`minDurationMs`field#

`maxSamples`field#

`maxDurationMs`field#

`suite`field#

`variant`field#

`caseName`field#

`parameters`field#

`baselineVariant`field#

`allocatedKb`field#

`retainedKb`field#

`p50Ms`field#

`p999Ms`field#

`frames`field#

`overBudget`field#

`abortSites`field#

`totalAborts`field#

`blacklisted`field#

`profilePath`field#

`p25Ms`field#

`p75Ms`field#

`iqrMs`field#

`trend`field#

`trendTau`field#

`trendPValue`field#

`trendDrift`field#

`concentration`field#

`outlierCount`field#

`outlierMaxRatio`field#

`forkCount`field#

`forkSummariesMs`field#

`intervalLowMs`field#

`intervalHighMs`field#

`intervalCoverage`field#

`intervalWithheld`field#

`SuiteCase`type#

`SuiteOptions`type#

`Variant`type#

`bench.case`function#

`bench.compare`function#

`bench.decodeBaseline`function#

`bench.format`function#

`bench.frames`function#