nupp.peg

nupp.peg compiles textual parsing-expression grammars into reusable typed matchers. The same grammar language works inside comptime and at runtime.

const Identifier = comptime do
    return nupp.peg.compile("[a-zA-Z_] [a-zA-Z_0-9]* !.")
end

assert(Identifier("item_2") == 7)
assert(Identifier("2_items") == nil)

A parsing-expression grammar describes a deterministic top-down parse. Its choice operator is ordered, so p / q tries q only if p fails, where a regular expression alternation may pick whichever arm makes the complete expression work.

All positions and classes are byte-oriented. The grammar does not decode UTF-8 codepoints: a literal UTF-8 character matches as its encoded byte sequence, and . consumes one byte rather than one Unicode character.

Compiling at either phase#

nupp.peg.compile(source, options?) is the only constructor. A constant call inside comptime is parsed and validated by the compiler, which derives nupp.peg.Peg<R...> from the validated capture shape:

const Word = comptime do
    return nupp.peg.compile("{ [a-z]+ } !.")
end

Recognizers, substring and position captures, and collections therefore need no result annotation. The emitted program receives an already-materialized matcher and does not carry the textual grammar parser unless some runtime call also needs it.

A normal call with literal grammar text gets the same inferred Peg<R...>. It still compiles at run time, though parsed plans are cached by grammar source and backend, so compiling the same grammar again avoids parsing, lowering and code generation. A genuinely dynamic string returns Peg<...any>, because its capture shape is not yet known:

local function loadMatcher(configuration: string): nupp.peg.Peg<...any>
    return nupp.peg.compile(configuration)
end

Neither form requires a separately installed LPeg: the resolved native effect asks the build for a host that supplies LPeg 1.1.

Prefer comptime for source-owned constant grammars. Runtime compilation is for configuration, plugins, user-selected formats, and other genuinely dynamic input. A bad expression fails during comptime for a static grammar and during compile for a runtime one, with a line and byte-column location either way.

Every Peg<R...> satisfies nupp.peg.Matcher<R...>, so a generic adapter can forward the complete result pack without collecting it into another value:

local function match<R...>(matcher: nupp.peg.Matcher<R...>, subject: string): ((R...) | (nil))
    return matcher:match(subject)
end

local Word = nupp.peg.compile("{ [a-z]+ }")
local word: string? = match(Word, "hello")

Matching and positions#

A matcher can be called directly or through match, which mean the same thing:

local Word = nupp.peg.compile("{ [a-z]+ }")
local result = Word("hello")
local same = Word:match("hello")

Matching begins at byte position 1 unless init is supplied. Positions are 1-based, a negative init counts from the end like Lua string operations, positions before 1 clamp to 1, and positions after #subject + 1 fail. Every operation on this page that takes an init normalizes it that way:

local Word = nupp.peg.compile("{ [a-z]+ }")
assert(Word("one two", 5) == "two")

A recognizer, meaning a grammar with no captures, returns the byte position immediately after its match rather than a boolean. Failure returns nil:

local Prefix = nupp.peg.compile("'get'")
assert(Prefix("getter") == 4)
assert(Prefix("setter") == nil)

Searching with find#

Use find when the grammar may begin after the starting position. It returns the first byte, the exclusive next byte, and every grammar result, without constructing a match record:

local Word = nupp.peg.compile("{ [a-z]+ }")
local first, nextPosition, value = Word:find("123 hello")
assert(first == 5 and nextPosition == 10 and value == "hello")

For Peg<R...>, success has the pack (integer, integer, R...) and failure has (nil, nil). The byte range is half-open, [first, nextPosition), so an empty match has equal positions. Test first for success rather than the value, because a grammar action may successfully answer nil or false. A recognizer's third result is the same next-byte position match would have answered.

Testing with isMatch#

Use isMatch when only existence matters. It performs the same search and returns only a boolean:

local Digits = nupp.peg.compile("[0-9]+")
assert(Digits:isMatch("room 42"))
assert(not Digits:isMatch("room"))
assert(not Digits:isMatch("42 rooms", 3))

The position after the last byte is included in the search, so an empty or end assertion can match there. Use match when the exact starting position is already known.

Repeated matching#

forEachMatch visits non-overlapping matches without constructing match records or an iterator closure. Its callback receives first, nextPosition, R..., the same values as find, and the call returns the number of visits:

local Word = nupp.peg.compile("{ [a-z]+ }")
local words: {string} = {}
local count = Word:forEachMatch("one, two, three", function(first: integer, nextPosition: integer, value: string)
    words[#words + 1] = value
end)

assert(count == 3)
assert(words[2] == "two")

The next search begins at the exclusive end of a consuming match. An empty match instead advances one byte, so an empty grammar cannot repeatedly report the same position. The boundary at #subject + 1 remains eligible and is visited at most once, which is why '' visits positions 1, 2 and 3 in a two-byte subject.

Text before init is not visited. The visitor's return value is ignored, so raising is the way to abort a traversal.

Replacement#

replace replaces the first match and replaceAll replaces every non-overlapping match. A string replacement is inserted literally, with no interpretation of $, %, or capture references:

local Digits = nupp.peg.compile("[0-9]+")
assert(Digits:replace("room 42, floor 3", "#") == "room #, floor 3")
assert(Digits:replaceAll("room 42, floor 3", "#") == "room #, floor #")

Use a typed callback when the replacement text depends on the match. It receives the raw positions followed by every grammar result and must answer a string:

local Word = nupp.peg.compile("{ [a-z]+ }")
local output = Word:replaceAll("one, two", function(first: integer, nextPosition: integer, value: string): string
    return "[" .. tostring(first) .. ":" .. tostring(nextPosition) .. " " .. value:upper() .. "]"
end)

assert(output == "[1:4 ONE], [6:9 TWO]")

Neither operation builds match records. A callback can use a substring capture as above, use another typed grammar result, or slice the original subject with the reported half-open range. When no match exists the original string is answered, and init leaves the prefix before it unchanged.

Empty matches insert without removing a byte, and replaceAll preserves that byte while advancing under the same progress rule forEachMatch uses. An empty grammar therefore turns "ab" into "-a-b-" when replacing with "-".

Expression syntax#

The notation is LPeg 1.1 re, operator for operator: the same forms parse the same way and capture the same things, and the test suite runs the official re module as a differential oracle. Read its reference for the syntax itself: https://www.inf.puc-rio.br/~roberto/lpeg/re.html. require("lpeg") returns native LPeg 1.1 and require("re") returns the bundled official frontend over it, so a grammar written for either works here unchanged.

What follows is only what Nupp adds to that notation or holds it to.

Captures become a typed result pack#

A matcher has one result pack, inferred from the grammar's captures. Adjacent captures are adjacent native Lua results, with no tuple and no table between them:

local Field = nupp.peg.compile("{ [a-z]+ } ':' {}")
local name, nextPosition = Field("size:")

That grammar is Peg<(string, integer)>. The parentheses delimit one explicit type-pack argument rather than constructing a tuple, and the compiler usually infers the pack, so the annotation is only needed at an API boundary.

Wrap repeated or open-ended captures in {| p |} or p -> {}, so the table allocation comes from the grammar rather than from the matcher:

const Fields = comptime do
    return nupp.peg.compile("{| { [a-z]+ } (',' { [a-z]+ })* |} !.")
end

Every arm of an ordered choice must produce the same capture shape. That is what keeps the inferred Peg<R...> true whichever arm matched.

Definitions a grammar names#

%name, -> name, => name, >> name, and ~> name each read a value the caller supplies. A runtime grammar takes them in CompileOptions.definitions:

local Integer = nupp.peg.compile("[0-9]+ -> integer !.", {
    definitions = {
        integer = function(text: string): integer
            return assert(tonumber(text)) as integer
        end,
    },
})

assert(Integer("42") == 42)

Integer is inferred as nupp.peg.Peg<integer> from the transformation's declared result pack. A transformation may answer several values, and they splice into the surrounding pack. Where the grammar or the definitions table is dynamic, annotate the result with what the application knows.

A static grammar cannot carry a runtime function, because a comptime value closes over nothing. It materializes as a factory instead, whose parameter is a closed record holding exactly the names the grammar used:

local record Definitions
    integer: function(string): integer
end

const IntegerFactory: function(Definitions): nupp.peg.Peg<integer> = comptime do
    return nupp.peg.compile("[0-9]+ -> integer !.")
end

local Integer = IntegerFactory(new Definitions(integer = function(text: string): integer
    return assert(tonumber(text)) as integer
end))

Every slot is required and an extra one is rejected, so a missing or misspelled name is a compile error rather than a nil call mid-parse.

What is rejected#

A grammar is refused with NUPP2417 when a reference resolves to nothing, a rule name repeats, a repetition body can match the empty string, the arms of one choice disagree about captures, a rule is directly or indirectly left-recursive, or nesting passes 256 levels. Rewrite a left-recursive list as a head followed by repetition:

value <- item (',' item)*

Backends#

CompileOptions.backend accepts "auto" or "lpeg". auto is the default: every static grammar becomes a validated canonical PEG graph, Nupp emits straight-line Lua for the few shapes it recognizes, namely fixed-width matches, repeated bytes and packed whole-input scans, and every other graph lowers directly to native LPeg patterns.

local Fast = nupp.peg.compile("[a-z]+ !.")
local General = nupp.peg.compile("[a-z]+ !.", {backend = "lpeg"})

lpeg disables Nupp's straight-line specializations and always lowers the graph to native LPeg, which is what makes a backend comparison possible. There is no Nupp PEG bytecode and no general-purpose interpreter.

Runtime textual grammars are compiled by LPeg's re module and cached by source. They do not invoke loadstring, and auto invokes it only when a static graph selects a Nupp specialization. A repeated byte or class plan also emits a direct byte-scanning forEachMatch loop, so a traversal does not re-enter LPeg for every match; a typed replacement callback keeps the general search loop.

The expression syntax is LPeg 1.1 re syntax: the same operators have the same parsing and capture meanings, and the test suite runs the official re module as a differential oracle. Direct require("lpeg") returns the native LPeg 1.1 module, and require("re") returns the bundled official Lua frontend over it. Nupp's declaration and operator checking track capture packs through ordinary LPeg composition, but the runtime object stays LPeg's pattern userdata.

type Backend = 'auto' | 'lpeg'

type CompileOptions = {
    --- Values named by `%name`, `-> name`, `=> name`, `>> name`, or `~> name`.
    definitions: Definitions?,

    --- Matcher implementation, `auto` when omitted.
    backend: Backend?
}

type Definitions = {[string]: any}

interface Matcher<R...> ...

match: function(self, subject: string, init: integer?): ((R...) | (nil))

find: function(self, subject: string, init: integer?): ((integer, integer, R...) | (nil, nil))

isMatch: function(self, subject: string, init: integer?): boolean

forEachMatch: function(
    self,
    subject: string,
    visitor: function(first: integer, nextPosition: integer, R...),
    init: integer?
): integer

replace: function(
    self,
    subject: string,
    replacement: string | function(first: integer, nextPosition: integer, R...): string,
    init: integer?
): string

replaceAll: function(
    self,
    subject: string,
    replacement: string | function(first: integer, nextPosition: integer, R...): string,
    init: integer?
): string

__call: function(self, subject: string, init: integer?): ((R...) | (nil))

record Peg<R...> is Matcher<R...> ...

match: function(self, subject: string, init: integer?): ((R...) | (nil))

local Word = nupp.peg.compile("{ [a-z]+ }")
assert(Word:match("one two", 5) == "two")
assert(Word("123") == nil)

find: function(self, subject: string, init: integer?): ((integer, integer, R...) | (nil, nil))

local Word = nupp.peg.compile("{ [a-z]+ }")
local first, nextPosition, value = Word:find("123 hello")
assert(first == 5 and nextPosition == 10 and value == "hello")

isMatch: function(self, subject: string, init: integer?): boolean

local Digits = nupp.peg.compile("[0-9]+")
assert(Digits:isMatch("room 42"))
assert(not Digits:isMatch("room", 2))

forEachMatch: function(
    self,
    subject: string,
    visitor: function(first: integer, nextPosition: integer, R...),
    init: integer?
): integer

local Word = nupp.peg.compile("{ [a-z]+ }")
local seen: {string} = {}
local count = Word:forEachMatch("one, two", function(_, _, word: string)
    seen[#seen + 1] = word
end)
assert(count == 2 and seen[2] == "two")

replace: function(
    self,
    subject: string,
    replacement: string | function(first: integer, nextPosition: integer, R...): string,
    init: integer?
): string

local Digits = nupp.peg.compile("[0-9]+")
assert(Digits:replace("room 42, floor 3", "#") ==
    "room #, floor 3")

replaceAll: function(
    self,
    subject: string,
    replacement: string | function(first: integer, nextPosition: integer, R...): string,
    init: integer?
): string

local Digits = nupp.peg.compile("[0-9]+")
assert(Digits:replaceAll("room 42, floor 3", "#") ==
    "room #, floor #")

__call: function(self, subject: string, init: integer?): ((R...) | (nil))

local compile: function(source: string, options: CompileOptions?): Peg<...any>

local Words = nupp.peg.compile(
    "{| { [a-z]+ } (',' { [a-z]+ })* |} !."
)
local values = assert(Words("red,green,blue")) as {string}
assert(values[2] == "green")

local General = nupp.peg.compile("[0-9]+ !.", {backend = "lpeg"})
assert(General("123") ~= nil)