# `nupp.text.utf8` `nupp.text.utf8` reads Unicode scalars out of strings and `nupp.io.ByteView` values. Reach for it to count, decode, validate or truncate text whose bytes came from somewhere else. ```nupp const utf8 = nupp.text.utf8 assert(utf8.length("A€") == 2) assert(utf8.encode(0x20ac) == "€") ``` `length` counts scalars, replacing each malformed byte with one replacement scalar rather than refusing the string. `encode` goes the other way, from one scalar value to its bytes. ## Byte offsets Offsets here are 1-based, so they compose with Lua string positions and with `string.sub` on the same text. This is the opposite of the zero-based offsets a buffer uses, because those are offsets into storage rather than positions in text. ## Decoding a scalar `decodeAt` decodes forward from an offset and answers the scalar with the offset after it. `decodeBefore` decodes the scalar ending before an offset and answers it with the offset it starts at, so the two walk in opposite directions over the same positions: ```nupp const utf8 = nupp.text.utf8 local codepoint, nextByte = utf8.decodeAt("A€", 2) assert(codepoint == 0x20ac and nextByte == 5) codepoint, nextByte = utf8.decodeBefore("A€", nextByte) assert(codepoint == 0x20ac and nextByte == 2) ``` A malformed byte answers `0xFFFD` and advances exactly one byte, so a walk over damaged text makes progress rather than stalling. Nil is the end of the value, and that is what stops the walk: ```nupp const utf8 = nupp.text.utf8 local at = 1 while true do local codepoint, nextAt = utf8.decodeAt(text, at) if codepoint == nil then break end at = nextAt end ``` One past the end is an accepted offset; anything further raises. ## Validating and truncating Validation is explicit, because decoding never refuses. `isValid` checks a complete byte sequence: ```nupp const utf8 = nupp.text.utf8 assert(utf8.isValid("café")) assert(not utf8.isValid("\xff")) ``` Overlong forms, surrogate halves and codepoints above the maximum are malformed even though their lead bytes are well formed, so they are rejected on the value rather than on the shape. `validPrefixLength(value, maxBytes)` answers the length of the largest valid prefix no longer than a byte budget, which is what keeps a fixed-width field from splitting a scalar in half. `truncate` applies that to a string and copies the prefix: ```nupp const utf8 = nupp.text.utf8 assert(utf8.validPrefixLength("A€B", 4) == 4) assert(utf8.truncate("A€B", 4) == "A€") ``` `truncate` takes a string, since it answers one. Everything else here takes a string or a byte view. ::: seealso - `nupp.io` for the byte views these operations accept - `nupp.codec.json` for JSON encoding and decoding ::: ## Functions ### `utf8.decodeAt` _function_ ```nupp function utf8.decodeAt(value: string | ByteView, byteOffset: integer): integer?, integer ``` Reads the codepoint starting at a byte offset, and where the next one starts. Walking a buffer is this call and nothing else, because a malformed byte answers `0xFFFD` and still advances: ```nupp const utf8 = nupp.text.utf8 local at = 1 while true do local codepoint, nextAt = utf8.decodeAt(text, at) if codepoint == nil then break end at = nextAt end ``` #### Arguments | Name | Type | Description | | --- | --- | --- | | `value` | `string | ByteView` | the bytes to read | | `byteOffset` | `integer` | the 1-based byte to start at; one past the end is allowed | #### Returns | Type | Description | | --- | --- | | `integer?` | the codepoint, or nil at the end of the value | | `integer` | the byte the next codepoint starts at | #### Raises - when byteOffset is not an integer, or is outside the value ### `utf8.decodeBefore` _function_ ```nupp function utf8.decodeBefore(value: string | ByteView, byteOffset: integer): integer?, integer ``` Reads the codepoint ending just before a byte offset, and where it starts. `decodeAt` and `decodeBefore` walk the same positions in opposite directions, so decoding forward and then back lands where it started: ```nupp const utf8 = nupp.text.utf8 local codepoint, nextByte = utf8.decodeAt("A€", 2) assert(codepoint == 0x20ac and nextByte == 5) codepoint, nextByte = utf8.decodeBefore("A€", nextByte) assert(codepoint == 0x20ac and nextByte == 2) ``` #### Arguments | Name | Type | Description | | --- | --- | --- | | `value` | `string | ByteView` | the bytes to read | | `byteOffset` | `integer` | the 1-based byte to look back from | #### Returns | Type | Description | | --- | --- | | `integer?` | the codepoint, or nil at the start of the value | | `integer` | the byte that codepoint starts at | #### Raises - when byteOffset is not an integer, or is outside the value ### `utf8.encode` _function_ ```nupp function utf8.encode(codepoint: integer): string ``` Encodes one codepoint as UTF-8 bytes. ```nupp const utf8 = nupp.text.utf8 assert(utf8.encode(0x20ac) == "€") ``` Surrogate halves encode rather than raise, the way a lone half decodes to the replacement rather than stopping a walk: what refuses them is [`nupp.text.utf8.isValid`](#nupp.text.utf8.isValid), on the bytes, and encoding one is how a caller reproducing damaged text writes it back out. #### Arguments | Name | Type | Description | | --- | --- | --- | | `codepoint` | `integer` | the codepoint to encode | #### Returns | Type | Description | | --- | --- | | `string` | its encoding | #### Raises - when the codepoint is not an integer, or is outside Unicode ### `utf8.isValid` _function_ ```nupp function utf8.isValid(value: string | ByteView): boolean ``` Reports whether the whole value is well-formed UTF-8. ```nupp const utf8 = nupp.text.utf8 assert(utf8.isValid("café")) assert(not utf8.isValid("\xff")) ``` #### Arguments | Name | Type | Description | | --- | --- | --- | | `value` | `string | ByteView` | the bytes to test | #### Returns | Type | Description | | --- | --- | | `boolean` | whether every sequence is well formed | ### `utf8.length` _function_ ```nupp function utf8.length(value: string | ByteView): integer ``` Counts the codepoints in the value. A malformed byte counts as one. ```nupp const utf8 = nupp.text.utf8 assert(utf8.length("A€") == 2) ``` #### Arguments | Name | Type | Description | | --- | --- | --- | | `value` | `string | ByteView` | the bytes to measure | #### Returns | Type | Description | | --- | --- | | `integer` | the codepoint count | ### `utf8.truncate` _function_ ```nupp function utf8.truncate(text: string, maxBytes: integer): string ``` Cuts text to at most `maxBytes`, never through a codepoint. ```nupp const utf8 = nupp.text.utf8 assert(utf8.truncate("A€B", 4) == "A€") ``` #### Arguments | Name | Type | Description | | --- | --- | --- | | `text` | `string` | the text to cut | | `maxBytes` | `integer` | the most bytes the answer may use | #### Returns | Type | Description | | --- | --- | | `string` | the cut text | ### `utf8.validPrefixLength` _function_ ```nupp function utf8.validPrefixLength(value: string | ByteView, maxBytes: integer): integer ``` Measures the longest well-formed prefix no longer than `maxBytes`. ```nupp const utf8 = nupp.text.utf8 -- "A€B" is 5 bytes; the 4-byte budget stops just short of "B" rather -- than splitting "€" in half. assert(utf8.validPrefixLength("A€B", 4) == 4) ``` #### Arguments | Name | Type | Description | | --- | --- | --- | | `value` | `string | ByteView` | the bytes to measure | | `maxBytes` | `integer` | the most bytes the prefix may use | #### Returns | Type | Description | | --- | --- | | `integer` | the prefix length, in bytes |