PiiCatalog.Patterns (pii_catalog v0.1.0)

Copy Markdown View Source

The catalog itself: every identifier as a PiiCatalog.Pattern.

This module is for maintainers and for readers who want to audit a regex. Use PiiCatalog.patterns/0 and the other functions in PiiCatalog to read the catalog.

Adding a pattern

  1. An entry below. Its name is final once released, as callers store it. Use <identifier>_<country> in lowercase ASCII, the identifier in its local name and the country as its ISO 3166 code (personnummer_se, nino_gb); no country for an identifier used across borders (iban).
  2. A validator in PiiCatalog.Validators where the identifier has a check digit or a structure the regex cannot express, with unit tests.
  3. A vector file in test/vectors/<name>.exs with valid values, values with the right shape that fail the check, and look-alikes. The catalog test fails without one.
  4. A benchmark with every pattern on, on prose and on number-heavy text, noted in the pull request. See "Scan cost" below.
  5. A line in CHANGELOG.md naming the identifier, and the pii_catalog.readme Mix task run to regenerate the README table.

Boundaries

Patterns are compiled without the u flag so that they match bytewise and hold on text that is not valid UTF-8. \b then reads each byte as Latin-1, so the bytes of a non-ASCII character next to a value would count as word characters and the boundary would not be seen. The boundary fragments below ask only about ASCII word characters (A-Z, a-z, 0-9 and _): a value directly after or before any other byte, including the bytes of a non-ASCII letter, is found.

Where the pattern itself begins or ends with a word character, half of the boundary is already given and only the other half is asserted. The one lookaround keeps the scan several times cheaper than the two-sided form, which is needed only where the neighbouring character may be punctuation: the start of an email address.

Scan cost

PiiCatalog.Scanner scans every pattern that starts with (?<![A-Za-z0-9_]) and then a digit or a capital letter in one shared pass, so such a pattern adds little to the scan. A pattern that cannot start that way should start with a literal or a keyword with a single first letter (cpr, national), which the regex engine finds without trying every position; a keyword with several first letters, or a leading character class, costs a full pass of its own. A literal every match contains goes in requires, and the pattern is skipped on text without it. A keyword-led pattern whose keywords start with different letters lists one fast regex per letter in requires instead (svnr_at, nif_pt).

Summary

Functions

Every pattern, in catalog order, with its regex compiled.

The pattern with this name, with its regex compiled, or nil. Builds only that pattern.

Every pattern name, in catalog order.

Functions

all()

@spec all() :: [PiiCatalog.Pattern.t()]

Every pattern, in catalog order, with its regex compiled.

Builds the list on each call. Use PiiCatalog.compile/1 once and keep the scanner rather than calling this on every scan.

get(name)

@spec get(atom()) :: PiiCatalog.Pattern.t() | nil

The pattern with this name, with its regex compiled, or nil. Builds only that pattern.

names()

@spec names() :: [atom(), ...]

Every pattern name, in catalog order.