RegexNer

Import

import { RegexNer } from '@stabrise/scaledp'

Group

Understand

Reads

document

Writes

ner

Needs

nothing

A NER model is good at names and poor at numbers, and numbers are exactly what redaction cannot afford to miss. A plain regex is no better on a scan: OCR reads 0 as O, 1 as l or |, drops a comma into a card number, and breaks a phone number across a line. One wrong character and an exact pattern sees nothing.

RegexNer matches loosely, then validates strictly. Every candidate is checked — Luhn for cards, the checksum and encoded birth date for PESEL, the calendar for dates, the allocation rules for SSN — and a failed check lowers the score.

import { Pipeline, RegexNer } from '@stabrise/scaledp'
import { PdfToImage } from '@stabrise/scaledp/pdf'
import { PaddleTextRecognizer } from '@stabrise/scaledp/ocr'

const rows = await new Pipeline([
    new PdfToImage({ resolution: 300 }),
    new PaddleTextRecognizer({ keepFormatting: true }),
    new RegexNer({ defaultCountry: 'PL' }),
]).transform(file)

for (const entity of rows[0].ner.entities) {
    console.log(entity.entity_group, entity.word, entity.score, entity.boxes)
}

It needs no model and no download. It reads the Document in text, the same column GlinerNer reads, and writes ner. Every entity carries source: 'pattern' and the OCR boxes it spans.

What it finds

Countries

Names come from the runtime's own Intl.DisplayNames, so countryLocales (default ['en', 'pl']) can be any language the browser knows and no list of names is shipped. Three-letter codes come from the ISO 3166 table.

A country is personal data mostly as a nationality or a country of birth, and a country list collides with ordinary text, so ambiguity is resolved by a label before it — Country:, Nationality:, Citizenship:, Obywatelstwo:, Kraj urodzenia::

Postal codes need context

Five bare digits are a US ZIP, and also a German, French, Italian or Spanish postal code — and a price, a house number, an invoice number. Such a code counts only with something around it that says what it is:

A Polish 00-000 or a ZIP+4 is distinctive by its shape and needs no context. UK and Canadian codes are read from the raw text, because their letters are part of the code and the lookalike repair would turn the B of 0B1 into an 8.

How it tolerates OCR

There are three layers, each for a different kind of mistake.

Lookalikes are repaired first. Inside a number, O o Q D read as 0; I l i | ! as 1; Z z as 2; S s as 5; G b as 6; B as 8; g q as 9. "Inside a number" means a run with at least one real digit, no letter touching it, and at least half real digits across the whole spaced-out number. So 4111 1lll 1111 1111 is repaired, while Bill, SOS and the 1s of 1st are not. A repair changes characters but never the length, so offsets still land on the right boxes. A repaired match scores a little lower.

Stray symbols are absorbed. Patterns run on a small approximate-regex engine, FuzzyRegex, so a match may absorb up to maxEdits characters — a comma in 123-45-67,89, or a * where a digit smudged. Only punctuation and symbols count. A stray letter or digit is never forgiven, because that is how one number becomes another. Each edit costs score, and a digit that had to be guessed can't be checksum-verified, so it scores as unverified.

Libraries add what the patterns don't know. When they are installed, libphonenumber-js finds phone numbers by each country's real numbering plan, and chrono-node finds dates in dateLocales. Both are optional peer dependencies, loaded lazily the first time the stage runs:

npm i libphonenumber-js chrono-node

Without them the built-in patterns still run. Set useLibraries: false to get the same results whatever is installed.

Phone numbers written without a country code, such as 600 123 456, need a defaultCountry before libphonenumber-js will read them. The built-in pattern finds them either way, at a lower score.

Scores and the threshold

The default threshold of 0.5 keeps the first four rows. A number grouped like a card is a card to a redactor, whatever its check digit says — one OCR misread is enough to break Luhn — so it is reported, plainly in doubt. Raise the threshold to 0.6 to keep only what checked out; lower it to see every failed check. An identifier that fails its check is still held against the other labels, so a card number with a typo is never reported as a phone number instead.

When two detectors claim the same digits, a reading that kept the text's own grouping beats one that had to absorb a character. After that, the more specific label wins: e-mail, URL, IBAN, card, PESEL, SSN, date, postal code, country, phone.

A match never starts straight after a country code. The digits in +48790844156 are eleven and PESEL-shaped, but the + says they are a phone number, and only the phone pattern, which reads the + itself, may claim them.

With GLiNER

The two are complementary: patterns for structured identifiers, the model for names and free text. Give one of them its own output column and draw both:

new Pipeline([
    new PdfToImage({ resolution: 300 }),
    new PaddleTextRecognizer({ keepFormatting: true }),
    new GlinerNer({ inputCol: 'text', labels: ['person', 'organization', 'address'] }),
    new RegexNer({ outputCol: 'ids' }),
    new ImageDrawBoxes({ inputCols: ['image', 'ner', 'ids'] }),
])

Parameters