NerMerger

Import

import { NerMerger } from '@stabrise/scaledp'

Group

Understand

Reads

ner, document

Writes

ner

Needs

nothing

Two kinds of pipeline end up with more than one NER result for a page:

• Two passes over a page. NER reads the text layer first, so entities appear at once, then reads only the text an embedded-image pass added (see DocumentDelta). Each reading's offsets point into the document it read, and neither is the merged page.
• Two recognisers over one text. GlinerNer and RegexNer both find an e-mail address, and the page should carry it once.

NerMerger puts every entity's start/end into documentCol's coordinates, then keeps an entity found twice (same span, same label) once, at its best score.

import { DocumentDelta, NerMerger, Pipeline } from '@stabrise/scaledp'

const pipeline = new Pipeline([
    // ... text layer into `document`, merged page into `merged`, NER on `document` into `ner`
    new DocumentDelta({ inputCols: ['document', 'merged'], outputCol: 'delta' }),
    { stage: new GlinerNer({ inputCol: 'delta', outputCol: 'delta_ner' }), when: { path: 'delta.bboxes.length', gt: 0 } },
    new NerMerger({ inputCols: ['ner', 'delta_ner'], documentCol: 'merged' }),
    new NerConsistency({ inputCols: ['ner', 'merged'], scope: 'document' }),
])

Why offsets have to be rebased

NerConsistency matches the entities it already has against the spans it is about to propagate by start and end, and skips a propagation that overlaps one. An entity whose offsets point into some other document overlaps the wrong span, and the pass tags a name twice or not at all.

Entities are placed by their boxes, not by searching for their text. The same name usually occurs several times on a page, and an entity's boxes say which occurrence it is. Boxes are absolute page coordinates, so they survive every pass unchanged. An entity whose boxes are not on the document falls back to the first occurrence of its word, and failing that keeps its offsets. A wrong offset costs a propagation, never a redaction, because the boxes are what get drawn.

Missing and failed inputs

A column no stage wrote, because a when skipped it, counts as no entities. One failed reading does not cost the page the others: the delta pass over a picture can fail while the text layer's entities stand. Only when every input failed does the output carry an exception.

Parameters