FitBoxesToInk

Import

import { FitBoxesToInk } from '@stabrise/scaledp'

Group

Transform

Reads

image, boxes

Writes

boxes

Needs

nothing

Every source of boxes reports them a little short of the glyphs. A text detector misses what is thin and isolated at a line's ends, such as the looping capital of handwriting, because those strokes score low on its probability map. A PDF text layer sizes boxes from font metrics, which a script font's flourishes and descenders overrun. For reading, a clipped capital turns Freya into reya. For redaction, a box short of its text leaves the edge of a name showing.

This stage checks each box against the pixels and grows it until the ink it belongs to is inside. Boxes only ever grow, and their text is untouched.

import { Pipeline, FitBoxesToInk } from '@stabrise/scaledp'
import { PdfToImage } from '@stabrise/scaledp/pdf'
import { DbnetOnnxDetector, PaddleRecognizer } from '@stabrise/scaledp/ocr'

const rows = await new Pipeline([
    new PdfToImage({ resolution: 200 }),
    new DbnetOnnxDetector({ outputCol: 'detected' }),
    // Before reading, so the crop contains the whole word.
    new FitBoxesToInk({ inputCols: ['image', 'detected'], outputCol: 'detected' }),
    new PaddleRecognizer({ inputCols: ['image', 'detected'] }),
    // After reading, so the final boxes cover the glyphs.
    new FitBoxesToInk({ inputCols: ['image', 'text'], outputCol: 'text' }),
]).transform(file)

It reads bboxes off a DetectorOutput or a Document and writes the same value back with the boxes grown, by default in place. The boxes must be in the image's own pixels: the embedded picture a detector ran on, or the page after PdfMergeImageText.

How a box is fitted

Ink is told from background locally. The background is the median brightness of a thin ring just outside the box, where it is the great majority even when a stroke escapes. Ink is anything that differs from it by at least half the contrast the box's own text has. So dark text on paper, white text on black and text on a coloured highlight all work, and a low-contrast box is left alone rather than guessed at.

Each edge steps outwards one pixel line at a time while the line just past it still holds ink:

Crossing small gaps along the line is what reaches a capital written a little apart from the rest of its word. Never crossing one across the line is what keeps two lines from merging. Two passes are made, so a swash leaving the box at a corner is reached along the text and then its top across it.

A box never grows into another box. Each edge may move at most halfway to the nearest box beyond it: along the line for left and right, among boxes overlapping it for top and bottom. Halfway, because both boxes may be growing towards each other: two neighbours can end up touching, never overlapping. In a PDF text layer each word's box runs almost to the next word's start, a gap of a few pixels, well inside the letter-gap tolerance, so ink alone would merge every word with the next. A box counts as beyond an edge only when its centre is, so an invisible text layer's box stacked on the same word doesn't pin the OCR box in place.

Thin rules don't count. A column needs a tenth of the line's height in ink to carry a box along, so an underline or a table rule running out of the box, one or two pixels thick, cannot drag it sideways.

Rotated boxes are fitted in their own frame, along and across their line, and keep their angle. Their neighbour limits are measured in that frame too: each neighbour is projected onto the box's own axes, so a box on the same turned line stops its ends and one in the same turned column stops its sides. Without that, a quarter-turned word would grow across a 3 px word gap into the next one.

Parameters