With fingerprints, you test for matching text in a document to determine:

1. the document's subtype, or "config", for standalone files
2. the document's page range in multi-document files, or "portfolios."

See the following table for more information:

You use different fingerprint syntaxes for the two use cases. For more information, see the following sections.

Standalone documents

Parameters

The following table shows fingerprint syntax for standalone documents:

Examples

The following example tests whether a standalone document is a Wells Fargo checking account statement, for a vendor-specific config in a bank statements document type.

/* Sensible uses JSON5 to support in-line comments */
{
  /* comment out preprocessors until you're done authoring fingerprints, then turn them back on in production
    preprocessors: []
  */ 
  "fingerprint": {
    /* optional. Sensible skips this config if the fingerprint fails, improving performance when you have multiple configs */
    "tests": [
      /* array of tests.
         The fingerprint passes if Sensible finds 50% or more of the elements ('matches') in the tests array. In this example, there are 3 elements in the tests array, so if 2 out of 3 passes, the config passes. The 3 elements are: a chained match, a Match object, and a string. */
      /* The first element is a chained match ([]), so Sensible scores it as one match. The chained match succeeds only if Sensible finds
         all its Match objects in order, on separate lines. The lines can be on different pages.
         If a Match object fails, the chained match fails */
      [ 
        {
          "type": "includes",
          "text": "wells fargo"
        },
        {
          "type": "startsWith",
          "text": "statement period"
        },
        {
          "type": "startsWith",
          "text": "transaction history"
        }
      ],
      /* element 2 succeeds if Sensible finds a line that starts with "account" anywhere in the document */
      {
        "type": "startsWith",
        "text": "account"
      },
      /* element 3 succeeds if Sensible finds a line that includes "checking" anywhere in the document.
         Sensible expands a string to a case-insensitive includes match */
      "checking"
    ]
  },
  "fields": []
}

The config preferentially runs if the fingerprint passes.

Portfolios

Parameters

A fingerprint for portfolios contains a Tests parameter that takes an array of test objects. The following table shows the parameters for each test object:

Examples

For an example of using fingerprints to extract multiple documents from a portfolio file, see Multi-document extraction.

Concepts

Fingerprint syntax conversion

If you use a config in both standalone and portfolio extraction requests, write its fingerprint in the stricter portfolio syntax. Sensible automatically converts between the two syntaxes as follows:

• To convert to portfolio syntax, Sensible expands standalone syntax into a test with "page": "any".
• To convert to standalone syntax, Sensible keeps the value of each test's Match parameter and ignores the Page and Offset parameters.

Fingerprint scoring

Sensible scores fingerprints differently for standalone documents and for portfolios.

Standalone documents using standalone syntax

For the config to pass, Sensible must find 50% or more of the elements, or "matches," in the Tests parameter.

Sensible searches the whole document for each element in the fingerprint. If more than one config passes, Sensible runs all the configs that pass and returns the extraction with the highest score. For information about configuring the strictness with which Sensible runs or skips configs, see Fingerprint mode.

Portfolios using portfolio syntax

To avoid incorrect segmentation, design the fingerprint so that every element, or "match," in the Match parameter of every test succeeds.

Sensible scores each test on each page of the portfolio. A test classifies a page as its page type, for example first or last, only if Sensible finds all the elements in the test's Match parameter on that page. A fingerprint that correctly identifies some of the page types and misses others can result in inaccurate segmentation.

For more information, see Troubleshooting portfolio fingerprints.

Chained matches

A chained match is an array of Match objects that Sensible scores as one match, no matter how many Match objects it contains. A chained match succeeds only if Sensible finds all its Match objects in order, on separate lines. You write a chained match differently in standalone syntax and in portfolio syntax.

The following table shows syntax and scoring for chained matches and other matches:

Notes

Troubleshooting portfolio fingerprints

If any element in a test fails on the page the test targets, Sensible doesn't classify the page's type. Sensible still segments the portfolio, but the page ranges can be wrong.

The following scenario shows how incorrect segmentation depends on the test's page type. The example portfolio contains three documents:

• pages 1–2: tax return for 2020
• pages 3–7: Wells Fargo bank statement
• page 8: pay stub

In the following table:

• Each row represents a separate portfolio extraction of the example portfolio.
• Each extraction specifies one document type that contains one config. The config's fingerprint includes a failing match.

Tests for other document types in the portfolio can hide these errors. For example, if the next document's first test passes, Sensible ends the previous document even if its last test failed. Design your fingerprints so that every match in every test succeeds. When troubleshooting, test one document type at a time.

Tips for authoring fingerprints

Use the following tips when you define fingerprints.

Alternate wordings

If a phrase varies slightly from document to document, match its alternate forms with an any Boolean match in a single test. For example, the following fingerprint's last test matches Preparer's name with either a straight or a curly apostrophe:

"fingerprint": {
  "tests": [
    {
      "page": "first",
      "match": [
        {
          "type": "includes",
          "text": "U.S. Individual Income Tax Return"
        }
      ]
    },
    {
      "page": "last",
      "match": [
        {
          /*
            The test passes on a page that contains either form of the phrase.
            In practice, you'd fingerprint on text that OCR reads reliably.
            This example uses an apostrophe for demonstration.
          */
          "type": "any",
          "matches": [
            {
              /* straight apostrophe */
              "type": "includes",
              "text": "Preparer's name"
            },
            {
              /* curly apostrophe */
              "type": "includes",
              "text": "Preparer’s name"
            }
          ]
        }
      ]
    }
  ]
}

In the following example portfolio, the fingerprint classifies pages 1–2 as a tax return:

Turn off preprocessors

Sensible runs fingerprints before preprocessors. While you write and troubleshoot fingerprints, comment out any preprocessors, so that the SenseML editor displays the lines of text exactly as Sensible recognizes them when it runs the fingerprint tests.

Standalone-specific tips

The following tips apply to fingerprints for standalone documents.

Test and verify fingerprints

In the Sensible app, you can test fingerprints for standalone documents against multiple reference documents at once. If you write the fingerprints with portfolio syntax, Sensible automatically converts them. For more information, see Visual troubleshooting in the SenseML editor.

Portfolio-specific tips

The following tips apply to fingerprints for portfolio documents.

Prefer match arrays to single-match tests

Unless the document contains highly unusual, characteristic text, put related matches in one test, rather than writing a separate test for each match. A passing single-match test classifies the page even if another single-match test fails on that page, so separate tests are less strict.

The following examples are fingerprints for the bank statement in this example portfolio, which contains a tax return on pages 1–2, a bank statement on pages 3–7, and a pay stub on page 8:

For example, don't write the following:

/* AVOID THIS SYNTAX */
"fingerprint": {
  "tests": [
    /* Each first-page test contains a single match, so Sensible scores each
       phrase independently instead of requiring the phrases together */
    {
      "page": "first",
      "match": [
        {
          "text": "Wells Fargo", /* distinctive phrase */
          "type": "includes",
          "isCaseSensitive": true
        }
      ]
    },
    {
      "page": "first",
      "match": [
        {
          "text": "Account", /* generic phrase. On its own, it can match pages in unrelated documents */
          "type": "includes",
          "isCaseSensitive": true
        }
      ]
    },
    {
      /* the statement's last page */
      "page": "last",
      "match": [
        {
          "text": "Minimum daily balance",
          "type": "includes",
          "isCaseSensitive": true
        }
      ]
    }
  ]
}

Instead, use one of the following alternatives:

/* ALTERNATIVE ONE for portfolios: less strict */
"fingerprint": {
  "tests": [
    /* one first-page test that contains an unordered match array, so the array succeeds if the matches are on the same page */
    {
      "page": "first",
      "match": [
        /* flat array: In a flat array ("match": [ ]), Sensible searches for each element independently, so the lines can occur in any order on the page, and more than one element can match the same line. They must all occur on the same page to classify the page type. If this fingerprint runs in a standalone-document extraction, Sensible scores each element as one match. For more information, see Fingerprint scoring */
        {
          "text": "Wells Fargo",
          "type": "includes",
          "isCaseSensitive": true
        },
        {
          "text": "Account",
          "type": "includes",
          "isCaseSensitive": true
        }
      ]
    },
    {
      /* the statement's last page */
      "page": "last",
      "match": [
        {
          "text": "Minimum daily balance",
          "type": "includes",
          "isCaseSensitive": true
        }
      ]
    }
  ]
}

or, for stricter syntax:

/* ALTERNATIVE TWO for portfolios: stricter */
"fingerprint": {
  "tests": [
    /* one first-page test that contains a chained match array, so the array succeeds if the matches are on the same page, with additional criteria */
    {
      "page": "first",
      "match": [[
        /* In a test's match parameter, a nested array ("match": [[ ]]) is a chained match: each element matches a separate line, and Sensible must find the lines in the same order as in the array. Sensible scores the whole chained match as one match. */
        {
          "text": "Wells Fargo",
          "type": "includes",
          "isCaseSensitive": true
        },
        {
          "text": "Account",
          "type": "includes",
          "isCaseSensitive": true
        }
      ]]
    },
    {
      /* the statement's last page */
      "page": "last",
      "match": [
        {
          "text": "Minimum daily balance",
          "type": "includes",
          "isCaseSensitive": true
        }
      ]
    }
  ]
}

In the example portfolio, both alternatives classify pages 3–7 as the bank statement. The fingerprint to avoid also classifies page 2 of the tax return as a separate bank statement, because its generic single-match test finds Account in the tax return's refund section.