API: Data Integrity

🚧

Data Integrity - Beta testing

Please note, this feature is in beta testing. We encourage you to provide any feedback you have on the product by selecting Submit idea here on our roadmap.

The Data Integrity API consists of the following endpoints:

  • Status endpoints: (one per datatype) exposes the information needed to usefully query results.
  • Summaries endpoints: (one per datatype) exposes summary results, queryable in a granular way.
  • Details endpoints: (one per datatype) exposes record by record information, queryable using the same parameters as the summary endpoint.

Matching process

Matching occurs each time a sync happens where data across all data types exist. There’s no mechanism to manually trigger a match because this is all done automatically.

Data sources and data types

Data Integrity requires Accounting and Banking data sources to be linked with the following data types enabled:

  • banking-accounts for the banking data source.
  • banking-transactions for the banking data source.
  • bankAccounts for the accounting data source.
  • accountTransactions for the accounting data source.

📘

Deprecation notice

Matching also works with the bankAccounts (banking data source) and bankTransactions (banking data source). Note that these data types will be deprecated in the future.

It is recommended that you use banking-accounts and banking-transactions data types to get the most out of Data Integrity.

Status

This endpoint exposes the status for the company’s match results between the data type provided in the URL and other data types with which Data Integrity supports matching. It will help you understand whether match data is available and, if so, how to usefully query it.

The response tells you whether match results are available, and, if they are:

  • When the results were generated, and their status.
  • The connection IDs, amounts and dates involved, to support useful querying.

📘

Overlapping dates

In the UK, Open Banking allows you to pull a maximum of 24 months of data but could be less with just 18 or 12 months available.

This means that companies can have data for different date ranges from different sources, which could distort results. For example, if the accounting package has data synced from 2018 and the banking source only has data synced from 2019, the match percentage will be distorted by all the ‘unmatched’ transactions from 2018.

It is recommended that you use match results only for the date range when data from both sources overlap; this is provided for you in the Overlapping dates part of the status response.

The endpoint is available in Swagger under Assess.

GET /data/companies/{companyId}/assess/dataTypes/{dataType}/dataIntegrity/status

Parameters

Parameter

Type

Description

Required

companyId

string

The ID of the company you want match results for.

Submit as route parameter.

Required

datatype

string

The data type you want match results for.

Accounting source:
bankAccounts
accountTransactions

Banking source:
banking-accounts
banking-transactions

Submit as route parameter.

Required

Data model

Field

Type

Description

type

string

The data type which the data type in the URL has been matched against. For example, if you've matched accountTransactions and banking-transactions, and you call this endpoint with accountTransactions in the URL, this property would be banking-transactions.

statusInfo

See Status info

connectionIds

See Connection ID

amounts

See Amounts

Only returned for transactions. For accounts, there is nothing returned.

dates

See Dates

Only returned for transactions. For accounts, there is nothing returned.

Status info

Field

Type

Description

lastMatched

string
See Date

The date the matching algorithm last ran against the company’s bank transactions.

currentStatus

string

One of the following:

  • Unknown
  • DoesNotExist - have never attempted a match run for this company as do not have datasets required
  • Error - something went wrong upon matching
  • Processing
  • CompleteWithWarning
  • Complete

statusMessage

string

Detailed explanation supporting the status value.

Connection ID

Field

Type

Description

source

array

An array of strings. The connection IDs for the type specified in the url.

target

array

An array of strings. The connection IDs for the type being matched to.

Amounts

Field

Type

Description

min

number

Lowest value of transaction set.

max

number

Highest value of transaction set.

Dates

Field

Type

Description

minDate

string
See Date

Earliest date of transaction set.

maxDate

string
See Date

Latest date of transaction set.

minOverlappingDate

string
See Date

Earliest date where transactions exist in both accounting and banking platforms.

maxOverlappingDate

string
See Date

Latest date where transactions exist in both account and banking platforms.

Sample response

//call with dataType = 'accountTransactions'
{
    "metadata":[
        {
            "type":"banking-transactions",
            "statusInfo":{
                "lastMatched":"2021-09-17T12:09:33.441Z",
                "currentStatus":"Unknown|DoesNotExist|Error|Processing|CompleteWithWarning|Complete",
                "statusMessage":"string"
            },
            "connectionIds":{
                "source": ["guid","guid","guid"],
                "target": ["guid","guid","guid"]
            },
            "amounts":{
                "min":130.0,
                "max":2450.0,
            },
            "dates":{
                "minDate":"2021-09-17T12:09:33.441Z",
                "maxDate":"2021-12-16T12:12:53.441Z",
                //Overlapping dates are dates where both 'sides' of the match have data 
                "minOverlappingDate":"2021-09-30T12:09:13.441Z",
                "maxOverlappingDate":"2021-11-27T12:19:33.441Z" 
            }   
        }
    ]
  }
  //(identically-formatted output if you call with dataType = banking-transactions except 
  //it will be keyed on accountTransactions)
  
  //call with dataType = 'banking-accounts'
  {
    "metadata":[
      {
        "type":"bankAccounts",
        "status":{
            "lastMatched":"2021-09-17T12:09:33.441Z",
            "status":"Unknown|DoesNotExist|Error|Processing|CompleteWithWarning|Complete",
            "statusMessage":"string"
        },
        "connectionIds":{
            "source" : ["guid","guid","guid"],
            "target":["guid","guid","guid"]
        }
          //Amount and date objects are null as not relevant to bank accounts
      }
    ]
  }
  //(identically-formatted output if you call with dataType = bankAccounts except 
  //it will be keyed on banking-accounts)

Summaries

This endpoint exposes a summary of match results for a given data type filtered by a query string in the Codat query language. Only the summary results are returned, no transactions.

So, for example, if you wanted to see summary match results only for transactions after 1 December 2020, you could send ‘query=date>2020-12-01’.

The endpoint is available in Swagger under Assess.

GET /data/companies/{companyId}/assess/dataTypes/{dataType}/dataIntegrity/summaries

Parameters

Parameter

Type

Description

Required

companyId

string

The ID of the company you want match results for.

Submit as route parameter.

Required

datatype

string

The data type you want match results for.

Accounting source:
bankAccounts
accountTransactions

Banking source:
banking-accounts
banking-transactions

Submit as route parameter.

Required

Query

string

You can query any properties in the response.
It can be left blank. E.g.
query=date>2020-12-01

Submit as query parameter.
This follows the standard Codat query language.

Data model

For transactions, the response contains summary statistics (such as match percentage) by both amount and count. For accounts, statistics based on amount are not meaningful, therefore we return only statistics based on count.

Field

Type

Description

type

string

The data type which the data type in the URL has been matched against. For example, if you've matched accountTransactions and banking-transactions, and you call this endpoint with accountTransactions in the URL, this property would be banking-transactions.

byAmount

See By amount

byCount

See By count

By amount

Field

Type

Description

matchPercentage

number

The percentage of the absolute value of transactions of the type specified in the route which have a match.

unmatched

number

The sum of the absolute value of transactions of the type specified in the route which don't have a match.

matched

number

The sum of the absolute value of transactions of the type specified in the route which have a match.

total

number

The total of unmatched and matched.

By Count

Field

Type

Description

matchPercentage

number

The percentage of records of the type specified in the route which have a match.

unmatched

number

The number of records of the type specified in the route which don't have a match.

matched

number

The number of records of the type specified in the route which do have a match.

total

number

The total of unmatched and matched.

Sample Response

//Call with banking-transactions 
{
    "summaries":[
      {
        "type":"accountTransactions",
        "byAmount":{
            "matchPercentage": 68.3,
            "unmatched":24871.5,
            "matched": 53587.5,
            "total":78459.0,
        },
        "byCount":{
            "matchPercentage": 79.3,
            "unmatched":253,
            "matched": 970,
            "total":1223
        }
      }
    ]
  }
  //(identically-formatted output if you call with dataType = accountTransactions except 
  //it will be keyed on banking-transactions)
  
  //Call with banking-accounts
  {
    "summaries":[
      {
        "type":"bankAccounts",
        //amount-related properties will be null as not relevant to bank accounts
        "byCount":{
          "matchPercentage": 71.4,
          "unmatched":2,
          "matched": 5,
          "total":7
        }
      }
    ]
  }
  //(identically-formatted output if you call with dataType = bankAccounts except 
  //it will be keyed on banking-accounts)

Reproducing the overall match percentage from the Portal

The match percentage you get back from our Summaries endpoint is for the data type you have specified in the url.

Consider a simple example where you have just three transactions:

  • Transaction A - Accounting, £3, no matches
  • Transaction B - Accounting, £1, matches transaction C
  • Transaction C - Banking, £1, matches transaction B

If you call the Summaries endpoint with data type = accountTransactions, you will get a match percentage of 25%:
match percentage = B/(A+B) = £1 / (£3 +£1)

If however you call the Summaries endpoint with data type = banking-transactions, you will get match percentage of 100%:
match percentage = C/C = £1 / £1

By contrast, in the Data Integrity page on Portal, the match percentage displayed is match percentage by amount across accounting and banking transactions. In our example, the match percentage displayed on the Portal would be 40%:
match percentage = (B+C)/(A+B+C) = (£1 + £1)/(£3 +£1 + £1)

942942

You can reproduce this match percentage yourself by fetching the summaries for accountTransactions and banking-transactions in separate API calls, and combining the results client-side, e.g. by taking a weighted average of the match percentages.

Note that by default the percentage on the Portal is also restricted to the overlapping date range; you can reproduce this yourself by getting the min and max overlapping dates from the Status endpoint, and then restricting your calls to the Summaries endpoint to these dates using the query parameter.

E.g. if the Status response contains this:

"dates":{
..
"minOverlappingDate":"2021-09-03T12:00:00.000Z",
"maxOverlappingDate":"2021-09-17T23:59:59.999Z" 
}

Then you would call each of the Summaries endpoints with (url-escaped) query=date>=2021-09-03T12:00:00.000Z&&date<=2021-09-17T23:59:59.999Z

Details

This endpoint exposes match results record by record for a given data type, filtered based on a query string in the same way as summary results. The results are paginated and support ordering, following the same conventions as our other data endpoints.

The endpoint is available in Swagger under Assess.

GET /data/companies/{companyId}/assess/dataTypes/{dataType}/dataIntegrity/details

Parameters

Parameter

Type

Description

Required

companyId

string

The ID of the company you want match results for.

Submit as route parameter.

Required

dataType

string

The data type you want match results for.

Accounting source:
bankAccounts
accountTransactions

Banking source:
banking-accounts
banking-transactions

Submit as route parameter.

Required

Query

string

Can query any property in response.

Submit as query parameter.
This follows the standard Codat query language.

page

number

Submit as query parameter. Defaults to 1.

pageSize

number

Submit as query parameter. Defaults to 100.

orderBy

string

State the property name by which you would like to order the response by.

Submit as query parameter.

Data model

Response for transactions

Element

Type

Description

type

string

The data type of the record.

connectionId

string

ID GUID representing the connection of the accounting or banking platform.

id

string

A concatenation of the accountId and transactionId, in the format accountId_transactionId. This is unique to data integrity.

date

date
See Date

The date of the transaction.

description

string

The transaction description.

amount

number

The transaction value.

currency

string

The currency of the transaction.

matches

array
See Transactions matches array

Refer to the matches array table below.

Transactions matches

This outlines the transaction(s) in which the original transaction has matched with its corresponding transaction in the other platform.

Element

Type

Description

type

string

The data type which the data type in the URL has been matched against. For example, if you've matched accountTransactions and banking-transactions, and you call this endpoint with accountTransactions in the URL, this property would be banking-transactions.

connectionId

string

ID GUID representing the connection of the accounting or banking platform.

id

string

A concatenation of the accountId and transactionId, in the format accountId_transactionId. This is unique to data integrity.

date

date
See Date

The date of the transaction.

description

string

The transaction description.

amount

number

The transaction value.

currency

string

The currency of the transaction.

Response for accounts

Element

Type

Description

type

string

The data type of the record.

connectionId

string

ID GUID representing the connection of the accounting or banking platform.

id

string

The account’s id.

accountName

string

The name of the account.

institution

string

The name of the financial institution.

matches

array
See Accounts matches array

Refer to the matches array table below.

Accounts matches

Element

Type

Description

type

string

The data type which the data type in the URL has been matched against. For example, if you've matched accountTransactions and banking-transactions, and you call this endpoint with accountTransactions in the URL, this property would be banking-transactions.

connectionId

string

ID GUID representing the connection of the accounting or banking platform.

id

string

The account’s id.

accountName

string

The name of the account.

institution

string

The name of the financial institution.

Sample Response

//Call with datatype = accountTransactions
{
    "results": [
      {
        "type":"accountTransactions",
        "connectionId": "guid",
        //Id field here and on matches will be accountId_transactionId.
        //This is because we ideally would like a single consistent model for
        //'transaction-related' datatypes, and e.g. invoices don't have an account ID.
        //We will make this clear in documentation. 
        "id": "string",
        "date": "2021-09-15T09:00:10.898Z",
        "description": "Buying toast",
        "amount": 236.94,
        "matches":[
          {
            "type":"banking-transactions",
            "connectionId": "guid",
            "id": "string",
            "date": "2021-09-15T09:00:10.898Z",
            "description": "Purchasing toast",
            "amount": 236.94,
          },
          {
            "type":"banking-transactions",
            "connectionId": "guid",
            "id": "string",
            "date": "2021-09-15T09:00:10.898Z",
            "description": "Investing in grilled bread",
            "amount": 288.81,
          }   
        ]
      }
    ],
    "pageNumber": 1,
    "pageSize": 50,
    "totalResults": 237
  }
  //(Calling with datatype = banking-transactions will produce results in an identical format,
  //except that the list of matches will all have a 'type' field of accountTransaction)
  
  //Call with datatype = bankAccounts
  {
    "results": [
      {
        "type": "bankAccounts",  
        "connectionId": "guid",
        "id": "string",
        "accountName": "string",
        "institution": "string",
        "matches": [
          {
            "type": "banking-accounts",  
            "connectionId": "guid",
            "id": "string",
            "accountName": "string",
            "institution": "string"
          }
        ]
      }
    ],
    "pageNumber": 1,
    "pageSize": 20,
    "totalResults": 23
  }
  //(Calling with datatype = banking-accounts will produce results in an identical format,
  //except that the list of matches will all have a 'type' field of bankAccounts)

Did this page help you?