Skip to main content
A pre-processor reshapes the data Twine has computed locally, just before it is converted to the target system’s format and sent. The most common use is constraining history: limiting the time span a target system receives, regardless of how much history Twine holds. The typical case is a customer who has moved to a new source system while keeping an existing payroll system. The new source may carry an imported history that nobody has verified, and writing it into payroll would overwrite records that are known to be correct. A pre-processor lets the payroll system receive only the part of the timeline that can be trusted.
A pre-processor changes nothing in Twine. It applies to a single domain mapping, in memory, on its way out - the data Twine holds, and what every other mapping sends, is untouched.

​
Where they run

Pre-processing happens after the entities for a run have been selected and filtered, and before anything is converted or sent:
  1. Twine loads the local entities for the mapping.
  2. Conditions exclude entities that should not be processed at all.
  3. Pre-processors reshape each remaining entity.
  4. The result is converted to the target system’s format and reconciled against what the target already has.
Because the pre-processor runs before conversion, previews reflect it too. Preview sync for all employees, and the preview for a single employee, show exactly what a pre-processor leaves behind - so a change can be checked before any sync runs.

​
What can be configured

Pre-processors are configured on the domain mapping itself, in the Employee Pre-Processors card. Support varies by domain: today they are available on domain mappings whose target domain is Employee, and the card does not appear elsewhere. Two slots are available, and each is optional: Configure one, both, or neither. With neither configured, the employee is sent untouched.

​
The dates are computed, not typed in

Neither slot holds a fixed date field. Each holds a small Data Engine graph that is evaluated once per employee, and whatever date it produces becomes that employee’s bound. A graph that returns nothing means no bound. That makes both the simple and the per-person case possible with the same mechanism:
  • A single Date node with a fixed value gives every employee the same cut-off - the usual setup for a migration.
  • Reading a property from the employee gives each person their own bound, for example a custom property holding the date their data was verified.
Press the pencil button on a slot to open the editor, or the funnel button to create one. The graph must produce a single date.

​
Range From Date

A from date can be applied in two ways, chosen with the buttons under the slot. The difference matters, and it is not a question of how much data is sent - it is a question of whether Twine is allowed to change the dates on what it sends. Slice timeline is the original behavior and stays selected on every mapping configured before the choice existed. Nothing changes until you change it.

​
An example

Take an employee hired on 2015-03-01, whose salary was set to 10 000 in 2019, corrected to 20 000 in December 2019, and raised to 30 000 in February 2020. The from date is 2020-01-01. Both modes leave out the 10 000, which had been replaced two weeks before the from date and was in effect nowhere inside the window. Only Slice timeline restates the remaining facts: the 20 000 is presented as if it began on the cut-off, and the employee as if they were hired on it. Three details of Drop superseded values are worth knowing:
  • A value that is still in effect is always kept, however old it is. A salary set in 2015 and never changed since is sent with its 2015 date. Nothing replaced it, so nothing is out of date.
  • A value replaced exactly on the from date is left out. It was in effect right up to the date and not on it, so it has nothing to say about the window.
  • Records with their own history are handled separately. Where an employee has several of something - two salary records, several employment periods - each one keeps its own latest value. A record last changed in 2018 is not dropped because a different record was changed more recently.

​
Range To Date

A to date is not a mirror of the from date. It means the employment ends on that date: values dated after it are left out, and the employee is presented to the target as having left, with a leave date and a terminated status derived from the bound. Leave the slot empty and the employment stays open. The to date behaves the same way whichever from mode is selected.

​
What a pre-processor does not do

  • It does not remove anything from the target system. Sending less history does not, on its own, delete records the target already holds. What happens to records outside the window is decided by the mapping’s per-section sync behavior - see the Diff Engine.
  • It does not choose which employees are sent. That is what conditions are for.
  • It does not change Twine’s own data. The full history stays in Twine and remains available to the API and to every other domain mapping.

Domain Mappings

Where pre-processors are configured, and the rest of a mapping’s behavior.