Sunil SarolkarProfessional profile ↗Profile ↗
← All projects

From the workbench

StructFormer

Transformer-based model built to automate structured data transformation from one schema to another

Explore the source ↗Read the project article ↗

Project overview

StructFormer explores a specific data-engineering question: can a sequence model learn to turn a structured validation error into a corrective statement? Its inputs combine identifiers and error descriptions; its targets are structured actions such as SQL updates and adjustment-log inserts.

The project uses a custom Transformer encoder–decoder with TensorFlow/Keras and a SentencePiece tokenizer. The repository contains preprocessing, training and inference code, making the transformation pipeline available to inspect rather than hiding it behind a hosted demo.

From an error record to an action

Financial and trading workflows can produce errors such as a missing quantity or an incorrect account type. The experiment frames correction as paired input and output text: an error record on one side, the intended adjustment on the other. The training data supplies the target; the model learns the sequence transformation.

Inside the implementation

  • SentencePiece converts domain text and identifiers into subword token IDs.
  • Preprocessing prepares encoder inputs, decoder inputs and target labels, with padding and sequence boundaries.
  • The encoder–decoder combines attention layers with feed-forward layers to predict the next output token.
  • Inference starts with a beginning-of-sequence token, selects the highest-scoring next token and continues until the end token or length limit, then decodes the result to text.

Ideas beyond the experiment

The original roadmap proposed a FastAPI interface, hosted inference, an optional LLM fallback and a dashboard for inspecting per-record changes. These are extension ideas, separate from the model and inference code described here.

How the transformation is represented

A conceptual reading of the repository's preprocessing and inference stages.

Preparing examples

  1. Paired records

    Structured input and its supplied target action.

  2. SentencePiece

    Convert both sequences into token IDs.

  3. Training inputs

    Prepare padded encoder inputs, decoder inputs and labels.

Generating a candidate

  1. Input tokens

    Represent the structured error as a sequence.

  2. Encoder–decoder

    Predict output tokens with the trained Transformer.

  3. Greedy decoding

    Choose one token at a time and decode to SQL text.

The output is a generated candidate statement, not a demonstrated database execution or correctness guarantee.

A paired example

This input and supplied target come from the repository README. They illustrate the training format, not a measured prediction.

Structured inputtext
TradeID=93297 AccountID=ACC1003 ErrorType=Missing Quantity
Supplied targetsql
UPDATE Trades SET Quantity=46 WHERE TradeID=93297;
INSERT INTO AdjustmentLog(ErrorID, AdjustedBy)
VALUES('ERR3008', 'User1');

Project notes

The historical articles and README report different accuracy figures. This page does not treat those figures as verified held-out accuracy or SQL correctness.