Getting Started

Summary

This guide walks you through running your first DataBridge Engine pipeline by copying data and the configuration file to the DataBridge Engine SFTP server.

The pipeline in this example will convert a file you provide in a JSON format to a CSV format. Normally, a pipeline would be configured to deliver data to a destination system but for purposes of demonstrating a successful pipeline execution we provide a simple example which does not require credentials for any external system.

Prerequisites

• SSH keys have been generated (see SFTP Setup) and shared with beqom
• A Tenant Name and a Username have been provided by beqom

Running a DataBridge Engine Pipeline

A DataBridge Engine Pipeline is triggered by running the following steps:

• Create a folder on the SFTP server (the "Drop Zone Folder")
  • An existing folder can be reused but this is discouraged
• Copy data files to the SFTP folder
• Copy a Configuration File (config.json) to the SFTP folder

In addition, the pipeline has access to secrets that the user has securely shared with the beqom support team when the DataBridge Engine account is being set up.

---

Configuration

Once the config.json file is detected in the Drop Zone Folder, the DataBridge Engine Pipeline is automatically triggered.

The DataBridge Engine Pipeline generates Results (results/results.json) during the execution of the pipeline and delivers pipeline information and errors to that file as well.

Create files locally

Create a folder on your workstation named databridge-getting-started. In this folder place the following files:

---

Name: employees.json

Contents:

[
  {
    "empid": 1,
    "gender": "male",
    "jobrole": "engineering",
    "salary": 1234
  },
  {
    "empid": 2,
    "gender": "female",
    "jobrole": "legal",
    "salary": 12350
  }
]

---

Name: config.json

Contents:

{
  "version": 1,
  "transformations": [
  ],
  "data_validations": [
  ],
  "destination_adapters": [
    {
      "adapter": "file_export",
      "configuration": {
        "data_source": "employees",
        "destination_file_name": "employees_out.csv",
        "format": "csv"
      }
    }
  ]
}

---

At this point, the folder structure on your local workstation should look like this:

databridge-getting-started
├── config.json
└── employees.json

In the configuration above, your employees.json file is automatically detected as the employeesData Frame and is referenced in a destination adapter by "data_source": "employees". If you were to provide a file named employees.xlsx it would also be detected as the employees dataset. Providing two files with identical names but different extensions is illegal.

Copy files to SFTP server

All data files shall be copied before the configuration file is copied. Follow these steps to copy the files:

export SFTP_TENANT_NAME=<tenant-name>
export SFTP_USER_NAME=<user-name>
export SFTP_SERVER_NAME=databridge-weu.beqom.io
export SSH_PRIVATE_KEY_PATH=~/.ssh/<private-key-filename>

local> sftp -i $SSH_PRIVATE_KEY_PATH bqmweudbrsprd1sftp.$SFTP_TENANT_NAME.$SFTP_USER_NAME@$SFTP_SERVER_NAME

sftp> mkdir 2025-04-01
sftp> cd 2025-04-01
sftp> put employees.json
sftp> put config.json

This triggers the DataBridge Engine Pipeline, which in this case takes a few seconds to run. After that, issue the following:

sftp> ls -l results # Repeat this until you see results.json and employees_out.csv
sftp> get results/results.json
sftp> get results/employees_out.csv

You can inspect the results.json file to see information about the pipeline and the employees_out.csv file contains a CSV version of the employees.json dataset provided above.

Conclusion

Congratulations, you have run your first DataBridge Engine Pipeline! The remaining sections of this documentation cover in more detail how to:

• Transform data (e.g. join datasets, map field names and values) before sending it to the destination
• Run per-row validations on datasets
• Configure destination adapters
• Secrets management