Configuration section: drop_zone_files

Description

The structure of delivered Source Data Files is described in the Drop Zone Folder section. The configuration for advanced pre-processing of Source Data Files (e.g. decryption, file pre-processing, field type coercion) is provided through the drop_zone_files section of the configuration.

Specification

file_name (string, required)

The name of the file this configuration is applied to, e.g. employees.xlsx

data_tables (array, required)

A "Data Table" can be either a single sheet in an Excel file or a single JSON or a CSV file. Therefore, an Excel file can contain one or more Data Tables but a JSON or a CSV file always contains a single Data Table.

The configuration for respective source files shall be as follows

• CSV and JSON files: A single entry in data_tables, has neither sheet_index nor sheet_name
• Excel files: One or more entries in data_tables, all entries have either sheet_index or sheet_name.

At least one data_tables entry is required; an empty array is invalid and will be rejected.

data_frame_name (string, required)

The name of the Data Frame which can be referenced in later steps of the pipeline

header_row (integer, optional)

The row number which contains the header (first row is indicated with 1). If not specified then 1 is assumed.

data_start_row (integer, optional)

The first row which contains data. If not specified then 2 is assumed.

sheet_name (string, optional)

The name of the Excel sheet this configuration shall be applied to. Only applicable for Excel files. Cannot be specified with sheet_index.

sheet_index (integer, optional)

The index of the Excel sheet this configuration shall be applied to (first sheet is indicated with 1). Only applicable for Excel files. Cannot be specified with sheet_name.

field_configurations (array, optional)

Per-field type declarations that override the pipeline's column type inference for the listed fields. Useful when inference would silently corrupt data — for example, a CSV column with values 001, 002, 003 is inferred as integer by default and arrives downstream as 1, 2, 3. Declaring the field as "str" preserves the leading zeros end-to-end.

field_configurations is opt-in per column: declared fields are coerced strictly to the declared type; fields not listed continue to use today's inference. A value that cannot be coerced to the declared type produces a FIELD_TYPE_MISMATCH error rather than a silent fallback.

field (string, required) - the name of the column in the source file.

configuration (object, required) - the coercion configuration to apply to this field. Currently contains a single key:

• type (string, required) - the type to coerce the field to. One of str, int, float, bool. To enforce null constraints, use a not_null entry under data_validations.

pgp_configuration (optional)

A single object with:

pgp_key_secret_name (string, required) - The name of the secret that contains the PGP private key.

pgp_key_passphrase_secret_name (string, optional) - The name of the secret that contains the password for the PGP private key.

excel_password_configuration (object, optional)

password_secret_name (string, required) - The name of the secret that contains the password for an encrypted Excel file.

Example

{
  "version": 1,
  "drop_zone_files": [
    {
      "file_name": "employees-1.xlsx",
      "data_tables": [
        {
          "data_frame_name": "employees_1",
          "sheet_index": 1
        }
      ]
    },
    {
      "file_name": "employees-2.xlsx",
      "excel_password_configuration": {"password_secret_name": "excel-pass"},
      "data_tables": [
        {
          "data_frame_name": "employees_2",
          "sheet_name": "Workers"
        }
      ]
    },
    {
      "file_name": "employees-3.xlsx.pgp",
      "pgp_configuration": {"pgp_key_secret_name": "pgp-key", "pgp_key_passphrase_secret_name": "pgp-pass"},
      "data_tables": [
        {
          "data_frame_name": "employees_3",
          "sheet_name": "Workers"
        }
      ]
    },
    {
      "file_name": "employees-4.csv",
      "data_tables": [
        {
          "data_frame_name": "employees_4",
          "field_configurations": [
            {"field": "emp_id", "configuration": {"type": "str"}}
          ]
        }
      ]
    },
    {
      "file_name": "employees-5.xlsx",
      "data_tables": [
        {
          "data_frame_name": "employees_5_a",
          "field_configurations": [],
          "header_row": 1,
          "data_start_row": 2,
          "transpose": false,
          "sheet_index": 1
        },
        {
          "data_frame_name": "employees_5_b",
          "field_configurations": [
            {"field": "employee_id", "configuration": {"type": "str"}},
            {"field": "years_of_service", "configuration": {"type": "int"}},
            {"field": "rate", "configuration": {"type": "float"}}
          ],
          "header_row": 2,
          "data_start_row": 3,
          "transpose": false,
          "sheet_index": 2
        }
      ]
    }
  ]
}