Pipeline results

Results from a DataBridge Engine Pipeline consist of:

• A Results File (results/results.json) which contains status and error message from the pipeline execution
• Optionally, data files that are generated by the file_export destination adapter.

Results Folder

All files generated by the DataBridge Engine Pipeline are left in the results/ subfolder from the Drop Zone Folder. This includes both the results.json file and any files generated by the File Export destination adapter

Results File

For every invocation of the DataBridge Engine Pipeline, a results.json file is generated in the results/ subfolder from the Drop Zone Folder. The file has the following format:

{
  "started_at": string, ISO 8601 formatted Date and time with offset,
  "completed_at": string, ISO 8601 formatted Date and time with offset,
  "is_complete": boolean,
  "is_success": boolean,
  "transformation_results": [
    {
      "success": boolean,
      "transformation_name": string, Name of the transformation,
      "error_detail": {
        "slug": string: An identifier for the error,
        "msg": string, A detailed description of what went wrong,
        "metadata": null or dictionary, Unstructured metadata with further error details
      }
    },
    ...
  ],
  "source_connector_results": [
    {
      "adapter": string, Name of adapter,
      "started_at": string, ISO 8601 formatted Date and time with offset,
      "completed_at": string, ISO 8601 formatted Date and time with offset,
      "result": {
        "success": boolean,
        "metadata": null or dictionary, Unstructured metadata with further error details
        "error": {
            "slug": string: An identifier for the error,
            "msg": string, A detailed description of what went wrong,
            "metadata": null or dictionary, Unstructured metadata with further error details
        }
      }
    }
  ],
  "destination_adapter_results": [
    {
      "adapter": string, Name of adapter,
      "started_at": string, ISO 8601 formatted Date and time with offset,
      "completed_at": string, ISO 8601 formatted Date and time with offset,
      "result": {
        "success": boolean,
        "metadata": null or dictionary, Unstructured metadata with further error details
        "error": {
            "slug": string: An identifier for the error,
            "msg": string, A detailed description of what went wrong,
            "metadata": null or dictionary, Unstructured metadata with further error details
        }
      }
    }
  ],
  "errors": [
    {
        "slug": string: An identifier for the error,
        "msg": string, A detailed description of what went wrong,
        "metadata": null or dictionary, Unstructured metadata with further error details
    },
    ...
  ]
}

The result folder and file are generated at the moment the pipeline starts. During the execution of the pipeline, the is_complete variable is set to false but once is_complete has switched to true, that signals that the conclusion of the pipeline execution. The results.json file is update continuously during the execution of the pipeline and can be downloaded multiple times during that time to monitor the status of the pipeline.

Exported files

The DataBridge Engine provides the option to deliver processed Data Frames in a data file. The following formats are available:

• CSV
• JSON

Refer to the File Export Destiation Adapter