ADF – DevOps CI/CD Pipeline Automation

Azure Data Factory is Azure’s cloud ETL service for scale-out serverless data integration and data transformation. Azure Data Factory utilizes Azure Resource Manager templates to store the configuration of your various ADF entities (pipelines, datasets, data flows, and so on).

The Promotion of data factory from one environment to another environment can be automated using Azure DevOps CI/CD Pipeline. Continuous Integration and Deployment for Azure Data Factory allows you to keep the state of your Data Factories consistent.

The below flowchart depicts the lifecycle of ADF- DevOps CI/CD pipeline.

• The Azure DevOps Git Repository can be connected to one of the ADF environments (DEV) using the Git configuration in the ADF.
• Create a feature branch in ADF Dev environment and design a pipeline to copy files or tables from one storage account/database to another.

• Raise a Pull request to merge it with Main Branch and Publish the pipeline in Master branch.

• ARM (Azure Resource Manager) Template will be generated and will be stored in DevOps git repository.
• Now Create a Release pipeline in Azure DevOps and add main branch as default branch.
• Create a job in the pipeline for UAT & PROD. Select “ARM Template deployment” and provide Name, Resource Manager Connection, Subscriptions, Resource group & Location.
• Select  the generated ARMTemplateForFactory.json in Template field & ARMTemplateParameteForFactory.json in Template parameter field.

• In Override Template parameters, add the below parameters and save the pipeline.

-factoryName “<UAT-ADFname” -<sourcestorageaccount>_connectionString “<sourceconnectionstring>” -<destinationstorageaccount> _connectionString “<destinationconnectionstring>”

• Execute the Release pipeline. Now the ADF pipelines, configurations & parameters from DEV environment will be deployed in UAT & PROD ADF environments using Devops Pipeline.

Using the above Azure Devops CI – CD pipeline automation technique, ADF pipelines & setup can be promoted from lower to higher environments seamlessly. This minimizes the dependency on manual changes in numerous environments and will enable a stable architecture. Please feel free to reach Cittabase for more information. Visit our blogs for more topics on DevOps & Power BI.

Flowchart credits : Microsoft