Seamless Collaboration: Azure DevOps CI for Enhanced dbt Cloud Workflows
In this blog post, we’ll explore the seamless integration of an existing dbt cloud project with Azure DevOps CI workflows. Discover how Azure-enabled DevOps, or any code repository, plays a crucial role in safeguarding your code. We’ll highlight the process and benefits of this integration, aiming to streamline your development process and protect your codebase.
Optimizing Data Projects with DBT Cloud:
DBT Cloud revolves around facilitating efficient and scalable data transformation and analytics processes. It provides a centralized platform for data engineers and analysts to collaboratively develop, test, and deploy data transformation pipelines. By leveraging DBT Cloud, organizations can streamline their data workflows, improve collaboration among team members, and ensure the reliability and consistency of their data transformations. Additionally, DBT Cloud offers features such as version control, scheduling, and monitoring, empowering teams to manage their data projects effectively and derive actionable insights from their data with confidence.
Azure DevOps: Enhancing Software Deployment:
Azure DevOps is to provide a comprehensive set of tools and services for software development teams to plan, collaborate, build, and deploy applications more efficiently. It offers a centralized platform where teams can manage their entire software development lifecycle, from planning and coding to testing, deployment, and monitoring. Azure DevOps facilitates seamless collaboration among team members, streamlines development processes through automation, and enables continuous integration and delivery (CI/CD) pipelines for faster and more reliable software releases.
Connect with Anblicks for dbt development and deployment
Benefits of Integrating dbt Cloud with Azure DevOps
-
- Native Integration: dbt Cloud seamlessly integrates with Azure DevOps, facilitating collaboration and enhancing productivity. This integration simplifies the setup of dbt projects, strengthens security measures, and unlocks new features within dbt Cloud.
- Enhanced Security: Users authenticate via OAuth with Azure DevOps, ensuring secure access to the dbt Cloud IDE. The integration enforces user permissions, adding an additional layer of security to the development environment.
- Continuous Integration: Azure DevOps integration enables continuous integration, automating the execution of dbt jobs. This streamlines the development process, ensuring that changes to dbt code are regularly tested and deployed in a controlled manner.
- Centralized Code Repository: Azure DevOps serves as a centralized code repository for dbt projects, facilitating version control and collaboration. This centralized approach improves code organization, enables efficient tracking of changes, and fosters collaboration among team members.
Step-by-Step Process to Connect dbt Cloud with Azure Devops
STEP – 1: Authenticate Dbt Cloud App with Azure DevOps
If we want to utilize the dbt Cloud IDE or dbt Cloud CLI for collaborative work on your team’s Azure DevOps dbt repository, it’s necessary to link your dbt Cloud profile to Azure DevOps. This linkage adds an additional layer of authentication.
Connect Azure Devops and dbt cloud profile :
To connect your dbt Cloud profile to Azure DevOps using OAuth, follow these steps:
-
- Click on the gear icon located at the top right corner and choose Profile settings.
- Select Linked Accounts.
- Click on Link next to Azure DevOps.
-
- You’ll be redirected to Azure DevOps. Sign in to your account.
- When prompted with the permission request screen from the Azure DevOps App, click Accept.
After being redirected back to dbt Cloud, your profile should be successfully linked. You are now prepared to commence development in dbt Cloud!
Additionally, once the Azure AD app is integrated into dbt Cloud, an account administrator must connect a service user via OAuth. This service user will be utilized to execute headless actions in dbt Cloud, including deployment runs and CI.
Connect a service account user :
-
- An admin must sign into the service user’s Azure DevOps account.
- Click “Link Azure Service User” in dbt Cloud.
- The admin will be directed to Azure DevOps and must accept the Azure AD app’s permissions.
- Once the admin completes the setup process, they will be redirected to dbt Cloud. From there, they will proceed to connect with the service user.
Connecting an Azure Service User
STEP – 2: Set up an Active Directory application and add it to dbt Cloud:
For seamless integration between Azure DevOps and dbt Cloud, create a dedicated Azure Active Directory (Azure AD) app, separate from any app used for Single Sign-On (SSO), as recommended
A. Update the permissions to newly created Azure app.
B. Modify the Redirect URI
C. Configure Connection between Azure DevOps and new app.
D. Integrate Azure Active Directory app to DBT Cloud
A. Registering an application in Azure Active Directory
To connect your dbt Cloud profile to Azure DevOps using OAuth, follow these steps:
-
- Navigate to the Azure portal, then proceed to Azure Active Directory, and select “App registrations” from the menu.
- Click “New registration” and provide a name for your app (like “dbt Labs Azure DevOps App”).
- Select “Accounts in any organizational directory (Any Azure AD directory – Multitenant)” from the options provided.
- Add a redirect URI: https://YOUR_ACCESS_URL/complete/azure_active_directory.
- Click “Register”.
B. Update Permissions for newly created Azure app
To grant your new app access to Azure DevOps, follow these steps:
-
- Go to Azure left navigation panel and select “API permissions”.
- Remove the permission labeled “Microsoft Graph / User Read”.
- Click “Add a permission”.
- Choose “Azure DevOps”.
- Select the “user_impersonation” permission.
C. Modify the redirect URI
To add another redirect URI to your Azure AD application for service user authentication:
-
- Navigate to your Azure AD application.
- Select the link next to “Redirect URIs”.
- Click “Add URI” and add the URI: https://example.com/complete/azure_active_directory_service_user with the appropriate Access URL for your region and plan).
- Click “Save”.
D. Configure Connection between Azure Devops and new app
If your Azure DevOps account is already connected to Active Directory, proceed to “Connecting a service user”. But if you’re setting up for the first time, follow these steps to connect Azure DevOps to the Active Directory App you created:
-
- Go to your Azure DevOps account and select “Organization settings” at the bottom left.
- Navigate to “Azure Active Directory”.
- Click “Connect directory”.
- Select the directory you want to connect.
- Click “Connect”.
E. Integrate Azure Active Directory App to dbt Cloud
To provide dbt Cloud with information about your Azure AD app and Azure DevOps connection, follow these steps:
-
- Go to account settings within dbt Cloud.
- Select “Integrations”.
- Scroll down to the Azure DevOps section.
- Complete the form as follows:
Adding an Active Directory App to dbt Cloud
Conclusion
Linking DBT Cloud with Azure DevOps streamlines the transfer of ETL tasks to the cloud, ensuring a smooth transition. At Anblicks, our expertise in Azure cloud services spans across industries, offering everything from consulting to implementation, all aimed at enhancing your business processes and operations. Reach out to explore our Azure-focused cloud data analytics solutions and how they can benefit your organization.
Transform your business with Anblicks – AWS Advanced Consulting Partner.
Kiran Kotha, is a Senior Software Engineer at Anblicks, he specializes in Modern Apps development, leveraging cloud-based technologies to drive innovative solutions.
