Connect Microsoft Azure Services to Vantage

Many Teradata customers have interest in connecting Teradata Vantage as a Service with Microsoft Azure First Party Services. This guide describes ‘high-level’ Vantage as a Service connectivity options with the Azure Services.

Although this approach has been implemented and tested internally by Teradata, it is offered on an as-is basis. Though, Azure connectivity options have been tested with Vantage as a Service, neither Microsoft nor Teradata provide validation given customer architecture and security requirements can vary greatly. Hence, guide should be viewed as a high-level reference only.

This guide includes content from both Microsoft Azure and Teradata product documentation.

Overview

This article describes how to connect various Azure Services to Vantage as a Service. You will need to first connect customer’s and Vantage as a Service virtual networks, then install Azure Services gateway and other components on Windows VM in the customer’s virtual network. Finally, depending on the service, create a private endpoint for the service to reach the gateway or customers virtual network to access Vantage as a Service.

Vantage as a Service on Azure supports multiple connectivity options depending on where the customer’s connections originate and the type and number of applications that need connectivity. 

This diagram displays the connectivity options.
 

• VNet Peering – VNet Peering is the preferred option when a Vantage instance in the Teradata VNet needs to initiate connections to multiple entities in the customer’s VNet (e.g., Azure Services offering).
• Azure Private Link – This is the preferred connectivity method when accessing Vantage from a customer VNet for installations that do not need connection initiation from the Teradata VNet into the customer’s VNet for any application access. Private Link also simplifies IP address planning, has higher network speeds, and keeps traffic within the Azure network. One Private Link connection is included in the price of a Vantage on Azure subscription.
• Azure ExpressRoute (not covered) – This is the preferred connectivity option when accessing Vantage from on-premises.
• VPN (not covered) – This is an alternate connectivity option when accessing Vantage from on-premises.

Note: We will cover VNet Peering and Private Link to access Vantage as a Service (SQLE) instance. Other connectivity and components (i.e. ViewPoint, QueryGrid Manager, etc) are not covered.

About Azure Services

The Azure cloud platform is more than 200 products and cloud services designed to help you bring new solutions to life—to solve today’s challenges and create the future. Build, run, and manage applications across multiple clouds, on-premises, and at the edge, with the tools and frameworks of your choice.

Many of the Azure Services support Teradata Vantage using a connector developed by Microsoft.  Azure Data Factory, Logic Apps, Power Apps, Power Automate and Power BI Service just to name a few. These Azure Services uses a Microsoft gateway component to access Teradata Vantage in the cloud and on-premise. 

For more information see documentation.

About Teradata Vantage as a Service

Teradata Vantage™ is our flagship analytic platform offering, which evolved from our industry-leading Teradata® Database. Until references in content are updated to reflect this change, the term Teradata Database is synonymous with Teradata Vantage.

Teradata Vantage is a data analytics platform for performing advanced analytics in the cloud or on-premises. With Vantage you can integrate analytic tools, languages, and engines to get insights from all your data.

With Vantage delivered as a service, Teradata manages the performance, security, availability, and operations of the Vantage platform as described in this service description.
Vantage on Azure

Deploying Vantage on Azure establishes a Vantage environment in a Teradata-owned Azure subscription. Customers can often subscribe to a Vantage environment in the same Azure region as their data.

With Vantage on Azure, customers can access the same software capabilities provided in an on-premises Vantage system but in an Azure environment.

Teradata provisions, configures, and provides customer access to an Azure environment in a supported region.

For more information see documentation. 

Prerequisites

You are expected to be familiar with the Azure networking concepts (VNet Peering, Private Link Services and Private Endpoints), Azure services and Vantage with the following accounts and systems.

• An Azure account
• A Teradata Vantage as a Service instance (version 17.0) or later

Procedure

Once you have met the prerequisites, choose virtual network connectivity option:

Virtual Network connectivity

1. Option 1: VNet Peering connectivity (recommended) 
2. Option 2: Private Link connectivity

Azure Services connectivity

1. Create Windows (VM) virtual machine
2. Install an Azure Service gateway and/or Teradata client software
3. Create Private Endpoint or requirements for an Azure Service
   1.  Azure Data Factory
   2. Azure Data Factory Managed Virtual Network
   3. Azure Blob Storage
   4. Azure Batch Service

Virtual Network connectivity

Customer and Vantage as a Service virtual networks need to be connected to access Vantage database. Network connectivity is something that’s done prior to handing over the system to a customer. Customers would decide what network connectivity they want during architecture discussions. After the system is provisioned, Cloud Ops team would set up the network connectivity based on the approved architecture of the customer.

If you do not have a virtual network, follow the steps in create virtual network (steps 1-6).

Ensure customer virtual network IP address range and subnet does not overlap with Vantage as a Service

Option 1 - VNet Peering connectivity (recommended)

Azure Virtual Network (VNet) Peering enables you to seamlessly connect Azure virtual networks. Once peered, the VNets appear as one, for connectivity purposes. The traffic between virtual machines in the peered virtual networks is routed through the Microsoft backbone infrastructure, much like traffic is routed between virtual machines in the same VNet, through private IP addresses only. No public internet is involved. You can peer VNets across Azure regions, too.

This diagram displays the VNet Peering connectivity

Figure 1: VNet Peering connectivity

The following steps are:

Step 1 – Acquire and add ‘shared’ service principal and assign network contributor role

Logon to the Azure portal with Azure Active Directory owner access. 

Browse to your VNet, and then click Access control (IAM) and assign Network Contributor role to the shared service principal provided by the cloud team (i.e. ICAzure_Peering_SP_Prod). 

Click Save.

Figure 2: Virtual network Access control (IAM) role assignment  

Customer can remove service principal after automation script completes VNet Peering.

Step 2 – Provide Tenant/Directory ID and VNet Resource ID to cloud team

Customer can find Tenant/Directory ID by clicking on Azure Active Directory --> Properties 

Figure 3: Azure Active Directory properties 

VNet Resource ID can found by clicking on Virtual Network --> Properties 

Figure 4: Virtual network properties   

Step 3 – Cloud team runs automation script to complete VNet Peering

Once automation script completes, click on Peerings under your Virtual Network and check status for Connected

Figure 5: Virtual network Peerings connections   

Step 4 – Use Vantage as a Service provided IP to access Vantage

Once virtual networks are peered, you can start using Vantage as a Service provided IP addresses.

Step 5 – Remove service principal from Azure Active Directory tenant

Browse to your VNet, click Access control (IAM) and then Role assignments.

Choose the shared service principal provided by the cloud team (i.e. ICAzure_Peering_SP_Prod).

Click Remove

Next, go to Azure Services connectivity section.

Option 2: Private Link connectivity

Azure Private Link service is the reference of a service or resource that is powered by Azure Private Link. In this case, the resource is Vantage as a Service running behind a Azure Standard Load Balancer can be enabled for Private Link access so that consumers can access it privately from their own VNets. Customers can create a private endpoint inside their VNet and map it to this service effectively bringing the Vantage as a Service resource into their VNet.

This diagram displays the Private Link connectivity

Similar to VNet Peering, Private Link service deployment would be performed by cloud team before system is handed over to a customer. This includes deploying Private Link and Standard ‘internal’ Load Balancer services in Vantage as a service virtual network. When completed, customer can then create a private endpoint in their VNet and map it to this service to the Vantage as a service resource.

Teradata recommends separate Private Link Service, Load Balancer and Private Endpoint for each Vantage as a Service offerings: SQLE, Viewpoint VM and QueryGrid

The following steps are:
Step 1 – Acquire Private Link service alias for the Vantage as a Service resource

Contact cloud team to acquire private link service alias for Vantage as a Service resource to complete private endpoint configuration.

Step 2 – Create Private Endpoint to connect to Private Link service 
 

Logon to the Azure portal and follow steps in create private endpoint. 

In the Basics tab of Create a private endpoint, enter the Subscription, Resource group, Name and Region customers virtual network to access Vantage as a Service.
 

Figure 7: Create a private endpoint Basics tab   

Click Next: Resource

In the Resource tab, for Connection Method choose Connect to an Azure resource by resource ID or alias and enter alias.

Figure 8: Create a private endpoint Resource tab   

Click Next: Configuration

In Configuration tab, enter Virtual network and Subnet

Figure 9: Create a private endpoint Configuration tab   

Click the Review + create tab or Review + create button at the bottom of the screen.

Step 3 – Receive approval from Private Link Service Vantage resource owner (cloud team)

Go to resources once deployment completes and check private endpoint approval status from cloud team.

Figure 10: Private endpoint resource overview   

Step 4 – Use Private Endpoint network interface IP to access Vantage

Once approved, click on private endpoint network interface to find IP address to connect to Vantage as a Service resource.

Figure 11: Private endpoint network interface resource overview   

Next, go to Azure Services connectivity section.

Azure Services connectivity

Some Azure Services like Power BI Service and Azure Data Factor requires a gateway component to access Vantage as a Service. Or to support data integration within your virtual network, you will need to deploy a Windows VM, install the appropriate gateway component and/or any Teradata client software.

Azure Data Factory Managed Virtual Network feature does not require a virtual network or a Self-hosted Integration Runtime gateway component. See Azure Data Factory Managed Virtual Network section for more information and skip steps 1-3 below.

This diagram displays the Azure Services and Vantage as a Service connectivity

Step 1 – Create Windows (VM) virtual machine

Deploy a Windows VM in customer’s virtual network VNet Peered or Private Endpoint connection to the Private Link Vantage as a Service resource to install the appropriate gateway component and/or any Teradata client software.

Logon to the Azure portal and follow steps in create a Windows virtual machine.

Ensure in Networking tab to choose virtual network peered or private link to Vantage as a Service

Next, Connect to virtual machine using RDP and perform the following: 

• Set Enhanced Security from On to Off
• Open Server Manager. Select Local Server in the left pane
• Click on IE Enhanced Security Configuration setting On to bring up UI and set to Off

• Download and install Chrome

Next, install an Azure Service gateway.

Step 2 – Install Azure Services gateway and/or Teradata client software 

Logon to your Windows VM machine and determine which gateway to install or both and/or any Teradata client software requirements.

• On-Premise Data Gateway – for Power BI Service, Logic Apps, Power Apps, Power Automate. 

Note gateway is required for cloud and on-premise Vantage access.

Prerequisite: 

• Download and install the latest Microsoft .NET Framework Runtime.
• Download and install .NET Data Provider for Teradata or see 3rd bullet point
• Self-hosted Integration Runtime – for Azure Data Factory. 

Note Self-hosted Integration Runtime is required for using a private IP and for on-premise Vantage access.

Prerequisite: 

• Create a Azure Data Factory as a Public Endpoint. 
• This ensures gateway can install, access and register with the service. Next section, we will set Data Factory service network access to private and create private endpoint for service to access Vantage as a Service using private IP.
• Logon to Azure portal from your Windows VM and open Azure Data Factory UI to install gateway.

Both gateways can co-exist on same Windows VM machine.

• Teradata Tools and Utilities – for Teradata client software
  • Download Teradata Tools and Utilities – Windows 
  • Install components for your use case (i.e. BTEQ, Teradata Parallel Transporter Base, Teradata Access Module for Azure, .NET Data Provider for Teradata, etc…)

Logoff Azure portal and Windows VM once installation is completed.
Next, create private endpoint for an Azure Service

Step 3 – Create Private Endpoint for an Azure Service

You will need to create a Private Endpoint, Subnet and NSG rules for some Azure Services in customer’s virtual network for services to access VM running gateway, services or other products to access Vantage as a Service.

Azure Data Factory 

Set Data Factory Network access to Private endpoint

Logon to the Azure portal and open your Data Factory instance and click on Networking on the left pane.

Ensure Network access is set to Private endpoint.
 

Figure 13: Azure Data Factory Network access settings   

Click Save

Create Private Endpoint for Data Factory

In the portal, go to your service instance and click Networking > Private endpoint connections and click + Private endpoint. Or on the upper-left side of the portal select Create a resource and search for Private endpoint in the search box and click Create.

In the Basic tab, enter customer Subscription, existing or new Resource group, Endpoint ‘Instance’ Name and Region.

Figure 14: Azure Data Factory Private endpoint Basic properties   

Click Next: Resource

In the Resource tab, choose Connection method: Connect to an Azure resource in my directory, Customer Subscription, Resource type: Microsoft.DataFactory/factories, your Data Factory Resource and Target sub-resource: dataFactory.

Figure 15: Azure Data Factory Private endpoint Resource properties   

Click Next: Configuration

In Configuration tab, choose the Virtual network that has been peered with Vantage as a Service and where the Windows VM is running the gateway component. Should be one of the same. Choose default Subnet.

Figure 16: Azure Data Factory Private endpoint Configuration properties

Click Next: Tags

Click Review+  create 

Click Create after Validation passed

Azure Data Factory Managed Virtual Network

When you create an Azure Integration Runtime (IR) within Azure Data Factory Managed Virtual Network (VNET), the integration runtime will be provisioned with the managed Virtual Network and will leverage private endpoints to securely connect to supported data stores.

Creating an Azure IR within managed Virtual Network ensures that data integration process is isolated and secure.

Currently, the managed Virtual Network is only supported in the same region as Azure Data Factory region. 
Prerequisite: Private Link connection to Vantage as a Service resource needs to exist.

This diagram displays the Azure Data Factory Managed Virtual Network and Vantage as a Service connectivity

Set Data Factory Network access to Private endpoint

Logon to the Azure portal and open your Data Factory instance and click on Networking on the left pane.

Ensure Network access is set to Private endpoint.
 

Figure 13: Azure Data Factory Network access settings   

Click Save

Task 1 – Acquire Private Link Service Resource ID from cloud team for your Vantage instance

You will need Resource ID when configuring your Managed Private Endpoint.

Task 2 – Create Azure Integration Runtime Managed Virtual Network

In the portal, open your Data Factory Studio UI and click on Manage icon. Next, click on Integration runtimes and click +New to create Azure Integration Runtime for a Managed Virtual Network.

Follow the Integration runtime setup and choose Azure, Self-Hosted>Azure click Continue.

Next, enter a Name and set Virtual network configuration to Enable and click Create.

Once created, click on your integration runtime to ensure it is in running status.

Task 3 – Create a Managed Private endpoint in Azure Data Factory

Next, click on Managed private endpoints icon and click +New.

In New managed private endpoint dialog choose Private Link Service and click Continue.

In the New managed private endpoint (Private Link Service) dialog enter a Name, enter manually Resource ID provided to you by cloud team (step 1) and enter a Fully qualified domain name which will be used when creating Linked services connections to Vantage. Click Create will send private endpoint request to owner of the data source for approval.

Task 4 – Receive approval from Private Link Service Vantage resource owner (cloud team)

Once the owner approves the connection, the private link is established.

Task 5 – Use fully qualified domain name to access Vantage

Open Linked services and create a connection to Vantage with the domain name you entered when creating your private endpoint.

Azure Blob Storage

Set Blob Storage access to Selected networks

Logon to the Azure portal and open your Blob Storage account and click on Networking on the left pane.

Ensure Network access is set to Selected networks.

Figure 17: Azure Blob Storage Network access settings   

Click Save

Create Private Endpoint for Blob Storage

In the portal, go to your service instance and click Networking > Private endpoint connections and click + Private endpoint. Or on the upper-left side of the portal select Create a resource and search for Private endpoint in the search box and click Create.

In the Basic tab, enter customer Subscription, existing or new Resource group, Endpoint ‘Instance’ Name and Region.

Figure 18: Azure Blob Storage Private endpoint Basic properties

Click Next: Resource

In the Resource tab, choose Target sub-resource: from the dropdown list (i.e. blob, file, table, web, queue)

Figure 19: Azure Blob Storage Private endpoint Resource properties

Click Next: Configuration

In Configuration tab, choose the Virtual network that has been peered with Vantage as a Service and where the Windows VM is running if required for access to your storage account. Should be one of the same. Choose default Subnet.

Figure 20: Azure Blob Storage Private endpoint Configuration properties

Click Next: Tags

Click Review+  create

Click Create after Validation passed

Azure Batch Service

You can provision the pool in a subnet of a virtual network. Before to adding a pool, the subnet must allow inbound and outbound communication from the Batch service to the compute nodes and other resources. For pools in the Virtual Machine configuration, see Network security groups: Batch default for NSG rules requirement.

• Inbound TCP traffic on ports 29876 and 29877 from Batch service IP addresses that correspond to the BatchNodeManagement service tag.
•  Inbound TCP traffic on port 22 (Linux nodes) or port 3389 (Windows nodes) to permit remote access. For certain types of multi-instance tasks on Linux (such as MPI), you will need to also allow SSH port 22 traffic for IPs in the subnet containing the Batch compute nodes. This may be blocked per subnet-level NSG rules.
• Outbound traffic on any port to the virtual network. This may be amended per subnet-level NSG rules.
• Outbound traffic on any port to the Internet. This may be amended per subnet-level NSG rules.

Create subnet to provision the pool in a virtual network.

Logon to the Azure portal and open your Virtual network and click on Subnets on the left pane.
Click + Subnet and enter a subnet Name (i.e. BatchSubnet) and choose Network security group (NSG) with required rules above.

Figure 21: Azure Virtual network add subnet settings  

Click Save.

Step 4 – Access Vantage

Depending on virtual network and services connectivity approach:

• For VNet Peering connectivivty use provided Vantage as a Service IP addresses.
• For Private Link Services connectivity use Private Endpoint network interface IP address.
• For Services with Managed VNet support (i.e. Data Factory) use fully qualified domain name.