This blog will help you if you are looking for managed big data platform by leveraging the Azure cloud. Azure Databricks platform help you to spin up a Spark cluster without any management and along with that its provide reporting as well.
We have done with the spark cluster but the biggest issue is how can we secure it by parsing the sensitive data in the codebase. Azur has a Key Vault service which basically stores the sensitive data with versioning.
Let’s integrate the Azure Key vault with Databricks. First, we need to create a Key Vault.
Post creating the key vault we need to define the Secret Scope for the key vault in Azure databricks. For that, you have used the below URL
https://<databricks-instance>#secrets/createScope. Post successful authentication you will get the databricks instance like below https://xxxxxxxxxxxx.x.azuredatabricks.net/#secrets/createScope
DNS Name and Resource ID will be available in the Key Vault Portal.
Now we have defined the Secret Scope and created the keys as well. Let’s access the same from databricks
import mysql.connectorUsername = dbutils.secrets.get(scope = "scope_name",key = "user")
Password = dbutils.secrets.get(scope = "scope_name",key = "password")mydb = mysql.connector.connect(
)mycursor = mydb.cursor()
mycursor.execute("SHOW DATABASES")for x in mycursor:
In the above code, we are using dbutils.secrets.get() function to get the key values from the key vault.