Over the weekend I was working on a big data platform & during the POC I found that in big data platform libraries are very important. If you used different version things will not work as expected.

This blog will help you to integrate apache spark with Azure blob storage as a data lake.

For this activity we need followings
- Azure Account(Blob Storage)
- Linux Instance
- Spark-3.1.2

Let’s start with Azure Blob Storage. For that, you need an Azure account and create a storage account like below

Ankur

DevOps Engineer with 10+ years of experience in the IT Industry. In-depth experience in building highly complex, scalable, secure and distributed systems.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store