DP-3014-A: Build machine learning solutions using Azure Databricks

Azure Databricks is a fully managed, cloud-based data analytics platform, which empowers developers to accelerate AI and innovation by simplifying the process of building enterprise-grade data applications. Built as a joint effort by Microsoft and the team that started Apache Spark, Azure Databricks provides data science, engineering, and analytical teams with a single platform for big data processing and machine learning. In this course, you’ll learn how to use Azure Databricks to train and deploy machine learning models.

Explore Azure Databricks
Azure Databricks is a cloud service that provides a scalable platform for data analytics using Apache Spark.

• Get started with Azure Databricks
• Identify Azure Databricks workloads
• Understand key concepts
• Data governance using Unity Catalog and Microsoft Purview
• Exercise – Explore Azure Databricks
• Module assessment

Use Apache Spark in Azure Databricks
Azure Databricks is built on Apache Spark and enables data engineers and analysts to run Spark jobs to transform, analyze and visualize data at scale.

• Get to know Spark
• Create a Spark cluster
• Use Spark in notebooks
• Use Spark to work with data files
• Visualize data
• Exercise – Use Spark in Azure Databricks

Train a machine learning model in Azure Databricks
Machine learning involves using data to train a predictive model. Azure Databricks support multiple commonly used machine learning frameworks that you can use to train models.

• Understand principles of machine learning
• Machine learning in Azure Databricks
• Prepare data for machine learning
• Train a machine learning model
• Evaluate a machine learning model
• Exercise – Train a machine learning model in Azure Databricks

Use MLflow in Azure Databricks
MLflow is an open source platform for managing the machine learning lifecycle that is natively supported in Azure Databricks.

• Capabilities of MLflow
• Run experiments with MLflow
• Register and serve models with MLflow
• Exercise – Use MLflow in Azure Databricks

Tune hyperparameters in Azure Databricks
Tuning hyperparameters is an essential part of machine learning. In Azure Databricks, you can use the Optune library to optimize hyperparameters automatically.

• Optimize hyperparameters with Optuna
• Review trials
• Scale hyperparameter optimization
• Exercise – Optimize hyperparameters for machine learning in Azure Databricks

Use AutoML in Azure Databricks
AutoML in Azure Databricks simplifies the process of building an effective machine learning model for your data.

• What is AutoML?
• Use AutoML in the Azure Databricks user interface
• Use code to run an AutoML experiment
• Exercise – Use AutoML in Azure Databricks

Train deep learning models in Azure Databricks
Deep learning uses neural networks to train highly effective machine learning models for complex forecasting, computer vision, natural language processing, and other AI workloads.

• Understand deep learning concepts
• Train models with PyTorch
• Distribute PyTorch training with TorchDistributor
• Exercise – Train deep learning models on Azure Databricks

Manage machine learning in production with Azure Databricks
Machine learning enables data-driven decision-making and automation, but deploying models into production for real-time insights is challenging. Azure Databricks simplifies this process by providing a unified platform for building, training, and deploying machine learning models at scale, fostering collaboration between data scientists and engineers.

• Automate your data transformations
• Explore model development
• Explore model deployment strategies
• Explore model versioning and lifecycle management
• Exercise – Manage a machine learning model

N/A

This learning path assumes that you have experience of using Python to explore data and train machine learning models with common open source frameworks, like Scikit-Learn, PyTorch, and TensorFlow.