Trigger an Azure Machine Learning job, including from Azure DevOps or GitHub

Azure Machine Learning provides a powerful platform for designing and implementing data science solutions on Azure. One of the key features it offers is the ability to trigger and automate machine learning jobs, allowing you to run experiments, train models, and deploy solutions at scale. In this article, we will explore how to trigger an Azure Machine Learning job, including using Azure DevOps or GitHub.

Triggering with the Azure Machine Learning Python SDK

Triggering an Azure Machine Learning job can be done through the Azure Machine Learning Python SDK. Let’s start by looking at how to trigger a job using the Python SDK:

1. Connect to your Azure Machine Learning workspace by creating an instance of the Workspace class:

from azureml.core import Workspace

ws = Workspace.get(
subscription_id='',
resource_group='',
workspace_name=''
)

2. Create an experiment within your workspace:

from azureml.core import Experiment

experiment = Experiment(workspace=ws, name='')

3. Set up the script run configuration:

from azureml.core import ScriptRunConfig

script_run_config = ScriptRunConfig(
source_directory='',
script='',
arguments=['', ''],
environment={'': '', '': ''}
)

4. Submit the job using the experiment and script run configuration:

run = experiment.submit(config=script_run_config)

With these steps, you have triggered an Azure Machine Learning job using the Python SDK. The job will be queued for execution and can be monitored using the run object. You can also retrieve its status, logs, and other details as necessary.

Triggering with Azure DevOps or GitHub

Azure Machine Learning integrates seamlessly with Azure DevOps and GitHub, allowing you to trigger and manage machine learning jobs as part of your CI/CD pipelines.

Using Azure DevOps, you can create a pipeline that executes your data science experiments or machine learning models whenever changes are made to your code repository. You can use Azure Machine Learning tasks provided by Azure DevOps to trigger and monitor jobs, upload datasets, and register models. Here’s an example of YAML-based pipeline code that triggers an Azure Machine Learning job:

trigger:
branches:
include:
- main

steps:
- task: AzureMachineLearningRun@v1
inputs:
azureSubscription: ''
mlWorkspaceName: ''
experimentName: ''
script: ''
arguments: '--arg1 --arg2 '
environment: ''

By defining this pipeline and configuring the necessary inputs, you can trigger the Azure Machine Learning job whenever changes are pushed to the main branch of your code repository.

Similarly, when using GitHub, you can create a GitHub Action workflow that integrates with Azure Machine Learning. This workflow automatically triggers a machine learning job whenever code changes are committed to your GitHub repository. Here’s an example of a GitHub Action workflow file that triggers an Azure Machine Learning job:

name: Trigger Azure Machine Learning job
on:
push:
branches:
- main

- name: Trigger Azure Machine Learning job
uses: azure/actions/ml-remote-run@v1
with:
azure_credentials: ${{ secrets.AZURE_CREDENTIALS }}
workspace: ''
experiment: ''
script: ''
arguments: '--arg1 --arg2 '
environment: ''

In this GitHub Action workflow file, the ml-remote-run action is used to trigger the Azure Machine Learning job. The required inputs such as the workspace, experiment, script, arguments, and environment are provided to the action for job execution.

By leveraging the integration with Azure DevOps or GitHub, you can easily trigger and automate Azure Machine Learning jobs, making it effortless to incorporate data science experiments and machine learning workflows into your CI/CD processes.

With the flexibility and power of Azure Machine Learning, combined with seamless integration with Azure DevOps and GitHub, you can design and implement robust data science solutions on Azure. Triggering and automating machine learning jobs has never been easier, allowing you to scale your data science workflows and accelerate your time to insight.

Start exploring Azure Machine Learning and unleash the potential of your data science projects on Azure!