CI/CD for Deploying Machine Learning Models on AWS

by Mayank Chamoli

The following is an example of building a simple MLOps pipeline using AWS resources. The pipeline is built with data scientists in mind. A large portion of data scientist’s work revolves around understanding data, building models to solve problems and refining models based on new or updated parameters. It is important for data scientists to be able to complete these primary tasks without worrying about redeploying the model each time it is altered.

version: 0.1

phases: 
  build: 
    commands: 
      - | 
         echo push artifacts to s3 
         zip -r notebook.zip . 
         aws s3 cp notebook.zip s3://s3-for-codepipeline/notebook/notebook.zip

artifacts: 
  files: 
    - "**/*"

Once the build phase in the CodePipeline is complete, the deploy phase is triggered. The deploy phase instructs the CloudFormation to delete any existing SageMaker instances before creating a new SageMaker instance using the latest code from the CodeCommit repository.

The stages in the CodePipeline and the CloudFormation template used in this case are shown below.

CodePipeline
|-Source (CodeCommit)
|-Build (CodeBuild)
|-Deploy (CloudFormation – StackName: sagemaker-test)
    |-delete_test_sagemaker
    |-deploy_test_sagemaker

sagemaker-template-test.yml

Description: "Forecast POC Notebook Instance"
Resources:
  ForecastPOCNotebookInstance:
    Type: "AWS::SageMaker::NotebookInstance"
    Properties:
      InstanceType: "ml.t2.xlarge"
      LifecycleConfigName: !Sub "${AWS::StackName}-lifecycle"
      RoleArn: !GetAtt ExecutionRole.Arn
  ExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                - "sagemaker.amazonaws.com"
            Action:
                "sts:AssumeRole"
      Path: "/"
      Policies:
        -
          PolicyName: "root"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action: "*"
                Resource: "*"
Outputs:
  BasicNotebookInstanceId:
    Value: !Ref ForecastPOCNotebookInstance

The CloudFormation uses the lifecycle script (below) in SageMaker while creating the notebook instance. The script copies the latest code from the S3 bucket to the notebook instance and unzips it. After granting the necessary permissions to the directory, the Jupyter notebook is executed. When the Jupyter notebook runs, it executes all of the steps in it, then deploys the model. Finally, the model can be tested for accuracy.

sagemaker-test-lifecycle:

#!/bin/bash

set -e

su ec2-user

mkdir -p /home/ec2-user/SageMaker/notebook-test-deploy/
aws s3 cp s3://s3-for-codepipeline/notebook/notebook.zip /home/ec2-user/SageMaker/notebook.zip
unzip /home/ec2-user/SageMaker/notebook.zip -d /home/ec2-user/SageMaker/notebook-test-deploy/

sudo chown -R ec2-user:ec2-user /home/ec2-user/SageMaker/notebook-test-deploy

source /home/ec2-user/anaconda3/bin/activate python3
 
nohup jupyter nbconvert --execute /home/ec2-user/SageMaker/notebook-test-deploy/Breast\ Cancer\ Prediction.ipynb --ExecutePreprocessor.kernel_name=python3 --ExecutePreprocessor.timeout=7200 &

next steps

The CodePipeline process can be further expanded to deploy to production by adding new stages. Once the model testing is complete in the test environment, and the results are as expected, the approval can be given for deployment to the production environment. An example of how CodePipeline would look with additional stages is shown below.

CodePipeline
|-Source (CodeCommit)
|-Build (CodeBuild)
|-Deploy (CloudFormation – StackName: sagemaker-test)
    |-delete_test_sagemaker
    |-deploy_test_sagemaker
|-Approve
|-Deploy (CloudFormation – StackName: sagemaker-prod)
    |-delete_prod_sagemaker
    |-deploy_prod_sagemaker

external references

ML Ops: Machine Learning as an Engineering Discipline

AWS re:Invent 2018: CI/CD for Your Machine Learning Pipeline with Amazon SageMaker (DVC303) (video)

CI/CD on AWS (video)