Seamless CI/CD Pipelines with GitHub Actions on GCP: Your Tools for Effective MLOps | by Paul Iusztin | Jun, 2023


CI/CD Pipeline Using GitHub Actions (free)

The GitHub Actions YAML files are under the .github/workflows directory.

Firstly, let me explain the main components you have to know about a GitHub Actions file πŸ‘‡

Using the “on -> push -> branches:” section, you specify which branch to listen to for events. In this case, the GitHub Action is triggered when new code is committed to the “main” branch.

In the “env: “section, you can declare the environment variables you need inside the script.

In the “jobs -> ci_cd -> steps:” section, you will declare the CI/CD pipeline steps, which will run sequentially.

In the “jobs -> ci_cd -> runs-on:” section, you specify the image of the VM you want the steps to run on.

Now, let’s take a look at some actual GitHub Action files πŸ”₯

ML Pipeline GitHub Actions YAML file

The action will be triggered when new code is committed to the “main” branch, except for the web app directories and the YAML and Markdown files.

We added environment variables that contain information about the GCP project and VM.

As for the CI/CD steps, we mainly do 2 things:

  1. configure the credentials & authenticate to GCP,
  2. connect with SSH on the given GCP VM and run a command that: goes to the code directory, pulls the latest changes, builds the Python packages, and deploys them to the PyPi registry. Now Airflow will use the new Python packages the next time it runs.

Basically, it does what you would have done manually, but now, everything is nicely automated using GitHub Actions.

Note that you don’t have to remember or know how to write a GitHub Actions file from scratch, as you can find already written templates for most of the use cases. For example, here is the google-github-actions/ssh-compute [11] repository we used to write the YAML file below.

You will find similar templates for almost any use case you have in mind.

Web App GitHub Actions YAML file

The Web App actions file is 90% the same as the one used for the ML pipeline, except for the following:

  • we ignore the ML pipeline files;
  • we run a docker command that builds and runs the web app.

But where does the “${{ vars… }}” weird syntax come from? I will explain in just a sec, but what you have to know now is the following:

  • β€œ${{ vars.<name> }}”:variables set inside GitHub;
  • β€œ${{ secrets.<name> }}”: secrets set inside GitHub. Once a secret is set, you can’t see it anymore (the variables you can);
  • ${{ env.<name> }}”: environment variables set in the “env:” section.

Important Observation

The YAML file above doesn’t contain the CI part, only the CD one.

To follow good practices for a robust CI pipeline, you should run an action that builds the Docker images and pushes them to a Docker registry.

Afterward, you would SSH to a testing environment and run your test suit. As a final step, you would SSH to the production VM, pull the images and run them.

The series got too long, and we wanted to keep it simple, but the good news is that you learned all the necessary tools and principles to do what we described above.



Source link

Leave a Comment