Change Python version in Vertex AI

Vertex AI is a bare-bones analytics environment in Google Cloud. As simple tasks as changing the Python version requires multiple steps.

Creating a new Python environment in Vertex AI

Here is how to change Python version in Vertex AI by creating a new Python environment:

Launch a new terminal in Vertex AI:

Run the lines one by one. This way you can react to command line prompts and possible warnings:

#Deactivate the existing Python environment
conda deactivate
#Create new Python environment using Python version 3.11
conda create -n python311 python=3.11 -y
#Active the new environment
conda activate python311
#This enables pip to install to conda environment
conda install pip
#Verify that you use conda pip to install
which -a pip
#Install Python packages from requirements.txt file - Make sure you are in the same folder
pip install --user -r requirements.txt
#Install ipykernel if it was not requirements.txt
pip install ipykernel
#Install the kernel so it becomes available in notebooks
ipython kernel install --name python311 --display-name "Python 3.11"  --user

Other useful commands for Jupyter kernels in Vertex AI

Show what kernels are installed:

ls ~/.local/share/jupyter/kernels

Change the display name later:

sudo nano  ~/.local/share/jupyter/kernels/python311/kernel.json

Open Vertex AI notebook with new Python version

After this you can use the new Python version and related packages in two ways. Either launch a new notebook or switch the kernel to an existing one.

The new Python kernel should appear in the launcher available for new notebooks.

Verify the Python and package versions in the notebooks.

You can see the Python version 3.11 on top right corner of the notebook. There you can switch the kernel for existing notebooks. Refresh the browser page if the new Python version is not visible.

Problem with Vertex AI Python environments

Vertex AI notebook platform offers only one Python environment by default.

What if you need Python 3.11 in one project instead the default 3.7?

You need to create a new environment!

When to create a new Python environment in Vertex AI?

Preferably each analytics project would have their own Python virtual environment. An environment has specific Python and package versions.

Kernel is different concept than Python virtual environment. But for basic Vertex AI usage can use consider to have one kernel and environment per one project. In this regard, it would be smart to name the kernel or environment by project rather than Python version.

In the most straightforward approach each project has their own git repository. Exact Python package version can be defined in the requirements.txt file.

Why not to have one environment for all projects?

At some point you start requiring different package versions depending on the task you work on. This happens when you project start to have plenty of code.

Keeping projects small enough is key to success in data science.