Handling Multiple Versions of a Package in Python

Photo by Chris Ried on Unsplash

Recently we faced a problem in our project for handling multiple versions of a package. Here is the requirement

An integrated script that will load a package(scikit-learn) with a specific version to use some inbuilt functions based on the version.

To achieve this, we need to install multiple versions of scikit-learn and have access to them whenever required

If you’ve used pip, you know the problem, it dosen’t support multiple version. If you try to install another version of package, it will replace the previous version instead of keeping both. So we needed an alternate way of doing this

If we just used the source code of multiple versions in different folders and try to import, we will have clash in dependencies versions

One simple way of doing this is installing on demand,

import pip
import sklearn
if sklearn.__version__==0.23.2:
    pass
else:
    pip.main(['install', 'scikit-learn==0.23.2'])
#or
#if you wanna use the module right away 
#import subprocess,sys
#subprocess.check_call([sys.executable, "-m", "pip", "install", "scikit-learn==0.23.2"])

You can see the problems in this, If two scripts/workers are running in parallel, they might clash with the different versions and it will be a mess

Server might also be blocked from downloading anything from internet if it’s inside a virtual network, so we need a solution that should not connect to pypi every time for a package

While researching about this, we found out about “ — target” option with pip install. This will install a package inside a folder along with dependencies

pip install scikit-leatn==0.23.0 --target=/absolute/path/to/a/folder

So we wrote a script that will install all the packages versions inside a specific folder

import sys
import subprocess
def install(pkg,path):
    return subprocess.check_call([sys.executable, "-m", "pip", "install", pkg,"--target={}".format(path)])
pkgs = [
'scikit-learn==0.24.2',
'scikit-learn==0.24.1',
'scikit-learn==0.24.0',
'scikit-learn==0.23.2',
'scikit-learn==0.23.1',
'scikit-learn==0.23.0'
]
base = 'somepath/'
for i in pkgs:
    [name,ver] = i.split('==')
    path = base+name+'/'+ver
    print(path)
    install(i, path)

Just change the packages inside the pkgs and base folder for creating the folders of packages along with their dependencies

This will create a structure of folders like below

somepath
 - scikit-learn
  -0.24.2
  -0.24.1
  -0.23.2
.....

Now we need to import them along with dependencies on demand

In python you can import a library from another directory by adding that directory to PYTHONPATH variable,

but this will not work as we will be having multiple version of same package and this might create some conflicts, so we need to do the same but without adding the path to global variable PYTHONPATH

Here we can use sys.path.insert for the same,this will temporarily add the path for import .Use the following at start of the script

import os
import sys
sys.path.insert(1,os.path.abspath('somepath/scikit-learn/0.24.2'))
import sklearn
print(sklearn.__version__)
#this will be 0.24.2 

NOTE : You should use absolute path for this to work

Now you can import any version along with dependencies without a clash

You can see we gave ‘1’ inside sys.path.insert .This will increase the priority of the path . To understand this lets take an example

lets say sklearn 0.24.2 uses numpy 1.21.1 , so your folder of 0.24.2 contains the dependency of numpy 1.21.1 .

If you install numpy==1.19.1 globally with pip install , after adding the sys.path.append , if you import numpy and see the version it will be 1.21.1

Numpy inside the path given in sys.path.insert will be given priority while importing. If it’s not available there, then the global path is scanned for the package

Edit : I’ve create pypi package to simplify the process — https://pypi.org/project/pypi-multi-versions/

Give a clap if this helped you. Thanks for reading