Python pip requirements.txt lock file

Overview

Opinions vary on how one should make use of lock files, depending on whether the project is the main application, or the project is actually a library that is meant to be consumed by an application or another library.

Lock files are unquestionably useful if you build any application. However, if you publish a library or CLI to a packing publisher like Pypi, lock files are never published. Meaning your users and you might use different versions of dependencies, and that is fine if you internally still use lock files.

Personally, I see library projects as no different to any other project, and all projects that build no matter where they get deployed should include a lock file for the sake of  reproducible builds and team member collaboration.

If you are familiar with other languages that use lock file such as Node.js, you might have asked at least once asked if Python can provide the same functionality.
The short answer is python has no concept of lock files, equally it can be argued python has no package dependency files at all and that's why there are many options outside the core python team like setup.py, Pipfile, and the most common requirements.txt as a pattern for Pip.

A long answer to the lock file question that you might have learned is that you must painstakingly write out the packages by hand and your packages whole dependency tree, and theirs, and so on. Locking the versions using == of all of these.
You might also be familiar with pip freeze to try make that easier, but everyone has encountered the  complications of skipping the hand crafted requirements after running pip freeze.

The Future

Pip are hard at work creating a new dependency resolver and will soon provide a python native solution to this age old problem. They are working with the various existing implementations (that use the current pip solution anyway) which has been represented in the solution iterations I've looked at. Pip is looking a lot like Pipenv.

If you have a free minute, run pip check on some of your more complex projects and provide the Pip team with your results for research. And consider a generous donation to support their efforts for the whole community and proprietary companies that rely on their work they give away for free.

Solution

While Pip are hard at developing their solution, I have written a very simple script that automates that painstaking process of locking versions of a whole project dependency tree.

Show me the code!

#!/usr/bin/env bash

CWD=$(pwd)
TMP_DIR=$1

if [[ $EUID -eq 0 ]]; then
   echo -e "${RED}x${NC} This script must not be run as root" 
   exit 1
fi
if [ -z $(which python3) ]; then
  echo "python3 not found"
  exit 1
fi
if [ -z $(which pip) ]; then
  echo "python3 pip not found"
  exit 1
fi
if [[ ! -f requirements.txt ]]; then
  echo "requirements.txt not found"
  exit 1
fi

if [[ -z "${TMP_DIR}" ]]; then
  TMP_DIR=/tmp/piplock.$(date +'%s%N')
fi
if [[ ! -z "$(which deactivate)" ]]; then
  deactivate
fi

mkdir -p ${TMP_DIR}
cd ${TMP_DIR}
python3 -m pip install -U pip
python3 -m pip install -U virtualenv
python3 -m venv .venv
source .venv/bin/activate
pip install -q -U --no-cache-dir --isolated --no-warn-conflict -r ${CWD}/requirements.txt
check=$(pip check --no-cache-dir --isolated)
check_exit=$?
if [[ $check_exit -ne 0 ]]; then
  echo ${check}
  exit 1
fi
LOCK="$(pip freeze)"
deactivate
cd ${CWD}
rm -rf ${TMP_DIR}
echo ${LOCK}

Example - locking requests dependency.

It is as easy, get the script;

wget -q https://gist.githubusercontent.com/chrisdlangton/905a14e7a42118e0d6b5a45c8ce1d3a0/raw/f0916b0b99187a9e839146fbd4e3d5bc26e5d97a/piplock.sh -O piplock.sh
chmod a+x piplock.sh
mv piplock.sh /usr/local/bin/piplock

Add some simple requirements;

mkdir -p test-piplock
cd test-piplock
echo -e "retry\nrequests" > requirements.txt

Running piplock produces;

certifi==2020.6.20
chardet==3.0.4
decorator==4.4.2
idna==2.10
py==1.9.0
requests==2.24.0
retry==0.9.2
urllib3==1.25.10

Conclusion

This flow allows you to maintain reproducible builds and consistent dependencies for development workflow, and at the same time enables developers to catch any potentially breaking changes for your library consumers—and all of this, while also keeping all of the developers on your team happy.