Setting up a Pip Installable Python 3 Git Repo

Recently, upon starting a new project, I was sketching out the repositories I would need to begin. These included several implementation repositories that can be packaged and deployed. However I realized that I may have common functions that live across the different implementation repositories. To combat this – I decided to create a few utility repositories to keep my implementation repositories DRY (Don’t Repeat Yourself). I wanted an easy way to be able to simply pip install my shared utility code without having to push my utility repositories to PyPi.

After doing research, I found that Bitbucket/GitHub can act as a PyPi server! Pip has functionality to install python packages directly from a git repo.

For Bitbucket users:

pip install git+ssh://git@bitbucket.org/<user_name>/<repo_name>.git@<tag_number | branch_name>

When setting up continuous integration for the shared repo, the git tag command can be used to freeze the code at a specific version and make that version pip installable. On the other hand, specifying a branch name like master would always provide the version of the code at the HEAD of the master branch. Using tag number is more stable than using branch name because the branch will constantly change its HEAD as developers commit to that branch.

git tag -a v0.0.1 -m "v0.0.1, first release to prod"

Remember to push the tags to origin!

git push origin --tags

Setting the tags in the git repo is not enough to make the repo pip installable. The repository needs to be buildable. Here is a sample of my setup.py :

#!/usr/bin/env python

from setuptools import setup, find_packages

setup(name='shared',
      version='0.0.1',
      description='This package has shared components.',
      author='Jane Doe',
      author_email='user@email.com',
      packages=find_packages(exclude=["*.tests", "*.tests.*", "tests.*", "tests"]),
      license='LICENSE.txt',
    )

To make sure that the repo is building properly, build the repo locally in a virtual environment.

pip install .

Once the package is installed, check to see that it is available to use in the environment:

pip freeze

To make sure that the package has been installed successfully, change directory so that the current working directory is not the root folder of the package, then import and test one of the functions in the package that was just installed. Hopefully it works!

You can add the package from a repo to requirements.txt file like so:

git+ssh://git@bitbucket.org/<user_name>/<repo_name>.git@<tag_number | branch_name>

Voila! A python package that is installable without using PyPi or a dedicated Python Package server.

If you run into permission issues, it can be because the ssh client of the host machine doesn’t have access to that repository. If you are using Bitbucket you may have a limited number of users that you can invite to the repository. Fortunately, it is possible to have an unlimited number of read-only access users. These can be registered under the Access Control section of the bitbucket repository settings.

Enjoy keeping your python code DRY!

GCP as a CDP

Create your own Customer Data Platform leveraging the power of Google Cloud A Customer Data Platform (CDP) is an important utility for analysts and especially

Read More...