Mike's corner of the web.

Adding git (or hg, or svn) dependencies in setup.py (Python)

Wednesday 29 May 2013 21:02

Update: the behaviour of pip has changed, meaning that the option --process-dependency-links is required when running pip install.

You can specify dependencies for your Python project in setup.py by referencing packages on the Python Package Index (PyPI). But what if you want to depend on your own package that you don't want to make public? Using dependency_links, you can reference a package's source repository directly.

For instance, mayo is a public package on PyPI, so I can reference it directly:

setup(
    install_requires=[
        "mayo>=0.2.1,<0.3"
    ],
    # Skipping other arguments to setup for brevity
)

But suppose that mayo is a private package that I don't want to share. Using the dependency_links argument, I can reference the package by its source repository. The only way I could get this working with git was to use an explict SSH git URL, which requires a small transformation from the SSH URLs that GitHub or BitBucket provide. For instance, if GitHub lists the SSH URL as:

git@github.com:mwilliamson/mayo.git

then we need to explictly set the URL as being SSH, which means adding ssh:// at the front, and replacing the colon after github.com with a forward slash. Finally, we need to indicate the URL is a git URL by adding git+ to the front. This gives a URL like:

git+ssh://git@github.com/mwilliamson/mayo.git

To use a specific commit, add an at symbol followed by a commit identifier. For instance, if we wanted to use version 0.2.1, which has the tag 0.2.1 in git:

git+ssh://git@github.com/mwilliamson/mayo.git@0.2.1

Then, we can use the URL in setup.py like so:

setup(
    install_requires=[
        "mayo==0.2.1"
    ],
    dependency_links=[
        "git+ssh://git@github.com/mwilliamson/mayo.git@0.2.1#egg=mayo-0.2.1"
    ]
    # Skipping other arguments to setup for brevity
)

Note that we depend on a specific version of the package, and that we use the URL fragment (the bit after #) to indicate both the package name and version.

Topics: Python