Yokadi 1.0.1, Python packaging woes
written on Saturday, December 12, 2015
(Sorry, this is a rant)
So I released Yokadi 1.0.0 last week, yay!
and then shortly after that I had to release Yokadi 1.0.1, meh...
I had to release this quick bug fix release because I messed up declaration of dependencies: pip install yokadi would not install anything :/
This is one of the few aspects of Python I struggle with: packaging. It's a mess.
There is distutils in Python itself, which does half of the job and has a painful-to-extend API. There is also setuptools which is slightly more powerful and is the recommended solution, but since it is backward compatible with distutils its API inherits the painfulness of distutils API. It's also not shipped in Python itself despite having been around for a few years now, so you have to either assume your users have it installed (likely, but ugly) or bootstrap it by bundling a part of it in your archive, which will download a matching version of setuptools. This is how setuptools documentation says one should proceed:
Your users might not have setuptools installed on their machines, or even if they do, it might not be the right version. Fixing this is easy; just download ez_setup.py, and put it in the same directory as your setup.py script. (Be sure to add it to your revision control system, too.) Then add these two lines to the very top of your setup script, before the script imports anything from setuptools:
import ez_setup
ez_setup.use_setuptools()
That’s it. The ez_setup module will automatically download a matching version of setuptools from PyPI, if it isn’t present on the target system. Whenever you install an updated version of setuptools, you should also update your projects' ez_setup.py files, so that a matching version gets installed on the target machine(s). Whenever you install an updated version of setuptools, you should also update your projects' ez_setup.py files, so that a matching version gets installed on the target machine(s).
Am I the only one who finds this hackish?
Back to my issue, what caused pip to fail to get dependencies? It was caused by me being too obsessed about warnings: when I built the archive using ./setup.py sdist I noticed a warning about an unsupported keyword argument passed to setup(): install_requires. This argument lists package requirements, but is a setuptools extension, which is why distutils does not recognize it. I foolishly thought that pip would get my requirements from the requirements.txt file, so I removed the argument. Always try to avoid repeating yourself, right? Except it was not a good idea: when installing a package pip only looks at the install_requires argument. There is a subtle difference between what should go in requirements.txt and what should go in install_requires, which I think is very easy to get wrong...
Another surprising aspect of source packaging is that you have to explicitly specify files to include in the source archives. If a file does not match any wildcard from the MANIFEST.in file at the root of your source tree, it does not go into the archive. I found out that although my 1.0.0 archive included everything needed to install and run Yokadi (except for a working dependency list...), it was missing some files which are present in the source repository. Building the archive with git archive does not sound like a good option because then the archive would lack the PKG-INFO file generated by ./setup.py sdist.
To make sure those problems do not come up again, I updated Yokadi release script to test pip install works and wrote a script called diffinst to compare the content of a git repository and a source archive, modulo some files to ignore, and report any differences (actually, I should include a call to diffinst in the release script)
I need to spend time to get more familiar with setuptools and use it in Yokadi and my other Python-based projects, but I hope this aspect gets more love in future versions of Python.