Replace Pyenv With a Nix Flake
Nix is a great tool for simplifying the pains of package management. Figuring out where Nix sits in your workflow, is not so simple. Nix’s flexibility and the wide variety of nix+python solutions make adoption overwhelming. Do not despair. In this post, we’ll walk through a minimal adoption of Nix by using a flake to replace Pyenv.
Why Use Nix Instead of Pyenv?
Why would you want this? Doesn’t Pyenv a great job of installing the different versions of python already? If your python use cases are simple and python only, this isn’t for you.
Others, unfortunately, have to interact with other languages and packages. A full stack might use python+nodejs, a data scientist might need python+java. Nix is a universal package manager. You can manage python and other languages. Never download another language version manager again.
Pyenv also can’t handle any C libraries you may need during installation.
You have to rely on homebrew, apt, or pacman for that.
This let’s you combine two tools into one.
I’m a solve once kind of guy. I like fixing leaks once and not having to figure it out again later. Nix’s reproducibility let’s me set it and forget it. Then I’m free to focus on what’s matters, delivering value.
Setting Up A Nix Flake
The Nix Wiki is a good reference when getting started with flakes. You may have to enable them depending on how Nix was installed. Right now ther are still considered experimental but there has been a larger amount of adoption.
To replace pyenv I have a single flake.nix that sits in my dotfiles.
Let’s start with a simplified version:
{
description = "My Python Flake Shells";
inputs = { nixpkgs.url = "nixpkgs/nixos-22.11"; };
outputs = { self, nixpkgs, ... }@inputs:
let
system = "x86_64-linux";
pkgs = import inputs.nixpkgs { inherit system; };
in {
devShells.x86_64-linux.python38 = pkgs.mkShell {
nativeBuildInputs = with pkgs;
let
devpython = pkgs.python38.withPackages
(packages: with packages; [ virtualenv pip setuptools wheel ]);
in [ devpython ];
};
};
}
In the inputs section I am specifying the version of nixpkgs to use.
To see the active versions of nixpkgs you can check the hydra jobsets.
This make switching between nixpkgs versions much less painful.
Now let’s talk about outputs.
I hardcore system to x86_64-linux because that’s what I use everywhere.
If you are using a M1 mac or some other system you will want to update that line.
The flake has an attribute called devShells.x86_64-linux.python38.
This is where python38 lives.
DevShells let you install packages temporarily for development.
Once you exit the shell the packages are no longer available.
After you create your flake.nix you can run:
`nix develop .#python38 -c $SHELL`
I use -c $SHELL because I’m using zsh and nix defaults to bash.
In your new shell you will have your new version of python:
$ python -V
Python 3.8.16
Adding More Versions
This pattern makes it easy to copy/paste different versions of python. What if the version of python you want doesn’t exist in the version of nixpkgs that you are using? Nix has the ability to pick and choose packages from different versions of nixpkgs.
{
description = "Python Flake Shells";
inputs = {
nixpkgs.url = "nixpkgs/nixos-22.11";
# This bad boy is the last one to support 3.6
nixpkgs-python36.url = "nixpkgs/nixos-21.05";
};
outputs = { self, nixpkgs, ... }@inputs:
let
system = "x86_64-linux";
pkgs = import inputs.nixpkgs { inherit system; };
pkgs-python36 = import inputs.nixpkgs-python36 { inherit system; };
in {
devShells.x86_64-linux.python36 = pkgs.mkShell {
nativeBuildInputs = with pkgs-python36;
let
devpython = python36.withPackages
(packages: with packages; [ virtualenv pip setuptools wheel ]);
in [ devpython ];
};
devShells.x86_64-linux.python38 = pkgs.mkShell {
nativeBuildInputs = with pkgs;
let
devpython = pkgs.python38.withPackages
(packages: with packages; [ virtualenv pip setuptools wheel ]);
in [ devpython ];
};
devShells.x86_64-linux.python39 = pkgs.mkShell {
nativeBuildInputs = with pkgs;
let
devpython = pkgs.python39.withPackages
(packages: with packages; [ virtualenv pip setuptools wheel ]);
in [ devpython ];
};
};
}
In our inputs section we’re importing an older version of nixpkgs that contains python3.6.
We’ve added a line to generate pkgs-python36 which gives us access to packages in 21.05.
Then our devShells.x86_64-linux.python36 can use those packages to create a shell that gives us python3.6.
You can test switching between the different dev shells with:
nix develop .#python36 -c $SHELL
nix develop .#python38 -c $SHELL
nix develop .#python39 -c $SHELL
Typing those commands out manually gets old fast. I use a function in my
~/.zshrc to do the heavy lifting with fzf:
lkjh () {
CHOICE=$(echo -ne "python39\npython38\npython36" | fzf)
[ -z "$CHOICE" ] && echo "no choice" && return 1
nix develop absolute/path/to/nix/flake/.#"$CHOICE" -c $SHELL
}
Once I pick the version of python I want, I use virtual environments and continue my normal workflow. The following aliases help quickly setup/destroy environments:
alias vin="virtualenv .venv && source .venv/bin/activate"
alias vout="deactivate && rm -rf .venv"
Adding Dev Libraries
This will work great until you have to deal with python packages that need require other libraries:
$ pip install --no-cache --no-binary :all: confluent-kafka==1.9.2
...
/tmp/nix-shell.xSjbfb/pip-install-q69u3269/confluent-kafka_3d7270483ea84b23bc350008d92a1abc/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
23 | #include <librdkafka/rdkafka.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/nix/store/dkw46jgi8i0bq64cag95v4ywz6g9bnga-gcc-wrapper-11.3.0/bin/gcc' failed with exit code 1
[end of output]
I’m using 1.9.2 to match the version of rdkafka in nixpkgs 22.11.
Right now pip doesn’t know where to find rdkafka’s files, which, it needs for compiling.
We can solve this by setting the appropriate environment variables.
In confluent-kafka’s case, we need to set C_INCLUDE_PATH as well as LIBRARY_PATH.
devShells.x86_64-linux.python38 = pkgs.mkShell {
nativeBuildInputs = with pkgs;
let
devpython = pkgs.python38.withPackages
(packages: with packages; [ virtualenv pip setuptools wheel ]);
in [ devpython ];
C_INCLUDE_PATH = "${pkgs.rdkafka}/include/";
LIBRARY_PATH = "${pkgs.rdkafka}/lib/";
};
Now when we try to install our library we get:
$ pip install --no-cache --no-binary :all: confluent-kafka==1.9.2
Collecting confluent-kafka==1.9.2
Downloading confluent-kafka-1.9.2.tar.gz (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.8/109.8 kB 1.8 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Skipping wheel build for confluent-kafka, due to binaries being disabled for it.
Installing collected packages: confluent-kafka
Running setup.py install for confluent-kafka ... done
Successfully installed confluent-kafka-1.9.2
[notice] A new release of pip available: 22.2.2 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
Hopefully you will simply be able to pull the wheels for your platform. But if you do have to go this route, you solve it once and it’s done.
Conclusion
This gives you an idea of where Nix could fit in your own development workflow. This approach is a very shallow adoption of using Nix. If it doesn’t work for you, uninstall it and forget it ever happened. Happy Hacking!