Have you had a task to fix a supposedly simple thing in a project, a code or configuration change, only to spend most of the time to set up the environment so you can do the fix in the first place?
The setup you just did for this now clutter your setup, and conflicts with other projects? In order to get back to your old working environment, you have to do some re-installs of software packages?
One approach is to use containers from Docker images to package and run the specific tools. However, that can sometimes be cumbersome to set up and combine with tools that you already have.
The approach here does not use containers, and it can just be a matter of a one-line command to create a new shell environment. The extra software needed is in place as long as the shell is active only.
First, I will show an example, then look at how to set it up and get started.
For a project, we want to install Node.js 14 + Yarn + git + ripgrep, plus cowsay for good measure. First, just check what is on the machine before. After that, install the required tools and verify that they are available in our new shell. When we are done, we exit the shell and get the environment back to its old state.
bash-4.4# # First let us create a work directory
bash-4.4# mkdir work && cd work
bash-4.4# pwd
/work
bash-4.4# # We want to have Node.js 14.x, yarn, git, ripgrep and cowsay
bash-4.4# which node yarn git rg cowsay
which: no node in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no yarn in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
/root/.nix-profile/bin/git
which: no rg in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no cowsay in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
bash-4.4#
bash-4.4# # None of the tools are installed
bash-4.4# # Now we will set up the tools in our ad hoc environment
bash-4.4# nix-shell -p nodejs-14_x yarn git ripgrep cowsay
these 25 paths will be fetched (71.04 MiB download, 327.01 MiB unpacked):
/nix/store/1l6jgfsgjmsrbrciz8r714dnkyngvpkb-zlib-1.2.11-dev
/nix/store/1v0pmb9f8jmn126nvbs2k2varzqfqczq-icu4c-69.1
/nix/store/2dd4n8cca16pa8yic1gh079hxb2x0ji4-yarn-1.22.10
/nix/store/3k69hbxg04sdxlgi1236ddggs346sxf3-stdenv-linux
/nix/store/4wf2bnddgwg17m465rg66j67gmcwy83q-bash-interactive-4.4-p23-info
/nix/store/4xkhd51k7y6g9vi2h9kfvcxd67k3v3f1-gawk-5.1.0
/nix/store/5ddb4j8z84p6sjphr0kh6cbq5jd12ncs-binutils-2.35.1
/nix/store/5wvf6qy2pjijd93n4dn6vvbn45ij9fdr-linux-headers-5.12
/nix/store/7al99yyrc2ix9c5idq2kf8zgksscaf9z-bash-interactive-4.4-p23-doc
/nix/store/88ghxafjpqp5sqpd75r51qqg4q5d95ss-gcc-wrapper-10.3.0
/nix/store/bjyyiipkwxgn8awrpp5aj3yp75l043i9-nodejs-14.17.6
/nix/store/cr23xzsbbvjgxaf1mchn2drhh6yvsq0q-cowsay-3.03+dfsg2
/nix/store/dr5yvz4dv2rq6mqr8pq5ix2g9wn4h5wp-icu4c-69.1-dev
/nix/store/k6sslkn2f764mgnd2a6q0wfwlnfwpybi-ripgrep-12.1.1
/nix/store/llywalnma9vv74z0xivwc87kxs6crk9b-binutils-wrapper-2.35.1
/nix/store/mbc7y2k96nhgazi7w7hls8y4pbma1y82-diffutils-3.7
/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev
/nix/store/q141hd8jl7in5223jmf7kmx9h517km4p-glibc-2.32-54-dev
/nix/store/sa90ixnafl08k8d9wwmzhr7br2dyjxp2-expand-response-params
/nix/store/sf6w5cyyksw7kacvhhw0aw0m1x74r524-ed-1.17
/nix/store/shcm9y3r9xlx6pksg3kkkpi56vm5cgqg-libuv-1.41.0
/nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0
/nix/store/wfzdk9vxayfnw7fqy05s7mmypg5a8lyr-gnumake-4.3
/nix/store/xfc0ffwz6s87mnz9x2k2qj6ykwxgwjxk-patchelf-0.12
/nix/store/ywkvpy8va67s7z8i1m32mklar9mii8qd-patch-2.7.6
copying path '/nix/store/7al99yyrc2ix9c5idq2kf8zgksscaf9z-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/4wf2bnddgwg17m465rg66j67gmcwy83q-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/5ddb4j8z84p6sjphr0kh6cbq5jd12ncs-binutils-2.35.1' from 'https://cache.nixos.org'...
copying path '/nix/store/cr23xzsbbvjgxaf1mchn2drhh6yvsq0q-cowsay-3.03+dfsg2' from 'https://cache.nixos.org'...
copying path '/nix/store/mbc7y2k96nhgazi7w7hls8y4pbma1y82-diffutils-3.7' from 'https://cache.nixos.org'...
copying path '/nix/store/sf6w5cyyksw7kacvhhw0aw0m1x74r524-ed-1.17' from 'https://cache.nixos.org'...
copying path '/nix/store/sa90ixnafl08k8d9wwmzhr7br2dyjxp2-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/4xkhd51k7y6g9vi2h9kfvcxd67k3v3f1-gawk-5.1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/wfzdk9vxayfnw7fqy05s7mmypg5a8lyr-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/1v0pmb9f8jmn126nvbs2k2varzqfqczq-icu4c-69.1' from 'https://cache.nixos.org'...
copying path '/nix/store/shcm9y3r9xlx6pksg3kkkpi56vm5cgqg-libuv-1.41.0' from 'https://cache.nixos.org'...
copying path '/nix/store/dr5yvz4dv2rq6mqr8pq5ix2g9wn4h5wp-icu4c-69.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/5wvf6qy2pjijd93n4dn6vvbn45ij9fdr-linux-headers-5.12' from 'https://cache.nixos.org'...
copying path '/nix/store/ywkvpy8va67s7z8i1m32mklar9mii8qd-patch-2.7.6' from 'https://cache.nixos.org'...
copying path '/nix/store/q141hd8jl7in5223jmf7kmx9h517km4p-glibc-2.32-54-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/xfc0ffwz6s87mnz9x2k2qj6ykwxgwjxk-patchelf-0.12' from 'https://cache.nixos.org'...
copying path '/nix/store/llywalnma9vv74z0xivwc87kxs6crk9b-binutils-wrapper-2.35.1' from 'https://cache.nixos.org'...
copying path '/nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/k6sslkn2f764mgnd2a6q0wfwlnfwpybi-ripgrep-12.1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/88ghxafjpqp5sqpd75r51qqg4q5d95ss-gcc-wrapper-10.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/1l6jgfsgjmsrbrciz8r714dnkyngvpkb-zlib-1.2.11-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/3k69hbxg04sdxlgi1236ddggs346sxf3-stdenv-linux' from 'https://cache.nixos.org'...
copying path '/nix/store/bjyyiipkwxgn8awrpp5aj3yp75l043i9-nodejs-14.17.6' from 'https://cache.nixos.org'...
copying path '/nix/store/2dd4n8cca16pa8yic1gh079hxb2x0ji4-yarn-1.22.10' from 'https://cache.nixos.org'...
[nix-shell:/work]#
[nix-shell:/work]# which node yarn git rg cowsay
/nix/store/bjyyiipkwxgn8awrpp5aj3yp75l043i9-nodejs-14.17.6/bin/node
/nix/store/2dd4n8cca16pa8yic1gh079hxb2x0ji4-yarn-1.22.10/bin/yarn
/nix/store/cpx9hzqcxci468qjwx1yk9k8rwmhlqxl-git-2.31.1/bin/git
/nix/store/k6sslkn2f764mgnd2a6q0wfwlnfwpybi-ripgrep-12.1.1/bin/rg
/nix/store/cr23xzsbbvjgxaf1mchn2drhh6yvsq0q-cowsay-3.03+dfsg2/bin/cowsay
[nix-shell:/work]# git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /work/.git/
[nix-shell:/work]# cowsay Now we have the tools for our environment in place
_______________________________
/ Now we have the tools for our \
\ environment in place /
-------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[nix-shell:/work]# exit
exit
bash-4.4# which node yarn git rg cowsay
which: no node in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no yarn in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
/root/.nix-profile/bin/git
which: no rg in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
which: no cowsay in (/root/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin)
bash-4.4#
bash-4.4# # We have left the shell we created with nix-shell, and the tools are not there anymore
bash-4.4# # One more time...
bash-4.4#
bash-4.4# nix-shell -p nodejs-14_x yarn git ripgrep cowsay
[nix-shell:/work]# cowsay All packages cached locally, so quicker this time
_________________________________________
/ All packages cached locally, so quicker \
\ this time /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[nix-shell:/work]# exit
exit
bash-4.4# So what is this nix-shell? It is a command-line tool which is part of a group of tools, under the name Nix. Nix is a few different things:
- A package manager
- A domain specific language for building, packaging and deploying software
- A Linux distribution using the Nix package manager, which is called NixOS.
Nix is very much about creating reproducible and declarative deployments and system configurations, with tooling to support these tasks. One outcome of this focus is to have immutable configurations - you are not destroying any particular resources, just adding to what is already there.
You do not have to run NixOS to take advantage of other features of Nix. In fact, the tools work on other Linux distributions, on macOS or in WSL 2 (Windows Subsystem for Linux) on Windows machines. You just install Nix to get a couple of command-line tools.
The Nix project has been around for a while, about 19 years. However, a large portion of that time, its usage was quite small. It has only grown more significantly in the last few years.
A good starting point is the NixOS website. The site covers Nix and a tool and you do not need to set up NixOS to get started with it. It outlines some of the use cases for Nix, of which ad hoc environments are just one of them.
You find the installation instructions for Nix here. This also includes running it as a container via Docker, which is a good way to get an idea of what it is all about, before actually installing the software itself. The example above was running in an instance of this docker image, in fact. The recommendation is to do a multi-user installation if you use Linux or macOS, while use a single-user installation under Windows WSL2.
It is essentially running a shell command on your local computer to download and run an installation script:
For macOS:
sh <(curl -L https://nixos.org/nix/install)For Windows (WSL2), single-user installation:
sh <(curl -L https://nixos.org/nix/install) --no-daemonFor Linux, single user installation is same as above. For multi-user installation:
sh <(curl -L https://nixos.org/nix/install) --daemonIf you want to run the docker image instead and have Docker desktop installed, you can run:
docker run -it nixos/nixor run it to expose a selected directory (workdir) on your local computer:
docker run -it -v $(pwd)/workdir:/workdir nixos/nixThe first time you run the command it will take a bit of time, because it will download the and setting up the specified packages from NixPkgs, which have over 80000 packages at this point in time. If you re-run the command again later, it will be faster, since the packages have been cached locally.
Tweak and persist the package information
For the simplest use cases, just specifying the package labels works fine. However, tweak it a bit more, depending on which packages to install and any additional settings that may be needed. For this purpose, there is also the Nix domain specific language, which is geared towards repeatable and/or reproducible setups.
Let us create a directory work, and in that directory, create a file mysetup.nix. The content will look like this:
let
getpkgs = import <nixpkgs>;
pkgs = getpkgs {};
in
pkgs.mkShell {
buildInputs = [
pkgs.git
pkgs.nodejs-14_x
pkgs.yarn
pkgs.ripgrep
pkgs.cowsay
];
shellHook = ''
cowsay "here is your shell environment"
'';
} What this piece of code will do is to retrieve information about all available packages in the latest update of NixPkgs. Using the retrieved data, it will create a shell environment (mkShell), where there packages we want to install/build are inputs. As a bonus here, we also add an execution of cowsay once the environment is ready.
Now we can set up the environment using the command
nix-shell mysetup.nixIn the work directory, we should see something like this:
bash-4.4# nix-shell mysetup.nix
________________________________
< here is your shell environment >
--------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
[nix-shell:/work]# This makes for a repeatable setup, but is not reproducible. This is because the latest versions of the different packages might be different if the latest versions of these packages are updated. To get the same versions each time, we need to be specific about which release of NixPkgs we are using. Nix has multiple ways to get a specific release of NixPkgs. Since NixPkgs information is available through a repository in GitHub, we can fetch a specific release and git commit from that repository. In that case, we will always get the same releases for the packages, as well as the same dependencies for all these packages. We cannot choose specific versions for each package, since there is an infinite number of combinations with these packages, including their dependencies in that case. Repeatable and reproducible environments are of higher importance here. We will get the specific versions of the packages and their dependencies that are in that specific release of NixPkgs.
Let us create a new file, this time called shell.nix. The content is adjusted slightly, so that we fetch the package data from the NixPkgs repository in GitHub, with a specific tag or branch and commit that is associated with that. One way to get the specific commit is to use the git ls-remote command:
git ls-remote --tags https://github.com/nixos/nixpkgs/ 21.11
506445d88e183bce80e47fc612c710eb592045ed refs/tags/21.11Instead of a general reference to “
let
nixpkgs = (fetchGit {
name = "nixos-release-21.11";
url = "https://github.com/nixos/nixpkgs/";
ref = "refs/tags/21.11";
rev = "506445d88e183bce80e47fc612c710eb592045ed";
});
getpkgs = import nixpkgs;
pkgs = getpkgs {};
in
pkgs.mkShell {
buildInputs = [
pkgs.git
pkgs.nodejs-14_x
pkgs.yarn
pkgs.ripgrep
pkgs.cowsay
];
shellHook = ''
cowsay "here is your shell environment"
'';
} Now, because we have named the file shell.nix, we do not need to specify the filename when running, we can simply run nix-shell. Again, the first time you run the command, it will take some time to fetch the package information (be patient!), but if that is run again, it will be cached and execution is much faster.
Final words
This was a quick introduction to a specific tool within Nix, the nix-shell. It is a nice way to create ad hoc environments in a repeatable and/or reproducible way. It is only scratching the surface of Nix, though, and there is a lot more to explore there.
I must stay though that the documentation around Nix and its usage leaves a bit to be desired. It can sometimes be a bit confusing to figure out what you can do and how to do it. Right now I hope to make some examples and descriptions as I learn and explore more of Nix, which I think is very promising, despite some challenges.