Essential pkgsrc - The Missing Mini Handbook

Hamza Sheikh

2017-03-25 19:29 (updated 2018-05-16 20:30)

pkgsrc is a cross operating system package manager. It supports -- among many others -- NetBSD, Minix, SmartOS, Linux, and macOS. I like it because of this portability. It also has the additional, and I would say the best, benefit of being installed in the home directory and run completely without needing root access. I also like that I don't have to depend on binary packages built by someone else, say Joyent, although there's absolutely nothing wrong with it. Finally, it provides a large number of different packages. I have never encountered a package that I needed but was not available. In short pkgsrc is a portable, featureful, and flexible package manager. What's not to like?

pkgsrc can sometimes be a little behind native package managers, such as MacPorts on macOS, but it catches up quickly. For my use case -- getting access to multiple versions of Python -- it works well enough if I closely follow its trunk branch.

There's generally good and detailed documentation available for pkgsrc but an introductory guide that pulled in some essential starter information was lacking. This guide fills that void by making it easy to get started with pkgsrc and learn about some of its core concepts. Thus, I dubbed it the mini handbook or the missing starter handbook.

I wrote this guide by running pkgsrc on macOS. The instructions should work on Linux as well but may need minor modifications.

I wrote a similar guide for Linux (pkgsrc on Linux - Quickstart Guide) a while ago. This guide is an updated and more general version of it.

Contents

Install

You need git to clone the pkgsrc repo from GitHub. I prefer a shallow clone to save disk space. Newer git versions can pull and push shallow clones just fine. I also prefer trunk branch since it has the latest goodies available. If you prefer stable branches then substitue trunk below with a quarterly release such as pkgsrc_2016Q4.

$ git clone https://github.com/NetBSD/pkgsrc.git -b trunk --single-branch ~/pkgsrc

I believe you may need build tools such as a compiler. Make sure you have these tools installed just in case.

Unlike the official install I prefer an unprivileged install because it gets installed in my home directory without affecting the rest of the system. As The pkgsrc guide puts it,

In unprivileged mode in contrast to privileged one all programs are installed under one particular user and cannot utilise privileged operations (packages don't create special users and all special file permissions like setuid are ignored).

If you're running on Linux (e.g. Ubuntu), run these steps first. Reason is an error message described on pkgsrc wiki (Shell's echo command is not BSD-compatible).

$ export SH=/bin/bash

Then run the bootstrap.

$ cd ~/pkgsrc/bootstrap
$ ./bootstrap --unprivileged

Environment Variables

Insert these lines in ~/.profile (macOS or Ubuntu) if they're not already present.

Set the PATH variable. I prefer to give priority to pkgsrc-installed packages.

export PATH
PATH="${HOME}/pkg/sbin:${HOME}/pkg/bin:${PATH}"

Set the MANPATH variable to access man pages that come with packages installed with pkgsrc.

export MANPATH
MANPATH="${HOME}/pkg/man:${MANPATH}"

NetBSD wiki recommends these settings for Unicode to work.

export LC_CTYPE
LC_CTYPE="en_US.UTF-8"
export LC_COLLATE
LC_COLLATE="C"
export LC_TIME
LC_TIME="C"
export LC_NUMERIC
LC_NUMERIC="C"
export LC_MONETARY
LC_MONETARY="C"
export LC_MESSAGES
LC_MESSAGES="en_US.UTF-8"
export LC_ALL
LC_ALL=""

Don't forget to source the file.

$ source ~/.profile  # macOS or Ubuntu

mk.conf

The bootstrap process creates ~/pkg/etc/mk.conf. mk.conf is a configuration file used when building packages. Familiarize yourself with it as many problems you will face as a novice will be solved by editing it.

You are encouraged to keep the pkgsrc directory clean by moving all work directories and distfiles in separate paths, such as ~/pkg/usr/work and ~/pkg/usr/distfiles respectively.

Create these directories.

$ mkdir -p ~/pkg/usr/{work,distfiles}

Add these two lines to mk.conf before the last line which most likely will be .endif # end pkgsrc settings.

WRKOBJDIR=${HOME}/pkg/usr/work
DISTDIR=${HOME}/pkg/usr/distfiles

Python

Since the pkgsrc install in this guide is custom and unprivileged you have to build packages from source. If you prefer not to do that then you must follow the official instructions to install pkgsrc. You can then use pkgin to manage packages.

Python 3.6 is the latest version available at the time of writing. It's available in trunk branch of pkgsrc.

Edit ~/pkg/etc/mk.conf and insert the following line before the last line which most likely will be .endif # end pkgsrc settings. This configures the build system to always use Python 3.6 as the default version.

PYTHON_VERSION_DEFAULT=36

To install packages from source navigate to the directory of the package and run bmake. Here we install Python 3.6.

$ cd ~/pkgsrc/lang/python36
$ bmake install

Confirm python3.6 was installed and your PATH is setup correctly.

$ which python3.6
~/pkg/bin/python3.6

Installing pip is a little trickier. There's ~/pkgsrc/devel/py-pip but it installs only one version of pip depending on how you set up PYTHON_VERSION_DEFAULT above. In our case it will install pip3.6. We'll discuss how to install multiple pip versions in the next section titled pkg_alternatives.

$ cd ~/pkgsrc/devel/py-pip
$ bmake install

Confirm pip3.6 was installed and your PATH is setup correctly.

$ which pip3.6
~/pkg/bin/pip3.6

Last thing we need is virtualenv-3.6.

$ cd ~/pkgsrc/devel/py-virtualenv
$ bmake install

Confirm virtualenv-3.6 was installed and your PATH is setup correctly.

$ which virtualenv-3.6
~/pkg/bin/virtualenv-3.6

We have all the things we need to create a virtualenv and start working with Python 3.6.

$ python3.6
Python 3.6.0 (default, Mar 25 2017, 00:15:25)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import virtualenv
>>> ^D

pkg_alternatives

Just like Debian pkgsrc has an alternatives system.

But what is an "alternatives system"? It is a framework that allows multiple packages providing similar functionality to be installed concurrently (by removing files with common names), and then using a utility to set up those common names with symlinks to the preferred program.

In other words, we can install, for example, pip2.7 and pip3.6 and set pip to point (symlink) to pip3.6. We can still use pip2.7 and pip3.6 independent of each other but when we use pip it will run pip3.6.

So let's try this and install Python 2.7 and pip2.7.

$ cd ~/pkgsrc/lang/python27
$ bmake install

When I initially tried to install an alternative pip I ran into the same problem as reported here: Can't have "pip" packages for python 2.7 and 3.4 installed at the same time. The solution provided by Mark Davies was

Don't install pip and use the ALTERNATES mechanism to create it. See the similar changes I did to the py-docutils and py-sphinx packages last night.

I asked the pkgsrc-users mailing list on how to do it. Thanks to Adam for pointing me to the solution.

The trick is to temporarily override PYTHON_VERSION_DEFAULT when you run bmake install like so.

$ cd ~/pkgsrc/devel/py-pip
$ bmake install PYTHON_VERSION_DEFAULT=27
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Checking for vulnerabilities in py27-pip-9.0.1
===> Installing binary package of py27-pip-9.0.1
pkg_add: no pkg found for '~/pkg/usr/work/devel/py-pip/work/work/.packages/py27-pip-9.0.1.tgz', sorry.
pkg_add: 1 package addition failed
*** Error code 1

Stop.
bmake[2]: stopped in ~/pkgsrc/devel/py-pip
*** Error code 1

Stop.
bmake[1]: stopped in ~/pkgsrc/devel/py-pip
*** Error code 1

Stop.
bmake: stopped in ~/pkgsrc/devel/py-pip

Uh oh. Where did this error come from?

From what I understand we need to clean the work directories from when we built pip3.6. Otherwise the install process looks for a pre-built package to install. In this case it finds py36-pip but not py27-pip and fails. Fair warning, this is just my very flawed understanding. I can fix the symptom but can't explain the cause.

The fix is to clean all work directories and run bmake install again.

$ rm -rf ~/pkg/usr/work/*
$ cd ~/pkgsrc/devel/py-pip
$ bmake install PYTHON_VERSION_DEFAULT=27

A better way to avoid the problem in the first place is to always run bmake install clean clean-depends which will build and install the package and clean the work directories for the package and its dependencies. The down side is that the next time you build the package it will re-do all the work.

Similar to how we override PYTHON_VERSION_DEFAULT with py27 we can use other Python versions like py35 and py34.

Let's look at what versions of pip we have installed.

$ ls -l ~/pkg/bin/pip?.?
~/pkg/bin/pip2.7* ~/pkg/bin/pip3.6*

To run say pip3.6 when you run ~/pkg/bin/pip you need to use the pkg_alternatives tool. Install it first.

$ cd ~/pkgsrc/pkgtools/pkg_alternatives
$ bmake install

Let's list all packages that provide alternatives.

$ pkg_alternatives list
pkg_alternatives: looking for alternatives in `~/pkg/pkgdb'
py27-pip-9.0.1
py36-pip-9.0.1
py36-virtualenv-15.1.0
python27-2.7.13nb1
python36-3.6.0nb2

Let's ensure pip points to pip3.6.

$ pkg_alternatives manual py36-pip
pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/pip'

Similarly we can use python3.6 as default Python for other binaries.

$ pkg_alternatives manual python36
pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/2to3'
pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/pydoc3'
pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/python'
pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/python3'

As always read the man page of pkg_alternatives for more information.

$ man -S 8 pkg_alternatives

Vulnerabilities

pkgsrc makes it simple to check for vulnerabilities in installed packages.

Fetch the latest list of known vulnerabilities.

$ pkg_admin -K ~/pkg/pkgdb fetch-pkg-vulnerabilities

Audit all installed packages for known vulnerabilities.

$ pkg_admin -v audit
No vulnerabilities found

Sometimes packages will not build because there are known vulnerabilities in them or their dependencies. For example,

$ cd ~/pkgsrc/devel/ncurses
$ bmake install
=> Bootstrap dependency digest>=20010302: found digest-20160304
===> Checking for vulnerabilities in ncurses-6.1
Package ncurses-6.1 has a null-pointer-dereference vulnerability, see https://nvd.nist.gov/vuln/detail/CVE-2018-10754
ERROR: Define ALLOW_VULNERABLE_PACKAGES in mk.conf or IGNORE_URL in pkg_install.conf(5) if this package is absolutely essential.
*** Error code 1

Stop.
bmake: stopped in /home/codeghar/pkgsrc/devel/ncurses

Sometimes you have no choice but to install the package despite the vulnerability. I prefer to ignore specific CVEs in ~/pkg/etc/pkg_install.conf file rather than ignore all CVEs in ~/pkg/etc/mk.conf file. In this example, the file would look like,

$ more ~/pkg/etc/pkg_install.conf
IGNORE_URL=https://nvd.nist.gov/vuln/detail/CVE-2018-10754

Now you can build the package successfully.

Upgrade Single Package

Follow these steps when you want to update a single package, say Python 3.6.

$ cd ~/pkgsrc/lang/python36
$ bmake update clean clean-depends

Upgrade All Packages

pkg_rolling-replace removes packages before installing them in a safer way. Read more on how to upgrade packages.

$ cd ~/pkgsrc/pkgtools/pkg_rolling-replace
$ bmake install clean clean-depends

Add PKGSRCDIR variable to ~/pkg/etc/mk.conf just before the last line which most likely will be .endif # end pkgsrc settings.

PKGSRCDIR=${HOME}/pkgsrc

Pull any upstream changes.

$ cd ~/pkgsrc
$ git pull

List available updates. Note the use of -n flag.

$ pkg_rolling-replace -u -n -v

Apply available updates. Note the absence of -n flag.

$ pkg_rolling-replace -u -v

List Installed Packages

Get a list of all installed packages with pkg_info. The -u flag does not list any dependencies. I prefer this format over using -a flag.

$ pkg_info -u

Read pkg_info man page for more information.

$ man -S 1 pkg_info

Delete Package

Use pkg_delete to remove a package.

$ pkg_delete py27-pip

You can uninstall a package and all its dependencies that are not needed by any other package with the -R flag.

$ pkg_delete -R py27-pip