Chapter 11. The build process

Table of Contents

11.1. Program location
11.2. Main targets
11.3. Other helpful targets

The basic steps for building a program are always the same. First the program's source (distfile) must be brought to the local system and then extracted. After any patches to compile properly on NetBSD are applied, the software can be configured, then built (usually by compiling), and finally the generated binaries, etc. can be put into place on the system. These are exactly the steps performed by the NetBSD package system, which is implemented as a series of targets in a central Makefile, pkgsrc/mk/bsd.pkg.mk.

11.1. Program location

Before outlining the process performed by the NetBSD package system in the next section, here's a brief discussion on where programs are installed, and which variables influence this.

The automatic variable PREFIX indicates where all files of the final program shall be installed. It is usually set to LOCALBASE (/usr/pkg), or CROSSBASE for pkgs in the “cross” category. The value of PREFIX needs to be put into the various places in the program's source where paths to these files are encoded. See Section 7.3, “patches/*” and Section 12.3.1, “Shared libraries - libtool” for more details.

When choosing which of these variables to use, follow the following rules:

  • PREFIX always points to the location where the current pkg will be installed. When referring to a pkg's own installation path, use “${PREFIX}”.

  • LOCALBASE is where all non-X11 pkgs are installed. If you need to construct a -I or -L argument to the compiler to find includes and libraries installed by another non-X11 pkg, use “${LOCALBASE}”.

  • X11BASE is where the actual X11 distribution (from xsrc, etc.) is installed. When looking for standard X11 includes (not those installed by a pkg), use “${X11BASE}”.

  • X11 based are special in that they may be installed in either X11BASE or LOCALBASE.

    Usually, X11 packages should be installed under LOCALBASE whenever possible. Note that you will need to set USE_X11 in them to request the presence of X11 and to get the right compilation flags.

    Even though, there are some packages that cannot be installed under LOCALBASE: those that come with app-defaults files. These packages are special and they must be placed under X11BASE. To accomplish this, set either USE_X11BASE or USE_IMAKE in your package.

    Some notes: USE_X11 and USE_X11BASE are mutually exclusive. If you need to find includes or libraries installed by a pkg that has USE_IMAKE or USE_X11BASE in its pkg Makefile, you need to use both ${X11BASE} and ${LOCALBASE}. To force installation of all X11 packages in LOCALBASE, the pkgtools/xpkgwedge is enabled by default.

  • X11PREFIX should be used to refer to the installed location of an X11 package. X11PREFIX will be set to X11BASE if xpkgwedge is not installed, and to LOCALBASE if xpkgwedge is installed.

  • If xpkgwedge is installed, it is possible to have some packages installed in X11BASE and some in LOCALBASE. To determine the prefix of an installed package, the EVAL_PREFIX definition can be used. It takes pairs in the format “DIRNAME=<package>”, and the make(1) variable DIRNAME will be set to the prefix of the installed package <package>, or “${X11PREFIX}” if the package is not installed.

    This is best illustrated by example.

    The following lines are taken from pkgsrc/wm/scwm/Makefile:

    EVAL_PREFIX+=           GTKDIR=gtk+
    CONFIGURE_ARGS+=        --with-guile-prefix=${LOCALBASE}        \
                            --with-gtk-prefix="${GTKDIR}"           \
                            --enable-multibyte
    

    Specific defaults can be defined for the packages evaluated using EVAL_PREFIX, by using a definition of the form:

    GTKDIR_DEFAULT= ${LOCALBASE}
    

    where GTKDIR corresponds to the first definition in the EVAL_PREFIX pair.

  • Within ${PREFIX}, packages should install files according to hier(7), with the exception that manual pages go into ${PREFIX}/man, not ${PREFIX}/share/man.

11.2. Main targets

The main targets used during the build process defined in bsd.pkg.mk are:

fetch

This will check if the file(s) given in the variables DISTFILES and PATCHFILES (as defined in the package's Makefile) are present on the local system in /usr/pkgsrc/distfiles. If they are not present, an attempt will be made to fetch them using commands of the form:

${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${site}${file} ${FETCH_AFTER_ARGS}

where ${site} varies through several possibilities in turn: first, MASTER_SITE_OVERRIDE is tried, then the sites specified in either SITES_file if defined, else MASTER_SITES or PATCH_SITES, as applies, then finally the value of MASTER_SITE_BACKUP. The order of all except the first can be optionally sorted by the user, via setting either MASTER_SORT_AWK or MASTER_SORT_REGEX.

checksum

After the distfile(s) are fetched, their checksum is generated and compared with the checksums stored in the distinfo file. If the checksums don't match, the build is aborted. This is to ensure the same distfile is used for building, and that the distfile wasn't changed, e.g. by some malign force, deliberately changed distfiles on the master distribution site or network lossage.

extract

When the distfiles are present on the local system, they need to be extracted, as they are usually in the form of some compressed archive format, most commonly .tar.gz.

If only some of the distfiles need to be uncompressed, the files to be uncompressed should be put into EXTRACT_ONLY.

If the distfiles are not in .tar.gz format, they can be extracted by setting either EXTRACT_SUFX, or EXTRACT_CMD, EXTRACT_BEFORE_ARGS and EXTRACT_AFTER_ARGS. In the former case, pkgsrc knows how to extract a number of suffixes (.tar.gz, .tgz, .tar.gz2, .tbz, .tar.Z, .tar, .shar.gz, .shar.bz2, .shar.Z, .shar, .Z, .bz2 and .gz; see the definition of the various DECOMPRESS_CMD variables bsd.pkg.mk for a complete list). Here's an example on how to use the other variables for a program that comes with a compressed shell archive whose name ends in .msg.gz:

     EXTRACT_SUFX=   .msg.gz
     EXTRACT_CMD=            zcat
     EXTRACT_BEFORE_ARGS=
     EXTRACT_AFTER_ARGS=     |sh
patch

After extraction, all the patches named by the PATCHFILES, those present in the patches subdirectory of the package as well as in $LOCALPATCHES/$PKGPATH (e.g. /usr/local/patches/graphics/png) are applied. Patchfiles ending in .Z or .gz are uncompressed before they are applied, files ending in .orig or .rej are ignored. Any special options to patch(1) can be handed in PATCH_DIST_ARGS. See Section 7.3, “patches/*” for more details.

By default patch(1) is given special args to make it fail if the patches apply with some lines of fuzz. Please fix (regen) the patches so that they apply cleanly. The rationale behind this is that patches that don't apply cleanly may end up being applied in the wrong place, and cause severe harm there.

configure

Most pieces of software need information on the header files, system calls, and library routines which are available in NetBSD. This is the process known as configuration, and is usually automated. In most cases, a script is supplied with the source, and its invocation results in generation of header files, Makefiles, etc.

If the program's distfile contains its own configure script, this can be invoked by setting HAS_CONFIGURE. If the configure script is a GNU autoconf script, GNU_CONFIGURE should be specified instead. In either case, any arguments to the configure script can be specified in the CONFIGURE_ARGS variable, and the configure script's name can be set in CONFIGURE_SCRIPT if it differs from the default “configure”. Here's an example from the sysutils/top package:

HAS_CONFIGURE=          yes
CONFIGURE_SCRIPT=       Configure
CONFIGURE_ARGS+=        netbsd13

If the program uses an Imakefile for configuration, the appropriate steps can be invoked by setting USE_IMAKE to “YES”. (If you only want the package installed in $X11PREFIX but xmkmf not being run, set USE_X11BASE instead!)

build

Once configuration has taken place, the software will be built by invoking $MAKE_PROGRAM on $MAKEFILE with $BUILD_TARGET as the target to build. The default MAKE_PROGRAM is “gmake” if USE_GNU_TOOLS contains “make”, “make” otherwise. MAKEFILE is set to “Makefile” by default, and BUILD_TARGET defaults to “all”. Any of these variables can be set in the package's Makefile to change the default build process.

install

Once the build stage has completed, the final step is to install the software in public directories, so users can access the programs and files. As in the build-target, $MAKE_PROGRAM is invoked on $MAKEFILE here, but with the $INSTALL_TARGET instead, the latter defaulting to “install” (plus “install.man”, if USE_IMAKE is set).

If no target is specified, the default is “build”. If a subsequent stage is requested, all prior stages are made: e.g. make build will also perform the equivalent of:

make fetch
make checksum
make extract
make patch
make configure
make build

11.3. Other helpful targets

pre/post-*

For any of the main targets described in the previous section, two auxiliary targets exist with “pre-” and “post-” used as a prefix for the main target's name. These targets are invoked before and after the main target is called, allowing extra configuration or installation steps be performed from a package's Makefile, for example, which a program's configure script or install target omitted.

do-*

Should one of the main targets do the wrong thing, and should there be no variable to fix this, you can redefine it with the do-* target. (Note that redefining the target itself instead of the do-* target is a bad idea, as the pre-* and post-* targets won't be called anymore, etc.) You will not usually need to do this.

reinstall

If you did a make install and you noticed some file was not installed properly, you can repeat the installation with this target, which will ignore the “already installed” flag.

deinstall

This target does a pkg_delete(1) in the current directory, effectively de-installing the package. The following variables can be used to tune the behaviour:

PKG_VERBOSE

Add a "-v" to the pkg_delete(1) command.

DEINSTALLDEPENDS

Remove all packages that require (depend on) the given package. This can be used to remove any packages that may have been pulled in by a given package, e.g. if make deinstall DEINSTALLDEPENDS=1 is done in pkgsrc/x11/kde, this is likely to remove whole KDE. Works by adding “-R” to the pkg_delete(1) command line.

update

This target causes the current package to be updated to the latest version. The package and all depending packages first get de-installed, then current versions of the corresponding packages get compiled and installed. This is similar to manually noting which packages are currently installed, then performing a series of make deinstall and make install (or whatever UPDATE_TARGET is set to) for these packages.

You can use the “update” target to resume package updating in case a previous make update was interrupted for some reason. However, in this case, make sure you don't call make clean or otherwise remove the list of dependent packages in WRKDIR. Otherwise you lose the ability to automatically update the current package along with the dependent packages you have installed.

Resuming an interrupted make update will only work as long as the package tree remains unchanged. If the source code for one of the packages to be updated has been changed, resuming make update will most certainly fail!

The following variables can be used either on the command line or in /etc/mk.conf to alter the behaviour of make update:

UPDATE_TARGET

Install target to recursively use for the updated package and the dependent packages. Defaults to DEPENDS_TARGET if set, “install” otherwise for make update. e.g. make update UPDATE_TARGET=package

NOCLEAN

Don't clean up after updating. Useful if you want to leave the work sources of the updated packages around for inspection or other purposes. Be sure you eventually clean up the source tree (see the “clean-update” target below) or you may run into troubles with old source code still lying around on your next make or make update.

REINSTALL

Deinstall each package before installing (making DEPENDS_TARGET). This may be necessary if the “clean-update” target (see below) was called after interrupting a running make update.

DEPENDS_TARGET

Allows you to disable recursion and hardcode the target for packages. The default is “update” for the update target, facilitating a recursive update of prerequisite packages. Only set DEPENDS_TARGET if you want to disable recursive updates. Use UPDATE_TARGET instead to just set a specific target for each package to be installed during make update (see above).

clean-update

Clean the source tree for all packages that would get updated if make update was called from the current directory. This target should not be used if the current package (or any of its depending packages) have already been de-installed (e.g., after calling make update) or you may lose some packages you intended to update. As a rule of thumb: only use this target before the first time you run make update and only if you have a dirty package tree (e.g., if you used NOCLEAN).

If you unsure about whether your tree is clean you can either perform a make clean at the top of the tree, or use the following sequence of commands from the directory of the package you want to update (before running make update for the first time, otherwise you lose all the packages you wanted to update!):

# make clean-update
# make clean CLEANDEPENDS=YES
# make update

The following variables can be used either on the command line or in /etc/mk.conf to alter the behaviour of make clean-update:

CLEAR_DIRLIST

After make clean, do not reconstruct the list of directories to update for this package. Only use this if make update successfully installed all packages you wanted to update. Normally, this is done automatically on make update, but may have been suppressed by the NOCLEAN variable (see above).

info

This target invokes pkg_info(1) for the current package. You can use this to check which version of a package is installed.

readme

This target generates a README.html file, which can be viewed using a browser such as www/mozilla or www/links. The generated files contain references to any packages which are in the PACKAGES directory on the local host. The generated files can be made to refer to URLs based on FTP_PKG_URL_HOST and FTP_PKG_URL_DIR. For example, if I wanted to generate README.html files which pointed to binary packages on the local machine, in the directory /usr/packages, set FTP_PKG_URL_HOST=file://localhost and FTP_PKG_URL_DIR=/usr/packages. The ${PACKAGES} directory and its subdirectories will be searched for all the binary packages.

readme-all

Use this target to create a file README-all.html which contains a list of all packages currently available in the NetBSD Packages Collection, together with the category they belong to and a short description. This file is compiled from the pkgsrc/*/README.html files, so be sure to run this after a make readme.

cdrom-readme

This is very much the same as the “readme” target (see above), but is to be used when generating a pkgsrc tree to be written to a CD-ROM. This target also produces README.html files, and can be made to refer to URLs based on CDROM_PKG_URL_HOST and CDROM_PKG_URL_DIR.

show-distfiles

This target shows which distfiles and patchfiles are needed to build the package. (DISTFILES and PATCHFILES, but not patches/*)

show-downlevel

This target shows nothing if the package is not installed. If a version of this package is installed, but is not the version provided in this version of pkgsrc, then a warning message is displayed. This target can be used to show which of your installed packages are downlevel, and so the old versions can be deleted, and the current ones added.

show-pkgsrc-dir

This target shows the directory in the pkgsrc hierarchy from which the package can be built and installed. This may not be the same directory as the one from which the package was installed. This target is intended to be used by people who may wish to upgrade many packages on a single host, and can be invoked from the top-level pkgsrc Makefile by using the “show-host-specific-pkgs” target.

show-installed-depends

This target shows which installed packages match the current package's DEPENDS. Useful if out of date dependencies are causing build problems.

check-shlibs

After a package is installed, check all its binaries and (on ELF platforms) shared libraries to see if they find the shared libs they need. Run by default if PKG_DEVELOPER is set in /etc/mk.conf.

print-PLIST

After a “make install” from a new or upgraded pkg, this prints out an attempt to generate a new PLIST from a find -newer work/.extract_done. An attempt is made to care for shared libs etc., but it is strongly recommended to review the result before putting it into PLIST. On upgrades, it's useful to diff the output of this command against an already existing PLIST file.

If the package installs files via tar(1) or other methods that don't update file access times, be sure to add these files manually to your PLIST, as the “find -newer” command used by this target won't catch them!

See Section 8.3, “Tweaking output of make print-PLIST for more information on this target.

bulk-package

Used to do bulk builds. If an appropriate binary package already exists, no action is taken. If not, this target will compile, install and package it (and it's depends, if PKG_DEPENDS is set properly. See Section 5.3.1, “Configuration”. After creating the binary package, the sources, the just-installed package and it's required packages are removed, preserving free disk space.

Beware that this target may deinstall all packages installed on a system!

bulk-install

Used during bulk-installs to install required packages. If an upto-date binary package is available, it will be installed via pkg_add(1). If not, make bulk-package will be executed, but the installed binary not be removed.

A binary package is considered “upto-date” to be installed via pkg_add(1) if:

  • None of the package's files (Makefile, ...) were modified since it was built.

  • None of the package's required (binary) packages were modified since it was built.

Beware that this target may deinstall all packages installed on a system!