Skip to content

lastsummer.de

Der Konflikt sind halt die anderen.

Menu and widgets

Deutsch:expand child menu

Schnappschüsse

Technologie

Neues

Zitate

Geschichten

Über Franco

English:expand child menu

Snapshots

Stories

Quotes

News

Technology

Poems

Songsexpand child menu

Listen!

of workloads and bliss

chasing tomorrow

better together

the NULL hypothesis

/lost+found

nothing’s changed

About Franco

Beefy feedback

Franco on About Franco Fichtner

Franco on About Franco Fichtner

Goran Mekić on About Franco Fichtner

Abdulkader Alrazj on About Franco Fichtner

peylight on About Franco Fichtner

Thanh on About Franco Fichtner

Fabien on About Franco Fichtner

Timely stuff

Eine Kreissäge weiter

Twitter, bye-bye

pfSense FreeBSD source and ports repositories are back

About Today

Cheap Shots from the Dull Side

About Remission

Dauerschweigesendung

Setting Up a Package Mirror on FreeBSD

Welcome back! Today we’ll look briefly into how to setup a pkg(8) mirror on FreeBSD. Essentially, we’ll only need a bunch of binary packages for the mirror, pkg-create(8), a running HTTP server and a configuration file for the new mirror on your target system. You must know how to build packages for your target system — if that is the case you are ready to dive in.

Assuming you have those prebuilt packages lying around in /tmp/packages, the following script will create a layout that is reminiscent of the upstream FreeBSD package mirror. I’ll explain why that’s important in a bit. First the script with a couple of annotations:

#!/bin/sh

PACKAGESDIR=/tmp/packages
STAGEDIR=/tmp/stage
SETSDIR=/tmp/sets

mkdir -p ${SETSDIR}
rm -f ${SETSDIR}/packages-*.tar

rm -rf ${STAGEDIR}
mkdir -p ${STAGEDIR}

# rebuild expected FreeBSD structure
mkdir -p ${STAGEDIR}/Latest
mkdir -p ${STAGEDIR}/All

# push packages to home location
cp ${PACKAGESDIR}/* ${STAGEDIR}/All

# needed bootstrap glue when no packages are on the system
cd ${STAGEDIR}/Latest && ln -s ../All/pkg-*.txz pkg.txz

# generate index files
cd ${STAGEDIR} && pkg repo .

tar -cf ${SETSDIR}/packages-`date '+%Y%m%d'`.tar *

Unfortunately, the bootstrap binary pkg-static(8) expects the mirror to point to a file Latest/pkg.txz, but pkg-create(8) itself only creates the index files digests.txz, packagesite.txz and meta.txz (new since version 1.3) for a given directory and its subdirectories. That also means you can shuffle your package mirror around and use custom layouts — it doesn’t matter where the packages are as long as there is a link to the current pkg package for bootstrapping. If you don’t provide a pkg package, you’ll either have a complimentary mirror with extra packages for FreeBSD, or you won’t be able to operate your package mirror replacement. ;)

The archive itself is not compressed since all the files already use a good compression algorithm. It would only waste time to try to compress it again. On the mirror, unpack the archive into an exposed HTTP directory. Your mirror is already fully set up!

Last stage is to add repository awareness to your target system. This is done by setting up a simple config file, either under /etc/pkg (assuming it’s your base system) or /usr/local/etc/pkg/repos (package-shipping-friendly). The file name itself doesn’t really matter, but good style is your repo name followed by the idiomatic dot-conf:

cat > /usr/local/etc/pkg/repos/my_repo.conf <<EOF
my_repo: {
url: "pkg+http://pkg.doe.it/${ABI}/latest",
mirror_type: "srv",
enabled: yes
}
EOF

It’s important to set the repository name in the file, because the file name itself is not authoritative. If you are trying to replace the stock mirror, either set enabled to no in /etc/pkg/FreeBSD.conf or delete the file. ${ABI} and latest aren’t really necessary in the URL: you’ll want ${ABI} when you need to support different architectures on the same mirror, and you’ll want latest if you have another type of packages that denote a more conservative update behaviour (or you simply wish to follow the current FreeBSD style). Generally, these are just names you may choose to use even for alternate meanings in your domain.

An invoke of pkg-update(8) will reconfigure your system to adjust to your repository settings. pkg-install(8) and friends will then be able to manage packages from your mirror. One use case idea is shipping a package that embeds a custom mirror for external plugins so that FreeBSD doesn’t have to worry about delivering them — they are most likely not essential but still nice to have for the user. Have some fun on your own. See you again soon.

From here on out things get a little funky with regard to package mirror priority in case (seemingly identical) packages are found on multiple mirrors. It is especially bad when you have different options for these same packages. Upstream has just released 1.4.3 to address two issues: priorities for repo configs and a CONSERVATIVE_UPGRADE switch that retains packages from the same mirror even when other mirrors have newer versions. Unfortunately, this also has security implications like hijacking a package… There are ways to only talk to a single mirror (option “-r”), but having a consistent mirror (or multiple complementary mirrors) is probably the most sensible course at this point.

Posted on December 28, 2014January 2, 2015Author FrancoCategories english, technology

5 thoughts on “Setting Up a Package Mirror on FreeBSD”

Daniel says:

January 2, 2015 at 2:27 pm

Where/how does ${SETSDIR} get set? It is first seen in the final line of your script.

Franco says:

January 2, 2015 at 2:37 pm

Hi Daniel, good catch. The code is copied from a larger base so I had to make a few simplifications and zapped too much it seems. Fixed up the article. Thanks!

darkfader says:

January 13, 2015 at 4:25 pm

Hi!

Sorry to ask questions when you already laid out such a nice article… but:

One thing I just can’t figure:

I’ve a build jail for old 9.1, say another for 9.2 and one for 9.3 and when it comes out I would roll one for 9.4.

(I think 9.1 and 9.2 are dead already, they’re here for sake of example)

$ABI will resolve to the very same for all of them, yet a build of i.e. bash in a 9.3 jail will produce a different binary than the one in the 9.4 jail.

I’d like to lay out my network pkg server as nicely as possible but right now I don’t see how…

I know there’s also going to be a 9.3-compat package once 9.4 is out. Is that the answer?

Would appreciate your thoughts, especially if I’m just not groking the thing and there’s nothing to worry :)

Franco says:

January 13, 2015 at 10:51 pm

Hi there,

well, every build will produce different binaries even on the same 9.x. The whole idea of STABLE is that the ABI doesn’t change, so that you can use all binaries on all of these versions interchangeably, hence ${ABI} is the same.

You want to, however, always build your packages against the latest STABLE you can get and use them for all earlier versions as well, so in your case use 9.4 exclusively for ports builds and use them on older 9.x versions.

For dynamic linking 9.x doesn’t matter, but if you have a port that statically links against a vulnerable library it’ll inherit this vulnerability. The latest STABLE is always your best bet for ports builds.

Hope that helps! :)

Cheers,

Franco

darkfader says:

January 14, 2015 at 9:06 pm

Thanks a lot!

This makes things easier. I’m only doing FreeBSD on the side and had forgotten some basics :)

I’ll set up one more build running on latest STABLE, makes sense.

And your script is stored now, will check it out, probably only for the custom packages, since I just finished moving away from tinderbox to poudriere and don’t wanna replace everything once more.

Greets

Florian

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Comment

Name *

Email *

Website

Save my name, email, and website in this browser for the next time I comment.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Post navigation

Previous Previous post: Creating Custom Packages on FreeBSD

Next Next post: Mach’s gut 2014

Proudly powered by WordPress