SubsetRepositoriesFedora14

Subset Repositories (Fedora 14)

Subset Repositories is a term I use for a local mirror of a limited subset of a large repository such as the Fedora Everything or Updates repositories. Using a Subset Repository in conjunction with the official repositories can be a big saving if you have multiple machines to install or update on a network, or if you can schedule downloads of updates for a time when your bandwidth is cheaper but can't afford the space and/or bandwidth for a full mirror.

Follow the instructions here for creating and using a base repository based on an ISO image of the DVD media stored on your hard disk, and a subset of the updates for many common packages (i.e. those on the DVD media).

Creating the Base Repository

The base distribution repository is created in the same way as it's always been done on the earlier distribution releases. The files can be located anywhere but for the purposes of this article, the directory /srv/nb/distros/fc14 is used as a base.

To support both 32-bit and 64-bit machines on the network, the ISO images for both the 32-bit and 64-bit releases are needed. If you only need one architecture, skip the parts pertaining to the other architecture.

The ISO images should be placed the repository area on the server that you are using, e.g.:

# mkdir -p /srv/nb/distros/fc14/os/i386/iso/
# cp Fedora-14-i386-DVD.iso /srv/nb/distros/fc14/os/i386/iso/

# mkdir -p /srv/nb/distros/fc14/os/x86_64/iso/
# cp Fedora-14-x86_64-DVD.iso /srv/nb/distros/fc14/os/x86_64/iso/

Create mount points so that the ISO images can be loopback-mounted:

# mkdir /srv/nb/distros/fc14/os/i386/dvd/
# chcon -t mnt_t /srv/nb/distros/fc14/os/i386/dvd

# mkdir /srv/nb/distros/fc14/os/x86_64/dvd/
# chcon -t mnt_t /srv/nb/distros/fc14/os/x86_64/dvd

Edit /etc/fstab and add entries for the DVD images to get them mounted at boot time:

/srv/nb/distros/fc14/os/i386/iso/Fedora-14-i386-DVD.iso /srv/nb/distros/fc14/os/i386/dvd iso9660 _netdev,ro,loop,fscontext=system_u:object_r:public_content_t:s0 0 0
/srv/nb/distros/fc14/os/x86_64/iso/Fedora-14-x86_64-DVD.iso /srv/nb/distros/fc14/os/x86_64/dvd iso9660 _netdev,ro,loop,fscontext=system_u:object_r:public_content_t:s0 0 0

<!> Note that these are just single long lines for each ISO image

It should now be possible to mount the ISO image(s) by issuing the command:

# service netfs start

It may seem strange to treat local loopback mounts as "network filesystems" but this is the point during the boot process where loopback mounts get done. Using service netfs start rather than plain mount -a checks that there are no SELinux issues (the former runs confined by SELinux, the latter does not). The long-winded mount options in the fstab entries prevent SELinux objecting if you serve out the repository using ftp, httpd, rsync etc.

If you are going to loopback-mount a large number of ISO images (perhaps earlier releases too), you may need to increase the number of loopback devices available in the kernel. The Fedora 14 kernel has the loopback module built-in rather than as a module, so append to the end of each kernel line in /etc/grub.conf:

max_loop=64

That will provide support for up to 64 loop devices (the default is 8 I believe). So a typical grub.conf entry would look like this:

title Fedora (2.6.35.11-83.fc14.x86_64)
        root (hd1,1)
        kernel /vmlinuz-2.6.35.11-83.fc14.x86_64 ro root=/dev/mapper/VgOSzion-root.b rd_LVM_LV=VgOSzion/root.b rd_MD_UUID=ddf68436:273c9c68:bfe78010:bc810f04 rd_LVM_LV=VgOSzion/swap.a rd_LVM_LV=VgOSzion/swap.b rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=uk rhgb quiet max_loop=64
        initrd /initramfs-2.6.35.11-83.fc14.x86_64.img

This change will, of course, not have any effect until the next reboot.

Using the Base Repository

Using the base repository is very straightforward.

Using your favourite editor, create a file /etc/yum.repos.d/fedora-dvd.repo:

[fedora-dvd]
name=Fedora Install Media
metadata_expire=-1
baseurl=file:///srv/nb/distros/fc$releasever/os/$basearch/dvd/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
cost=500
enabled=1

You should now be able to test your setup by installing some package. Packages on the media will be shown as coming from fedora-dvd and other packages will be shown as coming from fedora.

Creating the Updates Repository

The object here is to create an updates repository that contains only updates for the packages available on the original media. This will reduce the bandwidth and disk space needed to maintain the mirror.

The tool to be used is reposync, which is part of the yum-utils package.

The first step is to create lists of the packages on the media (and hence those we want to mirror updates for):

# cd /srv/nb/distros/fc14/os/i386
# ls dvd/Packages |
        awk '{ pkgname = $1; sub("-[^-]*-[^-]*$", "", pkgname); print "\t" pkgname }' |
        uniq > fedora-14-i386-dvd-pkgs.conf
# cd /srv/nb/distros/fc14/os/x86_64
# ls dvd/Packages |
        awk '{ pkgname = $1; sub("-[^-]*-[^-]*$", "", pkgname); print "\t" pkgname }' |
        uniq > fedora-14-x86_64-dvd-pkgs.conf

This process creates a sorted list of the packages on each DVD ISO (the lists are slightly different for each architecture), one per line, indented with a tab character at the start of each line. The tab character (or at least some form of leading space) is important for later use.

Now create a mirror configuration file, /etc/yum.repos.d/fedora-updates-mirror.repo:

[fedora-14-updates-subset-i386]
name=Fedora 14 - i386 - Updates Subset
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/updates/14/i386/
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f14&arch=i386
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/updates/14/i386/
#       http://download.fedoraproject.org/pub/fedora/linux/updates/14/i386/
#       http://mirrors.eu.kernel.org/fedora/updates/14/i386/
#       http://sunsite.mff.cuni.cz/MIRRORS/fedora.redhat.com/linux/updates/14/i386/
#       ftp://ftp.tudelft.nl/pub/Linux/download.fedora.redhat.com/linux/updates/14/i386/
includepkgs=
include=file:///srv/nb/distros/fc14/fedora-14-i386-dvd-pkgs.conf
        roundcubemail
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-primary
failovermethod=priority
metadata_expire=1

[fedora-14-updates-subset-x86_64]
name=Fedora 14 - x86_64 - Updates Subset
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/updates/14/x86_64/
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f14&arch=x86_64
#baseurl=http://download.fedora.redhat.com/pub/fedora/linux/updates/14/x86_64/
#       http://download.fedoraproject.org/pub/fedora/linux/updates/14/x86_64/
#       http://mirrors.eu.kernel.org/fedora/updates/14/x86_64/
#       http://sunsite.mff.cuni.cz/MIRRORS/fedora.redhat.com/linux/updates/14/x86_64/
#       ftp://ftp.tudelft.nl/pub/Linux/download.fedora.redhat.com/linux/updates/14/x86_64/
includepkgs=
include=file:///srv/nb/distros/fc14/fedora-14-x86_64-dvd-pkgs.conf
        roundcubemail
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-14-primary
failovermethod=priority
metadata_expire=1
  • {i} It's a good idea to replace the mirrorlist entries with baseurl entries pointing to one or more fast, nearby mirrors as shown in the commented-out baseurl entries above; not all mirrors are as up to date as each other, so with the mirrorlist you might find that one day's reposync run removes the updates downloaded in the previous day's run.

  • {i} If you want to include additional packages in your local updates repository for packages that were not included on the release media, you can add them (indented by a tab) one per line after the include= lines in the repository configuration, as I have shown in the examples above for roundcubemail.

Then create a script updates-sync, and put it somewhere in your PATH, such as /usr/local/bin:

#!/bin/bash

FEDORA_RELEASE=14
SUBSET_32_DIR=/srv/nb/distros/fc${FEDORA_RELEASE}/fedora-${FEDORA_RELEASE}-updates-subset-i386
SUBSET_64_DIR=/srv/nb/distros/fc${FEDORA_RELEASE}/fedora-${FEDORA_RELEASE}-updates-subset-x86_64

echo "Updating mirror of fedora-${FEDORA_RELEASE}-updates-subset-x86_64"
reposync -n -g -d -a x86_64 -r fedora-${FEDORA_RELEASE}-updates-subset-x86_64 -p /srv/nb/distros/fc${FEDORA_RELEASE} |
        grep -Ev "Skipping existing|%" | tr -d '\r'
createrepo -q -d --update ${SUBSET_64_DIR}
echo ""

# Hardlink 32-bit and noarch packages from 64-bit repo into 32-bit repo
common_arch_pkgs=$(echo $SUBSET_64_DIR/*.i686.rpm)
[ "$common_arch_pkgs" != "$SUBSET_64_DIR/*.i686.rpm" ] && ln -f $common_arch_pkgs $SUBSET_32_DIR/
common_noarch_pkgs=$(echo $SUBSET_64_DIR/*.noarch.rpm)
[ "$common_noarch_pkgs" != "$SUBSET_64_DIR/*.noarch.rpm" ] && ln -f $common_noarch_pkgs $SUBSET_32_DIR/

echo "Updating mirror of fedora-${FEDORA_RELEASE}-updates-subset-i386"
reposync -n -g -d -r fedora-${FEDORA_RELEASE}-updates-subset-i386 -p /srv/nb/distros/fc${FEDORA_RELEASE} |
        grep -Ev "Skipping existing|%" | tr -d '\r'
createrepo -q -d --update ${SUBSET_32_DIR}
echo ""

Skip the parts of the script for any architecture you're not interested in. The section in the middle hardlinking files between the x86_64 and i386 repos is only useful if you are mirroring both x86_64 and i386 updates.

The grep command here reduces the volume of output from reposync whilst leaving the interesting output alone (unlike the -q option of reposync). Don't forget to make the script executable:

# chmod 755 /usr/local/bin/updates-sync

To populate the repository (download the update packages), just run /usr/local/bin/updates-sync. This will create repositories in directories /srv/nb/distros/fc14/fedora-14-updates-subset-i386 and /srv/nb/distros/fc14/fedora-14-updates-subset-x86_64 for i386 and x86_64 respectively.

Using the Updates Repository

Create a new repository configuration file /etc/yum.repos.d/fedora-local-updates.repo as follows:

[updates-local]
name=Fedora $releasever - $basearch - Updates (local)
failovermethod=priority
baseurl=file:///srv/nb/distros/fc$releasever/fedora-$releasever-updates-subset-$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
metadata_expire=120
cost=500

{i} A low value for metadata_expire is used so that users of your local updates repo see the updates there very soon after you've sync'ed your mirror, and get fresh updates from there instead of downloading them themselves from the main updates repo over the Internet.

You should now be able to test your setup by running yum update. Packages in the local repository will be shown as coming from updates-local and other packages will be shown as coming from updates.

No/Slow Internet Connection?

Edit the configuration files:

  • /etc/yum.repos.d/fedora.repo

  • /etc/yum.repos.d/fedora-updates.repo

Change the line:

enabled=1

to:

enabled=0

for the network repositories and leave just the local repositories enabled by default. You should then be able to use the package management tools whilst offline, though you may find that things you'd like to install appear not to be available.

To enable the Internet repositories for a single yum run whilst you're online, use:

# yum --enablerepo=fedora --enablerepo=updates update

or

# yum --enablerepo=fedora --enablerepo=updates install somepackage

etc.


CategoryTip

Recent