PaulHowarth/Blog/2009-02-17

Tuesday 17th February 2009

Fedora Project

  • Updated gtkwave in Rawhide to 3.2.0

Local Packages

  • New package fuse-davfs2

  • Updated curl with patches from Fedora to fix NSS support; curl is now built using NSS rather than OpenSSL again for releases that support it

  • Updated gtkwave to 3.2.0

  • Updated perl-Params-Util to 0.38

Creating a Local Mirror of a WebDAV Site

I have over the years helped to maintain the Manchester City Supporters' website in conjunction with Svenn Hanssen at the University of Tromsø. This has been an ongoing project since 1994 and our methodologies reflect that - Svenn does most of the maintenance and I keep a local mirror of the site and push my updates after I've made them.

I used to use the traditional mirror script to pull my local copy from the site's FTP server, and later switched to using lftp, still using FTP. However, late last year the site moved over to using WebDAV rather than FTP for upload access, and this proved to be a problem for lftp; although there is support in lftp for WebDAV, it appears to use index pages served out by the web server rather than the WebDAV indexes, and thus doesn't see the whole site (and gets very confused by broken links but I need to fix those anyway). So I've been using cadaver to upload my files and not resyncing back Svenn's changes for a while.

Today I tried a different approach, and it works very nicely. I used davfs2 to mount the remote site as a local filesystem and then used rsync to update my local mirror. Not only does it work well, it's much faster than my old lftp approach :-)

I made fuse-davfs2 packages for Fedora 9, 10, and Rawhide based on some old packages from Dag's repository. Having installed that, I then set up the configuration files I needed, namely an entry in /etc/fstab for the mount: 

https://webdav.example.com/ /home/paul/webdav.example.com davfs noauto,user 0 0

and a credentials file ~/.davfs2/secrets: 

/home/paul/webdav.example.com username password

The ~/.davfs2/ directory is mode 0700 and the file ~/.davfs2/secrets is mode 0600.

I then added my account paul to the davfs2 group: 

# usermod -a -G davfs2 paul

I could then use this script to update my local copy of the site: 

#!/bin/bash

# Exit on failure
set -e

echo Mounting WebDAV filesystem
mount /home/paul/webdav.example.com

if [ ! -s /home/paul/webdav.example.com/index.html ]; then
        echo "sitesync: index.html missing from remote filesystem - aborting" 1>&2
        exit 1
fi

echo Syncing data from WebDAV filesystem to local copy
rsync -av --delete /home/paul/webdav.example.com/ /home/paul/mymirror

echo Unmounting WebDAV filesystem
umount /home/paul/webdav.example.com

{i} Since writing this article, I have renamed the fuse-davfs2 package to davfs2, as that is what the software is known as upstream and, as of March 2009, in Fedora.

Recent