PaulHowarth/Blog/2010-04

Paul's Blog Entries for April 2010

Thursday 1st April 2010

Local Packages

  • Updated dovecot 2.0 to 20100330 nightly build

  • Updated java-1.6.0-sun to 1.6.0.19 (thanks to Stephan Buchert for update), see SunJava6OnFedora for details

  • Updated perl-IPC-Run to 0.89 (revert extended tests to require "oslo consensus" variables; add IO::Pty to META.yml requirement)

  • Updated perl-Module-Build to 0.3605, running the signature test with the ${GNUPGHOME} environment variable used to specify key locations, so we can avoid writing files within ${HOME}

    • Added missing newline to "Changing sharpbang" messages (CPAN RT#54474)

    • Added 'beos' to list of Unix-like OS types (CPAN RT#53876)

    • Sets $ENV{HOME} to a temporary directory during testing

    • For VMS: fixed prefix handling plus other test fixes
    • Support anonymous array of directories for c_source

    • Small POD formatting fix
  • Updated xz to 20100401 snapshot

Friday 2nd April 2010

Local Packages

  • Updated perl-Module-Build to 0.3607

    • 'dist' action now always ensures a clean dist directory for tarball

    • Migrated repository to git and updated META.yml to match

    • Removed bugtracker URL (let search.cpan.org use default)
    • Disabled SIGNATURE generation

Sunday 4th April 2010

Local Packages

  • Updated getlive to 0.59, addressing the latest hotmail changes

  • Updated perl-FreezeThaw to 0.5001 (switch to a coarse version-check to recognize that regexps are first-level, avoiding the need to rely on Scalar::Util)

Monday 5th April 2010

Local Packages

Tuesday 6th April 2010

Local Packages

  • Updated dovecot 2.0 to 20100406 nightly build, fixing managesieve for an API change in dovecot

  • Updated perl-DateTime to update DateTime::TimeZone to 1.16 (based on version 2010h of the Olson database, with changes for Bangladesh, Palestine, and Russia)

  • Updated perl-Parse-RecDescent to 1.965001 (removed all references to /opt version of perl interpreter, and added Parse::RecDescent::redirect_reporting_to())

Wednesday 7th April 2010

Local Packages

  • Updated libcurl7112 to handle move of kerberos installation prefix in krb5 ≥ 1.8, enable built-in manual for non-compat build (adding buildreq groff), comment patches in spec file and also merge a couple of fixes from the EL-4 curl package:

    • fix zlib-compression causing curl to pass more than CURL_MAX_WRITE_SIZE bytes to write callback (CVE-2010-0734, Bug #563220)

    • add workaround for broken ProFTPD SSL auth (Bug #134133)

  • Updated libcurl7155 to handle move of kerberos installation prefix in krb5 ≥ 1.8, enable built-in manual for non-compat build (adding buildreq groff), comment patches in spec file and also merge a bunch of fixes from the EL-5 curl package:

    • fix zlib-compression causing curl to pass more than CURL_MAX_WRITE_SIZE bytes to write callback (CVE-2010-0734, Bug #563220)

    • fix curl-config missing vernum value (Bug #174556)

    • document lack of IPv6, FTPS and LDAP support if using a socks proxy (Bug #473128)

    • avoid tight loop if an upload connection is broken (Bug #479967)

    • add options --ftp-account/--ftp-alternative-to-user to program help (Bug #517084)

    • fix crash when reusing connection after negotiate-auth (Bug #517199)

    • add support for CRL loading from a PEM file (Bug #532069)

  • Updated ppp to add a manpage for pppoe-discovery

Thursday 8th April 2010

Local Packages

  • Updated perl-Compress-Raw-Bzip2 to 2.026 (no changes since 2.025)

  • Updated perl-Compress-Raw-Zlib to 2.026, reverting to building with the system zlib if it's version 1.2.1 or higher (required version 1.2.4 or higher with 2.025)

  • Updated perl-IO-Compress to 2.026; buildreq at least the same versions of Compress::Raw::Zlib and Compress::Raw::Bzip2

    • IO::Uncompress::Zip - some updates to documentation

    • IO::Uncompress::Zip - fixed default setting for ExtAttr

Monday 12th April 2010

Local Packages

  • New package perl-DBD-SQLite (1.29)

  • New package perl-DBI (1.609)

  • Updated perl-DBM-Deep to 1.0020, patching t/96_virtual_functions.t to skip everything due to its need for DBM::Deep::Engine::Test, which isn't shipped (CPAN RT#56512), adding another patch to support distributions with old Test::More versions (prior to 0.88, done_testing wasn't available), adding buildreqs perl(DBI) and perl(DBD::SQLite) and enabling SQLite tests

  • Updated perl-DateTime to update DateTime::TimeZone to 1.17 (updated Win32 to Olson name translation mapping - CPAN RT#56445)

Tuesday 13th April 2010

Local Packages

  • Updated kernel-advansys to 0.9.5

    • Add support for RHEL 5.4, RHEL 5.5
    • Add support for Fedora kernels up to 2.6.33
    • Note that for kernel 2.6.30 onwards, the firmware has been removed from the kernel module and is now found in the kernel-firmware package, which must be installed for the module to work)

  • Updated perl-Module-CoreList to 2.29 (updated for 5.12.0)

  • Updated perl-Test-CPAN-Meta to 0.17 (fixed qr// delimiters due to issues with the not sign symbol)

Wednesday 14th April 2010

Local Packages

  • Updated dovecot and pigeonhole (2.0 branch) to today's snapshots, as per Rawhide

  • Updated libssh2 to 1.2.5, adding buildreq /bin/hostname for build host detection and a patch to fix AES-CTR detection, though no Fedora/Red Hat distribution currently supports AES-CTR in OpenSSL

    • Add keep-alive support: libssh2_keepalive_config()/libssh2_keepalive_send()

    • Add libssh2_knownhost_addc(), libssh2_init() and libssh2_exit()

    • Add LIBSSH2_SFTP_S_IS***() macros

    • Fix memory leak in libssh2_session_startup()

    • Add missing error codes - shown as hangs in blocking mode
    • Fix memory leak in userauth_keyboard_interactive()

    • libssh2_knownhost_del: fix write to freed memory

    • Send and receive channel EOF before sending SSH_MSG_CHANNEL_CLOSE

    • Use AES-CTR from OpenSSL when available (this needed patching)

    • Fixed gettimeofday to compile with Visual C++ 6

    • NULL dereference when window adjusting a non-existing channel

    • Avoid using poll on Interix and MacOS X systems
    • Fix scp memory leak

    • Correctly clear blocking flag after sending multipart packet
    • Reduce used window sizes by factor 10
    • libssh2_userauth_publickey_fromfile_ex() handles a NULL password

    • sftp_init() deal with _libssh2_channel_write() short returns

  • Updated perl-DBM-Deep to add missing files for t/96_virtual_functions.t (CPAN RT#56512) and fix some spelling errors in the documentation (CPAN RT#56520)

Thursday 15th April 2010

Fedora Project

  • Updated perl-Sysadm-Install to 0.35 in devel branches:

    • Fix blurt_atomic on Win32 (CPAN RT#54885)

    • Fixed local caller_depth increments

    • Fixed printable() bug masking '-'

Local Packages

Friday 16th April 2010

Local Packages

  • Updated perl-Test-MinimumVersion to 0.101050, enabled RELEASE_TESTING, patched out overzealous requirement for ExtUtils::MakeMaker 6.31, and updated dependencies from META.yml

    • convert to dzil, fix a prereq num/str conversion stupidity

    • fix bug when comparing versions to undef

Monday 19th April 2010

Local Packages

  • Updated perl-DBM-Deep to 1.0021, dropping the upstreamed test suite and POD spelling patches

  • Updated perl-Test-MinimumVersion to 0.101080 (more "comparison to undef" bugs fixed)

  • Updated spamass-milter to fix a regression introduced with the update for CVE-2010-1132 in which the newly-added popenv() function spawned new processes that were never reaped, resulting in eventual resource starvation due to all of the zombie processes (Bug #583523, Debian Bug #575019)

Fedora Project

  • Updated spamass-milter as per my local package

Bacula Database Check

Yesterday's scheduled backup didn't happen automatically so I decided to kick it off manually, which worked fine. The backup database seems to be growing at quite a rate and little, if anything, seems to be being pruned. Given this background, I decided to give bacula's dbcheck database consistency check utility a whirl; I've not run this before even though I've been running bacula for years.

Anyway, I fired it up like this:

# dbcheck -c /etc/bacula/bacula-dir.conf -v -b -f

When it got to checking for orphaned Path entries it just seemed to hang, with one of the server's CPUs running mysqld at 100% for the best part of a day before I decided to kill dbcheck and restart mysqld. Googling around, it seemed that adding some indexes to some of the MySQL tables would improve things, so I did:

# mysql -u root --password='my-secret-password' bacula
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 5.1.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE INDEX file_pathid_idx on File(PathId);
mysql> CREATE INDEX file_filenameid_idx on File(FilenameId);
mysql> CREATE INDEX job_filesetid_idx on Job(FileSetId);
mysql> CREATE INDEX job_clientid_idx on Job(ClientId);
mysql> quit

The first of these index creation tasks also seemed to be taking forever, but not taking up any CPU time. I noticed in this morning's logwatch report that /var/lib/mysql had reached 100% and it dawned on me that mysqld was probably waiting on disk space being available rather than being busy creating the index. I increased the size of /var/lib/mysql using lvextend and resize2fs and the index creation completed shortly afterwards. The remaining indexes were created in a matter of minutes for File(FilenameId) and seconds for the others.

I then re-ran the dbcheck command and it ran interactively with virtually no delays of note, deleting over a million orphaned records. We'll see what effect, if any, that has on the catalog backup next time it runs.

<!> Update

The catalog backup was not noticeably smaller after deleting the orphaned records.

After updating to bacula 5.0.x, restores became horrendously slow, in particular the Building directory tree phase. It turned out that this was due to the extra indexes. Dropping the indexes on the File table again improved things to how they were with bacula 3.0.x:

# mysql -u root --password='my-secret-password' bacula
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 5.1.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show indexes from File; 
+-------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name            | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| File  |          0 | PRIMARY             |            1 | FileId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_pathid_idx     |            1 | PathId      | A         |      144231 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_filenameid_idx |            1 | FilenameId  | A         |      977569 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jobid_idx      |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx        |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx        |            2 | FilenameId  | A         |     8798128 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx        |            3 | PathId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
+-------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
7 rows in set (0.00 sec)

mysql> drop index file_pathid_idx on File;
drop index file_filenameid_idx on File;
Query OK, 17596256 rows affected (2 min 26.90 sec)
Records: 17596256  Duplicates: 0  Warnings: 0

mysql> drop index file_filenameid_idx on File;
Query OK, 17596256 rows affected (2 min 26.48 sec)
Records: 17596256  Duplicates: 0  Warnings: 0

mysql> show indexes from File; 
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name       | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| File  |          0 | PRIMARY        |            1 | FileId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jobid_idx |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx   |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx   |            2 | FilenameId  | A         |     8798128 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx   |            3 | PathId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
5 rows in set (0.00 sec)

mysql> quit

So next time I want to run dbcheck, I'll add the indexes back and then remove them again afterwards.

Tuesday 20th April 2010

Local Packages

  • Updated curl to sync patches with Rawhide and remove redundant compiler/linker flags from libcurl.pc

  • Updated perl-DateTime to update DateTime::TimeZone to 1.18

    • Based on version 2010i of the Olson database, with changes for Morocco, Taiwan (historical only), and Argentina
  • Updated perl-version to 0.82

    • Full version object support
    • Patch for bug in Perl 5.10.0 core version support
    • Sync with Perl 5.12.0
    • Vastly improved regex definitions
    • Completely rewrite pure Perl code to operate exactly like the XS code

Fedora Project

  • Raised Bug #584107 on curl: curl ≥ 7.20.0 introduced a --configure option in curl-config to display the program's build options, which is hard-coded into the script and results in a multilib file conflict in /usr/bin/curl-config; I attached a patch that resolves the issue in the usual way, putting the arch-specific bits into libcurl.pc and calling pkg-config from curl-config to display the text

Wednesday 21st April 2010

Local Packages

  • Updated perl-Module-CoreList to 2.31 (updated for 5.13.0)

  • Updated perl-Test-Pod to 1.43, dropping perl(Pod::Simple) version requirement to 3.05

Fedora Project

  • Took ownership of the orphaned EL-4 branch of perl-Params-Validate and updated it to 0.95

Thursday 22nd April 2010

Local Packages

  • Some updates for curl:

    • experimentally enable POSIX threaded DNS lookup instead of using c-ares (Rawhide builds only)

    • fix multilib conflict in curl-config --configure (Bug #584107)

    • tighten up dependency on libcurl from libcurl-devel to use %{?_isa}

    • replace Rawhide s390-sleep patch with a more targeted patch adding a delay after tests 513 and 514 rather than after all tests, significantly reducing the package build time

    • add patch disabling valgrind in test623 as it identifies a memory leak in libssh2 and breaks the build

  • Updated dovecot 2.0 to the 20100421 nightly build and added a new managesieve patch for a dovecot API change

    • mdbox: purge crashed if it purged all messages from a file
    • lib-storage: shared namespace's prefix_len updated after prefix truncated

    • imap-quota: iterate quota roots only once when replying to GETQUOTAROOT

    • idle: do cork/uncork when sending "OK Still here" notification
    • login: if proxy returns ssl=yes and no port, switch port to imaps/pop3s
  • Updated perl-MIME-tools to 5.428, adding buildreq perl(Test::Kwalitee) for additional test coverage

    • CPAN RT#56764: build release with a newer Module::Install

    • CPAN RT#52924: ensure we add <> around Content-id data

    • CPAN RT#48036: make mimesend example script a bit more useful

    • CPAN RT#43439: fix for parsing of doubled ; in multipart headers

    • CPAN RT#41632: if RFC-2231 and non-RFC-2231 parameters present, use only the RFC-2231 ones

    • CPAN RT#40715: reference Encode::MIME::Header in docs

    • CPAN RT#39985: correct POD typos

    • Only bind to localhost in smtpsend test, not all interfaces

Fedora Project

  • Became co-maintainer of curl and applied the package build speed-up patch in the devel branch

  • Updated perl-MIME-tools to 5.428 in devel branches

Friday 23rd April 2010

Local Packages

  • Updated perl-Archive-Tar to 1.60 (fix for CPAN RT#54850, making write() and create_archive() close only handles they opened)

Sunday 25th April 2010

Local Packages

  • Updated curl to apply upstream patches for failure of test536 with threaded DNS resolver and SSL handshake time-out underflow

Monday 26th April 2010

Local Packages

  • Updated perl-Compress-Raw-Bzip2, perl-Compress-Raw-Zlib, and perl-IO-Compress to 2.027 (update bundled zlib to 1.2.5, remove autoload code from Zlib.pm in Compress::Zlib - Perl RT#74088)

  • Updated tidyp to 1.00, adding the patch below to fix duplicate metatag insertion and adding a run of the self-tests during the build:

  • --- tidyp-1.00/src/lexer.c.orig 2010-04-25 06:49:19.000000000 +0100
    +++ tidyp-1.00/src/lexer.c      2010-04-26 12:13:55.239099722 +0100
    @@ -1373,7 +1373,7 @@
                         attval = TY_(AttrGetById)(node, TidyAttr_CONTENT);
     
                         if (AttrHasValue(attval) &&
    -                        TY_(tmbstrncasecmp)(attval->value, "HTML Tidy", 9) == 0)
    +                        TY_(tmbstrncasecmp)(attval->value, "tidyp ", 6) == 0)
                         {
                             /* update the existing content to reflect the */
                             /* actual version of Tidy currently being used */
  • Updated perl-HTML-Tidy to build against tidyp 1.00, adding the patch below to fix the broken test suite:

  • diff -ur HTML-Tidy-1.50.orig/t/roundtrip.t HTML-Tidy-1.50/t/roundtrip.t
    --- HTML-Tidy-1.50.orig/t/roundtrip.t   2010-02-16 15:48:40.000000000 +0000
    +++ HTML-Tidy-1.50/t/roundtrip.t        2010-04-26 12:01:59.946110426 +0100
    @@ -27,7 +27,7 @@
     is( scalar @messages, 0, q{The cleaned stuff shouldn't have any errors} );
     diag( 'But they do...', Dumper(\@messages) ) if @messages;
     
    -$clean =~ s/"HTML Tidy.+w3\.org"/"Tidy"/;
    +$clean =~ s/"(HTML Tidy|tidyp).+w3\.org"/"Tidy"/;
     
     my $expected = do { local $/ = undef; <DATA> };
     is( $clean, $expected, 'Cleaned up properly' );
    diff -ur HTML-Tidy-1.50.orig/t/unicode.t HTML-Tidy-1.50/t/unicode.t
    --- HTML-Tidy-1.50.orig/t/unicode.t     2010-04-26 11:54:07.607116011 +0100
    +++ HTML-Tidy-1.50/t/unicode.t  2010-04-26 12:02:47.593212175 +0100
    @@ -39,6 +39,7 @@
     ok(utf8::is_utf8($clean), 'cleaned output is also unicode');
     
     $clean =~ s/"HTML Tidy.+w3\.org"/"Tidy"/;
    +$clean =~ s/"(HTML Tidy|tidyp).+w3\.org"/"Tidy"/;
     is($clean, $reference, q{Cleanup didn't break anything});
     
     my @messages = $tidy->messages;
  • Updated perl-Test-Pod to 1.44 (use Module::Build::Compat's "traditional" configuration)

Wednesday 28th April 2010

Local Packages

  • Updated dovecot (2.0) to 20100427 nightly build:

    • doveconf <setting name> now prints only the one setting's value

    • mdbox: automatically delete old temp.* files from storage/ directory

    • mdbox: use flock locking by default

  • Updated libssh2 to get the OpenSSH test to run as part of the build and still work in an SELinux-aware environment (Bug #558911), adding buildreq openssh-server in the process

  • Updated perl-Pod-Simple 3.14

Thursday 29th April 2010

Local Packages

  • New package perl-Compress-Raw-Lzma (2.027); I had to patch this to get it to build against a recent xz (CPAN RT#57080)

  • New package perl-IO-Compress-Lzma (2.027)

I also updated curl to patch the test suite to get it to be able to start the OpenSSH server on a system with SELinux in enforcing mode (but only for an unconfined user) - Bug #521087. Now it might seem strange that this would be a problem for an unconfined user but OpenSSH is an SELinux-aware application, and takes different code paths depending on whether SELinux is enforcing or not. When starting a new session, it needs to set the correct SELinux security context for the session, and the possible options to choose from a different depending on the context OpenSSH is running in. When it's running unconfined, it doesn't get a valid context to start a session, and so it bombs out. So the trick is to get it to run in the regular sshd_t security context.

Processes can transition between SELinux domains when they exec another process but not all execs cause domain transitions. In the curl test suite running as unconfined_t, the sshd daemon (file content sshd_exec_t) is started but no domain transition is defined for that in policy so sshd continues running as unconfined_t. My patch adds an intermediate process (a perl script called initdaemon.pl, which is labelled with a file context of initrc_exec_t) that is used to start sshd. Now, when a process running in unconfined_t executes a file labelled initrc_exec_t, it transitions to the initrc_t domain, which is what system initscripts run as. And when that process subsequently runs sshd (file content sshd_exec_t), it transitions to sshd_t as desired.

That gets us part way there. The next problem is that sshd_t, being a confined domain, can no longer write its log files or PID files into the directory the curl build is running in (typically a user's home directory, or a subdirectory of it) because SELinux prevents the system process trampling over user data. So the other part of the patch is to:

  • Create a new subdirectory for the test suite to write PID files to, label it appropriately by using the same SELinux context type as the /var/run directory where system PID files usually reside, and persuade the test suite to put PID files in that directory

  • Make sure that the file that records the output of the sshd process is labelled with an SELinux context type that sshd_t can write to, for which I copied the context type of ~/.ssh (home_ssh_t on F-12)

The changes I've made don't seem to have broken anything else, so I'll look to push it upstream. First though, Kamil, the Fedora curl maintainer, will try it out on a variety of systems (both with and without SELinux) to check that it works as advertised and doesn't cause any regressions, which should happen some time next week.

Friday 30th April 2010

Local Packages

  • Updated dovecot (1.2.11) to update dovecot-sieve to 0.1.16:

    • extensions spamtest, spamtestplus and virustest implemented

    • vacation extension: from address set based on recipient alias

    • restructured and optimized the lexical scanner
    • added --with-docs configure option

    • fixed compile failures on certain systems
    • imapflags extension now deprecated

    • fixed various memset argument mixups in enotify extension

  • Updated perl-DBI to 1.611, adding a patch to fix the test suite for distributions with Test::More < 0.84 - the t/85gofer.t test uses the note function, which was introduced in Test::More 0.82 though the test requires 0.84 to be on the safe side; since note is just used for debugging logs, all the patch needed to do was to remove the strict version requirement and delete all the calls to note

Previous Month: March 2010
Next Month: May 2010

Recent