Paul's Blog Entries for April 2010

Thursday 1st April 2010

Local Packages

Friday 2nd April 2010

Local Packages

Sunday 4th April 2010

Local Packages

Monday 5th April 2010

Local Packages

Tuesday 6th April 2010

Local Packages

Wednesday 7th April 2010

Local Packages

Thursday 8th April 2010

Local Packages

Monday 12th April 2010

Local Packages

Tuesday 13th April 2010

Local Packages

Wednesday 14th April 2010

Local Packages

Thursday 15th April 2010

Fedora Project

Local Packages

Friday 16th April 2010

Local Packages

Monday 19th April 2010

Local Packages

Fedora Project

Bacula Database Check

Yesterday's scheduled backup didn't happen automatically so I decided to kick it off manually, which worked fine. The backup database seems to be growing at quite a rate and little, if anything, seems to be being pruned. Given this background, I decided to give bacula's dbcheck database consistency check utility a whirl; I've not run this before even though I've been running bacula for years.

Anyway, I fired it up like this: 

# dbcheck -c /etc/bacula/bacula-dir.conf -v -b -f

When it got to checking for orphaned Path entries it just seemed to hang, with one of the server's CPUs running mysqld at 100% for the best part of a day before I decided to kill dbcheck and restart mysqld. Googling around, it seemed that adding some indexes to some of the MySQL tables would improve things, so I did: 

# mysql -u root --password='my-secret-password' bacula
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 5.1.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE INDEX file_pathid_idx on File(PathId);
mysql> CREATE INDEX file_filenameid_idx on File(FilenameId);
mysql> CREATE INDEX job_filesetid_idx on Job(FileSetId);
mysql> CREATE INDEX job_clientid_idx on Job(ClientId);
mysql> quit

The first of these index creation tasks also seemed to be taking forever, but not taking up any CPU time. I noticed in this morning's logwatch report that /var/lib/mysql had reached 100% and it dawned on me that mysqld was probably waiting on disk space being available rather than being busy creating the index. I increased the size of /var/lib/mysql using lvextend and resize2fs and the index creation completed shortly afterwards. The remaining indexes were created in a matter of minutes for File(FilenameId) and seconds for the others.

I then re-ran the dbcheck command and it ran interactively with virtually no delays of note, deleting over a million orphaned records. We'll see what effect, if any, that has on the catalog backup next time it runs.

<!> Update

The catalog backup was not noticeably smaller after deleting the orphaned records.

After updating to bacula 5.0.x, restores became horrendously slow, in particular the Building directory tree phase. It turned out that this was due to the extra indexes. Dropping the indexes on the File table again improved things to how they were with bacula 3.0.x: 

# mysql -u root --password='my-secret-password' bacula
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 5.1.45 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show indexes from File; 
+-------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name            | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| File  |          0 | PRIMARY             |            1 | FileId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_pathid_idx     |            1 | PathId      | A         |      144231 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_filenameid_idx |            1 | FilenameId  | A         |      977569 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jobid_idx      |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx        |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx        |            2 | FilenameId  | A         |     8798128 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx        |            3 | PathId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
+-------+------------+---------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
7 rows in set (0.00 sec)

mysql> drop index file_pathid_idx on File;
drop index file_filenameid_idx on File;
Query OK, 17596256 rows affected (2 min 26.90 sec)
Records: 17596256  Duplicates: 0  Warnings: 0

mysql> drop index file_filenameid_idx on File;
Query OK, 17596256 rows affected (2 min 26.48 sec)
Records: 17596256  Duplicates: 0  Warnings: 0

mysql> show indexes from File; 
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name       | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| File  |          0 | PRIMARY        |            1 | FileId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jobid_idx |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx   |            1 | JobId       | A         |         411 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx   |            2 | FilenameId  | A         |     8798128 |     NULL | NULL   |      | BTREE      |         |
| File  |          1 | file_jpf_idx   |            3 | PathId      | A         |    17596256 |     NULL | NULL   |      | BTREE      |         |
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
5 rows in set (0.00 sec)

mysql> quit

So next time I want to run dbcheck, I'll add the indexes back and then remove them again afterwards.

Tuesday 20th April 2010

Local Packages

Fedora Project

Wednesday 21st April 2010

Local Packages

Fedora Project

Thursday 22nd April 2010

Local Packages

Fedora Project

Friday 23rd April 2010

Local Packages

Sunday 25th April 2010

Local Packages

Monday 26th April 2010

Local Packages

Wednesday 28th April 2010

Local Packages

Thursday 29th April 2010

Local Packages

I also updated curl to patch the test suite to get it to be able to start the OpenSSH server on a system with SELinux in enforcing mode (but only for an unconfined user) - Bug #521087. Now it might seem strange that this would be a problem for an unconfined user but OpenSSH is an SELinux-aware application, and takes different code paths depending on whether SELinux is enforcing or not. When starting a new session, it needs to set the correct SELinux security context for the session, and the possible options to choose from a different depending on the context OpenSSH is running in. When it's running unconfined, it doesn't get a valid context to start a session, and so it bombs out. So the trick is to get it to run in the regular sshd_t security context.

Processes can transition between SELinux domains when they exec another process but not all execs cause domain transitions. In the curl test suite running as unconfined_t, the sshd daemon (file content sshd_exec_t) is started but no domain transition is defined for that in policy so sshd continues running as unconfined_t. My patch adds an intermediate process (a perl script called initdaemon.pl, which is labelled with a file context of initrc_exec_t) that is used to start sshd. Now, when a process running in unconfined_t executes a file labelled initrc_exec_t, it transitions to the initrc_t domain, which is what system initscripts run as. And when that process subsequently runs sshd (file content sshd_exec_t), it transitions to sshd_t as desired.

That gets us part way there. The next problem is that sshd_t, being a confined domain, can no longer write its log files or PID files into the directory the curl build is running in (typically a user's home directory, or a subdirectory of it) because SELinux prevents the system process trampling over user data. So the other part of the patch is to:

The changes I've made don't seem to have broken anything else, so I'll look to push it upstream. First though, Kamil, the Fedora curl maintainer, will try it out on a variety of systems (both with and without SELinux) to check that it works as advertised and doesn't cause any regressions, which should happen some time next week.

Friday 30th April 2010

Local Packages

Previous Month: March 2010
Next Month: May 2010