Don't Leave the Barn Door Open: Troubleshooting SELinux Issues
Recently while I was working on strengthening the security of an online password reset tool, I ran into an a pair of file permission issues related to SELinux. I've witnessed many system administrators and engineers throw up their hands when they encounter a problem like this, and turn off SELinux policy enforcement, weakening the security posture of the system. Instead of doing that, I added a new SELinux policy to allow just the operations my new application needed.
I was busy integrating cracklib into a password changing application written in PHP, to be deployed on a Red Hat Enterprise Linux 5 server running Apache 2.2. Cracklib is a library that helps protect you from weak passwords, by performing dictionary checks and other transformations to your clear text password before it gets stored in an authentication system. When I ran the password checking functions from the command line, the program had expected results, but I had multiple problems getting the functions to run from within the web server.
The first problem was that the functions in the PHP extension crack.so were not found by PHP when running as part of the web server, despite being installed by pear. The web server error log had this message, shedding some light on the problem:
[Fri Jan 08 17:19:45 2010] [error] [client 127.0.0.1] PHP Fatal error: Call to undefined function crack_opendict() in /var/www/html/password/include/backend_functions.inc.php on line 270, referer: https://www.example.com/password/
I knew that this system was running SELinux, and it was turned on, in "Enforcing" mode. When run from the command line as my user, the program was run in the context of an unconfined process, but when run as the web server, it had to comply with SELinux restrictions.
SELinux, a mandatory access control technology originally introduced by the National Security Agency, is now implemented in many Linux distributions, such as Fedora Linux and its offshoots, including Red Hat Enterprise Linux. SELinux can protect your server from unknown attacks, as I demonstrated in a Linux Journal article, Mambo Exploit Blocked by SELinux.
The basic idea behind SELinux is that the system maintains a set of security labels for files and processes, and a set of security policies that define what operations and state transitions are allowed. The security policies can prevent processes from reading or altering files that they normally would not have any reason to touch. When a process attempts to use a file, the system determines whether the currently running policy allows it.
To avoid the trap where you shut off SELinux at the first permission denied error, you must understand how to troubleshoot and mitigate SELinux permission denied issues. This process consists of examining the audit logs and file security labels, and taking actions that might include resetting SELinux security labels, running tools that provide additional information, and if required, altering the security policies of the system to allow additional state transitions from one security label to another to complete.
The first step is to examine system log files for clues. Whenever you get a mysterious permission denied error on a system running SELinux, it pays to check the system audit log, in addition to the usual web server error logs and /var/log/messages. The SELinux system audit log, /var/log/audit/audit.log had this message:
type=AVC msg=audit(1262987478.811:653174): avc: denied { read } for pid=2806 comm="httpd" name="crack.so" dev=dm-0 ino=18253929 scontext=system_u:system_r:httpd_t:s0 tcontext=user_u:object_r:tmp_t:s0 tclass=file
This message indicates that the httpd process could not use a file that had a label with the tmp_t type. On inspection, it turned out that the crack.so shared object did not have the correct SELinux security label to allow it to be loaded by httpd. The pear command used to compile and load the extension was not SELinux-aware, and moved the file into place from some temporary directory. Here's how I inspected the labels and fixed the label on crack.so:
[root@localhost ~]# cd /usr/lib64/php/modules
[root@localhost modules]# ls -laZ
drwxr-xr-x root root system_u:object_r:lib_t .
drwxr-xr-x root root system_u:object_r:lib_t ..
-rwxr-xr-x root root user_u:object_r:tmp_t crack.so
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t dbase.so
[root@localhost modules]# restorecon crack.so
[root@localhost modules]# ls -lZ crack.so
-rwxr-xr-x root root system_u:object_r:textrel_shlib_t crack.so
Once this was done, the PHP script still did not run correctly. The second problem was that once the system could load the crack.so cracklib libraries, it could not open the dictionary files. The web server error log had this clue:
[Mon Jan 11 09:48:25 2010] [error] [client 173.79.175.45] PHP Warning: crack_opendict() [<a href='function.crack-opendict'>function.crack-opendict</a>]: Could not open crack dictionary: /usr/share/cracklib/pw_dict in /var/www/html/password/include/backend_functions.inc.php on line 270, referer: https://www.example.com/password/
Since SELinux permissions had already been an issue in getting this to work, I inspected the audit.log file once again. Sure enough, it had an entry like this:
type=AVC msg=audit(1263219675.417:656468): avc: denied { search } for pid=7014 comm="httpd" name="cracklib" dev=dm-0 ino=6030531 scontext=user_u:system_r:httpd_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir
This indicated that the httpd process was having trouble reading files with the crack_db_t security label. That is how the cracklib dictionary files are labeled:
[someuser@localhost ~]$ ls -laZ /usr/share/cracklib
drwxr-xr-x root root system_u:object_r:crack_db_t .
drwxr-xr-x root root system_u:object_r:usr_t ..
-rw-r--r-- root root system_u:object_r:crack_db_t cracklib.magic
-rw-r--r-- root root system_u:object_r:crack_db_t pw_dict.hwm
-rw-r--r-- root root system_u:object_r:crack_db_t pw_dict.pwd
-rw-r--r-- root root system_u:object_r:crack_db_t pw_dict.pwi
Relabeling these files in order to get around the SELinux issue might have caused other things to break, so I decided to attempt to extend the SELinux policy so that the web server could perform the operations it needed on the files and directories labeled "crack_db_t".
I then did an experiment in setting the SELinux mode to "Permissive" temporarily to see if that would allow the program to run. That can also provide useful log entries for audit2allow, a tool that gives policy change suggestions when fed audit log entries. Be aware that changing the settings from "Enforcing" to "Permissive" on a production system is a security risk, and if you must do that, you should strive to minimize the amount of time that the system is set to "Permissive" and then examine the audit logs for that period carefully to check for signs of unauthorized activities. Another good strategy to help mitigate this problem is to attempt to debug the problem on a test system similar to the target system that is not connected to the Internet, so that you can avoid changing SELinux out of "Enforcing" mode.
After configuring SELinux to be in permissive mode ("/usr/sbin/setenforce Permissive"), the application ran as expected. I immediately reset the security context to enforcing ("/usr/sbin/setenforce 1") and proceeded to analyse the audit log output, by feeding it to the audit2allow program:
[root@localhost ~]# audit2allow < /var/log/audit/audit.log
#============= httpd_t ==============
allow httpd_t crack_db_t:dir search;
allow httpd_t crack_db_t:file { read getattr };
#============= named_t ==============
allow named_t named_zone_t:dir write;
#============= system_mail_t ==============
allow system_mail_t httpd_tmp_t:file { read write };
These policy suggestions related to more than just the httpd program, but since the application was a PHP web application, it is safe to ignore the named and mail related policy suggestions. I then created a type enforcement SELinux source file from the httpd_t suggestions, and called it httpdcracklib.te:
require {
class dir { search };
class file { read getattr };
type crack_db_t;
type httpd_t;
};
#============= httpd_t ==============
allow httpd_t crack_db_t:dir search;
allow httpd_t crack_db_t:file { read getattr };
Note that I had to enumerate the classes and files that the policy required in a "require" stanza before the policy suggestions. Without this, the policy compiler won't be able to resolve those symbols.
The SELinux policy development files, in the RPM selinux-policy-devel, have a Makefile that simplifies the creation and maintenance of these policies. To use it, you can run "make -f /usr/share/selinux/devel/Makefile" in the directory that has your policy files in it. To simplify this further, I created a Makefile that I could check into version control:
all:
@make -f /usr/share/selinux/devel/Makefile $@
clean:
@rm -rf *.fc *.if *.pp tmp
Then, running 'make' creates the policy file:
[someuser@localhost selinux]$ make
make[1]: Entering directory `/home/someuser/src/password_change/selinux'
Compiling targeted httpdcracklib module
/usr/bin/checkmodule: loading policy configuration from tmp/httpdcracklib.tmp
/usr/bin/checkmodule: policy configuration loaded
/usr/bin/checkmodule: writing binary representation (version 6) to tmp/httpdcracklib.mod
Creating targeted httpdcracklib.pp policy package
rm tmp/httpdcracklib.mod tmp/httpdcracklib.mod.fc
make[1]: Leaving directory `/home/someuser/src/password_change/selinux'
[someuser@localhost selinux]$ ls -l
total 44
-rw-rw-r-- 1 someuser someuser 0 Jan 11 12:10 httpdcracklib.fc
-rw-rw-r-- 1 someuser someuser 0 Jan 11 12:10 httpdcracklib.if
-rw-rw-r-- 1 someuser someuser 7803 Jan 11 12:10 httpdcracklib.pp
-rw-r--r-- 1 someuser someuser 280 Jan 11 09:09 httpdcracklib.te
-rw-rw-r-- 1 someuser someuser 88 Dec 10 21:25 Makefile
drwxrwxr-x 2 someuser someuser 4096 Jan 11 12:10 tmp
It is also possible to shortcut this process by running "audit2allow -m httpdcracklib </var/log/secure", and audit2allow will generate the require stanza for you, and compile the module for you. However, that does not give you the level of control and repeatability you get if you keep the policy sources in version control, and drive the policy creation through a Makefile.
Installing the policy module is pretty easy then
$ sudo /usr/sbin/semodule -i httpdcracklib.pp
After that, the PHP application behaved as expected, even with SELinux set to enforcing mode.
Going through these troubleshooting steps does take more time than disabling SELinux, but it is often worth the effort, given the higher levels of protection that SELinux can give a server.
- rbulling's blog
- Login to post comments
