Are Your Servers Secure???

Written By : Blessen Cherian

Published In : Print media-Magazine: DeveloperIQ
                            linuxgazette.net
                            http://www.linuxgazette.net/111/cherian.html

                                 In a word, No. No machine connected to the internet is 100% secure. This doesn't mean that you are helpless. You can take measures to avoid hacks, but you cannot avoid them completely. This is like a house — when the windows and doors are open then the probability of a thief coming in is high, but if the doors and windows are closed and locked the probability of being robbed is less, but still not nil.

1 What is Information Security?

For our purposes, Information Security means the methods we use to protect sensitive data from unauthorized users.

2 Why do we need Information Security?

The entire world is rapidly becoming IT enabled. Wherever you look, computer technology has revolutionized the way things operate. Some examples are airports, seaports, telecommunication industries, and TV broadcasting, all of which are thriving as a result of the use of IT. "IT is everywhere."

A lot of sensitive information passes through the Internet, such as credit card data, mission critical server passwords, and important files. There is always a chance of some one viewing and/or modifying the data while it is in transmission. There are countless horror stories of what happens when an outsider gets someone's credit card or financial information. He or she can use it in any way they like and could even destroy you and your business by taking or destroying all your assets. As we all know "An ounce of prevention beats a pound of cure," so to avoid such critical situations, it is advisable to have a good security policy and security implementation.

3 Security Framework

The following illustrates the framework needed to implement a functioning security implementation:

	 [ Risk Analysis ]   [ Business Requirements ]

                         |

                [ Security Policy ]

                         |

       [ Security Service, Mechanisms, and Objects ]

                         |

[ Security Management, Monitoring, Detection and Response ]

This framework shows the basic steps in the life cycle of securing a system. "Risk Analysis" deals with the risk associated with the data in the server to be secured. "Business Requirements" is the study which deals with the actual requirements for conducting business. These two components cover the business aspects of the security implementation.

The "Security Policy" covers 8 specific areas of the security implementation, and is discussed in more detail in section 4 below. "Security Service, Mechanisms and Objects" is actually the implementation part of security. "Security Management, Monitoring, Detection and Response" is the operational face of security, where we cover the specifics of how we find a security breach, and how we react if a breach is found.

4 Security Policy

The Security Policy is a document which addresses the following areas:

1. Authentication: This section deals with what methods are used to determine if a user is real or not, which users can or cannot access the system, the minimum length of password allowed, how long can a user be idle before he is logged out, etc.
2. Authorization: This area deals with classifying user levels and what each level is allowed to do on the system, which users can become root, etc.
3. Data Protection: Data protection deals with the details like what data should be protected and who can access which levels of data on the system.
4. Internet Access: This area deals with the details of the users having access to the internet and what they can do there.
5. Internet Services: This section deals with what services on the server are accessible from the internet and which are not.
6. Security Audit: This area addresses how audit and review of security related areas and processes will be done.
7. Incident Handling: This area addresses the steps and measures to be taken if there is a breach of security. This also covers the steps to find out the actual culprit and the methods to prevent future incidents.
8. Responsibilities: This part covers who will be contacted at any given stage of an incident and the responsibilities of the administrator(s) during and after the incident. This is a very important area, since the operation of the incident handling mechanism is dependent on it.

5 Types of Information Security

There are 2 types of security. (1) Physical security / Host Security and (2) Network security. Each of these sections has 3 parts:

1. Protection: Slow down or stop intrusions or damage
2. Detection: Alert someone if a breach (or attempted breach) of security occurs, and quantify and qualify what sort of damage occurred or would have occurred.
3. Recovery: Re-secure the system or data after the breach or damage and where possible, undo whatever damage occurred

5.1 Host Security / Physical Security

Host Security / Physical Security means securing the server from unauthorized access. For that we can password protect the box with such steps as setting up a bios password, placing the computer box in a locked room where only authorized users have access, applying OS security patches, and checking logs on regular basis for any intrusion and attacks. In Host security we check and correct the permissions on all OS related files.

5.2 Network security

Network security is one of the most important aspects of overall security. As I mentioned earlier, no machine connected to the internet is completely secure, so security administrators and server owners need to be alert, and make sure that they are informed of all new bugs and exploits that are discovered. Failure to keep up with these may leave you at the mercy of some script kiddy.

5.3 Which operating system is the most secure?

Every OS has its own pros and cons. There are ways to make Windows more secure, but the implementation is quite costly. Linux is stable and reasonably secure, but many companies perceive it as having little vendor support. My vote for the best OS for security purposes goes to FreeBSD, another free Unix-like OS, but not many people are aware of its existence.

6 Is a firewall the final solution to the Network Security problem?

No, a firewall is just a part of the security implementation. Again, we will use the example of a house. In a house all the windows and doors can be closed but if the lock on the front door of the house is so bad that someone can put just any key-like thing in and open it, then what is the use of the house being all closed up? Similarly, if we have a strong firewall policy, it will restrict unauthorized access, but if the software running on the box is outdated or full of bugs then crackers can use it to intrude into the server and gain root access. This shows that a firewall is not the final solution. A planned security implementation is the only real quality solution to this issue.

7 Security is a continuous process

Continuing security is a on-going process. Security administrators can only conduct their work on the basis of the alerts and bugfixes released up to the date of securing, so in order to accommodate all of the fixes for the latest bugs, security work has to be done on a regular basis.

8 Does Security implementation create overhead and/or reduce performance?

Yes, Security implementation creates a small amount of overhead, but it need not reduce overall performance drastically. In order to take care of such things, a well done security implementation has an optimization section where the security administration gives priority to both performance and security. While securing any software, we should secure it in such a way that it provides maximum performance.

9 Security Audits - What Should be Checked

A security audit is a part of security implementation where we try to find out the vulnerabilities of the system and suggest actions to improve the security. In a normal audit, the points below should be checked, and a report with the results of that audit should be created.

1. Check intrusion detection. Use chkrootkit or rkhunter for this purpose.
2. Check for known bugs in the software installed on the server - the kernel, openssl, openssh, etc.
3. Scan all network ports and find out which ports are open. Report the ports that should not be open and what program is listening on them.
4. Check whether /tmp is secured.
5. Check for hidden processes.
6. Check for bad disk blocks in all partitions. (This is just to make sure that the system is reasonably healthy.)
7. Check for unsafe file permissions.
8. Check whether the kernel has a ptrace vulnerability.
9. Check the memory (Another system health check.)
10. Check if the server is an open e-mail relay.
11. Check if the partitions have enough free space.
12. Check the size of the log files. It's better that the log size remains in megabytes.

10 How to know if you are being hacked?

To find out if your box is compromised or not, follow these steps. These are the steps which I used to do and will be handy in most of the situations.

10.1 Check your box to see if your performance has degraded or if your machine is being over used.

For that, use the commands

vmstat

Displays information about memory, cpu and disk.

Ex: bash# vmstat 1 4 (where 1 is delay and 4 is count)

mpstat

Displays statistics about cpu utilization. This will help us to see if your cpu is over worked or not.

Ex: bash# mpstat 1 4 (where 1 is delay and 4 is count)

iostat

This command displays statistics about the disk system.

Useful options:

-d - Gives the device utilization report.

-k - Display statistics in kilobytes per second.

Ex: bash# iostat -dk 1 4 (where 1 is delay and 4 is count)

sar

Displays overall system performance.

10.2 Check to see if your server has any hidden processes running.

ps

Displays the status of all known processes.

lsof

List all open files. In Linux everything is considered a file, so you will be able to see almost all of the activity on your system with this command.

10.3 Use Intrusion Detection Tools

• rkHunter ( http://www.rootkit.nl/ )
• chkrootkit ( http://www.chkrootkit.org/ )

10.4 Check your machine's uptime.

If the uptime is less than it should be, this can mean that your machine's resources are being used by someone. Linux doesn't crash or reboot under normal conditions because it is such a stable OS. If your machine has been rebooted try to find out the actual reason behind it.

10.5 Determine what your unknown processes are and what they are doing.

10.5.0.1 Use commands like the following to take apart unknown programs

readelf

This command will display what the executable's program is performing.

ldd

This command will show the details of libraries used by a executable.

string

This command will display the strings in the binary.

strace

This command will display the system calls a program makes as it runs.

11 Hardening Methodology

1. Read all security related sites and keep up to date. This is one of the main things a security administrator or server owner should do. Server owners should be made aware of security and its importance. Security training is an important part of an overall security package.
2. Create a good security policy. Conduct security audits on the basis of this policy.
3. Keep your OS updated by applying all patches.
4. Install a custom kernel with all unwanted services removed and patched with either grsecurity or openwall.
5. Disable all unwanted services and harden the services you leave running; Change file and directory permissions so that security is tightened.
6. Install a firewall and create good rule sets.
7. Test and audit the server on regular basis
8. Install an intrusion detection system, log monitor, all of the Apache security modules, bfd, faf and tmp monitor. Make your partitions secure.
9. Run a good backup system to recover data in case of an intrusion, crash, or other destructive incident.
10. Install a log analyzer and check your logs for any suspicious entries.
11. Install scripts to send out mail or enable notifications when a security breach occurs.
12. After a security breach try to find out how, when and through what the breach occurred. When you find a fix for it, document the details for future reference.

12 Summary

Now lets conclude by covering the main steps by which a hosting server can be secured.

12.1 Determine the business requirements and risk factors which are applicable to this system

12.2 Devise a security policy with the above data in mind. Get management's approval and signoff on this security policy.

12.3 On approval of the policy, do a security audit on any existing systems to determine the current vulnerabilities and submit a report regarding this to the management.

The report should also cover the methods needed to improve existing security. A quick checklist:

• Software Vulnerabilities.
• Kernel Upgrades and vulnerabilities.
• Check for any Trojans.
• Run chkrootkit.
• Check ports.
• Check for any hidden processes.
• Use audittools to check system.
• Check logs.
• Check binaries and RPMS.
• Check for open email relays.
• Check for malicious cron entries.
• Check /dev /tmp /var directories.
• Check whether backups are maintained.
• Check for unwanted users, groups, etc. on the system.
• Check for and disable any unneeded services.
• Locate malicious scripts.
• Querylog in DNS.
• Check for the suid scripts and nouser scripts.
• Check valid scripts in /tmp.
• Use intrusion detection tools.
• Check the system performance.
• Check memory performance (run memtest).

12.4 Implement the security policy

12.4.1 Correct all known existing software vulnerabilities either by applying patches or by upgrading the software.

12.4.2 Implement host security

• Protect your systems with passwords
• Check the file systems and set correct permissions and ownerships on all directories and files

chmod -R 700 /etc/rc.d/init.d/*

Use rpm -Va to find out if an rpm is modified

• Apply security patches to vulnerable software (ie. patch -p1 < patch file)
• Remove all unneeded ttys and console logins by removing the entry from /etc/securetty
• Check system logs (eg: /var/log/messages, /var/log/secure, etc.)
• Set a password on the boot loader (lilo and grub both support this)
• Monitor the system (nagios or big brother)

12.4.3 Implement Network security

• Remove all unwanted users and groups.
• Use custom security scripts which will send out notification when sshing as root or while creating a user with uid of 0, etc.
• Require passwords with 16 characters (can be done by making changes in login.def).
• Disable unwanted services using tcpwrapper (unwanted services can also be disabled through xinet.d or xinetd.Conf).
• Set up an idle timeout, so that idle users will be logged out after a certain amount of time.
• Disable all console program access (eg: rm -rf /etc/security/console.app/<service name >.)
• Enable nospoof option in /etc/host.conf.
• Specify the order in which domain names should be resolved (eg: order bind hosts).
• Lock the /etc/services file so that no one can modify it.
• Restrict direct root login (comment out the PermitRootLogin login option in sshd_config).
• Restrict su, so that only wheel group members are able to su. (can use pam or disable the permission of other for the su binary).
• Limit users resources (using pam, specify the limits for each user in /etc/security/limit.conf).
• Secure /tmp (mount /tmp with noexec,nodev,nosuid).
• Hide the server details. Remove /etc/issues and /etc/issues.net.
• Disable unwanted suid and sgid files (eg: find -type -perm -04000 -o perm 02000.)

Examples of these: gpasswd, wall, and traceroute

• Using iptables, allow only pings from a specific locations (for monitoring systems to work).
• Take preventive measures against DOS, "ping of death" attacks, etc.
• Install a firewall (eg: apf and iptables) and only allow ports to operate which the box needs for its normal functions; block all other ports to prevent mischief.

Links: http://rfxnetworks.com/ and http://yolinux.com/TUTORIALS/LinuxTutorialIptablesNetworkGateway.html

• Install intrusion detection (eg: install tripwire or aide).

Links: http://www.cs.tut.fi/rammer/aide.html and http://redhat.com/docs/manuals/linux/RHL-9-Manual/ref-guide/ch-tripwire.html

• Install sxid to keep an eye on suid and sgid scripts.

Link: http://linux.cudeso.be/linuxdoc/sxid.php

• Restrict ssh to specific IP addresses and specific users (I suggest key authentication using passphrase).
• Install logcheck to check the logs.
• Install tmpwatch to delete the unused files from /tmp directory.
• Install and setup portsentry and configure it to use iptables to block IPs.
• Install mod_security and mod_dosevasive to safe guard apache.
• Delete files with nouser and nogroup.
• Deleted unwanted files/folders in htdocs, disable directory indexing.
• Check for unwanted scripts in /root, /usr/local, /var/spool/mbox.
• Install BFD and FAF for additional security.
• Disable open email relaying.
• Submit a status report to management detailing all discovered vulnerabilities and fixes.

12.5 Testing phase

Use tools like nessus, nikto, and nmap to do a penetration test and see how well your server is secured. Also do a stress test.

Security is of utmost importance to a server, compromising security is compromising the server itself. Hence, an understanding of the same is a prerequisite to server ownership and administration.

By
Blessen Cherian ,
Member of Eecutive team,
Poornam Info Vision Pvt Ltd
Bobcares.com, Poornam.com, Blessen.com