1. 2 servers with similar hard disk setup (These will be used to create a redundant nfs server)
2. atleast 1 server where the nfs share will be mounted.
3. Static IPs
4. Basic knowledge of vi (:q! = quit, :wq = write and then quit, i = insert mode, esc = leave insert mode, dd = delete line when not in insert mode)

First off, install CentOS on both machines.  During the install process, create a separate blank partition on both machines to be used as your nfs mount.  Set the mount point to /data during installation.

From this point on i’m going to be referring to both nfs servers by their IPs and hostnames.  Server1 will be nfs1 with ip 10.132.196.221 and server2 will be nfs2 with ip 10.132.196.222.  Your private IPs might be different so make sure to put in the correct IPs where necessary within this tutorial.

Do the following on nfs1(10.132.196.221) and nfs2(10.132.196.222):

vi /etc/fstab

This will give you the mount points and devices on your system.  Look for the /data mount point and comment it out to prevent it form automatically being mounted on boot.  Take note of the device for the /data mount point.  Here is what my fstab looks like.

/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
#/dev/VolGroup00/LogVol02 /data                   ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

I’m using LVM so my device names are weird.  Yours might just be /dev/sda1, /dev/sda2, etc…

Now lets go ahead an unmount that data partition because heartbeat will take care of mounting the partition on the live nfs server.

umount /data

Make sure that ntp and ntpdate are installed on both of the nfs servers.

yum install ntp ntpdate

The time on both servers must be identical.

Now lets check and make sure that the nfs service is not running on startup and that selinux is also turned off.

setup

Go down to “Firewall Configuration” and disable selinux and the firewall.  Next, lets go to “System Services” and make sure that the nfs service is not enabled.  You will need to reboot your machines for these settings to take affect.

Lets go ahead and create our exports for the nfs so that the nfs share can be mounted on other machines in your network.

vi /etc/exports

Add the following line to your exports file but make sure to replace the IP.  My private IPs are in the form of 10.132.196.1. Yours may be 192.168.0.1.

/data/export/ 10.132.196.0/255.255.255.0(rw,no_root_squash,no_all_squash,sync)

The above line in my exports file will allow me to mount the nfs share anywhere within my local network.  If you only want to allow a specific machine to be able to mount the nfs share then use a specific IP instead of the 0 at the end.  For example, here is an exports file that allows only 10.132.196.24 to mount the nfs share.

/data/export/ 10.132.196.24/255.255.255.0(rw,no_root_squash,no_all_squash,sync)

Now we need to install DRBD and the DRBD kernel module.

yum install drbd kmod-drbd

After installing drbd we need to setup the config file for it.

vi /etc/drbd.conf

Let me show you my config file and then we’ll go over it.

common {
protocol C;

syncer {
rate         15M;
al-extents    257;
}
}

resource r0 {

handlers {
pri-on-incon-degr    “halt -f”;
}

disk {
on-io-error    detach;
}

startup {
degr-wfc-timeout 120;
}

on nfs1 {
device        /dev/drbd0;
disk        /dev/VolGroup00/LogVol02;
address        10.132.196.221:7789;
meta-disk    internal;
}

on nfs2 {
device        /dev/drbd0;
disk        /dev/VolGroup00/LogVol02;
address        10.132.196.222:7789;
meta-disk    internal;
}
}

So lets start from the top.

• Protocol – This is the method that drbd will use to sync both of the nfs servers.  There are 3 available options here, Protocol A, Protocol B and Protocol C.Protocol A is an asynchronous replication protocol.  The drbd.org manual states, “local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has been placed in the local TCP send buffer. In the event of forced fail-over, data loss may occur. The data on the standby node is consistent after fail-over, however, the most recent updates performed prior to the crash could be lost.”Protocol B is a memory synchronous (semi-synchronous) replication protocol.  The drbd.org manual states, “local write operations on the primary node are considered completed as soon as the local disk write has occurred, and the replication packet has reached the peer node. Normally, no writes are lost in case of forced fail-over. However, in the event of simultaneous power failure on both nodes and concurrent, irreversible destruction of the primary’s data store, the most recent writes completed on the primary may be lost.”
  Protocol C is a synchronous replication protocol.  The drbd.org manual states, “local write operations on the primary node are considered completed only after both the local and the remote disk write have been confirmed. As a result, loss of a single node is guaranteed not to lead to any data loss. Data loss is, of course, inevitable even with this replication protocol if both nodes (or their storage subsystems) are irreversibly destroyed at the same time.

  You may choose your desired protocol but Protocol C is the most commonly used one and it is the safest method.

• rate – The rate is the maximum speed at which data will be sent from one nfs server to the other while syncing.  This should be about a third of your maximum write speed.  In my case, I have only a single disk that can write about 45mb/sec so a third of that would be 15mb.  This number will usually be much higher for people with raid setups.  In some large raid setups, the bottleneck would be the network and not the disks so set the rate accordingly.
• al-extent – This data on the disk are cut up into slices for synchronization purposes.  For each slice there is an al-extent that is used to indicate any changes to that slice.  Larger al-extent values make synchronization slower but benefit from less writes to the metadata partition.  In my case, I’m using an internal metadata which means the drbd metadata is written to the same parition that my nfs data is on.  It would benefit me to have less metadata writes to prevent the disk arm from constantly moving back and forth and degrading performance.  If you are using a raid setup and a separate partition for the metadata then set this number lower to benefit from faster synchronization.  This number MUST be a prime to gain the most possible performance because it is used in specific hashes that benefit from prime number sized structures.
• pri-on-incon-degr – The “halt -f” command is executed if the node is primary, degraded and if the data is inconsistent.  I use this to make sure drbd is halted when there is some sort of data inconsistency to prevent a major mess from occuring.
• on-io-error – This allows you to handle low level I/O errors.  The method I use is the “detach” method.  This is the recommended option by drbd.org. On the occurrence of a lower-level I/O error, the node drops its backing device, and continues in diskless mode.
• degr-wfc-timeout – This is the amount of time in seconds that is allowed before a connection is timed out.  In case a degraded cluster (cluster with only one node left) is rebooted, this timeout value is used instead of wfc-timeout, because the peer is less likely to show up in time, if it had been dead before.

The rest of the config is pretty self explanatory.  Replace nfs1 and nfs2 with the hostnames of your nfs servers.  To get the hostnames use the following command on both servers:

uname -n

Then replace the disk value with the device name from your fstab file that you commented out.  Enter the IP address of each server and use port 7789.  The last part is the meta-disk.  I used an internal meta-disk because I only have one hard disk in the server and it would not give me any benefit to create a separate partition for the metadata.  If you have a raid setup or a separate disk from your data partition that you can use for the meta data than go ahead and create a 150mb partition.  Replace the word “internal” in the config file with your device name that you used for the meta data partition.

Now that we finally have our drbd.conf file ready we can move on.  Lets go ahead and enable the drbd kernel module.

modprobe drbd

Now that the kernel module is enabled lets start up drbd.

drbdadm up all

This will start drbd, now lets check its status.

cat /proc/drbd

You can always use the above command to check the status of drbd.  The above command should show you something like this.

0: cs:Connected st:Secondary/Secondary ld:Inconsistent
ns:0 nr:0 dw:0 dr:0 al:0 bm:1548 lo:0 pe:0 ua:0 ap:0
1: cs:Unconfigured

You should get some more data before it but the above part is what we are interested in.  If you notice it shows that drbd is connected and both nodes are in secondary mode.  This is because we have not assigned which node is going to be the primary yet.  It also says the data is inconsistent because we have not done the initial sync yet.

I am going to set nfs1 to be my primary node and nfs2 to be my secondary node.  If nfs1 fails, nfs2 will takeover but if nfs1 comes back online then all the data from nfs2 will be synced back to nfs1 and nfs1 will take over again.

First of all lets go ahead and delete any data that was created on the /data partition that we setup during our intial OS installation.  Be very careful with the command below.  Make sure to use the appropriate device because all data on that device will be lost.

dd if=/dev/zero bs=1M count=1 of=/dev/VolGroup00/LogVol02; sync

Instead of “/dev/VolGroup00/LogVol02″, replace it with your device for the /data parition.  Now that the partition is completely erased on both servers, lets create the meta data.

drbdadm create-md r0

Do the following ONLY on nfs1(10.132.196.221)

Now that the metadata is created, we can move onto assigning a primary node and conducting the initial sync.  It is absolutely important that you only execute the following command on the primary node.  It doesn’t matter which node you choose to be the primary since they should be identical.  In my case, I decided to use nfs1 as the primary.

drbdadm — –overwrite-data-of-peer primary r0

Ok, now we just have to sit back and wait for the initial sync to finish.  This is going to take some time to finish even though there is no data on each device, drbd has to sync every single block on /data partition from nfs1 to nfs2.  You can check the status by using the following command.

cat /proc/drbd

Do the following on nfs1(10.132.196.221) and nfs2(10.132.196.222):

After the initial sync is finished, “cat /proc/drbd” should show something like this.

0: cs:Connected st:Primary/Secondary ld:Consistent
ns:37139 nr:0 dw:0 dr:49035 al:0 bm:6 lo:0 pe:0 ua:0 ap:0
1: cs:Unconfigured

If you notice, we are still connected and have a primary and secondary node with consistent data.

Do the following ONLY on nfs1(10.132.196.221):

Now lets make an ext3 file system on our drbd device and mount it.  Since drbd is running, the ext3 file system will also be created on the secondary node.

mkfs.ext3 /dev/drbd0

The above command will create an ext3 file system on the drbd device.  Now lets go ahead and mount it.

mount -t ext3 /dev/drbd0 /data

NFS has important information that it stores in /var/lib/nfs that is required to function properly.  In order to preserve file locks and other such information, we need to have that data stored on the drbd device so that if the primary node failes, NFS on the secondary node will continue from right where the primary node left off.

mv /var/lib/nfs/ /data/
ln -s /data/nfs/ /var/lib/nfs
mkdir /data/export
umount /data

So lets go over what we just did.  We moved the nfs folder from /var/lib to /data.  Then we created a symbolic link from /var/lib/nfs to /data/nfs since the operating system is still going to look for /var/lib/nfs when nfs is running.  Then we created an export directory in /data to store all the actual data that we are going to use for our nfs share.  Finally, we un-mounted the /data partition since we finished what we were doing.

Do the following ONLY on nfs2(10.132.196.222):

Since we moved the nfs folder to /data, that was synced over to the secondary node as well.  We just need to create the symbolic link so that when the /data partition is mounted on nfs2 we have a link to the nfs data.

rm -rf /var/lib/nfs/
ln -s /data/nfs/ /var/lib/nfs

So we removed the nfs folder and created a symbolic link from /var/lib/nfs to /data/nfs.  The symbolic link will be broken since the /data parition is not mounted.  Don’t worry about that because in the event of a failover that partiton will be mounted and everything will work just fine =).

Do the following on nfs1(10.132.196.221) and nfs2(10.132.196.222):

Now onto heartbeat.  Heartbeat is going to make sure partitions are umount/mount and services are started/stopped in the even of a fail over.  So lets get to it.

yum install heartbeat

Ok, now that we have heartbeat installed, lets go ahead and create our 3 necessary config files.

vi /etc/ha.d/ha.cf

Paste the following data into ha.cf and save it (:wq).

logfacility     local0
keepalive 2
deadtime 10
bcast   eth0
node nfs1 nfs2

Replace nfs1 and nfs2 with your server hostnames.  You can retrieve the hostname for each server by executing the following command.

uname -n

Now lets create our resource config file.

vi /etc/ha.d/haresources

Put the following data in there and save it.

nfs1  IPaddr::10.132.196.220/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfslock nfs

The first word is the hostname for the primary server and should be identical on both servers.  I have chosen nfs1 to be my primary server.  The net part is the virtual IP.  This is the virtual IP we are going to use for the live nfs server, whether it be nfs1 or nfs2.  This ip can be any IP that is not being used within your network.  For example, if your nfs servers have IPs 192.168.0.11 and 192.168.0.12 then maybe you can use 192.168.0.10 as your virtual IP.  Its up to you.

Finally lets create our authentication file.

vi /etc/ha.d/authkeys

Put the following data in that file and save it.

auth 3
3 md5 somepassword12345

Replace “somepassword12345″ with your own password.  This will be used by both of the heartbeat daemons on nfs1 and nfs2 to authenticate each other.  The filw should be read-only by root so lets go ahead and do that.

chmod 600 /etc/ha.d/authkeys

Thats it!  Lets just start drbd and heartbeat on both servers now.

/etc/init.d/drbd start
/etc/init.d/heartbeat start

Now we have a redundant NFS server running!  Lets do a couple tests on the primary nfs server.

ifconfig

We should see our virtual IP address show up.  Mine looks like this.

eth0      Link encap:Ethernet  HWaddr 00:14:22:7C:65:6B
inet addr:10.132.196.221  Bcast:10.132.196.255  Mask:255.255.255.0
inet6 addr: fe80::214:22ff:fe7c:656b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:897038670 errors:0 dropped:367 overruns:0 frame:0
TX packets:1204564630 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:245125213396 (228.2 GiB)  TX bytes:1194659917566 (1.0 TiB)
Interrupt:169

eth0:0    Link encap:Ethernet  HWaddr 00:14:22:7C:65:6B
inet addr:10.132.196.220  Bcast:10.132.196.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Interrupt:169

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:372 errors:0 dropped:0 overruns:0 frame:0
TX packets:372 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:50327 (49.1 KiB)  TX bytes:50327 (49.1 KiB)

If you notice, eth0:0 has the virtual IP that I used for the nfs servers.  You should only see this on the nfs server that is live.

Now lets check our partitions.

df -h

The primary nfs server should show the /data mounted while the secondary nfs should not show the drbd device mounted.  My primary nfs server looks like this.

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
15G  2.5G   11G  19% /
/dev/sda1              99M   19M   76M  20% /boot
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/drbd0            259G  172G   75G  70% /data

Now lets go ahead and mount the nfs share on another server.  First lets create the /data folder on the new server.  This doesn’t have to be named “data” and can be named whatever you like.

mkdir /data

Now lets setup our nfs mount.

vi /etc/fstab

At the end of the file, add the following line.

10.132.196.220:/data/export  /data    nfs          rw            0    0

Replace the IP address with the virtual IP address that you chose for your nfs servers.  Now lets mount the partition for the first time.

mount /data

Thats it!  Now you have a redundant nfs server and client actually using a redundant nfs server.  Just to simulate a failover lets test some stuff out.  Go ahead and create some files in the “/data” folder from your nfs client machine.

cd /data
touch testfile1.txt
mkdir testdirectory

Now that we have some data in the “/data” folder we can simulate a failed nfs server.  If we did everything right, the data that we just created was created on the primary nfs server and synced to the secondary nfs server via drbd.  Lets stop heartbeat on nfs1 so that nfs2 thinks that nfs1 has failed.

/etc/init.d/heatbeat stop

Now that heartbeat is stopped on nfs1 run the following commands to make sure that the /data partition was unmounted and the virtual IP is gone.

df -h
ifconfig

When you check the same thing on nfs2, you should see that the /data partition has been mounted and the virtual IP is now live.

Now if you go back to your nfs client machine and do an “ls” in the /data directory, you should see that your data is still there.  Lets change our test data around.

cd /data
mv testfile1.txt testfile2.txt
rm -rf testdirectory

Now lets go back to nfs1 and start up heartbeat again.

/etc/init.d/heatbeat start

Give it a couple seconds and check the partition and virtual IP.

df -h
ifconfig

You should see that the partition is mounted again and the virtual IP is also live on nfs1.  If you check the same thing on nfs2, the partition will be un-mounted and the virtual IP should be gone.  If you check the /data directory on your nfs client machine, you should see that the “testfile2.txt” file is still there.

Congratulations, you have a fully functional and highly available nfs server!  Check out http://www.drbd.org for more information on DRBD.

This totally caught me by surprise as I had never seen this before.  I had just finished installing awstats and wanted to run an update for the first time.  The logs were gathered from 6 webservers and each webserver had about 225 logs.

The awstats log merging script tried to open these log files (225×6 = 1,350 logs) and merge them to read the data but it kept crashing.  The reason it kept crashing was because the limit for the maximum number of files that you are allowed to open in a shell was set to 1024.

To check what your limits are just type in the following command:

ulimit -a

To change the limit for the number of files you are allowed to open, change the “open files” limit.  To do so, type in the following command:

ulimit -n3000

This will set the limit to 3000 files.  You can set it to whatever number you need to.

After changing my “open files” limit, I was able to run my initial awstats update without any issues.

1.  Create a text file with the following information in it.

UPDATE mysql.user SET Password=PASSWORD(‘NEW_PASSWORD’) WHERE User=’root’;
FLUSH PRIVILEGES;

Replace “NEW_PASSWORD” with your desired mysql password.  Save the file to /etc/mysql-pass-reset

2.  Stop MySQL

/etc/init.d/mysqld stop

3.  Start mysqld_safe with the –init-file option like so:

mysqld_safe –init-file=/etc/mysql-pass-reset &

This will start mysql and execute the query in the text file you created.

4.  Now stop and start mysql back up normally without the –init-file option.

/etc/init.d/mysqld stop
/etc/init.d/mysqld start

Don’t forget to delete the “mysql-pass-reset” file when you are done!

rm /etc/mysql-pass-reset

First you will need to setup Java on your computer so lets go ahead and do that. Open up the console and login as super user.

su

Now that we have full privileges, lets edit the sources.list file so that we can fetch the much needed java packages.

pico /etc/apt/sources.list

This will bring up your sources.list file in the editor. You should see something that looks like this:

deb http://debian.lcs.mit.edu/debian etch main
deb-src http://debian.lcs.mit.edu/debian etch main

Add the words “contrib non-free” at the end of both lines so it looks like the following:

deb http://debian.lcs.mit.edu/debian etch main contrib non-free
deb-src http://debian.lcs.mit.edu/debian etch main contrib non-free

Now hit “Ctrl + O” to write the changes to the file. Then hit enter to accept the changes. Now hit “Ctrl + X” to exit the editor. That was the hard part . Lets update our package listing and download the java packages by running the following commands:

apt-get update
apt-get install sun-java5-jre sun-java5-jdk

Follow the onscreeen instructions to complete the installation of the java packages. Its pretty much gonna ask you to agree to the licenses. After you are done you can close the console window.

Now log in to your TOS account and download their software for Linux. Save it to your desktop. Once the download is complete you should have a shell script file sitting on your desktop. The extension of the file should be .sh. Go ahead and open your console again.

Type in the following command to change your current working directory to your desktop directory:

cd Desktop

Now lets make the installation script executable.

chmod +x thinkorswim_installer.sh

We are ready to install the software now. Just type in the following command.

./thinkorswim_installer.sh

If you get an xserver error just run the command again with the “-c” argument like this:

./thinkorswim_installer.sh -c

Now just follow the on screen instructions. Let it install in the default directory “/usr/local/thinkorswim”. Once that is done installing we need to change the permissions on the installation folder so that it doesn’t stall on the update page. To do this we need super user privileges.

su

Now lets change the permissions on the entire thinkorswim folder recursively.

chmod 777 /usr/local/thinkorswim -R

Thats it, you are all set now. You can run the program by navigating to /usr/local/thinkorswim and clicking on the thinkorswim executable file.

You can also create a shortcut in your start menu. In KDE just right click on the kmenu and go to edit. Add a new menu item and enter the following in the path field:

/usr/local/thinkorswim/thinkorswim.sh

Thats it!

ls -l

As you can see, the first piece of information on each line are the permissions.  There are 3 sets of permissions put together all into one line.  The very first character denotes whether the file is a directory, a link or a file.

- = file
l = symbolic link
d = directory

The 3 characters following the first character represent the permission information for the file owner.  The 3 characters following that denote the permissions for the group and the last 3 characters are the file permissions for everyone else.  A key for what the characters mean is listed below:

- = no read, write, or execute permissions
r = read
w = write
x = execute

For example, lets say we have the following permission information on a file.

drwxr-xr-x

Since the first character is “d” we know that its a directory.  Then next 3 characters are “rwx” which means that the owner has read, write, and execute permissions.  The next set of characters are “r-x” which means the group has read and execute permissions but no write permissions.  The last 3 characters are “r-x” which means that everyone else has read and execute permissions but no write permissions.
You can easily change file permissions by using the “chmod” command.  Using numbers to change permissions on a file is very quick, easy, and precise.  A key for what each number represents is listed below.

4 = read
2 = write
1 = execute

You have to set permissions for the owner, group, and everyone else all at the same time when using the number method so that means there will be 3 numbers.  The first number is the permission for the owner, the second is for the permission of the group and the last number is the file permission for everyone else.

To give the owner, group, and everyone else only read permissions then the command would be the following.

chmod 444 filename

To give the owner, group, and everyone else read and execute permission then you just add the numbers together.  So 4(read) + 1(execute) = 5.

chmod 555 filename

The numbers don’t have to be the same, you can give different permissions to the owner, group and everyone else all in one shot.  So lets say you want the file owner to have read, write and execute permissions but you want the group and everyone else to have only read and execute permissions.  The command to do this would be the following.

chmod 755 filename

If you are applying permissions to a folder and would like to recursively change the permissions of the contents as well then don’t forget to use the capital ‘R’ switch:

chmod -R 755 somedir

Remember, the first digit is for the owner, the second digit is for the group and the last digit is for everyone else.