So if you are attempting to serve up a lot of home directories, don't use a 3511. There's probably a reason that Sun doesn't sell these anymore.

After further testing we discovered that this appears to the typical behavior for this array.  Up to a certain threshold point, it works just fine, fast reads and writes.  But once you cross the threshold, its performance radically degrades, to the point it is basically unusable.

• Fedora 7 and CentOS 5 to authenticate against Active Directory
• A CentOS 5 linux server to serve up home directories (including roving profiles) to Linux, Unix, and Windows clients with AD Kerberos
• Sun Ray Server Software (running Sun Ray thin clients) on CentOS 5 working with AD and Linux NFS home directories

And you know what? CentOS 5, serving NFS home directories off of a SATA array, is between 6 and 20 times faster than our (decent) Solaris SPARC servers with fibre channel array using Solaris 10. They were here before I was, but they're only like 10 months old. They should be *FASTER* than Linux.

More soon.

These arrays have an ethernet management port on them... But of course, these were never configured.

Sun makes array management software that you can optionally load on a directly attached host. The software actually talks to the array across the FC connection (they call this in-band). Of course, none of our hosts had this software loaded.

I always make sure that I have the latest firmware, and the latest management software. The firmware on disk arrays is especially important, as old firmware has bugs that cause I/O to hiccup, controllers to hang, etc.

Of course, our arrays have the firmware they shipped with (never been updated). It is so old now that a direct upgrade is not possible. You have to do an incremental upgrade. Follow these directions.

There are 6 different array components that have firmware on them, you have to do them all, and in the right order. If you fail to follow the directions, you could easily kill the array. Even if you do follow the directions, you may lose your disk configuration, FC configuration, etc. So make sure you either have it all backed up, or you are willing to reset the array to factory defaults.

Assuming you have the latest firmware and management software, then you can talk to the array from an attached host with the sccli command line tool. It is a little tricky, but it works.

Basic Disk Operations

Use this book to see sccli command syntax. Basically to use the array to add disks to your hosts you have to:

1. See if there are physical disks available in the array
2. Create a logical drive
3. Map the logical drive to a host FC channel
4. Reboot the host, and use host OS commands to manipulate the new disk

See if there are physical disks available in the array

So for this example, we are starting with an unconfigured array, no partitions, no maps, etc. You can see this stuff with sccli commands, like so:

[root@jedi ~]# sccli
sccli: selected device /dev/sg1 [SUN StorEdge 3511 SN#008AAD]
sccli> show disks
Ch     Id      Size   Speed  LD     Status     IDs                      Rev  
----------------------------------------------------------------------------
2(3)   0  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPPDC  
WWNN 22E4000A33007849
2(3)   1  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGWHDAC  
WWNN 22E4000A3300784A
2(3)   2  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYWS4C  
WWNN 22E4000A3300784B
2(3)   3  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYWH7C  
WWNN 22E4000A3300784C
2(3)   4  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGY7HYC  
WWNN 22E4000A3300784D
2(3)   5  372.61GB   200MB  NONE   USED       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYM8YC  
WWNN 22E4000A3300784E
2(3)   6  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPY6C  
WWNN 22E4000A3300784F
2(3)   7  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYKE3C  
WWNN 22E4000A33007850
2(3)   8  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPZ4C  
WWNN 22E4000A33007851
2(3)   9  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGXT6AC  
WWNN 22E4000A33007852
2(3)  10  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPS0C  
WWNN 22E4000A33007853
2(3)  11  372.61GB   200MB  NONE   FRMT       HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPM2C  
WWNN 22E4000A33007854
sccli> show logical-drives
sccli> show maps
sccli>

Create a logical drive

The syntax is fairly straight forward. In this example, my array has 12 disks in it. I'm going to set one as the global hot spare (this step is optional), and then use all of the rest for one big logical drive of 11 physical drives in RAID-5. Note that you specify physical disks in sccli in <channel>.<disk number>, so for example I'll use disk 11 for the spare below, so I'll specify 2.11.

sccli> configure global-spare 2.11
sccli> show disk
Ch     Id      Size   Speed  LD     Status     IDs                      Rev  
----------------------------------------------------------------------------
<snip>
2(3)  11  372.61GB   200MB  GLOBAL STAND-BY   HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPM2C  
WWNN 22E4000A33007854
sccli> create logical-drive raid5 2.0-10
sccli: created logical drive 651F6A8B
sccli> show logical-drive
LD    LD-ID        Size  Assigned  Type   Disks Spare  Failed Status     
------------------------------------------------------------------------
ld0   651F6A8B   3.64TB  Primary   RAID5  11    1      0      Good  I 
Write-Policy Default          StripeSize 128KB
sccli> show disks
Ch     Id      Size   Speed  LD     Status     IDs                      Rev  
----------------------------------------------------------------------------
2(3)   0  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPPDC  
WWNN 22E4000A33007849
2(3)   1  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGWHDAC  
WWNN 22E4000A3300784A
2(3)   2  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYWS4C  
WWNN 22E4000A3300784B
2(3)   3  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYWH7C  
WWNN 22E4000A3300784C
2(3)   4  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGY7HYC  
WWNN 22E4000A3300784D
2(3)   5  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYM8YC  
WWNN 22E4000A3300784E
2(3)   6  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPY6C  
WWNN 22E4000A3300784F
2(3)   7  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYKE3C  
WWNN 22E4000A33007850
2(3)   8  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPZ4C  
WWNN 22E4000A33007851
2(3)   9  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGXT6AC  
WWNN 22E4000A33007852
2(3)  10  372.61GB   200MB  ld0    ONLINE     HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPS0C  
WWNN 22E4000A33007853
2(3)  11  372.61GB   200MB  GLOBAL STAND-BY   HITACHI HDS7240SBSUN400G A47A 
S/N KRFS06RAGYPM2C  
WWNN 22E4000A33007854

Map the logical drive to a host FC channel

OK, so now the array knows it has a logical drive. However the controller doesn't know who to give it to (where to present it). So a map says basically to present a given logical drive to a particular host.

sccli> show channels
Ch  Type    Media   Speed   Width  PID / SID
--------------------------------------------
0  Host    FC(L)   2G      Serial  40 / N/A
1  Host    FC(L)   2G      Serial  N/A / 42
2  DRV+RCC FC(L)   2G      Serial  14 / 15
3  DRV+RCC FC(L)   2G      Serial  14 / 15
4  Host    FC(L)   2G      Serial  44 / N/A
5  Host    FC(L)   2G      Serial  N/A / 46
6  Host    LAN     N/A     Serial  N/A / N/A

The "host channels" basically correspond to the physical FC ports on the back of the controller. In my example, my one and only host is plugged into port 0 on the back of the controller. In sccli-speak, that is 0.40 - <channel>.<PID>. I will present my new logical drive as LUN 0 to 0.40.

sccli> show partition
LD/LV    ID-Partition        Size
-------------------------------------
ld0-00   651F6A8B-00       3.64TB
sccli> map 651F6A8B-00 0.40.0
sccli: mapping ld 651F6A8B-00 to 0.40.0
sccli> show map
Ch Tgt LUN   ld/lv   ID-Partition  Assigned  Filter Map
---------------------------------------------------------------------
0  40   0   ld0     651F6A8B-00   Primary   

Now reboot the host to get it to see the newly presented drive.

Use host OS commands to manipulate the new disk

These commands are specific to your OS. Google around for a howto.

[root@jedi ~]# fdisk -l
Disk /dev/sda: 73.4 GB, 73407868928 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        8924    71577607+  8e  Linux LVM
Disk /dev/sdb: 3998.1 GB, 3998146887680 bytes
255 heads, 63 sectors/track, 486080 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
[root@jedi ~]# parted /dev/sdb
GNU Parted 1.8.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)                                                                         

Install tomcat5

SRSS needs tomcat installed. No yum packages for CentOS 4.3, boo. Followed the instructions at http://www.tummy.com/Community/Articles/tomcat-centos/tomcat5.html. This appears to rebuild certain pre-installed rpm's with the right versions to get tomcat to install. On my server, an x4200 (x86_64 AMD), you need to get the jdk-1_5_0_11-linux-amd64.bin file from somwehere. I googled around and got it here. The examples are for i586, so in lots of places you have to substitute the right architecture (x86_64) in place of i586.

It was pretty hairy, but it did seem to work. Who knows if it is REALLY working, but /sbin/service tomcat5 start didn't complain when I ran it.

Install the right version of java

The install docs say Java 1.5 or better. CentOS 4.4 comes with 1.4. However, whatever I did with all the crazy rpm building stuff above magically updated the version. Yay. However the utinstall SRSS installer needs to know where the java is installed, and I don't have a clue. The manual also says explicitly to use the 32 bit version. I'm pretty sure I installed a 64 bit server version. The SRSS zip file comes with a 32 bit jre1.5.0_11 installer in srss_4.0/Supplemental/Java_Runtime_Environment/Linux. I installed it in /usr/share.

Install SRSS

So next, I googled around for a howto of SRSS on CentOS 4.4. This one seemed promising.

Downloaded SRSS from Sun into /tmp. Unzipped them. Cd'd into srss_4.0. Started utinstall. Told it where the java was (see above, /usr/share/jre1.5.0_11). utinstall ran without apparent complaint!

#./utinstall
# utinstall   Version: 4.0     Thu Jul 19 14:35:37 PDT 2007
GDM 2.4.4.7.2                            not installed
Your system currently has gdm-2.6.0.5-7.rhel4.15 installed.
This rpm needs to be removed in order to install a new version of gdm.
Before removing, make sure gdm is not managing any displays and is stopped.
Remove gdm-2.6.0.5-7.rhel4.15? ([Y]/N): y
+++ gdm-2.6.0.5-7.rhel4.15
Sun Ray Server Software 4.0              not installed
Sun Ray Data Store 3.0                   not installed
Kiosk Mode                               not installed
Enter Java v1.5 (or later) location [/usr/j2se]: /usr/share/jre1.5.0_11
About to carry out the following operations:
Install  [ GDM 2.4.4.7.2 ]
Install  [ Sun Ray Server Software 4.0 ]
Install  [ Sun Ray Data Store 3.0 ]
Skip     [ Sun Ray Server Software 4.0 French Admin GUI is not available ]
Skip     [ Sun Ray Server Software 4.0 Japanese Admin GUI is not available ]
Skip     [ Sun Ray Server Software 4.0 Simplified Chinese Admin GUI is not available ]
Skip     [ Sun Ray Server Software not available ]
Install  [ Kiosk Mode 4.0 ]
Install  [ data for utslaunch ]
Continue? ([Y]/N): y
Installing GDM version 2.4.4.7.2 ...
+++ gdm-2.4.4.7.2-12.i386.rpm
Installing Sun Ray Server Software version 4.0 ...
+++ SUNWuti-4.0-26.i386.rpm
+++ SUNWuto-4.0-26.i386.rpm
+++ SUNWutfw-4.0-26.i386.rpm
+++ SUNWutr-4.0-26.i386.rpm
+++ SUNWutu-4.0-26.i386.rpm
+++ SUNWuta-4.0-26.i386.rpm
+++ SUNWutsto-4.0-26.i386.rpm
+++ SUNWutstr-4.0-26.i386.rpm
+++ SUNWutps-4.0-26.i386.rpm
+++ SUNWutkau-4.0-26.i386.rpm
+++ Making and installing Sun Ray Audio module ...
rm -rf .tmp_versions *.cmd *.ko *.o *.mod.c .*.cmd
make -C /lib/modules/2.6.9-55.0.2.ELsmp/build SUBDIRS=/usr/src/SUNWut/utadem modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-55.0.2.EL-smp-x86_64'
CC [M]  /usr/src/SUNWut/utadem/utadem.o
Building modules, stage 2.
MODPOST
CC      /usr/src/SUNWut/utadem/utadem.mod.o
LD [M]  /usr/src/SUNWut/utadem/utadem.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.9-55.0.2.EL-smp-x86_64'
install -d /lib/modules/2.6.9-55.0.2.ELsmp/misc
install -c utadem.ko /lib/modules/2.6.9-55.0.2.ELsmp/misc
+++ Done.
+++ SUNWutio-4.0-26.i386.rpm
+++ Making and installing Sun Ray UTIO module ...
rm -rf .tmp_versions *.cmd *.ko *.o *.mod.c .*.cmd
make -C /lib/modules/2.6.9-55.0.2.ELsmp/build SUBDIRS=/usr/src/SUNWut/utio modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-55.0.2.EL-smp-x86_64'
CC [M]  /usr/src/SUNWut/utio/utio.o
Building modules, stage 2.
MODPOST
CC      /usr/src/SUNWut/utio/utio.mod.o
LD [M]  /usr/src/SUNWut/utio/utio.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.9-55.0.2.EL-smp-x86_64'
install -d /lib/modules/2.6.9-55.0.2.ELsmp/misc
install -c utio.ko /lib/modules/2.6.9-55.0.2.ELsmp/misc
+++ Done.
+++ SUNWutdsk-4.0-26.i386.rpm
+++ Making and installing Sun Ray Mass Storage modules ...
rm -rf .tmp_versions *.cmd *.ko *.o *.mod.c .*.cmd
make -C /lib/modules/2.6.9-55.0.2.ELsmp/build SUBDIRS=/usr/src/SUNWut/utdisk modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-55.0.2.EL-smp-x86_64'
CC [M]  /usr/src/SUNWut/utdisk/utdisk.o
CC [M]  /usr/src/SUNWut/utdisk/utdiskctl.o
Building modules, stage 2.
MODPOST
CC      /usr/src/SUNWut/utdisk/utdisk.mod.o
LD [M]  /usr/src/SUNWut/utdisk/utdisk.ko
CC      /usr/src/SUNWut/utdisk/utdiskctl.mod.o
LD [M]  /usr/src/SUNWut/utdisk/utdiskctl.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.9-55.0.2.EL-smp-x86_64'
install -d /lib/modules/2.6.9-55.0.2.ELsmp/misc
install -c utdisk.ko utdiskctl.ko /lib/modules/2.6.9-55.0.2.ELsmp/misc
+++ Done.
+++ SUNWutgsm-4.0-26.i386.rpm
+++ SUNWutref-4.0-26.i386.rpm
+++ SUNWlibusbut-4.0-26.i386.rpm
Installing Sun Ray Data Store version 3.0 ...
+++ SUNWutdso-3.0-03.i386.rpm
+++ SUNWutdsr-3.0-03.i386.rpm
Installing Sun Ray Server Software version 4.0 manpages ...
+++ SUNWutm-4.0-26.i386.rpm
+++ SUNWutwa-4.0-26.i386.rpm
+++ SUNWutwar-4.0-26.i386.rpm
+++ SUNWutwh-4.0-26.i386.rpm
+++ SUNWutwl-4.0-26.i386.rpm
Installing Kiosk Mode version 4.0 ...
+++ SUNWkior-4.0-26.i386.rpm
+++ SUNWkio-4.0-26.i386.rpm
+++ SUNWkiom-4.0-26.i386.rpm
Installation of Sun Ray Server Software has completed.
The system must be rebooted in order to complete this installation and
before starting the Sun Ray Server Software.

Restarted the box. Did the shared subnet config for my subnet, assigned a few ip's to DHCP for the Sun Rays. Restarted the box. Connected a Sun Ray and success!

Get the admin GUI working

The manual talks a lot about the admin gui. It didn't seem to be running. Read the install guide, didn't see anything obvious. Googled a bit, hits said to run utconfig, which I don't think was obviously in the manual. I did, like so:

[root@csl3 SUNWut]# utconfig
Configuration of Sun Ray Core Services Software
This script automates the configuration of the Sun Ray Core Services
software and related software products.  Before proceeding, you should
have read the Sun Ray Core Services 4.0 Installation Guide and filled out
the Configuration Worksheet.  This script will prompt you for the values
you filled out on the Worksheet.  For your convenience, default values
(where applicable) are shown in brackets.
Continue ([y]/n)? y
Enter Sun Ray admin password:
Re-enter Sun Ray admin password:
Configure Sun Ray Web Administration? ([y]/n)? y
Enter Apache Tomcat installation directory [/opt/apache-tomcat]: /usr/share/tomcat5
Enter HTTP port number [1660]:
Enable secure connections? ([y]/n)?
Enter HTTPS port number [1661]:
Enter Tomcat process username [utwww]: tomcat
Enable remote server administration? (y/[n])? y
Configure Sun Ray Kiosk Mode? (y/[n])? n
Configure this server for a failover group? (y/[n])? n
About to configure the following software products:
Sun Ray Data Store 3.0
Hostname: csl3.csc.calpoly.edu
Sun Ray root entry: o=utdata
Sun Ray root name: utdata
Sun Ray utdata admin password: (not shown)
SRDS 'rootdn': cn=admin,o=utdata
Sun Ray Web Administration
Apache Tomcat installation directory: /usr/share/tomcat5
HTTP port number: 1660
HTTPS port number: 1661
Tomcat process username: tomcat
Remote server administration: Enabled
Sun Ray Core Services 4.0
Failover group: no
Sun Ray Kiosk Mode: no
Continue ([y]/n)?
Updating Sun Ray Data Store schema ...
Updating Sun Ray Data Store ACL's ...
Creating Sun Ray Data Store Datastore ...
Restarting Sun Ray Data Store ...
Starting Sun Ray Data Store daemon .
Thu Jul 19 19:58 : utdsd starting
Loading Sun Ray Data Store ...
Executing '/usr/bin/ldapadd -h csl3.csc.calpoly.edu -x -p 7012 -D cn=admin,o=utdata' ...
adding new entry "o=utdata"
adding new entry "o=v1,o=utdata"
adding new entry "utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=desktops,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=users,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=logicalTokens,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=rawTokens,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=multihead,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=container,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=properties,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "cn=utadmin,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=smartCards,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utordername=probeorder,utname=smartCards,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=policy,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=resDefs,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utname=prefs,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utPrefType=resolution,utname=prefs,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
adding new entry "utPrefClass=advisory,utPrefType=resolution,utname=prefs,utname=csl3.csc.calpoly.edu,o=v1,o=utdata"
Added 18 new LDAP entries.
Creating Sun Ray Core Services Configuration ...
Adding user account 'tomcat' to group 'utadmin'... done
Sun Ray Web Administration enabled to start at system boot.
Sun Ray Web Administration not running - /opt/SUNWut/webadmin/tmp/utwebadmin.pid does not exist
Starting Sun Ray Web Administration...
See /var/opt/SUNWut/log/utwebadmin.log for server logging information
Unique "/etc/opt/SUNWut/gmSignature" has been generated.
Restarting Sun Ray Data Store ...
Stopping Sun Ray Data Store daemon
.Sun Ray Data Store daemon stopped
Starting Sun Ray Data Store daemon .
Thu Jul 19 19:58 : utdsd starting
Adding user admin ...
User(s) added successfully!
***********************************************************
The current policy has been modified.  You must restart the
authentication manager to activate the changes.
***********************************************************
Configuration of Sun Ray Core Services has completed.  Please check
the log file, /var/log/SUNWut/utconfig.2007_07_19_19:50:29.log, for errors.

I then did a cold restart (utrestart –c). I could then get to localhost:1660 and localhost:1661.

It seems to be working... Time to kick the tires some more. Stay posted.

A little about Rocks

Rocks clustering software is based on CentOS, a redistribution of Red Hat Enterprise Linux.  Rocks itself uses iptables to do both firewalling and network address translation (NAT). By default a Rocks cluster consists of a head node, that is more or less the controller of the cluster, and multiple compute nodes. The head node has 2 network interfaces cards in it. One of these cards is set up as a private network (10.0.0.0 "fake" subnet) that the compute nodes reside on, and a "real" network interface to talk to the outside world. The head node provides the compute nodes with most common network services (DNS, NTP, etc.). It also uses NAT to allow the compute node on their private subnet to access the internet. So even if you don't really need a firewall, you need iptables running to do the NAT on the head node.

No... Don't click that!

At some point a few weeks back, I figured that I didn't need the firewall, and I disabled it with some clicky Gnomish gui widget. I didn't realize about the NAT, or that the proper way to administer iptables was from the command line. I was in a hurry, so I took the "short cut". Yeah right.

Quite a few users on the cluster are mechanical or aeronautical engineers running computation fluid dynamic modeling using a commercial package called Fluent. So a few days after I monkeyed with the firewall, users were complaining that they couldn't run their Fluent jobs. Turns out that the College of Engineering has a Fluent license server, and that Fluent jobs need to talk to the license server (on a different subnet) before starting. Since I had inadvertently turned NAT off, compute nodes couldn't connect to anything outside of their private subnet, no licenses, no jobs.

What's done is done

"So no problem", I thought to myself, "I'll just use the clicky widget to turn it back on". Well, you don't do iptables like that. It didn't work of course. I'm no iptables expert, but it appears that there are basically 2 config files to make iptables work. There's /etc/sysconfig/iptables-config, which is the setup file for iptables itself, and then /etc/sysconfig/iptables which are the actual rules. Supposedly. I didn't have /etc/sysconfig/iptables at all. Iptables wouldn't start because it had no rules. It *USED* to start. It *USED* to NAT. So not only did the gui widget disable the firewall, it ate my /etc/sysconfig/iptables.

What to do

So at this point I had a couple of choices. I could send email of my silliness to the Rocks maintainers and hope they would have mercy on my soul. Or maybe I find some unused system laying around here, load a default load of Rocks on it, and copy the default /etc/sysconfig/iptables file back to the real head node. What I decided to do was to blow away one of the compute nodes, load it as if it were a new head node, copy the file I needed off of it, and then reload it as a compute node. Since Rocks makes it trivial to load a compute node, this wasn't as hard as it sounded. Once the "fake" head node was up, I jammed a USB drive it it, copied my file to the USB, and then rebooted the node and told it to PXE boot from the head node (which auto-magically reloads a compute node with no user interaction).

I then copied the "stock" /etc/sysconfig/iptables to the head node, and started iptables. NAT was back.

So like others have said "note: using other utilities to modify the firewall such as 'setup' or 'lokkit' will break Rocks". I should have listened.

OBTW, here what a "stock" Rocks 4.2.1 /etc/sysconfig/iptables looks like:

Linux

After some googling, I found Scott Lowe's blog on Active Directory Integration. Scott has basically done all of the heavy lifting, and has howtos that work. We sat up a test load of Fedora 7 on a VMware virtual machine, and followed his Linux howto. Basically you prepare the AD controllers to supply Unix attributes to Unix clients (in general this is a one time thing, once it's done, it's done). Then you set up the ability on the Unix machine to do LDAP queries against AD, so that clients can look up user attributes. Then you enable Kerberos, so that client mmachines and domain users can get and use valid AD Kerberos tickets. It is pretty straightforward, it worked, and we had no substantial issues with it. It is pretty disconcerting to see a Windows user log into a Linux machine with their Windows user name and password, but it seems to work fine.

Solaris

Next I tried this Solaris howto on a Solaris virtual machine. The Solaris howto is very similar to the Linux howto. The AD setup was already completed (since we already had performed it to get the Linux integration to work). So then I configured Solaris for LDAP and Kerberos. Surprisingly, I could get the Kerberos to work (Kerberos is well known to be complicated and cranky), but the LDAP configuration wouldn't work to save my life. It just silently did nothing. No error messages. Nothing of consequence in the logs on either the AD machines or the Solaris clients. It. Just. Didn't. Work. Grrr. I could do LDAP queries from the command line, but I'll be damned if I could get Solaris to use LDAP to look up user attributes. I spent something like 30 hours on it and finally gave up.

Solaris using Windows "Server for NIS"

The fallback idea was to use Windows "Server for NIS" that comes bundled with 2003 R2. I enabled and configured this. It really does do NIS, and Solaris clients can use it for authentication. I had 2 issues with it. The first was that the initial load of NIS with AD passwords only occurs when the AD password is changed. There's no way to force a copy of the existing passwords. So that means if you activate Windows Server for NIS, you have to tell everyone to change their AD passwords to get them into NIS. That sucks. The other issue I had was that although Solaris could use Server for NIS for secure shelll sessions, I couldn't get it working for GUI desktop sessions (CDE or Solaris Java Desktop). I'm guessing it was a PAM thing, but I couldn't figure it out.

Hmm. Server for NIS doesn't work for GUI sessions. That wouldn't work for my Sun Ray users. And I didn't feel like telling hundred, maybe thousands of people to change their passwords. I was stuck...

Solaris Redux

Well, Scott Lowe to the rescue again. He wrote up an updated howto for Solaris, that uses samba to help do some of the heavy lifting. It worked! I could use LDAP to query AD, and I could use samba to join my Solaris machines to the domain. I tell you it's a bit weird to see Solaris and Linux machines appear in the Windows GUI "Active Directory Users and Computers", but they do! The only hiccup I had with this updated howto was minor. Solaris 10 keeps it's Kerberos keytab in a different place than samba does. So although samba will join your machine to the domain (by getting a keytab and storing it in /etc/krb5.keytab), Solaris expects proper keytabs to be in /etc/krb5/krb5.keytab. Good old Solaris, gotta be different to confuse the ignorant. The symptoms are this - LDAP queries work (getent passwd <AD username> works, and kinit works (users can get tickets from the domain), but ssh sessions fail with "file not found". At first I thought the "file not found" error was a home directory thing, but I believe that it was really sshd whining that it couldn't find a keytab where it expected to.

Just copying the keytab to the place solaris expected it to be didn't seem to work either. So I used the ktutil command to read the keytab and write it to the new location like so:

# ktutil
ktutil: rkt /etc/krb5.keytab
ktutil: wkt /etc/krb5/krb5.keytab

After that, logins worked fine.

Next stop, unified home directories for all users, regardless of OS, via NFS and SMB.

In previous assignments, I used Big Brother to monitor my systems and networks. At that time ( a few years ago now...) Big Brother was open source. Big Brother, was good, fairly easy to configure, and worked well. At some point the rights to Big Brother were acquired by a commercial firm (Quest Software, Inc.) who closed the project. I soured a bit on Big Brother after that.

At another time we used the supposedly "industrial strength" Foglight and Spotlight by Quest. These were billed as "enterprise" monitoring software packages. Spotlight was pretty cool. It has lots of pretty lights to impress pointy haired bosses with. Foglight was not particularly impressive. First of all it ran on top of an Access database. Yup, nothing says "enterprise" more than having an Access backend. We found the configuration of monitored client systems to be cumbersome and not flexible enough to give us what we wanted in every situation. Did I say these cost thousands of dollars to purchase, and thousands of dollars to keep in maintenance? Yes, thousands. So with Foglight we had less than what we had with Big Brother and paid for it. Paid a lot for it. They say that later versions of Foglight are "New! Improved!" Well maybe they are. I'm not buying it.

In my new position there was no monitoring infrastructure. With 2 people and dozens of (undocumented) systems and servers, we had no idea which servers were up or down. Customers would tell us when things were broken.

I needed a monitoring tool that was:

• Free, both as in money and as in speech
• Capable of monitoring any type of system (Unix, Windows, Oracle, etc.)
• Rapid to install, rapid to deploy

After some research, I found Nagios. Nagios seemed almost as a spiritual heir to Big Brother. For someone that grew up with Big Brother, Nagios has a familar feel to it. The default install was straightforward (at least on CentOS 4). The install instructions were accurate and good.

When it comes to telling Nagios about the clients to be monitored, though, there are so many options that it was complicated almost to the point of incomprehensibility. I got a few of the server sided tests to work, but many of the more useful tests run on the client (invoked on demand by the Nagios server). In addition there are non-default options in Nagios that give more information, as well as add on packages to extend Nagios (by adding the graphing of trends, say). I didn't have time time to figure it all out from scratch.

Time for a big fat book.

I was in a hurry, so I headed down to the local Barnes and Noble, and lo and behold I found Nagios System and Network Monitoring
by Wolfgang Barth. (Which was kind of amazing, since the computer section has shrunk over time to maybe 20% of it's former size.) For me this was a great book. Basically Barth explains Nagios from entry level concepts and configuration, and then works his way up to the more complicated. I basically just followed along, and after a while (a long while) I had my monitoring set up in a useful fashion. I now monitor about 100 servers, which host about 400 different monitored services. Not a huge installation, but not a trivial one either. I'd say I put in about 80 hours of configuration and scripting to get this going. Now, I don't spend much time on it. Basically when a host is added or removed, you have to tweak the configuration to reflect the changes to your topology.

One thing that I suggest is that you always install a pair of Nagios servers. One will be the primary, the main one that watches everything else. The second one only watches the primary. If you don't do this, then when the primary fails, there's no one left to report the failure or any subsequent failure of another system. Your monitoring just silently goes off line. Not good.

A wise person once said that "Linux is only free if your time has no value". That's certainly true about Nagios. On the other hand, I had no budget for monitoring, so all I had was time. And like I said, in the past when we bought Foglight we spent something like $25,000 on the initial install, and then thousands, every year, forever, for maintanance. So 80 hours of labor doesn't seem too bad.

Nagios works better anyway.

An explanation of HP firmware (PDC and ISL) and how a HP boots. May be of use to someone, especially someone trying to load linux on HP hardware.

How to load HP-UX 11.00. Explains how to load HP-UX 11.00, step by step. Intended for newbies.

How to install MirrorDiskU/X. MirrorDisk is HP's implementation of software mirroring, which I believe they basically licensed from Veritas. This guide takes you through setting up mirroring step-by-step. Intended for newbies.

How to update HP firmware. If you ever have to do this, it's nice to have someone hold your hand.