SAN Storage Configuration in Solaris

Storage area networks have moved from the cutting edge to almost commonplace. But, if this is your first time connecting remote storage to Solaris, there are a number of configuration tasks that will be new to you if you have only worked with directly attached SCSI arrays or Sun's legacy Fibre Channel arrays such as the A5200 series. In this article, I will outline steps to configure Solaris to attach to remote disk storage from HP, IBM, and Network Appliance SANs. The platform configurations have much in common, but there are some differences in the patch sets and software libraries required. In my environment, I have used a few different SAN host bus adapter cards from Emulex and JNI, and we installed Veritas Volume Manager on the systems after the SAN devices were attached.

A SAN is a network that provides access to remote devices through a serial SCSI protocol such as Fibre Channel or iSCSI. This differs from NAS (network attached storage), which uses SMB/CIFS and NFS protocols. A SAN is much more than just a Fibre loop to remote disk drives -- tape robots and other hosts can transfer data directly between nodes and thereby decrease the load on the production Ethernet interfaces, where most of the user application traffic is directed. Using a SAN fabric of Fibre Channel switches, dozens of servers from multiple platforms can be interconnected to share the same cabinet of disks for data and share the same tape devices for enterprise backups.

Cost justification for investing in a SAN might be found by comparing the cost of directly attached storage to that of a centralized pool of storage. In my environment, many of my legacy directly attached storage arrays were underutilized. Although most of my old disk arrays still had room for growth, this storage space could not be effectively shared among servers. Pooling all disk storage into one large SAN array allowed for less waste and more efficient storage expansion as needed. Another advantage of migrating our storage to SANs has been increased I/O speed. Many of our older servers use 80 MB/s SCSI or 100 MB/s Fibre Channel for local disk storage.

Migrating storage to the SAN using dual 2-Gb Fibre Channel connections on each server has brought new life to some of our older, slower machines. Our older disk arrays had little or no cache to speed I/O transactions unlike our SAN storage cabinets, which have several gigabytes of I/O cache. In practice, we found some of our heavily I/O-bound applications, such as database loads, to run in half their previous time after migrating to the SAN. There have been numerous other advantages to our SAN, such as new tools for monitoring space, tools for trend analysis, and a significant consolidation of rack space in our data center.

Preparing Solaris for SAN Connectivity:

Regardless of the brand of SAN equipment used, the place to start is with Solaris patches. We can't stress enough the importance of applying Sun's recommended patch clusters immediately after installing Solaris. We found that some features didn't work as expected until the recommended patch set and additional patches below were applied. We started with Sun's Recommended and Security patch cluster, which can be downloaded from the Web or via ftp from:

http://sunsolve.sun.com

For Solaris 8, we downloaded the 8_Recommended.zip file and unzipped this file on our boot disk where we had plenty of free space. We shut down to single user mode with init s before running the install_cluster script. We issued the uname -a command before and after the patches to ensure the kernel version had been upgraded. The kernel version is the number after the dash in the output, for example, 108528-20 is Solaris 8 at kernel version 20. Sun's best practice is to keep up to date with the latest kernel release by installing the full Recommended and Security patch cluster. The patch cluster includes the basic patches that are recommended for all systems, although additional patches may be required for specific applications.

We then downloaded the Sun StorEdge SAN 4.2 software. This can be found at www.sun.com/storage/san, and will require a sunsolve account login. The file to download is Solaris_8_SFK_packages.tar.Z. We uncompressed and tar extracted this file and noted that it contained three small Solaris packages: SUNWsan, SUNWcfpl, and SUNWcfplx. We used the pkgadd utility to install all three of these with pkgadd -d . Solaris_8_SFK_packages.

The next step was to review individual patch versions on our box and determine whether additional patches were required. We used the showrev command to list installed patches and install higher versions as needed. For example, showrev -p | grep 109524 will search for the ssd driver patch at any version. We wanted to see version two, 109524-02, or higher. The best practice is to make a list of the versions you have and compare them to the latest available versions on sunsolve.sun.com. Below are the specific patches we applied for Fibre channel and general SAN readiness on our servers. These patches are all installed separately with the patchadd command; refer to the readme files that come with these patches for specific installation instructions. After all the patches were installed, we rebooted and checked /var/adm/messages for any errors before continuing:

109524-15 /kernel/drv/ssd patch
109657-09 isp driver patch
108974-27 dada, uata, dad, sd and scsi drivers patch
109529-06 luxadm, liba5k and libg_fc patch #1
111413-08 luxadm, liba5k and libg_fc patch #2 (requires 109529-06)
110380-04 ufssnapshots support, libadm patch
110934-13 pkgtrans, pkgadd, pkgchk

Installing Host Bus Adapters

We used dual JNI (http://www.jni.com) host bus adapters to connect to our HP and IBM SANs, and we used Emulex (http://www.emulex.com) adapters to connect to our Network Appliance SAN. Thus, we can provide an example for both of these common cards. The choice of interface card will depend on vendor support and certification of card models and specific levels of firmware. Be sure to check the vendor Web sites for latest firmware releases. Power off your server and install the interface cards in appropriate slots. Our test servers were a Sun 220R system (with PCI slots) and a Sun Ultra-2 server (with SBUS slots). We installed two adapters for each of the installations below so we would have some hardware redundancy and be able to take advantage of multipathing software. Some of Sun's Enterprise servers can accept either SBUS or PCI cards by making use of Sun's I/O boards with the appropriate SBUS or PCI slots.

Network Appliance SAN Configuration

In the first installation, we connected a Sun 220R server to our Network Appliance SAN filer model F940. We installed two PCI-based Emulex LP9002L host bus adapters in our Sun server and studied /var/adm/messages to be sure we had no boot errors. The software driver for the Emulex cards was provided to us by Network Appliance on CD, although updates can be downloaded from their Web site (http://www.netapp.com). Network Appliance has repackaged the Emulex drivers to include additional tools beyond the generic drivers obtainable at http://www.emulex.com. We installed the drivers by uncompressing and tar extracting the file "ntap_solaris_fcp_1_0.tar.Z" and then running the install script in the resulting directory. The script installed the Emulex lpfc drivers and utilities, Network Appliance sanlun utility, and updated parameters in /etc/system. There was only one question to answer regarding the use of Veritas with multipathing, which we will discuss later. We rebooted after running the script.

The Network Appliance SAN filer has a worldwide node name (WWNN), which should be bound to your Sun server. Using persistent bindings between the filer (known as the target) and the Sun server's Fibre Channel cards (the initiators) means you will always get the same SCSI target IDs for your SAN disks after every reboot. Without persistent bindings, the SCSI ID could change. To set up these bindings we telneted into the SAN filer and issued the command fcp nodename. This provided us the Fibre Channel node name in two formats, with and without colons, similar to the following example:

filer> fcp nodename
Fibre Channel nodename: 50:a9:80:00:55:CC:66:CD (50a9800055CC66CD)

Once we installed the two Emulex host bus adapter cards in the Sun server, they were named by Solaris to be lpfc0 and lpfc1. We ran the newly installed lputil command (installed under the /usr/sbin/lpfc directory) to create persistent bindings. The utility is a text-based menu, so we selected the "Persistent Bindings" and "Bind Target Manually" options. We were shown a list of HBA cards in our server (lpfc0, lpfc1), entered 0 to select the first card, and selected "Node Name". We pasted in our Network Appliance Fibre Channel nodename discovered above as a block of 16 characters with no colons or dashes. We entered 1 for the target number. The target number could be set to any number from 0 to 511 and uniquely identifies the filer in the event you want to attach more than one SAN filer to the Sun server. We repeated these steps for the second HBA card lpfc1, entering the same Fibre Channel nodename and filer target 1. We performed a reconfiguration reboot at this point with reboot -- -r. We think it's a good idea to go back into the lpfc menu after the system reboots to verify that the persistent bindings were preserved and that everything looks correct.

LUN Configuration

The next step is to create LUNs on the SAN filer and make them available to the Sun server, or more specifically, make them available to the HBA cards in our Sun server. The term LUN is derived from SCSI unit number and, in this context, we are referring to SCSI disk devices, although they are virtual disks and not physical SCSI disk drives. A virtual disk is created by combining slices of physical disks into one logical volume. The physical disk structure may use mirroring or RAID5, but the resulting LUN can be treated as one unit of disk in Solaris. In our environment, the SAN provides storage for several operating systems and is not managed by the Unix administrators. How to configure and make available sets of disks or LUNs on the SAN arrays is beyond the scope of this article, but we can summarize the steps. We created an initiator group on the filer, which is a list of worldwide port names (WWPNs). Network Appliance provides a tool to make this step easier. We ran the following sanlun command on our Sun server and provided the output to our SAN systems administrator:

# /usr/sbin/sanlun fcp show adapter -c

Enter this filer command to create an initiator group for this system:

igroup create -f -t solaris "sunhost1" 10000000c9307031 10000000c930694c

The second line of output provided us with the Network Appliance igroup command to be used on the filer to create the initiator group. The name of the group is the same as the Sun host name, in this example sunhost1. The following numbers are the WWPNs of the two HBA cards in this Sun server, and the igroup command is used to add these cards into the sunhost1 igroup on the filer. We provided this information to our SAN administrator along with our request for LUNs. He created three LUNs for us and mapped them to this initiator group for our use.

On the Solaris side, we needed to configure the /kernel/drv/sd.conf file, which is used to define and configure SCSI disk storage. Each Sun server already has a generic version of this file, however, we need to expand upon it to tell Solaris to probe for new LUNs at boot time. We added the following lines to the bottom of our sd.conf file to configure three additional LUNs. Note that target 1 matches the target we mapped to our filer when configuring the HBA cards with the lputil menu:

name="sd" parent="lpfc" target=1 lun=0;
name="sd" parent="lpfc" target=1 lun=1;
name="sd" parent="lpfc" target=1 lun=2;

If you anticipate needing to add storage space to this system in the future, add plenty of extra entries to the sd.conf file. The sd.conf is only read by Solaris during a reboot, so to avoid reboots, you must define several more LUNs than you currently need by incrementing the lun number. As long as these definitions were read during the last reboot, you will be able to add LUNs on the fly with the devfsadm command.

After our sd.conf was configured and the server was rebooted, we saw the new devices listed in the Solaris format utility. We noticed warnings regarding the new LUNs indicating they had corrupt labels. When we selected a disk (LUN) by number, we received the message that it was formatted but not labeled and were asked, "Label it now?" Responding with a "y", we labeled the LUN and repeated the process for all the new LUNs. Along with the format command, we found Network Applicance's sanlun utility useful for displaying the status of our LUNs. For example, "sanlun lun show filer1" lists all the LUNs on SAN filer1, showing the Solaris device filename, LUN size, and state.

We installed two HBAs in our server so we had two paths to each LUN; for example, we had one set of disks on controller 2 and another set on controller 3. Our first disk, c2t0d0, is the same physical LUN as c3t0d0 and only the first instance needs to be labeled. Veritas will take advantage of the second path to the LUN if you install Veritas Volume Manager with Dynamic Multipathing. At this point, the LUNs looked like standard Solaris disks, and we were ready to create new file systems on them or initialize them with Veritas Volume Manager.

HP SAN Configuration

Our second installation was performed connecting an HP SAN model XP1024 to the Sun 220R server. Starting from a clean installation of Solaris on the server, we applied the Sun patches as indicated above and then installed two PCI-based JNI host bus adapters. We performed a reconfiguration reboot and again examined the /var/adm/messages to ensure we had no boot errors. Even before any software drivers for the cards were installed, we could verify that Solaris was aware of the JNI hardware by issuing the command prtconf | grep JNI. The output showed one line per card and indicated that a JNI device was installed but no driver was attached. We downloaded the drivers for the JNI cards directly from their Web site (http://www.jni.com). Our cards were model FCX-6562, and the files to download for Solaris were the driver package "JNIC146x.pkg" and the EZ Fibre GUI configuration software "EZF_22j.tar". You need to be root to perform these installs, although you do not need to be in single user mode. We installed the drivers with pkgadd -d jnic146x.pkg. In the pkgadd menu, we selected "all" to install both the JNI HBA driver and the associated libraries for Solaris. At this point, the command prtconf | grep JNI showed one line for each card indicating instance 0 and instance 1, an indication the device drivers were attached.

One difference between the Emulex and JNI products is the addition of a configuration GUI provided by JNI. We configured the sd.conf with LUN definitions and relied on the EZ Fibre GUI to perform these edits. The GUI also made tuning parameter changes for the JNI drivers, which are stored in the /kernel/drv/jnic146x.conf file. To install the EZ Fibre software, we tar extracted the ezf_22j.tar file, changed into the resulting EZF_22 directory, and ran the install.sh script. This is an X Window application, and it popped up a series of windows leading us through the installation. We accepted the license agreement and the default install directory /opt/jni/ezfibre/standalone. After installation, we started the EZ Fibre GUI by changing to the ezfibre/standalone directory and executing the ezf script. The GUI provided a view of our system, listing each JNI card and providing the card status and WWPN information.

We used the EZ Fibre configuration GUI to look up the worldwide port names for each JNI HBA card and provided these WWPNs to our SAN administrator. The card parameter changes are stored in the jnic146x.conf file. The defaults may be correct for most installs, but we hard-coded the link speed to 2 GB, disabled IP protocol, and set the topology to use a fabric instead of a private Fibre loop. Changes here were made to both HBA cards separately before rebooting.

Our HP SAN administrator uses a Web-based tool from HP called the StorageWorks command view. Using this tool, he created LUNs within the HP SAN array and assigned them to our HBA WWPNs. This created a soft zone to map the LUNs to our HBAs and is analogous to using the igroup command in the Network Appliance installation above. After the LUNs were assigned to the cards in our Sun server, we could see the LUNs as available to us in the "LUN-Level Zoning" tab in the EZ Fibre GUI. Checking them off and accepting them in the GUI made the proper edits to our sd.conf file. We repeated the process of attaching LUNs to both HBA cards. We committed changes for each card separately in the GUI and then rebooted the server. Note that persistent bindings are in effect since Dynamic Binding in the GUI is disabled by default. After exiting the EZ Fibre GUI, we examined the sd.conf file and saw the LUNs added at the bottom in the following format:

name="sd" class="scsi" target=0 lun=1;
name="sd" class="scsi" target=0 lun=2;
name="sd" class="scsi" target=0 lun=3;

Although using the GUI to add LUNs to sd.conf was convenient, it did not provide a way to add extra LUN definitions for future disk expansion. Thus, we used a text editor to edit the sd.conf file and add several more lines, incrementing the LUN number. The sd.conf file is only read at boot time, and we wanted to be able to add more disk space on the fly without having to reboot in the future. We can have up to 256 LUNs per target number. If our SAN manager provides another LUN on the fly, we can run the devfsadm command to create the Solaris device for the predefined LUN without rebooting.

IBM Shark SAN Configuration

We connected an older Sun Ultra-2 server to our Shark SAN ESS 2105-F20 using a pair of SBUS-based JNI cards (model FCE1473). Again, we downloaded the card drivers from the JNI Web site, along with the JNI EZ Fibre utility, and there were no significant differences from our previous JNI installations. This Ultra-2 was installed with Solaris 9 and, unlike with Solaris 8, we could achieve connectivity to the Shark SAN without additional patches after the base Solaris install. However, the best practice is to always install the cluster of patches 9_Recommended.zip.

We downloaded the Sun StorEdge SAN 4.2 software, Solaris_9_SFK_packages.tar.Z, from http://www.sun.com/storage/san. This is essentially the same as for Solaris 8 and contains the Solaris packages SUNWsan, SUNWcfpl, and SUNWcfplx for Solaris 9. We confirmed through the /var/adm/messages file that our hardware was running cleanly before setting up LUNs with the EZ Fibre GUI. After rebooting, we saw the familiar message about corrupt labels on our LUNs and used the Solaris format utility to label each LUN with a default Solaris label. Although not required for our installation, enhanced functionality is available through IBM's Subsystem Device Drivers (SDD) for Solaris. These drivers support the multipath configuration within the Shark and allow for load balancing multiple paths.

Veritas Volume Manager and DMP

In each of our SAN installations, every LUN that was provided to our Sun server became a Solaris disk device in the familiar Sun convention of c#t#d#, where c = controller, t = target, d = disk. The LUNS as defined in sd.conf as 0, 1, 2 became Solaris disk devices c2t0d0, c2t0d1, c2t0d2. We saw them again as controller 3 (c3t0d0, c3t0d1, c3t0d2) since we have two HBA paths to the same devices. We worked with our SAN administrator to create LUNs that were roughly 7 Gb in size. Although we could create LUNs many times this size, we thought it would be less wasteful to add disks in increments of 7 Gb, one or two LUNs at a time, as our applications required more space.

We needed a utility that allowed us to grow and expand file systems on the fly without reboots. With this setup, we could add LUNs on the fly if they were already reserved in the sd.conf file, however, we could not resize an existing LUN. Although our SAN administrator could resize a LUN on the backend, it would require us to create a new file system on the Solaris device. We could back up, resize, and rebuild the LUN and Solaris disk device, and then restore our data. But we could not keep our file system online during that process.

Veritas Volume Manager allows us to create Unix file systems that span multiple LUNs and also provides tools to resize Unix file systems without taking them offline. If one of our mounted file systems is running out of space, we can import another LUN, initialize it as a Veritas disk, and extend the file system onto this disk. It is not necessary to unmount the file system during the process.

In our three SAN environments, we installed Veritas Volume Manager version 3.5. Veritas recommends also installing their maintenance pack update MP1 for 3.5. (At the time of writing, MP1 consisted of Sun patches 113210-02 for Solaris 8, 112392-04 for vxvm, and three for vea, 113203-02, 113595-02, and 13596-02. On our older Veritas 3.2 installations, we downloaded 113201-02 for vxvm, 111904-06 for vmsa, and 110435-08 for vxfs.) There was nothing unusual about the installation in a SAN environment compared to legacy disk arrays. Once the LUNs are visible as Solaris devices under the format utility and have been labeled, they can be brought under Veritas control. Veritas should see the disks after a reconfiguration reboot or, if you have added LUNs on the fly, you can run the Veritas vxdctl enable command to make them visible to Veritas. The vxdisk list command is convenient for checking whether all your disk devices are known to Veritas. We use the vxdiskadm menu to initialize all our LUN disks and then the Veritas vea (previously vmsa) GUI to create volumes and file systems.

As mentioned previously, we have dual HBAs in each server so there is one instance of each disk under each controller c2 and c3. Veritas is installed with Dynamic Multipathing by default and will configure itself to show only one instance of each disk as seen in the vxdisk list output. We verified that Veritas is aware of the second path to each of these disks by listing the detailed information on each disk. For example, vxdisk list c2t0d0s2 will show a page of detail about the disk, including multipath information near the bottom:

Multipathing information:
numpaths: 2
c2t0d0s2 state=enabled
c3t0d0s2 state=enabled

An enabled state means that DMP has been enabled in Veritas, but it is not an immediate indicator of the physical connection status. To certify our servers during the installation process, we watched a tail -f /var/adm/messages on our system and unplugged one Fibre HBA connection at a time to observe the failure and reconnection of each path to the LUN devices.

In our HP SAN environment, we downloaded an additional software package, the Veritas Array Support Library. Starting with Volume Manager version 3.2, Veritas introduced the Device Discovery Layer (DDL), which enhances the Veritas configuration daemon to discover multipathing attributes of disks. Support for some disk arrays is built in before adding these libraries and vxddladm listsupport will identify them. However, check the Veritas Web site at support.veritas.com to see whether a library is available for your specific array. In the case of our HP SAN, the array support library enabled Veritas to correctly identify the array and provide additional advanced functionality relative to command devices, which are reserved devices for each host server to communicate with the array. We did not install Veritas support libraries for our Shark and Network Appliance SANs; however, we issued the command vxddladm addjbod vid=NETAPP for the Network Appliance SAN to identify NETAPP to the Veritas DDL.

Summary

We have shown examples of configuring Solaris to connect to three different vendors' SANs. Although there are differences in how to configure the HBA software drivers for various types of cards, the steps to achieve connectivity are similar. Before beginning an installation, we ensure that our system has been upgraded to the latest patch sets available. Patch releases are frequently released, and time spent researching the latest versions on sunsolve before installing is time well spent. Once the system is running with the latest patches, SAN software, and HBA drivers, we provide the WWPN numbers of our HBA cards to our SAN administrator. He creates virtual disks and puts them into a group or soft zone, which restricts their use only to our HBAs.

We then edit our Solaris sd.conf file to add these new LUNs and instruct Solaris to define them as disk devices during the next reconfiguration reboot. We configure the HBA driver software to persistently bind the SCSI targets for the new LUNs so they will be consistent across reboots. The new disk devices are labeled with the Solaris format utility and initialized with Veritas Volume Manager. We install enhanced libraries for Veritas DDL if they exist for our vendor's SAN and, finally, test the functionality of dynamic multipathing. Our first attempt to connect Solaris to a SAN was challenging because we needed to learn several new concepts, such as configuring the sd.conf file and how to bind WWPN numbers in our environment. We hope we have provided enough of an overview of these concepts here to make this process easier for others.