How to Dynamically Add and Remove SCSI Devices on Linux (文档 ID 603868.1)

In this Document

APPLIES TO:

GOAL

This article is primarily intended for Linux System and Storage Administrators.
It aims to list and describe the various, low-level methods available to identify, add and remove SCSI devices from a Linux system.

Examples used are based on Oracle Linux Server 5 GA (2.6.18-8.el5).

SOLUTION

1. SCSI Device Addressing

A four-part addressing scheme is used to define the location of SCSI devices within a system. The attributes include:

<H>ost: Instance of hostadapter to which device is attached
<B>us: SCSI Bus or Channel on the hostadapter
<T>arget: SCSI Id assigned to an individual device
<L>un: Logical unit number on the device

Each attribute, <H> <B> <T> <L>, refers to a part of the device location, similar to how number, street, suburb and state all form an address.

References to device addresses are readily visible from system logs and various command output, though the availability of certain commands or utilities depends on the distribution and operating system version used. The lsscsi(8) utility, for example, is natively available on Oracle Linux 5.

The following denotes the use of several native commands to describe a relatively simple SCSI system.

Above, it's evident that four devices are attached to two hostadapters, both of type ata_piix i.e. host0: sda sdb, host1: sdc sr0.

Note too, there is sufficient overlap between various command output that device addresses and naming can be easily identified and mapped.

2. SCSI Device Naming

The name assigned to a SCSI device is completely independent to it's SCSI address. In fact, taking the linux 2.6 kernel as an example, the device naming system used, udev(8), dynamically allocates device names upon each boot.

During system initialisation, hardware is scanned and devices are named according to their discovery order. This means, however, that the same device may not always be assigned the same name. This may have implications on some systems, especially those that solely rely on device names such as file /etc/fstab for mounting filesystems.

For such cases, explicit udev(8) configuration may be required to guarantee cross-reboot persistency of device naming. Alternatively, other methods, such as mount-by-label or mount-by-UUID (where supported), may be employed to ensure only intended devices are selected and used, regardless of their arbitrary device name. udev(8), though mentioned here, is not described in any detail - it's relevance, however, becomes more apparent later. Refer to the udev(8) and mount(8) man pages and references below for more information.

3. Scanning, Adding and Removing Devices

From time to time, it may be necessary to add, remove, replace or even reorganise SCSI devices in a system. Broadly speaking, there are two main approaches to how this can be achieved - offline and online.

The method one might choose to add/remove devices usually depends on several factors, such as:

• distribution, operating system and version
• hardware type, driver version/capability
• system availability requirements
• storage availability requirements
• availability of backup/redundant systems
• familiarity of system, I/O stack and storage
• acceptance of associated risk

Following are several methods of device addition and removal. Regardless of which method you use, always ensure to perform thorough testing before use within a production environment.

3.1 System reboot

Adding and removing devices whilst a system is shutdown is considered the simplest and safest method. Clean filesystem unmount and ordered shutdown of all of components/layers/drivers involved in the I/O path avoids the potential risks associated with online (dynamic) device removal. On boot, ordered driver load allows newly added/removed devices to be correctly discovered and identified. This method, obviously, necessitates total system unavailability.

3.2 Reload hostadapter driver

The installation (loading) of common host bus adapter (HBA) driver modules initiates a scan (or rescan) of the associated device, resulting in the (re)discovery of any newly added/removed devices. Whilst performed with the system online, to be able to reload the hostadapter module, it must first be unloaded. This, therefore, means that any (all) filesystem, volume or array on devices associated with the driver must first be offlined (in the case of arrays or logical volumes) and/or unmounted. In fact, depending on the complexity of the I/O stack involved, this method is likely to also require the shutdown of related storage services and the unloading of other related or dependent modules e.g. multipathing.

3.3 procfs /proc/scsi/scsi (2.4 kernel)

Linux provides the ability to dynamically interact with the running kernel via the /proc (procfs) filesystem. Dynamically adding or removing devices can be accomplished via the /proc/scsi/scsiinterface i.e.:

To remove a specific device:

where <H> <B> <T> <L> refers to Host, Bus, Target and Lun

To add a specific device:

where <H> <B> <T> <L> refers to Host, Bus, Target and Lun

This method allows for all filesytems, volumes and arrays, except those immediately involved in specific device removal, to remain online and mounted. However, this method carries a high element of risk. Strong knowledge of the system and all storage layers/devices involved in the I/O path is required. Removal of an incorrect device or premature removal of an intended device may result in volume corruption or, in the presence of Clusterware (e.g. Oracle Cluster File System 2 (OCFS2) or Real Application Clusters (RAC)), could result in node fence (eviction).

Use this method as required when using a 2.4 kernel. The 2.6 kernel provides an improved sysfs interface (/sys, described below) for managing devices.

The following example illustrates the removal, then addition of a SCSI device:

Note that the removed device, originally known as /dev/sdc, was added back to the system, but as /dev/sdd. This highlights the potential risk associated of solely relying on arbitrary kernel-assigned device file names.

3.4 sysfs /sys/class/scsi_host/ (2.6 kernel)

The 2.6 kernel provides the /sys (sysfs) interface for interacting and managing system devices. In the case of SCSI devices, the /sys/class/scsi_host/ interface can be used to dynamically rescan a hostadapter, as well as add or remove specific devices.

To rescan a hostadapter:

where <H> refers to the hostadapter or instance of hostadapter where multiple adapters of the same type exist on the system

To remove a specific device:

where <H> <B> <T> <L> refers to Host, Bus, Target and Lun

To add a specific device:

where <H> <B> <T> <L> refers to Host, Bus, Target and Lun

Like the /proc/scsi/scsi interface, the /sys/class/scsi_host/ interface similarly allows for all filesytems, volumes and arrays, except those immediately involved in specific device removal, to remain online and mounted. Again, this method carries a high level of risk and requires a strong knowledge of the system and all storage layers/devices involved in the I/O path.

Where Fiber Channel (FC) Host Bus Adapters (HBA) are used, separate procfs and/or sysfs entries are created in various locations depending on Operating System and driver type and version used. In such cases, FC HBA driver level re-scan should precede SCSI Bus rescan. For example, for QLogic (qla2xxx):

Enterprise Linux 4:

Enterprise Linux 5:

The following example illustrates (non-FC HBA) SCSI hostadapter rescan, then removal and addition of a device:

Once again, note that the name of the re-added device (/dev/sdd) differs to it's original name (/dev/sdc).

The entries and names beneath /sys/class/scsi_host/ may vary depending on the operating system, kernel version and type of devices in use on the system. For example, if using iSCSI devices, additional directory entry session exists:

3.5 Other Native and Vendor Supplied Scripts

Some hostadapter/driver vendors supply their own scripts that can be used to scan, add and remove devices. These scripts may be used in favour of manual device interaction via native kernel interfaces. The scripts should, however, be thoroughly tested before their reliance in Production. Naturally, where provided by a third-party vendor, any issues arising from their use should be referred to the originating supplier.

Examples of vendor supplied scripts include:

• scsi-rescan/rescan-scsi-bus.sh (Linux sg3_utils package)
• ql-dynamic-tgt-lun-disc.sh (QLogic)
• qlun_disc.sh (QLogic)
• lun_scan (Emulex)
• hp_rescan (HP)
• proprietary and custom others

Contact your hostadapter vendor for latest available and recommended scripts.

4. Removing a Multipath Device by Example

Let's look at a more complex, yet typical scenario involving multipathing (device-mapper-multipath).
In this instance, explicitly white-listed iSCSI-served partitioned target devices (LUNs) are multipathed on the initiator using user-defined names; ocr1, voting1, etc.

Below, unused multipath device ocr3 is dynamically removed from the system, as are its associated underlying devices. The devices were verified to no longer be in use or required by any program or service before their removal.

Note, whether the /sbin/multipath or /sbin/dmsetup command is used, the result is the same. However, when using the dmsetup command to remove partitioned multipathed devices, the multipath aliases for all of its partitions must first be removed before the multipath alias of the device itself can be removed, otherwise the command fails with a 'device or resource busy error' message. When using the multipath command to remove partitioned devices, removal of the multipath alias of the device will automatically remove all multipath aliases for all of its partitions.

Note that the device-mapper devices previously associated with multipath device ocr3 (/dev/dm-7, /dev/dm-11) are removed, however the underlying device paths (/dev/sdn, /dev/sdo) remain until explicitly removed from the system.

REFERENCES