Guidelines for OpenVMS Cluster Configurations

The OpenVMS system attempts to prevent low-numbered targets from being completely blocked by monitoring the amount of time an I/O request takes. If the request is not completed within a certain period, the OpenVMS system stops sending new requests until the tardy I/Os complete. While this algorithm does not ensure that all targets get equal access to the bus, it does prevent low-numbered targets from being totally blocked.

A.7.5.2 Solutions for Resolving Arbitration Problems

If you find that some of your disks are not being serviced quickly enough during periods of heavy I/O, try some or all of the following, as appropriate for your site:

Assign the highest ID numbers to those disks that require the fastest response time.
Spread disks across more SCSI buses.
Keep disks that need to be accessed only by a single host (for example, page and swap disks) on a nonshared SCSI bus.

Another method that might provide for more equal servicing of lower and higher ID disks is to set the host IDs to the lowest numbers (0 and 1) rather than the highest. When you use this method, the host cannot gain control of the bus to send new commands as long as any disk, including those with the lowest IDs, need the bus. Although this option is available to improve fairness under some circumstances, Digital considers this configuration to be less desirable in most instances, for the following reasons:

It can result in lower total throughput.
It can result in timeout conditions if a command cannot be sent within a few seconds.
It can cause physical configuration difficulties. For example, StorageWorks shelves such as the BA350 have no slot to hold a disk with ID 7, but they do have a slot for a disk with ID 0. If you change the host to ID 0, you must remove a disk from slot 0 in the BA350, but you cannot move the disk to ID 7. If you have two hosts with IDs 0 and 1, you cannot use slot 0 or 1 in the BA350. (Note, however, that you can have a disk with ID 7 in a BA353.)

A.7.5.3 Arbitration and Bus Isolators

Any active device, such as a DWZZA, that connects bus segments introduces small delays as signals pass through the device from one segment to another. Under some circumstances, these delays can be another cause of unfair arbitration. For example, consider the following configuration, which could result in disk servicing problems (starvation) under heavy work loads:

Although disk 5 has the highest ID number, there are some circumstances under which disk 5 has the lowest access to the bus. This can occur after one of the lower-numbered disks has gained control of the bus and then completed the operation for which control of the bus was needed. At this point, disk 5 does not recognize that the bus is free and might wait before trying to arbitrate for control of the bus. As a result, one of the lower-numbered disks, having become aware of the free bus and then submitting a request for the bus, will gain control of the bus.

If you see this type of problem, the following suggestions can help you reduce its severity:

Try to place all disks on the same bus segment.
If placing all disks on the same bus segment is not possible (for example if you have both some RZ28 disks by themselves and an HSZ40), try to use a configuration that has only one isolator between any pair of disks.
If your configuration requires two isolators between a pair of disks (for example, to meet distance requirements), try to balance the number of disks on each bus segment.
Follow the suggestions in Section A.7.5.2 to reduce the total traffic on the logical bus.

A.7.6 Removal and Insertion of SCSI Devices While the OpenVMS Cluster System is Operating

With proper procedures, certain SCSI devices can be removed from or inserted onto an active SCSI bus without disrupting the ongoing operation of the bus. This capability is referred to as hot plugging. Hot plugging can allow a suitably configured OpenVMS Cluster system to continue to run while a failed component is replaced. Without hot plugging, it is necessary to make the SCSI bus inactive and remove power from all the devices on the SCSI bus before any device is removed from it or inserted onto it.

In a SCSI OpenVMS Cluster system, hot plugging requires that all devices on the bus have certain electrical characteristics and be configured appropriately on the SCSI bus. Successful hot plugging also depends on strict adherence to the procedures described in this section. These procedures ensure that the hot-plugged device is inactive and that active bus signals are not disturbed.

Hot Plugging for SCSI Buses Behind a Storage Controller
This section describes hot-plugging procedures for devices that are on the same SCSI bus as the host that is running OpenVMS. The procedures are different for SCSI buses that are behind a storage controller, such as the HSZ40. Refer to the storage controller documentation for the procedures to hot plug devices that they control.

A.7.6.1 Terminology for Describing Hot Plugging

The terms shown in bold in this section are used in the discussion of hot plugging rules and procedures.

A SCSI bus segment consists of two terminators, the electrical path forming continuity between them, and possibly, some attached stubs. Bus segments can be connected together by bus isolators (for example, DWZZA), to form a logical SCSI bus or just a SCSI bus.
There are two types of connections on a segment: bussing connections, which break the path between two terminators, and stubbing connections, which disconnect all or part of a stub.
A device is active on the SCSI bus when it is asserting one or more of the bus signals. A device is inactive when it is not asserting any bus signals.
The segment attached to a bus isolator is inactive when all devices on that segment, except possibly the bus isolator, are inactive.
A port on a bus isolator has proper termination when it is attached to a segment that is terminated at both ends and has TERMPWR in compliance with SCSI--2 requirements.

A.7.6.2 Rules for Hot Plugging

Follow these rules when planning for and performing hot plugging:

The device to be hot plugged, and all other devices on the same segment, must meet the electrical requirements described in Annex A, Section A.4, of the SCSI-3 Parallel Interface (SPI) Standard, working draft X3T10/855D.¹ The SPI document places requirements on the receivers and terminators on the segment where the hot plugging is being performed, and on the transceivers, TERMPWR, termination, and power/ground/signal sequencing, of the device that is being hot plugged.
All the devices in Table A-2, except the DWZZA, meet these requirements. The DWZZA's transceivers do not meet the requirements for error-free powering on and off. The rules and procedures in this section have been adjusted to compensate for this fact.
Hot plugging must occur only at a stubbing connection.
This implies that a hot-plugged device can make only one connection to the SCSI bus, the device must not provide termination for the SCSI bus, and the device's connection must not exceed the maximum stub length, as shown in Figure A-3. An example of a SCSI bus topology showing the valid hot plugging connections is illustrated in Figure A-15.
Figure A-15 SCSI Bus Topology
Take precautions to ensure that electrostatic discharge (ESD) does not damage devices or disrupt active signals on the SCSI bus. You should take such precautions during the process of disconnecting and connecting, as well as during the time that SCSI bus conductors are exposed.
Take precaution to ensure that ground offset voltages do not pose a safety hazard and will not interfere with SCSI bus signaling, especially in single-ended configurations. The procedures for measuring and eliminating ground offset voltages are described in Section A.7.8.
The device that is hot plugged must be inactive during the disconnection and connection operations. Otherwise, the SCSI bus may hang.²

Note
Ideally, a device will also be inactive whenever its power is removed, for the same reason.

The procedures for ensuring that a device is inactive are described in Section A.7.6.3.
A quorum disk must not be hot plugged. This is because there is no mechanism for stopping the I/O to a quorum disk, and because the replacement disk will not contain the correct quorum file.
The OpenVMS Cluster system must be reconfigured to remove a device as a quorum disk before that device is removed from the bus. The procedure for accomplishing this is described in OpenVMS Cluster Systems.
An alternate method for increasing the availability of the quorum disk is to use an HSZ40 mirror set as the quorum disk. This would allow a failed member to be replaced while maintaining the quorum disk functionality.
Disks must be dismounted logically before removing or replacing them in a hot-plugging operation. This is required to ensure that the disk is inactive and to ensure the integrity of the file system.
The DWZZA must be powered up when it is inserted into an active SCSI bus and should remain powered up at all times while it is attached to the active SCSI bus. This is because the DWZZA can disrupt the operation of the attached segments when it is powering up or down.
The segment attached to a bus isolator must be maintained in the inactive state whenever the other port on the bus isolator is terminated improperly. This is required because an improperly terminated bus isolator port may pass erroneous signals to the other port.
Thus, for a particular hot-plugging operation, one of the segments attached to a bus isolator must be designated as the (potentially) active segment, and the other must be maintained in the inactive state, as illustrated in Figure A-16. The procedures for ensuring that a segment is inactive are described in Section A.7.6.3.
Figure A-16 Hot Plugging a Bus Isolator

Note that, although a bus isolator may have more than one stubbing connection and thus be capable of hot plugging on each of them, only one segment can be the active segment for any particular hot-plugging operation.
Take precautions to ensure that the only electrical conductor that contacts a connector pin is its mate. These precautions must be taken during the process of disconnecting and connecting as well as during the time the connector is disconnected.
Devices must be replaced with devices of the same type. That is, if any system in the OpenVMS Cluster configures a SCSI ID as a DK or MK device, then that SCSI ID must contain only DK or MK devices, respectively, for as long as that OpenVMS Cluster member is running.
Different implementations of the same device type can be substituted (for example, an RZ26L can be replaced with an RZ28B). Note that the system will not recognize the change in device type until an attempt is made to mount the new device. Also, note that host-based shadowing continues to require that all members of a shadow set be the same device type.
SCSI IDs that are empty when a system boots must remain empty as long as that system is running. This rule applies only if there are multiple processors on the SCSI bus and the MSCP server is loaded on any of them. (The MSCP server is loaded when the MSCP_LOAD system parameter is set to 1).
This is required to ensure that nodes on the SCSI bus use their direct path to the disk rather than the served path. When the new device is configured on a system (using SYSMAN IO commands), that system serves it to the second system on the shared SCSI bus. The second system automatically configures the new device by way of the MSCP served path. Once this occurs, the second system will be unable to use its direct SCSI path to the new device because failover from an MSCP served path to a direct SCSI path is not implemented.

A.7.6.3 Procedures for Ensuring That a Device or Segment is Inactive

Use the following procedures to ensure that a device or a segment is inactive:

To ensure that a disk is inactive:
1. Dismount the disk on all members of the OpenVMS Cluster system.
2. Ensure that any I/O that can occur to a dismounted disk is stopped, for example:
  - Disable the disk as a quorum disk.
  - Allocate the disk (using the DCL command ALLOCATE) to block further mount or initialization attempts.
  - Disable console polling by all halted hosts on the logical SCSI bus (by setting the console variable SCSI_POLL to OFF and entering the INIT command).
  - Ensure that no host on the logical SCSI bus is executing power-up or initialization self-tests, booting, or configuring the SCSI bus (using SYSMAN IO commands).

To ensure that an HSZ40 controller is inactive:

Dismount all of the HSZ40 virtual disks on all members of the OpenVMS Cluster system.
Shut down the controller, following the procedures in the HS Family of Array Controllers User's Guide.
Power down the HSZ40 (optional).

To ensure that a host adapter is inactive:

Halt the system.
Power down the system, or set the console variable SCSI_POLL to OFF and then enter the INIT command on the halted system. This ensures that the system will not poll or respond to polls.

To ensure that a segment is inactive, follow the preceding procedures for every device on the segment.

A.7.6.4 Procedure for Hot Plugging StorageWorks SBB Disks

To remove an SBB disk from an active SCSI bus, use the following procedure:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance.
Follow the procedure in Section A.7.6.3 to make the disk inactive.
Squeeze the clips on the side of the SBB, and slide the disk out of the StorageWorks shelf.

To plug an SBB disk into an active SCSI bus, use the following procedure:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance.
Ensure that the SCSI ID associated with the device (either by jumpers or by the slot in the StorageWorks shelf) conforms to the following:
- The SCSI ID is unique for the logical SCSI bus.
- The SCSI ID is already configured as a DK device on all of the following:
  - Any member of the OpenVMS Cluster system that already has that ID configured
  - Any OpenVMS processor on the same SCSI bus that is running the MSCP server
Slide the SBB into the StorageWorks shelf.

Configure the disk on OpenVMS Cluster members, if required, using SYSMAN IO commands.

A.7.6.5 Procedure for Hot Plugging HSZ40s

To remove an HSZ40 controller from an active SCSI bus:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance.
Follow the procedure in Section A.7.6.3 to make the HSZ40 inactive.
The HSZ40 can be powered down, but it must remain plugged in to the power distribution system to maintain grounding.
Unscrew and remove the differential triconnector from the HSZ40.
Protect all exposed connector pins from ESD and from contacting any electrical conductor while they are disconnected.

To plug an HSZ40 controller into an active SCSI bus:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance. Also, ensure that the ground offset voltages between the HSZ40 and all components that will be attached to it are within the limits specified in Section A.7.8.
Protect all exposed connector pins from ESD and from contacting any electrical conductor while they are disconnected.
Power up the HSZ40 and ensure that the disk units associated with the HSZ40 conform to the following:
- The disk units are unique for the logical SCSI bus.
- The disk units are already configured as DK devices on the following:
  - Any member of the OpenVMS Cluster system that already has that ID configured
  - Any OpenVMS processor on the same SCSI bus that is running the MSCP server
Ensure that the HSZ40 will make a legal stubbing connection to the active segment. (The connection is legal when the triconnector is attached directly to the HSZ40 controller module, with no intervening cable.)
Attach the differential triconnector to the HSZ40, using care to ensure that it is properly aligned. Tighten the screws.

Configure the HSZ40 virtual disks on OpenVMS Cluster members, as required, using SYSMAN IO commands.

A.7.6.6 Procedure for Hot Plugging Host Adapters

To remove a host adapter from an active SCSI bus:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance.
Verify that the connection to be broken is a stubbing connection. If it is not, then do not perform the hot plugging procedure.
Follow the procedure in Section A.7.6.3 to make the host adapter inactive.
The system can be powered down, but it must remain plugged in to the power distribution system to maintain grounding.
Remove the "Y" cable from the host adapter's single-ended connector.
Protect all exposed connector pins from ESD and from contacting any electrical conductor while they are disconnected.
Do not unplug the adapter from the host's internal bus while the host remains powered up.
At this point, the adapter has disconnected from the SCSI bus. To remove the adapter from the host, first power down the host, then remove the adapter from the host's internal bus.

To plug a host adapter into an active SCSI bus:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance. Also, ensure that the ground offset voltages between the host and all components that will be attached to it are within the limits specified in Section A.7.8.
Protect all exposed connector pins from ESD and from contacting any electrical conductor while they are disconnected.
Ensure that the host adapter will make a legal stubbing connection to the active segment (the stub length must be within allowed limits, and the host adapter must not provide termination to the active segment).
Plug the adapter into the host (if it is unplugged).
Plug the system into the power distribution system to ensure proper grounding. Power up, if desired.
Attach the "Y" cable to the host adapter, using care to ensure that it is properly aligned.

A.7.6.7 Procedure for Hot Plugging DWZZAs

Use the following procedure to remove a DWZZA from an active SCSI bus:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance.
Verify that the connection to be broken is a stubbing connection. If it is not, then do not perform the hot plugging procedure.
Do not power down the DWZZA. This can disrupt the operation of the attached SCSI bus segments.
Determine which SCSI bus segment will remain active after the disconnection. Follow the procedure in Section A.7.6.3 to make the other segment inactive.
When the DWZZA is removed from the active segment, the inactive segment must remain inactive until the DWZZA is also removed from the inactive segment, or until proper termination is restored to the DWZZA port that was disconnected from the active segment.

The next step depends on the type of DWZZA and the segment that is being hot plugged, as follows:

DWZZA Type	Condition	Action
DWZZA-VA	Single-ended segment will remain active.	Squeeze the clips on the side of the SBB, and slide the DWZZA-VA out of the StorageWorks shelf.
DWZZA-VA	Differential segment will remain active.	Unscrew and remove the differential triconnector from the DWZZA-VA.
DWZZA-AA	Single-ended segment will remain active.	Remove the "Y" cable from the DWZZA-AA's single-ended connector.
DWZZA-AA	Differential segment will remain active.	Unscrew and remove the differential triconnector from the DWZZA-AA.

Protect all exposed connector pins from ESD and from contacting any electrical conductor while they are disconnected.

To plug a DWZZA into an active SCSI bus:

Use an ESD grounding strap that is attached either to a grounding stud or to unpainted metal on one of the cabinets in the system. Refer to the system installation procedures for guidance. Also, ensure that the ground offset voltages between the DWZZA-AA and all components that will be attached to it are within the limits specified in Section A.7.8.
Protect all exposed connector pins from ESD and from contacting any electrical conductor while they are disconnected.
Ensure that the DWZZA will make a legal stubbing connection to the active segment (the stub length must be within allowed limits, and the DWZZA must not provide termination to the active segment).
The DWZZA must be powered up. The SCSI segment that is being added must be attached and properly terminated. All devices on this segment must be inactive.

The next step depends on the type of DWZZA, and which segment is being hot plugged, as follows:

DWZZA Type	Condition	Action
DWZZA-VA	Single-ended segment is being hot plugged.	Slide the DWZZA-VA into the StorageWorks shelf.
DWZZA-VA	Differential segment is being hot plugged.	Attach the differential triconnector to the DWZZA-VA, using care to ensure that it is properly aligned. Tighten the screws.
DWZZA-AA	Single-ended segment is being hot plugged.	Attach the "Y" cable to the DWZZA-AA, using care to ensure that it is properly aligned.
DWZZA-AA	Differential segment is being hot plugged.	Attach the differential triconnector to the DWZZA-VA, using care to ensure that it is properly aligned. Tighten the screws.

If the newly attached segment has storage devices on it, then configure them on OpenVMS Cluster members, if required, using SYSMAN IO commands.

A.7.7 OpenVMS Requirements for Devices Used on Multihost SCSI OpenVMS Cluster Systems

At this time, the only devices approved for use on multihost SCSI OpenVMS Cluster systems are those listed in Table A-2. While not specifically approved for use, other disk devices might be used in a multihost OpenVMS Cluster system when they conform to the following requirements:

Support for concurrent multi-initiator I/O.
Proper management for the following states or conditions on a per-initiator basis:
- Synchronous negotiated state and speed
- Width negotiated state
- Contingent Allegiance and Unit Attention conditions
Tagged command queuing. This is needed to provide an ordering guarantee used in OpenVMS Cluster systems to ensure that I/O has been flushed. The drive must implement queuing that complies with Section 7.8.2 of the SCSI--2 standard, which says (in part):
- "...All commands received with a simple queue tag message prior to a command received with an ordered queue tag message, regardless of initiator, shall be executed before that command with the ordered queue tag message." (Emphasis added.)
Support for command disconnect.
A reselection timeout procedure compliant with Option b of Section 6.1.4.2 of the SCSI--2 standard. Furthermore, the device shall implement a reselection retry algorithm that limits the amount of bus time spent attempting to reselect a nonresponsive initiator.
Automatic read reallocation enabled (ARRE) and automatic write reallocation enabled (AWRE) (that is, drive-based bad block revectoring) to prevent multiple hosts from unnecessarily revectoring the same block. To avoid data corruption, it is essential that the drive comply with Section 9.3.3.6 of the SCSI--2 Standard, which says (in part):
- "...The automatic reallocation shall then be performed only if the target successfully recovers the data." (Emphasis added.)
Storage devices should not supply TERMPWR. If they do, then it is necessary to apply configuration rules to ensure that there are no more than four sources of TERMPWR on a segment.

Finally, if the device or any other device on the same segment will be hot plugged, then the device must meet the electrical requirements described in Section A.7.6.2.

A.7.8 Grounding Requirements

This section describes the grounding requirements for electrical systems in a SCSI OpenVMS Cluster system.

Improper grounding can result in voltage differentials, called ground offset voltages, between the enclosures in the configuration. Even small ground offset voltages across the SCSI interconnect (as shown in step 3 of Table A-9) can disrupt the configuration and cause system performance degradation or data corruption.

Table A-9 describes important considerations to ensure proper grounding.

Table A-9 Steps for Ensuring Proper Grounding
Step Description

1 Ensure that site power distribution meets all local electrical codes.

2 Inspect the entire site power distribution system to ensure that:

All outlets have power ground connections.
A grounding prong is present on all computer equipment power cables.
Power-outlet neutral connections are not actual ground connections.
All grounds for the power outlets are connected to the same power distribution panel.
All devices that are connected to the same circuit breaker as the computer equipment are UL® or IEC approved.

3 If you have difficulty verifying these conditions, you can use a hand-held multimeter to measure the ground offset voltage between any two cabinets. To measure the voltage, connect the multimeter leads to unpainted metal on each enclosure. Then determine whether the voltage exceeds the following allowable ground offset limits:

Single-ended signaling: 50 millivolts (maximum allowable offset)
Differential signaling: 800 millivolts (maximum allowable offset)

The multimeter method provides data for only the moment it is measured. The ground offset values may change over time as additional devices are activated or plugged into the same power source. To ensure that the ground offsets remain within acceptable limits over time, Digital recommends that you have a power survey performed by a qualified electrician.

4 If you are uncertain about the grounding situation or if the measured offset exceeds the allowed limit, Digital recommends that a qualified electrician correct the problem. It may be necessary to install grounding cables between enclosures to reduce the measured offset.

5 If an unacceptable offset voltage was measured and a ground cable was installed, then measure the voltage again to ensure it is less than the allowed limits. If not, an electrician must determine the source of the ground offset voltage and reduce or eliminate it.

**Table A-9 Steps for Ensuring Proper Grounding**
Step	Description
1	Ensure that site power distribution meets all local electrical codes.
2	Inspect the entire site power distribution system to ensure that: All outlets have power ground connections. A grounding prong is present on all computer equipment power cables. Power-outlet neutral connections are not actual ground connections. All grounds for the power outlets are connected to the same power distribution panel. All devices that are connected to the same circuit breaker as the computer equipment are UL® or IEC approved.
3	If you have difficulty verifying these conditions, you can use a hand-held multimeter to measure the ground offset voltage between any two cabinets. To measure the voltage, connect the multimeter leads to unpainted metal on each enclosure. Then determine whether the voltage exceeds the following allowable ground offset limits: Single-ended signaling: 50 millivolts (maximum allowable offset) Differential signaling: 800 millivolts (maximum allowable offset) The multimeter method provides data for only the moment it is measured. The ground offset values may change over time as additional devices are activated or plugged into the same power source. To ensure that the ground offsets remain within acceptable limits over time, Digital recommends that you have a power survey performed by a qualified electrician.
4	If you are uncertain about the grounding situation or if the measured offset exceeds the allowed limit, Digital recommends that a qualified electrician correct the problem. It may be necessary to install grounding cables between enclosures to reduce the measured offset.
5	If an unacceptable offset voltage was measured and a ground cable was installed, then measure the voltage again to ensure it is less than the allowed limits. If not, an electrician must determine the source of the ground offset voltage and reduce or eliminate it.

Note

¹ Referring to this draft standard is necessary because the SCSI--2 standard does not adequately specify the requirements for hot plugging.

² OpenVMS will eventually detect a hung bus and reset it, but this problem may first temporarily disrupt OpenVMS Cluster operations.

  6318P012.HTM
  OSSG Documentation
  26-NOV-1996 11:20:31.15

Legal