AIX平台下磁盘的PVID对ASM磁盘的破坏

In this Document

APPLIES TO:

DESCRIPTION

Assigning a Physical Volume ID (PVID) to an existing ASM disk will destroy the ASM disk header rendering the
ASM disk unusable.

Various documents including the 10gR1 and 10gR2 installation instructions for AIX platforms suggest to assign
a PVID to disks to be used for ASM using the following command:
  

  # /usr/sbin/chdev -l hdiskn -a pv=yes

These documents furthermore suggest that this command is to be run on ALL nodes of a RAC cluster. This
does not present a problem as long as the disks have not yet been used by ASM. If however the disk are 
already in use and above command is issued against an ASM disk the file header will be destroyed. 
This is likely to happen if a new node is added to an existing RAC cluster as the documentation seems
to imply this has to be done on all nodes.

To check if a device has an associated PVID , use lspv:
EXAMPLE:


# lspv

hdisk0 0003286f04bc73ee rootvg active
hdisk1 0003286f867d77e1 rootvg active
hdisk2 0003286fb3470dae vg01 active
hdisk3 0003286fb3474190 vg01 active
hdisk4 0003286fb34747d1 vg01 active
hdisk5 0003286fb3474dff vg01 active
hdisk6 0003286fb3475428 vg01 active
hdisk7 0003286fb347607d vg01 active
hdisk8 0003286fb34766f3 vg01 active
hdisk9 0003286fb3476d70 vg01 active
hdisk10 0003286fb34773d5 vg01 active
hdisk11 0003286fb34780b8 vg01 active
hdisk12 0003286fb347872f vg01 active
hdisk13 0003286fb347940c vg01 active
hdisk14 0003286fb3479a7b vg01 active

The second column is the PVID.

OCCURRENCE

This is more likely to happen in a RAC environment, specifically if a new node is added to an existing
cluster.

SYMPTOMS

If the 'chdev' command is run while ASM instances have the disk mounted nothing will be noticed immediately 
as the disk header is only read when the disk is mounted. If however the diskgroup is unmounted and re-mounted
(e.g. ASM instance restart) the disk is no longer recognized as an ASM disk and the diskgroup mount will fail
with 
ORA-15063 "diskgroup \"%s\" lacks quorum of %s PST disks; %s found"
or
ORA-15063: ASM discovered an insufficient number of disks for diskgroup s%
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "%" is missing

WORKAROUND

Do not assign a PVID to ASM disks, contrary to the documentation PVIDs are not required for ASM disks 
as ASM uses the ASM disk header to discover it's disks. 
This has been addressed in (Documentation)  Bug 3636335 which states:

       "This is a doc. bug and we are going to clearly document not to put PVIDs on
        disks given to ASM. The idea here is that ASM is the one which manages the
        disk and not any OS / vendor volume managers etc., PVIDs are needed for volume
        groups to work. For ASM to work, PVIDs are not needed. ASM has its own headers
        to identify the disk which is what is getting written here. "

As long as there is still an ASM instance which has the disk(group) mounted the 
contents may be backed up via RMAN as soon as possible.

Also the action plan from the Document 750016.1 can be applied. Also recommend to raise an SR with Oracle Support.

HISTORY

 Checked for relevance on 18-APR-2013

REFERENCES

In this Document

APPLIES TO:

GOAL

You have created a diskgroup with disks having PVID and the diskgroup is in use. There is no diskgroup metadata corruption reported yet. You now know that ASM Disk should not have PVID as alerted in MetaLink Note 353761.1 

This note will give the steps to clear the PVID of these ASM Disks.

SOLUTION

When the PVID is set to a disk in a volume group, the PVID is stored in two locations. In Physical disk header ( within first 4K )and in AIX's system object database, called ODM ( Object Data Manager ). 

When the diskgroup is created, the disk header information of PVID is overwritten. However, with reboot the OS, from ODM, AIX might try to restore the PVID information onto the disk header, 
there by destroying the ASM metadata. 

If the ASM disk header Metadata has not been over written by PVID from ODM ( before a reboot ), then you can follow the following steps to update the ODM not to have PVID for the disks:

1] Do not reboot any node.

1.1] Drop one disk at a time from the diskgroup.

1.2] Clear the PVID of the dropped disk

# chdev -l hdisk5 -a pv=clear

Run this on ALL the nodes in case of RAC.

1.3] Check the disk does not have the PVID from ALL the nodes

# lspv

1.4] Add the disk back to the diskgroup

1.5] Do this for all the disks having PVID in the diskgroup, one by one. Take care that the rebalance is complete from the drop/add disk command before going for the next disk.

OR

2] This needs downtime:

2.1] Take 'dd' backup of the disk headers

# dd if=/dev/hdisk5 of=/tmp/d5.txt bs=1024 count=1024

2.2] Shutdown ASM instance ( on ALL the nodes in RAC setup ).

2.3] Clear the PVID

# chdev -l hdisk5 -a pv=clear

Run this on ALL the nodes in case of RAC.

2.4] Check the disk does not have the PVID from ALL the nodes

# lspv

2.5] Start the ASM Instance(s) and mount the diskgroup on ALL the nodes


WARNING:
Point-2 commands overrides the content of the disk header and so could be destructive if not correctly used. If you have any doubt, raise an SR with Oracle Support before any action.