11g生产数据库crsctl start has无法启动故障处理

简介:

以下内容为模拟操作,因为客户核心数据库是不允许把日志拿出来的哈。
不过处理过程和思路几乎是一样的。

一、故障描述
11G RAC -单机ADG,备端HAS服务无法启动。

二、现象

查看了集群的各种日志,均无任何日志输出。

[root@roidb2 bin]# pwd
/u01/app/11.2.0/grid/bin
[root@roidb2 bin]# ./crsctl start has
[root@roidb2 bin]# --无输出,不提示报错,也不提示成功启动
[root@roidb2 bin]# 

怎么办,怎么办?第一次遇到这样的问题。问了客户,说了周五做了搬迁工作,难道是磁盘出了问题,还是权限出了问题。按照这个思路查了一遍,也没有什么发现。回过头来,整理了一下思路,使用strace来看一下,也许会有意想不到的收获。

[root@roidb2 bin]# strace ./crsctl start has
execve("./crsctl", ["./crsctl", "start", "has"], [/* 28 vars */]) = -1 ENOEXEC (Exec format error)  --格式错误
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff367e29000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: Exec format error\n", 32strace: exec: Exec format error
) = 32
close(3)                                = 0
munmap(0x7ff367e29000, 4096)            = 0
exit_group(1)                           = ?
[root@roidb2 bin]# 

为什么会是这样的报错,难道是文件又问题?继续往下查..........

[root@roidb2 bin]# ls -l crsctl
-rwxr-xr-x 1 root root 0 Dec 11 20:54 crsctl
[root@roidb2 bin]# file crsctl
crsctl: empty  --竟然是空文件!!!!!!!
[root@roidb2 bin]# 

怎么办,怎么办?我们知道这是一个脚本文件,那么,我们从其他节点copy一个文件怎么样呢?

三、处理过程

--远程传输一个文件过来呗
<roidb1:+ASM1:/home/grid>$scp /u01/app/11.2.0/grid/bin/crsctl root@192.168.1.212:/u01/app/11.2.0/grid/bin/
root@192.168.1.212's password: 
crsctl                                                                                 100% 8574     8.4KB/s   00:00    
<roidb1:+ASM1:/home/grid>$
[root@roidb2 bin]# file crsctl         
crsctl: POSIX shell script text executable
[root@roidb2 bin]# ./crsctl start has  
CRS-4123: Oracle High Availability Services has been started.
[root@roidb2 bin]# 
--搞定
--学习官方都怎么写脚本
[root@roidb2 bin]# cat crsctl
#!/bin/sh
#
# Copyright (c) 2001, 2013, Oracle and/or its affiliates. All rights reserved. 
# Notes:
#   - This script should only use clsecho.bin directly and not clsecho(which is
#     this same script).
#   - FIXME: crswrap should process hostname locally as well just like init.ohasd.

### Main ###
ORA_CRS_HOME=/u01/app/11.2.0/grid
MY_HOST=roidb1
ORACLE_USER=grid
ORACLE_HOME=/u01/app/11.2.0/grid
CRF_HOME=/u01/app/11.2.0/grid
export ORA_CRS_HOME ORACLE_HOME CRF_HOME

#limits
CRS_LIMIT_CORE=unlimited
CRS_LIMIT_MEMLOCK=unlimited
CRS_LIMIT_OPENFILE=65536
CRS_LIMIT_STACK=2048

#export the limit variables
export CRS_LIMIT_CORE CRS_LIMIT_MEMLOCK CRS_LIMIT_OPENFILE CRS_LIMIT_STACK

#listener
CRS_LSNR_STACK=10240
export CRS_LSNR_STACK

# Unset env var ORACLE_BASE before spawning any processes.
unset ORACLE_BASE

[ -z "$PERL" ] && PERL="/u01/app/11.2.0/grid/perl/bin/perl -I${ORA_CRS_HOME}/perl/lib"
LOGMSG="/bin/logger -puser.err"
CLSECHO="/u01/app/11.2.0/grid/bin/clsecho.bin"

PLATFORM=`/bin/uname`
case $PLATFORM in
Linux) 
       ORACLUSTER_LIB=/etc/ORCLcluster/lib
       LD_LIBRARY_PATH=/u01/app/11.2.0/grid/lib:$ORACLUSTER_LIB
       export LD_LIBRARY_PATH 

       # forcibly eliminate LD_ASSUME_KERNEL to ensure NPTL where available
       LD_ASSUME_KERNEL=
       export LD_ASSUME_KERNEL

       LOGGER="/usr/bin/logger"
       if [ ! -f "$LOGGER" ];then
        LOGGER="/bin/logger"
       fi
       LOGMSG="$LOGGER -puser.err"
       ;;
HP-UX) MACH_HARDWARE=`/bin/uname -m`
       if [ "$MACH_HARDWARE" = "ia64" ]; then
          SO_EXT=so
          NMAPIDIR_64=/opt/nmapi/nmapi2/lib/hpux64
          NMAPIDIR_32=/opt/nmapi/nmapi2/lib/hpux32
       else
          SO_EXT=sl
          NMAPIDIR_64=/opt/nmapi/nmapi2/lib/pa20_64
          NMAPIDIR_32=/opt/nmapi/nmapi2/lib
       fi 

       case $0 in
           */lsnodes|lsnodes)
               if [ ! -f $NMAPIDIR_64/libnmapi2.so -a ! -f $NMAPIDIR_32/libnmapi2.so ]; then
                   /bin/echo "No vendor clusterware installed."
                   exit 1
               fi
               ;;
       esac

       LD_LIBRARY_PATH=/u01/app/11.2.0/grid/lib:$NMAPIDIR_64:/usr/lib:$LD_LIBRARY_PATH
       SHLIB_PATH=/u01/app/11.2.0/grid/lib32:$NMAPIDIR_32:$SHLIB_PATH
       export LD_LIBRARY_PATH
       export SHLIB_PATH 
       ;;
SunOS) ARCH_NAME=`/bin/uname -p`
       if [ "${ARCH_NAME}" = "sparc" ]; then
           LD_LIBRARY_PATH_64=/u01/app/11.2.0/grid/lib:/opt/ORCLcluster/lib:/usr/lib/sparcv9:/usr/ucblib/sparcv9:$LD_LIBRARY_PATH_64
       else
           LD_LIBRARY_PATH_64=/u01/app/11.2.0/grid/lib:/opt/ORCLcluster/lib:/usr/lib/amd64:/usr/ucblib/amd64:$LD_LIBRARY_PATH_64
       fi
       LD_LIBRARY_PATH=/u01/app/11.2.0/grid/lib:/opt/ORCLcluster/lib:/usr/lib:/usr/ucblib:$LD_LIBRARY_PATH
       export LD_LIBRARY_PATH_64
       export LD_LIBRARY_PATH
       GREP='/usr/bin/grep'
       /usr/bin/coreadm | $GREP  'process core dumps' | $GREP  'enabled' > /dev/null
       STATUS1=$?
       /usr/bin/coreadm | $GREP  'global core dumps' | $GREP 'enabled' > /dev/null
       STATUS2=$?
       if [ "$STATUS1" != "0" ] && [ "$STATUS2" != "0" ];
       then
           /usr/bin/coreadm -e global > /dev/null 2>&1
       fi

       /usr/bin/coreadm | $GREP  'process setid' | $GREP 'enabled' > /dev/null
       STATUS1=$?
       /usr/bin/coreadm | $GREP  'global setid' | $GREP 'enabled' > /dev/null
       STATUS2=$?
       if [ "$STATUS1" != "0" ] && [ "$STATUS2" != "0" ];
       then
           /usr/bin/coreadm -e global-setid > /dev/null 2>&1
       fi

       # Solaris allows partitioning of resources by Projects.
       # On Solaris, start crsd/ohasd using the default Project of
       # the owner of the Grid Home. See bugs 9442360 / 5629487.
       PROJECT=`/usr/bin/projects -d $ORACLE_USER` 
       # If no project is set use the default root project
       if [ "$PROJECT" = "" ]; then
           PROJECT="user.root"
       fi
       ;;

AIX)   ORACLUSTER_LIB=/opt/ORCLcluster/lib 
       LIBPATH=/u01/app/11.2.0/grid/lib:$ORACLUSTER_LIB:/usr/lib
       LD_LIBRARY_PATH=$LIBPATH:$LD_LIBRARY_PATH
       AIXTHREAD_SCOPE=S
       export LIBPATH
       export LD_LIBRARY_PATH
       export AIXTHREAD_SCOPE
       ;;
*)     /bin/echo "ERROR: Unknown Operating System"
       exit -1
       ;;
esac

# enable GIPCHA consistently along with root scripts
case $PLATFORM in
  Linux)
    GIPCD_PASSTHROUGH=false
    export GIPCD_PASSTHROUGH
    ;;
  HP-UX)
    GIPCD_PASSTHROUGH=false
    export GIPCD_PASSTHROUGH
    ;;
  SunOS)
    GIPCD_PASSTHROUGH=false
    export GIPCD_PASSTHROUGH
    ;;
  AIX)
    GIPCD_PASSTHROUGH=false
    export GIPCD_PASSTHROUGH
    ;;
  OSF1)
    ;;
esac

case $0 in
*.bin) 
    ORASYM=/u01/app/11.2.0/grid/bin/`basename $0 .bin`
    ;;
*)     
    ORASYM=$0.bin
    ;;
esac

export ORASYM

case $ORASYM in
*ocrpatch*)
     if [ ! -x $ORASYM ]
     then
       /bin/echo "NOTE:"
       /bin/echo "The ocrpatch binary is not part of the software distribution;"
       /bin/echo "ocrpatch can only be obtained and used by Oracle Support."
       exit -1
     fi
     ;;
*ocssd*)
     if [ "$PLATFORM" = "AIX" ]
     then
       UID=`id -u`
       if [ $UID -eq 0 ]; # do not want to do su in SIHA
       then
         SU='/bin/su'
         $SU $ORACLE_USER -c "/bin/sh -c 'ulimit -c unlimited; $ORASYM $@'"
         exit 0
       fi
     fi
     ;;
*ohasd*)
    CRSWRAPEXECE="/u01/app/11.2.0/grid/bin/crswrapexece.pl"
    ENV_FILE="${ORA_CRS_HOME}/crs/install/s_crsconfig_${MY_HOST}_env.txt"
    export ENV_FILE

    if [ ! -f "$CRSWRAPEXECE" ]
    then
      $LOGMSG "$CRSWRAPEXECE script is not found"
      exit 1;
    fi

    # we attempt to set limits here and check if return code is 0
    # if not we generate an alert using clsecho
    # see init.ohasd.sbs for a full rationale

    #STACK_SIZE limit. The goal is to reduce thread usage across the grid
    #infrastructure bottom up from the ohasd wrapper (Bug 9154152).
    #Only the soft limit is set so that any process even unpriviledged can
    #reincrease it up to the administrator set hard limit
    ulimit -Ss 2048
    if [ "$?" != "0" ]
        then
        $CLSECHO -p has -f crs -l -m 6021 "Ss" "2048"
    fi    

    case $PLATFORM in
    Linux) 
        # MEMLOCK limit is for Bug 9136459
        ulimit -l unlimited
        if [ "$?" != "0" ]
        then
            $CLSECHO -p has -f crs -l -m 6021 "l" "unlimited"
        fi
        ulimit -c unlimited
        if [ "$?" != "0" ]
        then
            $CLSECHO -p has -f crs -l -m 6021 "c" "unlimited"
        fi
        ulimit -n 65536
        if [ "$?" != "0" ]
        then
            $CLSECHO -p has -f crs -l -m 6021 "n" "65536"
        fi
        ;;
    *) 
        ulimit -c unlimited
        if [ "$?" != "0" ]
        then
            $CLSECHO -p has -f crs -l -m 6021 "c" "unlimited"
        fi
        ulimit -n 65536
        if [ "$?" != "0" ]
        then
            $CLSECHO -p has -f crs -l -m 6021 "n" "65536"
        fi
        ;;
    esac

    $LOGMSG "exec $PERL /u01/app/11.2.0/grid/bin/crswrapexece.pl $ENV_FILE $ORASYM \"$@\""
    exec $PERL /u01/app/11.2.0/grid/bin/crswrapexece.pl $ENV_FILE $ORASYM "$@"

    # Reached here only if exec fails
    /bin/echo "Failed to execute \"exec $PERL /u01/app/11.2.0/grid/bin/crswrapexece.pl $ENV_FILE $ORASYM \"$@\""
    $LOGMSG "Failed to execute \"exec $PERL /u01/app/11.2.0/grid/bin/crswrapexece.pl $ENV_FILE $ORASYM \"$@\""
    exit 1;
    ;;
*)
    if [ "$PLATFORM" = "AIX" ]
    then
      # Prevents the setting of RT_GRQ for non-ocssd and non-cssagent processes
      # RT_GRQ is turned on globally for all processes in the environment file
      # generated by s_crsconfig_lib.pm during install setup, for AIX platform.
      # This should prevent rdbms RT processes from inheriting this attribute
      # since crsd will not have RT_GRQ set.
      #
      # NOTE: cssdagent and monitor does not need a special case since they
      #       do not use this wrapper script. So the '*)' case here does not
      #       apply and they *will* inherit RT_GRQ attribute, as intended
      RT_GRQ=
      export RT_GRQ
    fi
    ;;
esac

# Solaris allows partitioning of resources by Projects.
# On Solaris, start crsd/ohasd using the default Project of
# the owner of the Grid Home. See bugs 9442360 / 5629487.
case $PLATFORM in
SunOS)
    case $ORASYM in
    *ohasd*|*crsd*)
         exec /usr/bin/newtask -p $PROJECT $ORASYM "$@"
         ;;
    *)
         exec $ORASYM "$@"
         ;;
    esac
    ;;
*)
    exec $ORASYM "$@"
    ;;
esac 
[root@roidb2 bin]# 

小结:
1.数据库、主机的启停一定要正常步骤进行,切记直接断电。
2.搬迁之前,做好备份工作,移动安装设备要注意轻拿轻放。










本文转自 roidba 51CTO博客,原文链接:http://blog.51cto.com/roidba/2049554,如需转载请自行联系原作者

目录
相关文章
|
7月前
|
Kubernetes 关系型数据库 MySQL
ChaosBlade常见问题之数据库进行故障注入报错ibdata1文件异常如何解决
ChaosBlade 是一个开源的混沌工程实验工具,旨在通过模拟各种常见的硬件、软件、网络、应用等故障,帮助开发者在测试环境中验证系统的容错和自动恢复能力。以下是关于ChaosBlade的一些常见问题合集:
154 1
|
5月前
|
SQL druid Java
线程池相关故障问题之Druid数据库连接池中,为何需要设置TransactionTimeout
线程池相关故障问题之Druid数据库连接池中,为何需要设置TransactionTimeout
155 0
|
2月前
|
存储 Oracle 关系型数据库
数据库数据恢复—Oracle ASM磁盘组故障数据恢复案例
Oracle数据库数据恢复环境&故障: Oracle ASM磁盘组由4块磁盘组成。Oracle ASM磁盘组掉线 ,ASM实例不能mount。 Oracle数据库故障分析&恢复方案: 数据库数据恢复工程师对组成ASM磁盘组的磁盘进行分析。对ASM元数据进行分析发现ASM存储元数据损坏,导致磁盘组无法挂载。
|
3月前
|
存储 数据挖掘 数据库
服务器数据恢复—raid磁盘故障导致数据库数据损坏的数据恢复案例
存储中有一组由3块SAS硬盘组建的raid。上层win server操作系统层面划分了3个分区,数据库存放在D分区,备份存放在E分区。 RAID中一块硬盘的指示灯亮红色,D分区无法识别;E分区可识别,但是拷贝文件报错。管理员重启服务器,导致离线的硬盘上线开始同步数据,同步还没有完成就直接强制关机了,之后就没有动过服务器。
|
4月前
|
存储 Oracle 安全
服务器数据恢复—Raid故障导致数据库数据丢失的数据恢复案例
一台光纤存储中有一组由16块硬盘组成的raid。 该存储出现故障导致数据丢失。RAID中2块盘掉线,还有1块盘smart状态为“警告”。
|
4月前
|
监控 数据可视化 前端开发
基于python django生产数据与计划大屏,可链接数据库
本文介绍了一个基于Python Django框架开发的生产数据与计划大屏系统,该系统能够实时采集和展示生产数据,支持数据可视化和实时更新,以提高生产监控的效率和质量。
|
7月前
|
存储 关系型数据库 数据处理
OceanBase数据库常见问题之start ob出错如何解决
OceanBase 是一款由阿里巴巴集团研发的企业级分布式关系型数据库,它具有高可用、高性能、可水平扩展等特点。以下是OceanBase 数据库使用过程中可能遇到的一些常见问题及其解答的汇总,以帮助用户更好地理解和使用这款数据库产品。
|
4月前
|
数据库
【计算机三级数据库技术】第11章 数据库的故障管理--附思维导图
文章概述了数据库故障类型及其解决办法、数据库恢复技术、数据转储、日志文件的使用与格式、硬件容错方案(包括RAID技术和服务器容错技术)、以及数据库镜像与容灭策略。
59 2
|
5月前
|
Oracle 关系型数据库 数据库
关系型数据库Oracle 故障转移能力
【7月更文挑战第10天】
64 2
|
6月前
|
SQL 关系型数据库 MySQL
使用mysql数据库的binlog应对故障
【6月更文挑战第1天】本文介绍`mysql的 binlog`工具用于解析MySQL的二进制日志,转换为可执行的SQL语句,主要用于数据库主从复制和增量恢复。定期备份和binlog推送能实现故障时的数据恢复。
259 9
使用mysql数据库的binlog应对故障