案例信息
环境介绍
11.2.0.4 oracle 集群 。
1 使用scan地址无法连接数据库。且listener为启动状态。而且监听一场。
2 连接数据库报错,无法提供堆栈程序错误,数据库连接数未达到上线。
回复业务措施 。
1 将节点1 监听进程使用kill -9命令删除。
2 重启scan地址
srvctl stop scan_listener -i 1
srvctl start scan_listener -i 1
3 但是启动节点1的listener发现如下报错。
lsnrctl startError listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12555: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00525: Insufficient privilege for operation
Linux Error: 1: Operation not permitted
) = 443
集群节点1报错。
1 使用strace命令进行跟踪发现问题所在。
使用strace命令进行跟踪
strace srvctl start listener -l listener 未跟踪到任何有用信息。
strace lsnrctl start 跟踪到对应信息如下
nanosleep({0, 320000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 10000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 20000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
brk(0xa7e000) = 0xa7e000
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 100000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 200000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 10000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 20000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 40000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 80000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 160000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 320000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 10000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
access("/var/tmp/.oracle/sLISTENER", F_OK) = 0
nanosleep({0, 20000000}, 0x7ffd881c3f50) = 0
close(5) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 5
connect(5, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sLISTENER"}, 110) = -1 ECONNREFUSED (Connection refused)
查看对应文件的权限
[grid@orcl01 .oracle]$ ls -lrt sLISTENER
srwxrwxrwx 1 oracle oinstall 0 Oct 9 16:22 sLISTENER
我们知道listener使用grid用户管理的。于是尝试socket文件属主修改为grid
[root@orcl01 .oracle]# chown grid:oinstall sLISTENER
在启动listener发现启动成功
[grid@orcl01 ~]$ srvctl start listener -l listener
[grid@orcl01 ~]$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-OCT-2022 10:21:49
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 10-OCT-2022 10:21:44
Uptime 0 days 0 hr. 0 min. 4 sec
Trace Level off
Security ON: Password or Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2/grid/network/admin/listener.ora
Listener Log File /u01/app/11.2/grid/log/diag/tnslsnr/orcl01/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.28.24.207)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.28.24.209)(PORT=1521)))
Services Summary...
Service "orcl" has 1 instance(s).
Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "orclXDB" has 1 instance(s).
Instance "orcl1", status READY, has 1 handler(s) for this service...
Service "bptest" has 1 instance(s).
Instance "bppttest1", status READY, has 1 handler(s) for this service...
Service "bptestXDB" has 1 instance(s).
Instance "bppttest1", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@orcl01 ~]$
[grid@orcl01 ~]$
[grid@orcl01 ~]$
[grid@orcl01 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
ora.LISTENER.lsnr
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
ora.LISTENERV6.lsnr
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
ora.OCR.dg
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
ora.ORADATA.dg
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
ora.asm
ONLINE ONLINE orcl01 Started
ONLINE ONLINE orcl02 Started
ora.gsd
OFFLINE OFFLINE orcl01
OFFLINE OFFLINE orcl02
ora.net1.network
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
ora.ons
ONLINE ONLINE orcl01
ONLINE ONLINE orcl02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE orcl02
ora.orcl.db
1 ONLINE ONLINE orcl01 Open
2 ONLINE ONLINE orcl02 Open
ora.orcl01.vip
1 ONLINE ONLINE orcl01
ora.orcl02.vip
1 ONLINE ONLINE orcl02
ora.bppttest.db
1 ONLINE ONLINE orcl01 Open
2 ONLINE ONLINE orcl02 Open
ora.cvu
1 ONLINE ONLINE orcl02
ora.oc4j
1 ONLINE ONLINE orcl02
ora.scan1.vip
1 ONLINE ONLINE orcl02
而且资源也正常了。