CRS Stack Fails to Start After Reboot ORA-29702 CRS-0184

Problem Description
An attempt to manually start the instance fails with,

ORA-29702: error occurred in Cluster Group Service operation

crs_stat produces the error,

/oracle/crs/bin/crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

CSS stack did not come up as the following command issued out of init.cssd
crsctl check boot
failed with error message
Failed 3 to bind listening endpoint: (ADDRESS=(PROTOCOL=tcp)(HOST=node2priv))
and return code 11.
The error message was also written to /var/log/messages on this node.

Note that in 10gR2 the message no longer gets written to /var/log/messages, instead it will be written to /tmp/crsctl.


Cause of the Problem
Error message 'Failed 3 to bind listening endpoint: (ADDRESS=(PROTOCOL=tcp)(HOST=node2priv))' indicated that CSS had a problem creating the listening endpoint for hostname node2

In node1 within the /etc/hosts file the name of the private interconnect exist. But on node2 somehow it is missing the private interconnect in the /etc/hosts file. That's why the problem occurs.

Solution of The Problem

1. Add the missing hostnames to /etc/hosts

2. Stop and start CRS stack