ERROR OGG-01032 There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. Length is 3 - 000000: 46 45 00 |FE. |.

After the Server 2 crashed and brought up the pumps on Server 1 sending trails to Server2 was abending with below error

ERROR   OGG-01032  There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. Length is 3 - 000000: 46 45 00                                        |FE.             |.

Cause

Trail file system on target goes out of space causing the pump  to hang. The issue is ultimately due to an  interrupted rmt trail writing caused by connection issues because the target file system cannot be written to.

Note that this is a single instance of a broader issue. Anytime the collector server fails for any reason, it will look like a TCP or network issue to the pump. This can be space or security or path issues or anything else.

Solution

Fix any target space issue.

1) Go the source server and login into ggsci

- Execute the below command
alter <extract pump> etrollover

start <extract pump>

(ETROLLOVER will force the pump to write on a fresh trail)

3) Go the target server and check if the new trail files are being generated.

4) Verify if the first record in the new trail file is indeed the next record in the local extract trail. (It can be done by looking at the RBA value through the logdump utility)

5) Alter the replicat to the new trail file
alter <replicat> extseqno <new trail file> extrba 0

6) Start the Replicat process

7) Verify if all the processes are running fine. 

Oracle : Determine Backups/Restore Time

col OPNAME for a30

select OPNAME,SOFAR/TOTALWORK*100 PERCENT_SOFAR, trunc(TIME_REMAINING/60) TIME_REMAINING,

trunc(ELAPSED_SECONDS/60) TIME_CONSUMED

from v$session_longops where TOTALWORK>0 and OPNAME like '%RMAN%';

Output for above query

OPNAME                         PERCENT_SOFAR TIME_REMAINING TIME_CONSUMED

------------------------------ ------------- -------------- -------------

RMAN: incremental datafile bac           100              0             0

kup

RMAN: full datafile backup               100              0             0

RMAN: incremental datafile bac           100              0             0

kup

RMAN: full datafile backup               100              0             0

RMAN: full datafile backup               100              0             0

RMAN: full datafile backup               100              0             0

RMAN: archived log backup                100              0             0

RMAN: archived log backup                100              0             0

RMAN: archived log backup                100              0             0

RMAN: archived log backup                100              0             0

Understanding the Output

About v$session_longops : Please click here 

OPNAME : Shows the Brief description of the operation

PERCENT_SOFAR : Percentage completed sofar

TIME_REMAINING : Estimated remaining time to complete the job

TIME_CONSUMED : Elapsed Time for the job

You can alter the above query to find the time remaining for 'restore' work as well by substituting 'RMAN' with 'RESTORE'

Oracle : ORA-06570: shared pool object does not exist, cannot be pinned/purged

SQL> select ADDRESS, HASH_VALUE from GV$SQLAREA where SQL_ID like '8xzc%';

ADDRESS HASH_VALUE
---------------- ----------
00000067B31EB568  1105770

Running from first node

SQL> execute dbms_shared_pool.purge('00000067B31EB568,1105770','C');
BEGIN dbms_shared_pool.purge('00000067B31EB568,1105770','C'); END;

*
ERROR at line 1:
ORA-06570: shared pool object does not exist, cannot be pinned/purged
ORA-06512: at "SYS.DBMS_SHARED_POOL", line 48
ORA-06512: at "SYS.DBMS_SHARED_POOL", line 115
ORA-06512: at line 1

As the details were obtained from GV$SQLAREA and it is RAC database, the purge should be done on another node as it might not be on the library cache on the instance you are connected.

Now, running it from another node

SQL> execute dbms_shared_pool.purge('00000067B31EB568,1105770','C');

PL/SQL procedure successfully completed.