Automatic Shared Memory Management problem ?

From time to time one of out 10g databases (10.2.0.5) seems to 'hang'
Our monitoring shows a 'time out' on different checks and when trying to connect using sql, the sql session is hanging. No connection is possible.
A few days ago, something like this happened again. Instead of bouncing the database, I decided to look for clues to find out why the database was 'hanging'.
The server itself did not show anything. Nothing strange was going on at that time, also the alert.log of the database did not show anything. Nothing strange before the hang and nothing during the hang. Also the other 'dump' directories did not help me in any way.
At that time I noticed the database was responding again.

The next day I decided the run an AWR report using a snapshot before and one after the hanging. I noticed the wait event 'SGA: allocation forcing component growth' in the top 3 of wait events. I also remembered that sometimes the mmon proces is using 100% cpu during the hang period. I did not notice that this time, but decided to look closer at this event.

I found the following going on at the time of the 'hanging' of the database (using the v$sga_resize_ops)
We noticed the hanging around  4:05 pm....The v$sga_resize_ops shows..

WHEN                                COMPONENT                     OPER_TYPE
-------------------------           -------------------------              -------------
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK

The sga was resized  406 times (!!!) in one second. From 16:08:12 the resizing looked normal (?) again..

Mrt-12:16:08:11           shared pool                         GROW
Mrt-12:16:08:12           DEFAULT buffer cache     SHRINK
Mrt-12:16:08:12           shared pool                        GROW
Mrt-12:16:08:12           DEFAULT buffer cache     SHRINK
Mrt-12:16:08:12           shared pool                        GROW
Mrt-12:16:08:14           DEFAULT buffer cache     SHRINK

I have to check this next time one off our databases is hanging, but this does not look good to me.
I noticed some bugs on MOS, but with no solution. A workaround solution is to disable ASMM by setting the sga_target to 0.

Missing controlfiles ?

During the backup check this morning, I noticed something strange..
The backup logfile was showing the following error :

Starting Control File and SPFILE Autobackup at 10-12-2012 01:46:29

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 12/10/2012 01:46:31

ORA-19504: failed to create file "+FRA/ngp01/controlfile/c-1446976194-20121210-02"

ORA-17502: ksfdcre:4 Failed to create file +FRA/ngp01/controlfile/c-1446976194-20121210-02

ORA-15173: entry 'ngp01' does not exist in directory '/'

I fist checked if the directory really did not exist in ASM.

Using the ASMCMD, it seems the directory is not there.

ASMCMD> ls FRA
NGP01I/
NGP02I/

But checking the status of the controlfile in the database :

SQL> select name from v$controlfile;

NAME
--------------------------------------------------------------------------------
+NGPDATA/ngp01/controlfile/current.263.796827271
+FRA/ngp01/controlfile/current.3434.796827271
 
Same directory that does not exist according to RMAN (and I can't see it neither in ASM)

Still, the database is running.... No errors in the alert about missing controlfiles. How can the database be running normally with controlfiles that we and RMAN cannot see ??

Update
Could it be possible the files still exist in the ASM cache ? The database is running fine, I also executed a backup controlfile to trace, and this file looks good. So, it seems the controlfile is online for the database  (11.2.0.1.0), but it's no longer present on disk. So, what is the database using and updating ?

Oracle OEM 12c unzipping.....

I decided to download the OEM12c software from Oracle to install the new OEM.
After downloading the software, I had some trouble unzipping the two files.
The files downloaded where :
- em12_linux_x86_disk1of2.zip
- em12_linux_x86_disk2of2.zip

When unzipping the files, the following error occurred :

[oracle@localhost software]$ unzip em12_linux_x86_disk2of2.zip
Archive: em12_linux_x86_disk2of2.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of em12_linux_x86_disk2of2.zip or
em12_linux_x86_disk2of2.zip.zip, and cannot find em12_linux_x86_disk2of2.zip.ZIP, period.

The same error occured at both zip files, so I decided to download them again.... :(
After the second download, the unzip still showed the same errors. I downloaded the unzip from the download site...still no change. Then I downloaded the unzip for large files, the LINUX Intel (32bit and 64bit - Large file support) unzip tool. This time no errors...

Oralce eBS R12 Financials Cookbook

Last week I was reading a Oracle eBS R12 Financials Cookbook.

This book is written for Oracle functional eBS specialists.

The book describes the most important processes in financials, and how to execute those processes. For example, voiding a payment. The book is showing screen prints of how to do this in R12. Not only is the book using screen pints from R12, but also a step by step explanation is given.

Each process is divided into a few steps. First step is getting ready, next one is 'How to do it...' followed by a 'How it works' and sometimes a 'See also' at the end of each process. So, each process in financials is clearly presented by using screen prints of that process and the steps to follow.

Other important processes that are mentioned in the cookbook are :

- Entering a prepayment

- Defining the chart of accounts

- Defining E-Business Tax

- Paying invoices

- Creating Items (inventory)

more processes are displayed in the cookbook. A must have for everybody who is working with the Oracle E-Business Suite R12.

Interested in this book ? Check http://link.packtpub.com/2QoJty

Also I am able to give away 3 e-copies of this book to anyone who can answer the following eBS question.

'What is the realtionship between the operating unit and the set of books in Oracle eBS ?'

Leave your email and name and a e-copy will be sent to you.

Oem Agent not starting anymore..

This morning I noticed a database down in our OEM application. I also received an email from the inicident, so I decided to look into this problem. I found a sollution very quick. It seemed the dbsnmp password of the target database was not valid anymore. So I changed the password in OEM, configured the target again en the problem was solved...

But then the agent stopt running and was producing al lot of errors like ..

EMAgent exited with signal 6

EMAgent is Thrashing. Exiting watchdog

oem agent Segmentation Fault

Could this have something to do with the password reset I did before ? I looked into the log/trace files but could not find anything special, except these errors. I decided to change the dbsnmp password again and not to change it in OEM, so the agent could not connect to the target. This time the agent started.

Chekcing the target again, I found the follwoing when executing sqlplus as system...

ORA-28002: the password will expire wuthin 7 days.

The target is a 11g database, could this be the problem ? I decided to edit the 'DEFAULT' profile.

alter profile default limit password_life_time unlimited;

Changed the system password back to is't previous value (so nothing changed...) and everything was fine.
The target is up and running in OEM..

Concurrent request error

A concurrent request was running into an error. The following error was shown in the logfile of the request.

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application to which the concurrent program executable &EXECUTABLE belongs.

I noticed the $CUSTOM_TOP variable was not set, but after setting this variable the same error occurred. 

Then I noticed something strange, the $XXX_TOP was not set in the environment. After adding this top to the adovars.env (to test the request), the concurrent request ended without error.

Configure Oracle dbconsole...

For a 10g database I wanted to check something in the database console. When trying to start this console, I received the following error : OC4J Configuration issue. Oracle/ora102/oc4j/j2ee/OC4J_DBConsole_.local_ not found

The ORACLE_HOME is a shared one, meaning different databases use the same ORACLE_HOME. So, because of this there were some other directories but not the one I needed. I decided to configure the dbconsole for this database using the emca tool. ./emca -config dbcontrol db

STARTED EMCA at Sep 27, 2010 10:38:38 AM

EM Configuration Assistant, Version 10.2.0.1.0 Production

Copyright (c) 2003, 2005, Oracle. All rights reserved. Enter the following information:

Database SID:

Listener port number: 1521

Password for SYS user:

Password for DBSNMP user:

Password for SYSMAN user:  

Email address for notifications (optional):

Outgoing Mail (SMTP) server for notifications (optional): ----------------------------------------------------------------- You have specified the following settings Database ORACLE_HOME ................ /oracle/ora102 Database hostname ................

Listener port number ................ 1521

Database SID ................

Email address for notifications ...............

Outgoing Mail (SMTP) server for notifications ............... -----------------------------------------------------------------

Do you wish to continue? [yes(Y)/no(N)]: Y

Sep 27, 2010 10:43:24 AM

oracle.sysman.emcp.EMConfig perform

INFO: This operation is being logged at /oracle/ora102/cfgtoollogs/emca/wtcgo/emca_2010-09-27_10-38-38-AM.log. Sep 27, 2010 10:43:27 AM oracle.sysman.emcp.util.DBControlUtil startOMS

INFO: Starting Database Control (this may take a while) ... Sep 27, 2010 10:45:02 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration

INFO: Database Control started successfully Sep 27, 2010 10:45:02 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration

INFO: >>>>>>>>>>> The Database Control URL is http://host.domain:5503/em <<<<<<<<<<<

Enterprise Manager configuration completed successfully

FINISHED EMCA at Sep 27, 2010 10:45:02 AM

After this the dbconsole was running and I was able to check something for the customer.