Automatic Shared Memory Management problem ?

From time to time one of out 10g databases (10.2.0.5) seems to 'hang'
Our monitoring shows a 'time out' on different checks and when trying to connect using sql, the sql session is hanging. No connection is possible.
A few days ago, something like this happened again. Instead of bouncing the database, I decided to look for clues to find out why the database was 'hanging'.
The server itself did not show anything. Nothing strange was going on at that time, also the alert.log of the database did not show anything. Nothing strange before the hang and nothing during the hang. Also the other 'dump' directories did not help me in any way.
At that time I noticed the database was responding again.

The next day I decided the run an AWR report using a snapshot before and one after the hanging. I noticed the wait event 'SGA: allocation forcing component growth' in the top 3 of wait events. I also remembered that sometimes the mmon proces is using 100% cpu during the hang period. I did not notice that this time, but decided to look closer at this event.

I found the following going on at the time of the 'hanging' of the database (using the v$sga_resize_ops)
We noticed the hanging around  4:05 pm....The v$sga_resize_ops shows..

WHEN                                COMPONENT                     OPER_TYPE
-------------------------           -------------------------              -------------
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK
Mrt-12:16:08:11             shared pool                          GROW
Mrt-12:16:08:11             DEFAULT buffer cache       SHRINK

The sga was resized  406 times (!!!) in one second. From 16:08:12 the resizing looked normal (?) again..

Mrt-12:16:08:11           shared pool                         GROW
Mrt-12:16:08:12           DEFAULT buffer cache     SHRINK
Mrt-12:16:08:12           shared pool                        GROW
Mrt-12:16:08:12           DEFAULT buffer cache     SHRINK
Mrt-12:16:08:12           shared pool                        GROW
Mrt-12:16:08:14           DEFAULT buffer cache     SHRINK

I have to check this next time one off our databases is hanging, but this does not look good to me.
I noticed some bugs on MOS, but with no solution. A workaround solution is to disable ASMM by setting the sga_target to 0.

Missing controlfiles ?

During the backup check this morning, I noticed something strange..
The backup logfile was showing the following error :

Starting Control File and SPFILE Autobackup at 10-12-2012 01:46:29

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 12/10/2012 01:46:31

ORA-19504: failed to create file "+FRA/ngp01/controlfile/c-1446976194-20121210-02"

ORA-17502: ksfdcre:4 Failed to create file +FRA/ngp01/controlfile/c-1446976194-20121210-02

ORA-15173: entry 'ngp01' does not exist in directory '/'

I fist checked if the directory really did not exist in ASM.

Using the ASMCMD, it seems the directory is not there.

ASMCMD> ls FRA
NGP01I/
NGP02I/

But checking the status of the controlfile in the database :

SQL> select name from v$controlfile;

NAME
--------------------------------------------------------------------------------
+NGPDATA/ngp01/controlfile/current.263.796827271
+FRA/ngp01/controlfile/current.3434.796827271
 
Same directory that does not exist according to RMAN (and I can't see it neither in ASM)

Still, the database is running.... No errors in the alert about missing controlfiles. How can the database be running normally with controlfiles that we and RMAN cannot see ??

Update
Could it be possible the files still exist in the ASM cache ? The database is running fine, I also executed a backup controlfile to trace, and this file looks good. So, it seems the controlfile is online for the database  (11.2.0.1.0), but it's no longer present on disk. So, what is the database using and updating ?

Oracle OEM 12c unzipping.....

I decided to download the OEM12c software from Oracle to install the new OEM.
After downloading the software, I had some trouble unzipping the two files.
The files downloaded where :
- em12_linux_x86_disk1of2.zip
- em12_linux_x86_disk2of2.zip

When unzipping the files, the following error occurred :

[oracle@localhost software]$ unzip em12_linux_x86_disk2of2.zip
Archive: em12_linux_x86_disk2of2.zip
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of em12_linux_x86_disk2of2.zip or
em12_linux_x86_disk2of2.zip.zip, and cannot find em12_linux_x86_disk2of2.zip.ZIP, period.

The same error occured at both zip files, so I decided to download them again.... :(
After the second download, the unzip still showed the same errors. I downloaded the unzip from the download site...still no change. Then I downloaded the unzip for large files, the LINUX Intel (32bit and 64bit - Large file support) unzip tool. This time no errors...

Oralce eBS R12 Financials Cookbook

Last week I was reading a Oracle eBS R12 Financials Cookbook.

This book is written for Oracle functional eBS specialists.

The book describes the most important processes in financials, and how to execute those processes. For example, voiding a payment. The book is showing screen prints of how to do this in R12. Not only is the book using screen pints from R12, but also a step by step explanation is given.

Each process is divided into a few steps. First step is getting ready, next one is 'How to do it...' followed by a 'How it works' and sometimes a 'See also' at the end of each process. So, each process in financials is clearly presented by using screen prints of that process and the steps to follow.

Other important processes that are mentioned in the cookbook are :

- Entering a prepayment

- Defining the chart of accounts

- Defining E-Business Tax

- Paying invoices

- Creating Items (inventory)

more processes are displayed in the cookbook. A must have for everybody who is working with the Oracle E-Business Suite R12.

Interested in this book ? Check http://link.packtpub.com/2QoJty

Also I am able to give away 3 e-copies of this book to anyone who can answer the following eBS question.

'What is the realtionship between the operating unit and the set of books in Oracle eBS ?'

Leave your email and name and a e-copy will be sent to you.

Oem Agent not starting anymore..

This morning I noticed a database down in our OEM application. I also received an email from the inicident, so I decided to look into this problem. I found a sollution very quick. It seemed the dbsnmp password of the target database was not valid anymore. So I changed the password in OEM, configured the target again en the problem was solved...

But then the agent stopt running and was producing al lot of errors like ..

EMAgent exited with signal 6

EMAgent is Thrashing. Exiting watchdog

oem agent Segmentation Fault

Could this have something to do with the password reset I did before ? I looked into the log/trace files but could not find anything special, except these errors. I decided to change the dbsnmp password again and not to change it in OEM, so the agent could not connect to the target. This time the agent started.

Chekcing the target again, I found the follwoing when executing sqlplus as system...

ORA-28002: the password will expire wuthin 7 days.

The target is a 11g database, could this be the problem ? I decided to edit the 'DEFAULT' profile.

alter profile default limit password_life_time unlimited;

Changed the system password back to is't previous value (so nothing changed...) and everything was fine.
The target is up and running in OEM..

Concurrent request error

A concurrent request was running into an error. The following error was shown in the logfile of the request.

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application to which the concurrent program executable &EXECUTABLE belongs.

I noticed the $CUSTOM_TOP variable was not set, but after setting this variable the same error occurred. 

Then I noticed something strange, the $XXX_TOP was not set in the environment. After adding this top to the adovars.env (to test the request), the concurrent request ended without error.

Configure Oracle dbconsole...

For a 10g database I wanted to check something in the database console. When trying to start this console, I received the following error : OC4J Configuration issue. Oracle/ora102/oc4j/j2ee/OC4J_DBConsole_.local_ not found

The ORACLE_HOME is a shared one, meaning different databases use the same ORACLE_HOME. So, because of this there were some other directories but not the one I needed. I decided to configure the dbconsole for this database using the emca tool. ./emca -config dbcontrol db

STARTED EMCA at Sep 27, 2010 10:38:38 AM

EM Configuration Assistant, Version 10.2.0.1.0 Production

Copyright (c) 2003, 2005, Oracle. All rights reserved. Enter the following information:

Database SID:

Listener port number: 1521

Password for SYS user:

Password for DBSNMP user:

Password for SYSMAN user:  

Email address for notifications (optional):

Outgoing Mail (SMTP) server for notifications (optional): ----------------------------------------------------------------- You have specified the following settings Database ORACLE_HOME ................ /oracle/ora102 Database hostname ................

Listener port number ................ 1521

Database SID ................

Email address for notifications ...............

Outgoing Mail (SMTP) server for notifications ............... -----------------------------------------------------------------

Do you wish to continue? [yes(Y)/no(N)]: Y

Sep 27, 2010 10:43:24 AM

oracle.sysman.emcp.EMConfig perform

INFO: This operation is being logged at /oracle/ora102/cfgtoollogs/emca/wtcgo/emca_2010-09-27_10-38-38-AM.log. Sep 27, 2010 10:43:27 AM oracle.sysman.emcp.util.DBControlUtil startOMS

INFO: Starting Database Control (this may take a while) ... Sep 27, 2010 10:45:02 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration

INFO: Database Control started successfully Sep 27, 2010 10:45:02 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration

INFO: >>>>>>>>>>> The Database Control URL is http://host.domain:5503/em <<<<<<<<<<<

Enterprise Manager configuration completed successfully

FINISHED EMCA at Sep 27, 2010 10:45:02 AM

After this the dbconsole was running and I was able to check something for the customer.

Oracle webcache and Oracle eBS 11i....

I have installe Oracle webcache 10.1.2.3.0 to act as a reverse proxy server for our Oracle eBS environments.

After installing the webcache, I used note 380486.1 to configure webcache for our R12 environments. Now, we can use the webcache url to enter the R12 application.

For our 11i environments I used note 306653.1

Using the webcache I am able to login the 11i application, but when starting a forms session I see the following error

'You don't have permission to access /pls//fnd_icx_launch.launch on this server.'

What is going on here ???

Read/Write NTFS on my MacBook Pro

Today I tried to start my virtual linux machines (created in vmware workstation on windows to install/upgrade eBS environments) on my MacBook Pro. I downloaded a trial version of Vmware Fusion. When trying to start a virtual machine, Vmware would show me the following error 'Read only file system' and the machine would not start. It seemed I could not write to the folders containing the Vmware files. So, this is a Mac problem ?

It seems my Mac cannot work with the NTFS filesystem on my external disk. To solve the problem I downloaded the following file 'ntfs-3g-2010.5.22-macosx.dmg' from http://www.macupdate.com/info.php/id/24481/ntfs-3g.
After installing the file, I was able to open my virtual machines on my MacBook Pro.

Agent blocked....

In our 11g Grid Control I noticed an agent that was not uploading any data to the oms anymore.
When checking the status of the agent I noticed the following:

Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 199
Size of XML files pending upload(MB) : 50.03
Available disk space on upload filesystem : 28.80%
Collection Status : Disabled by Upload Manager
Last attempted heartbeat to OMS : 2010-08-06 11:11:46
Last successful heartbeat to OMS : unknown

So, it seems to be disabled by the upload manager ??? When forcing an upload using the './emctl upload agent' it showed the 'oms version not checked' error. Because this a very common error, I decided the focus on the 'disabled by upload manager' error.

It seemed the agent was blocked for some reason. To fix this problem I did the following :

1. In Grid Control 11g go to Setup->agents-> Misconfiguration Alerts and Blocked Agents

You will see the 'blocked' agent in this section.

2. Go the home page of the blocked agent and in the right upper corner press the button 'Resynchronize'
3. When succesful go back to the setup page in step 1.
4. Unblock the agent
5. force an upload on the command line..

./emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

The agent is up and running again.

Incomplete restore....

Yesterday I was asked to help during a restore/recovery operation of the eBS datbase. Before restoring the dba'er removed the original database (rm *). After restoring the database files from the backup abd trying to start the database, we noticed the database did not to start because he could not find his redo log files. It seemed the redo log files were not in the backup, and therefore not restored.

Database was brought down before the backup (cold backup), no redo logs and no archivelogs available.
Because the dba already tried to start the database, we had to restore it again.
After the second restore I tried to open the database using a new control file. I used the backup controlfile to trace option to create the controlfile creation script.
After the controlfile was created and the database mounted, I still could not see the new logfiles on disk ? I noticed the redolog files are created when opening the database using the resetlogs option ( alter database open resetlogs).
So, database is up again.

Oracle.exe on Windows

So, I solved my last problem. Deleting the registry entry of NLS_LANG solved the problem. After this, the open database also showed me the NLS error again. I decided to start again with another environment, maybe the source was the problem ;)

With a new set, I ran into another problem...the database service cannot be started.

DIM-00019: Create Service Error

The Oracle Service service terminated with service-specific error 997

So, why can't I create this service anymore ???
It does create an entry in the registry....

Cloning eBS environment on windows....

For a new project I am working with Oracle eBS 11i on the windows platform. When cloning an environment using rapidclone, the database clone is showing me an error. When the clone is trying to create a new controlfile, the following error is shown :

The log information will be written to
"F:\oracle\behdb\9.2.0\appsutil\log\beh_smdbp02\adcrdb.txt"
ECHO is off.
ECHO is off.
Creating the control file for beh database ...
ECHO is off.
SQL*Plus: Release 9.2.0.6.0 - Production on Wed Mar 31 12:51:21 2010
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
ERROR:
ORA-12705: invalid or unknown NLS parameter value specified

ORA-01031: insufficient privileges

The Oracle service on windows is started, but when I try to connect using sql..

F:\oracle\behdb\9.2.0>sqlplus /nolog
SQL*Plus: Release 9.2.0.6.0 - Production on Wed Mar 31 14:03:12 2010
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
SQL> conn / as sysdba
ERROR:
ORA-12705: invalid or unknown NLS parameter value specified

So what is going wrong here ?? I have checked the NLS settings. They look the same as the ones in the source db. In the registry only the NLS_LANG is set.
I also tried creating the Oracle service by starting it with the pfile, but still the same error...
I am still looking for a solution...

Happy Xmas to everybody !!!!!

Login problems R12

On our R12 eBS environment we are facing a problem when loggin in. It does not happen all the time, because we use more then one web node (loadbalancer), but when trying to acces the login page the following error is shown...

"Unable to generate forwarding URL. Exception: oracle.apps.fnd.cache.CacheException"

or a blank page is shown instead of the login page. In the applications.log file ($INST_TOP/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1), I found the following error :

09/11/10 08:04:38.130 html: chain failed
javax.servlet.ServletException: java.lang.RuntimeException: Unable to create user session. Please contact your System Administrator.
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.EvermindPageContext.handlePageThrowable(EvermindPageContext.java:899)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.EvermindPageContext.handlePageException(EvermindPageContext.java:816)
at _AppsLocalLogin._jspService(_AppsLocalLogin.java:303)
at com.orionserver[Oracle Containers for J2EE 10g (10.1.3.4.0) ].http.OrionHttpJspPage.service(OrionHttpJspPage.java:59)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:379)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:594)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:518)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.EvermindFilterChain.doFilter(EvermindFilterChain.java:15)
at oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.java:318)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:621)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:37 )
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.4.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:619)

Searching Metalink did not solve this. I could only find something about APPS_TS_ARCHIVE tablespace being full...
What could cause this problem in R12 ??

It seems it is unable to create a user session at that moment.

Missing AppsLogin.jsp...

I am still facing the same problem with my R12 upgrade.
When running the post install checks using Rapidwiz , only the JSP and the Login page show errors.
For jsp I see 'JSP not responding, waiting 15 seconds and retesting'
and the Login Page shows 'RW-50016: Error. -{0} was not created. File= {1}'

The strange thing is that all other checks are oke. Even the /OA_HTML/help check !
So, the problem is not in the /OA_HTML/ part I think. But when trying to open the help or AppsLogin.jsp in a browser shows me the same error. '/OA_HTML/AppsLogin.jsp not found on this server'

To be continued...

R12 upgrade

I Finally upgraded my 11.5.10.2 environment to R12.
I followed the steps mentioned in the different upgrade guides. What do I have runing right now ?

- Oracle eBS 12.0.6
- Oracle database 10.2.0.4
- Oracle tech stack (old ora directory) 10.1.2.3.0
- Oracle tech stack (old iAS directory) 10.1.3.4.0

So, having no problems during the upgrade proces, I finished starting al services. When trying to login my new upgraded eBS I recieved an error saying the /OA_HTML/AppsLogin.sjp could not be found on this server...

At first I thought some config must be missing, but before looking at those conifg files I first checked if the file AppsLogin.jsp was in the $OA_HTML directory. I was suprised to see the jsp was not in the $OA_HTML ??? Were is the jsp ? And why is it not in this directory ?
I used the same stage area before to perform an fresh install of R12, and the AppsLogin.jsp was always in the right place !

So, this is the stage I am in at this moment. Upgrade finished with no errors od problems o nthe way. But my AppsLogin.jsp is missing !

To be continued...

R12 upgrade

Finally I started my owm R12 uprgade last week.
I installed 11.5.10.2 on Oracle Linux 5, so I could upgrade this environment to R12.
At this moment I am installing the upgrade pacth for R12.

After installing the vision environment 11i, I used the rapidwiz to create the R12 environment on that same machine.
I will post al the upgrade steps when I am finished upgrading...

Removing Credentials from a Cloned EBS Database

During the post clone actions, changing the apps pasword went wrong. Because of that the applsys and apps password were not the same anymore. Trying to fix the problem using FNDCPASS was useless, FNDCPASS was not working anymore.
We saw 'APP-FND-02704: Unable to alter user APPS to change password.' in the log file. After trying it again, 'APP-FND-02704: Unable to alter user APPLSYS to change password.' appeared in the logfile. At t his moment both passwords could not be changed anymore using FNDCPASS.

The first thing we tried was restoring FND_USER and FND_ORACLE_USERID from our production database. Exporting the data from both tables and importing it in the cloned database did not solve the problem. This should work normally, but not if you have already run autoconfig....and we did before the passwords became out of sync.
Note 419475.1 from Metalink solved the problem. Resetting all passwords in the database (eBS and non eBS database users), did solve this !

OEM problem solved...

The OEM problem with the http server not startign is solved. We solved the problem by installing an older version of the package libdb.so.2 (for Suse 10 install older version of 1_db).
After this the http server started and I was able to continue the install.

At the end of the install again a http error...

[Wed Apr 22 11:41:33 2009] [error] mod_ossl: Unknown error
[Wed Apr 22 11:41:37 2009] [error] mod_ossl: Init: Server oracle-grid-control.waterschapsgis.nl:1159: SSL setup failed: call to nzos_RetrievePersonaCopy returned error 28788
[Wed Apr 22 11:41:37 2009] [error] mod_ossl: Unknown error

I could not remember seeing this error before, but I found a sollution anyway. I solved it by adjusting the httpd_em.conf and changing the SSLWallet file and passsword. These values must be the same as in the ssl.conf file.

So, after some problemns I managed to install the OEM 10.2.0.1 on Suse 10 linux.
After this , I upgraded to OEM 10.2.0.5 (certified on Suse 10) and I upgraded the OEM repository to 10.2.0.4.
Now, the different targets can be configured for monitoring.