Patching Oracle e-Business Suite

this is my fourth week at my new assignment and already I am busy applying all kinds of patches. Because they have a real big environment here, it's important to know what options to provide adpatch with.
The options I mostly work with are :

- hotpatch, I don't want to put the database in maintenance mode every time
- noautoconfig, because of the multi node installation over here don't run autoconfig

But there are many more options to use. Check the following post for more options..
http://appsdbablog.com/blog/patching/

optimize parameters

At the customer I working right now, they asked me to look at a query that did run in a test EBS database, but wouldn't run in a developer EBS database.
When running the query in the dev database, the session seems to hang and no records are returned.
So, first I looked at the explain plan from the query running in the test database. It looked like this..

Operation Object Name Rows Bytes Cost Object Node In/Out PStart PStop
SELECT STATEMENT Optimizer Mode=CHOOSE 339 K 103359
NESTED LOOPS 339 K 8 M 103359
TABLE ACCESS BY INDEX ROWID CSI.CSI_T_EXTEND_ATTRIBS 339 K 6 M 103358
INDEX RANGE SCAN CSI.CSI_T_EXTEND_ATTRIBS_N02 339 K 4066
INDEX UNIQUE SCAN CSI.CSI_T_TXN_LINE_DETAILS_U01 1 6

Then I tried running the query in the dev database and the explain plan looked like..

Operation Object Name Rows Bytes Cost Object Node In/Out PStart PStop
SELECT STATEMENT Optimizer Mode=CHOOSE 339 K 56396
HASH JOIN 339 K 8 M 56396
TABLE ACCESS BY INDEX ROWID CSI.CSI_T_EXTEND_ATTRIBS 339 K 6 M 7
INDEX RANGE SCAN APPS.XXX_CSI_T_EXTEND_ATTRIBS_N1 339 K 2
TABLE ACCESS FULL CSI.CSI_T_TXN_LINE_DETAILS 11 M 64 M 51170

It seems in the dev database the query is doing a full table scan, instead of indexed scan in the test database. Also in the test database a NESTED LOOP is used, while the dev database uses a HASH JOIN ??
I focussed on the last finding and searched for a reason why the query should use HASH JOIN instead of a NESTED LOOP.
I found out the following two init.ora parameters were different in the dev database, the optimizer_index_caching parameter and the optimizer_index_cost_adj parameter. So, I tried running the query after I changed my session settings..

SQL> alter session set optimizer_index_caching=x;

SQL> alter session set optimizer_index_cost_adj=x;

And this time the query returned the desired rows ! Next step is to edited the init.ora so the whole dev environment can use these settings...

Changed Oracle EBS Dba OCP program

It seems Oracle has changed their OCP program for EBS Dba's
At the end of 2006 we were able to participate in the beta program of EBS OCP for dba's. I completed two exams, the Oracle 11i Install Patch and Maintain Applications beta exam and the Oracle 11i System Administration beta exam.
Now Oracle has changed the OCP program and added two new exams to the list. They also deleted two exams, the Oracle 11i System Administration and Implement Oracle Workflow 11i were removed out of the OCP program and they added Applications DBA Fundamentals I and II. Also, when you are already an OCP 9i or 10g, you only have to pass the 1Z0-233 Oracle 11i Install Patch and Maintain Applications exam to become a certified EBS Dba professional.

Also, the results of the beta exams are supposed to be known in April...

See also http://www.oracle.com/global/us/education/certification/appsdba.html

Lost root password

For a colleague I had to install Oracle Infrastructure on Suse Linux. The installation went very smoothly, but at the end I was facing a big problem. As you know, at the end of any Oracle installation you are asked to run a script called root.sh as root. So, you must have access to the root password for that host. Normally that is not a problem, but this time I did not know the password. When asking other colleague's, I found out that the password was once changed, and it seems nobody knew what the new password was. So, who to reset or change the root password when you don't know the old one ?
Another colleague of mine showed me some Google results, and we tried to change a unknown root password. This is what we did..

- We restarted the operating system (Suse..) and interrupted the start up so we could use a command line
- we typed in : init=/bin/bash and resumed startup
- on the following command prompt : mount -o remount,rw /
- at this point you can enter passwd and change the root password without knowing the old one !

Forum ?

I noticed a lot of visitors of this blog come to this blog with a particular question. Using a search engine, maybe a word you searched for is also present at my blog. But maybe not the way you searched for. Because I want to know what kind of problem you have with the e-Business Suite, I have opened a forum for EBS dba's. I am not sure if it's gonna work, but I will give it a try.
So, if you have any question and you cannot find the answer on the site, post a question on the forum and I will try to answer the question !

http://www.createforum.com/oracleebsdba/

Oracle Applications Manager in R12

A few post ago, I noticed there was no OAM link in the system administrator responsibility in the new Oracle EBS R12.

I found the OAM when accessing the workflow manager...

This problem exist on my new EBS R12 install with an empty database. Now that I have installed R12 with a vision database, sysadmin has got an 'Oracle Applications manager' entry in the system administrator responsibility. So, it seems a problem when installing an empty database instead of the vision database ??

Date and time on virtual machine....

I had this problem with date/time synchronizing on a virtual machine with its host machine. Result of this was that the clock on my virtual machines ran slower than the real clock. So my date and time was always behind. Because I did not want to restart the timeserver (ntp server) every time from the command line (when restarting the ntp server, the date/time was correct for a few minutes....), a colleague of mine suggested to put the restart in a cronjob. Great plan and also tested on a few other virtual machines...
I placed the following lines in a fiel called time.cron

[root@vamisux32 ~]# crontab -l
# ntp update statement - workaround
0-59 * * * * /usr/sbin/ntpdate amisnt10 >>/var/log/ntpdate 2>&1

But on my Oracle Linux virtual machines, the cron job did not run. So, the virtual clock was still running slower than the real clock.

In the logfile of the crontab, the following error was shown

"ntpdate[23109]: the NTP socket is in use, exiting"

So it seems the cronjob cannot connect to the external time server, because the NTP socket is already in use.
I found out that Oracle Linux starts a NTPD process when booted. This NTPD process is blocking my NTP socket....So, I stopped the NTPD process and made sure it would not start up again when booting the virtual machine. Checked the date and time and everything seems oke now. Even the crontab logfile is without errors...

"ntpdate[18258]: step time server 10.252.252.10 offset...."

So, don't run ntpd and ntpdate next to each other...

Oracle E-Business Suite R12 with OEM10G

After configuring the Oracle EBS 11.5.10 environment in OEM10G, it was time to also configure the new Oracle EBS R12 environment in OEM10G. Before I could configure R12 in OEM10G, I first had to upgrade the Oracle Applications plugin for EM10GR3. I downloaded the patches p5489352_R12_LINUX.zip and installed this patch on the OMS_HOME and in the local AGENT_HOME's (11i and R12 agent). For R12 also patch p5843357_R12[1].OAM.A_R12_GENERIC.zip was installed...After applying the patch, my first 11.5.10 configuration didn't work anymore, so I had to rediscover both R12 and 11i again. After rediscovering both environments, I was able to drop the old 11i target. So, now I have both environments in OEM10G. The services to be monitored look a little bit different for R12, because of the 10g iAS...Also OAM is integrated into OEM10G, so you can drill down into the EBS R12. How to configure R12 and 11i with the new Oracle Application plugin is documented in a Getting Started.pdf (can be downloaded from Metalink). So, now I am ready to monitor....

Oracle EBS 11i with 10G OEM

For our remote administration of databases, I was asked to configure our own internal EBS 11.5.10 environment in Oracle Enterprise Manager 10g.

First the Oracle EBS plugin was installed on the Oracle Management Server.

Second I had to install an agent on the EBS machine, and also apply the plugin patch into the Intelligent Agent home (patch p4678538_11i_LINUX.zip).

After the plugin patch I also applied the OAM integration patch (p3410599_11i_GENERIC.zip).

After patching the Agent home and defining targets in OEM, the Oracle EBS environment was configured in OEM. There was still one problem, when trying to configure the database target I could not connect with the sys account. Every time I tried to connect with sys, the following error occurred :

ORA-01017: invalid username/password; logon denied ORA-27037: unable to obtainfile status Compaq Tru64 UNIX Error: 2: No such file or directory Additionalinformation: 3

I found out that all I needed was a password file.....

Following note 359979.1 from Metalink helped me.

1. set REMOTE_LOGIN_PASSWORDFILE to exclusive (static parameter..)

2. Create password file using orapwd utility

3. restart database

After this you can rerun the database configuration in Grid Control....

First patch on Oracle E-Business Suite R12

This morning I downloaded my first patch for the new Oracle EBS R12. Already there are 2 high priority patches for the new EBS, and I downloaded one of them (patch p5856453_R12[1].AD.A_R12_LINUX.zip).
I could not help feeling a little bit excited, because patching is changed in EBS R12.
The manual steps to be taken before patching are removed from the readme and placed into an xml file. Before patching you are supposed to run a new perl script called $AD_TOP/bin/admsi.pl

This script creates an install_{patch_number}.html.
After performing the steps in the html file, you have to run $AD_TOP/bin/admsi.pl –done (to let the system know you have performed the manual steps….).
The admsi.pl also creates a custom readme which you have to read before patching.

I was a little bit disappointed, because the patch I downloaded is maybe to small. Nothing about admsi.pl in the readme…Tried to run admsi.pl anyway

cd /ebs/patches/5856453/ad/patch/115/manualsteps
[oracle@vamisux34 manualsteps]$ $AD_TOP/bin/admsi.pl
Invoking Oracle Patch Application Assistant….
Exception in thread "main" java.lang.ExceptionInInitializerError
at com.jgoodies.looks.plastic.PlasticLookAndFeel.(PlasticLookAndFeel.java:122)
at oracle.apps.ad.msi.LaunchUI._useOracleLookAndFeel(LaunchUI.java:69)
at oracle.apps.ad.msi.LaunchUI.(LaunchUI.java:29)
Caused by: java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
at sun.awt.HeadlessToolkit.getScreenResolution(HeadlessToolkit.java:189)
at com.jgoodies.looks.LookUtils.isLowResolution(LookUtils.java:436)
at com.jgoodies.looks.LookUtils.(LookUtils.java:180)
… 3 more

Maybe with X-manager and $DISPLAY set ? No, admsi.pl returns a java exception…

Invoking Oracle Patch Application Assistant….
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1762)
at oracle.apps.ad.msi.LaunchUI.main(LaunchUI.java:107)

Running admsi.pl like this.....

[oracle@vamisux34 manualsteps]$ perl $AD_TOP/bin/admsi.pl -cli

Gathering Information..
Enter the patch_top : /ebs/patches/5856453
Enter the APPS user password:
Logfile for this session is located at admsi.log
Generating installation instructions for patch 5856453..
Updating database....
install_5856453.html generated successfully
install_5856453.txt generated successfully
[oracle@vamisux34 manualsteps]$

The install_5856443.html is the new readme. Because there are no manual actions to perform, the new generated readme is the same as the old one and I can apply the patch using adpatch..

oracle@vamisux34 appl]$ cd /ebs/patches
[oracle@vamisux34 patches]$ ls
5856453 p5856453_R12[1].AD.A_R12_LINUX.zip
[oracle@vamisux34 patches]$ cd 5856453/
[oracle@vamisux34 5856453]$ adpatch options=hotpatch
************* Start of AutoPatch session *************
AutoPatch version: 12.0.0
AutoPatch started at: Tue Feb 27 2007 11:05:47

…..

AutoPatch is complete.
AutoPatch may have written informational messages to the file/appl/apps/apps_st/appl/admin/PROD/log/u5856453.lgi
Errors and warnings are listed in the log file/appl/apps/apps_st/appl/admin/PROD/log/u5856453.log
and in other log files in the same directory.

The patch finished without problems….

OAUG meeting in Amsterdam

This afternoon I was present at the OAUG netherlands. The meeting was held in the Nationaal Lucht -en Ruimtevaartlaboratorium in Amsterdam and the topic was Oracle ASM with the Oracle EBS.

So, first an Oracle certified master, Rene Kundersma, told something about Oracle ASM. Next topic was supposed to be about the Oracle EBS using ASM, but the speaker had to cancel...so Rene could tell us more about Oracle ASM without Oracle EBS. To bad, because the room was filled with Oracle EBS people, and a lot of questions were asked concerning the Oracle EBS with ASM. But, unfortunately all these questions remain unanswered...But the story Rene Kundersma told us was very interesting also !

Oracle Partner Release 12 Update seminar

This morning I was present at the Release 12 Partner update Seminar at Oracle in the Netherlands. Before the lunch there were some presentations, and one of them was very good. It was a presentation from Karsten Roigk (VP Applications), about the Oracle strategy and also some new features in Oracle General Ledger. Very interesting...

After the lunch I went to the Applications Technology meeting. This seminar was also an overview, but still there were some interesting issues. For example, in R12 the Integration Repository is part of the EBS, instead of the static html page it was in EBS 11i (http://irep.oracle.com/index.html). And now that it's part of the EBS, when a patch updates a api, the irep is also updated !!! The irep, by the way, is used for integration with the Oracle EBS. It shows us how you can integrate with the EBS. Making irep a part of the EBS R12 is also a part of the Oracle Fusion development.

Oracle Applications Manager in Oracle EBS R12

I tried to find the new OAM, but could not find the responsibility for it.

When login in a 11i environment, you could choose to login into the application or login the web OAM. In R12 you cannot make this choice anymore. So where is the new web OAM ??

When logged in as 'sysadmin' the only applications manager responsibility is ' Workflow : Applications Manager'

When you choose that responsibility, it takes you to the workflow part of OAM. From there you can go to the Applications dashboard tab and you are in the known OAM webpage...

You enter the new web OAM through the following url : http://vamisux34.amis.local:8000/OA_HTML/weboam/oam/oamApps$target=PROD

and in the 11i environment the url looked like

http://vamisux32.amis.local:8000/oa_servlets/weboam/oam/oamApps$target=oatest

So it changed from oa_servlets to OA_HTML.....

The OAM main page looks the same, but there is a new tab.. 'Software Updates'. Opening that page shows us the following...

The page schows us the Maintenance Activities like 'AD Administration -Disable Maintenance Mode'....

Also there is a Applied Bug Fix Check and a patch recommendation Request...

The old Applied Patches info is also available...

Guest password in Oracle EBS

Last Friday I received a message from an international EBS user. The GUEST password was changed in the profile option GUEST_USER_PASSWORD (default GUEST/ORACLE). After the change, they could not login the Oracle EBS anymore. She asked me for some advice, how to solve this problem since it was their production environment.

What the current password is, is shown by the following sql

SQL> select profile_option_value from fnd_profile_option_values
2 where profile_option_id = 3805;

To see if the login password for user GUEST is the default password...

select fnd_web_sec.validate_login('GUEST','ORACLE)
from dual;

Because autoconfig was no option at that time, I suggested to change the profile back to it's default by using the SAVE function from the FND_PROFILE package.
DECLARE
stat boolean;
BEGIN
dbms_output.disable;
dbms_output.enable(100000);
stat := FND_PROFILE.SAVE('GUEST_USER_PWD', 'GUEST/ORACLE', 'SITE');
IF stat
THEN
dbms_output.put_line( 'Stat = TRUE - profile updated' );
ELSE
dbms_output.put_line( 'Stat = FALSE - profile NOT updated' );
END IF;
commit;
END;
After running this function the password was set to ORACLE. The GUEST user password is also mentioned in the dbc file and in the xml file. If all the entry's are the same, everything should work fine.

Well, at Sunday we lost contact. But last Monday I received an email saying the problem was solved. I asked what solved the problem in the end, but I did not receive an answer yet.

Install Oracle EBS 12i

So, I finally installed the new EBS 12i release. I had to install with an empty database, because of disk space limits...
But the important thing is that the installation went well and was successful.








Some first impressions...
- the look and feel changed (we had already seen some previews..)
- some strange options when logging in the application ??


- the database is twice as big (10g vs 9i)
- new technology stack
- new instance top

I also faced a first problem. I could not startup the application in my Microsoft explorer..The java console showed the login was hanging on the appslogo_new.gif ??
I found out that when I deactivated my Oracle Dba Toolbar in the explorer, I could enter the application without any problem. But when activating the toolbar again..the login process was hanging again on the same gif. Strange...the toolbar was downloaded from OTN and is something Oracle brought to my attention... I wonder if this is a known issue....

I also saw the dbc file is now in the instance top directory. Strange, because the path to the dbc file seems the same, /appl/fnd/12.0.0/secure....but this is not in the APPL_TOP, but in the INSTANCE_TOP (inst/apps/PROD_vamisux34).
A simple ls command in the INSTANCE_TOP shows..

[oracle@vamisux34 PROD_vamisux34]$ ls

admin appltmp conf_PROD.txt logs ora pids rgf

appl certs conf_PROD.txt.bac ocm out portal temp

The admin directory contains the scripts directory, where all the start/stop scripts are located.

The appl directory shows us

[oracle@vamisux34 appl]$ ls

admin fnd

where fnd holds the dbc file and the admin directory contains the xml file.

More to come....

Sizing Oracle EBS 12i

I wanted to find out how ‘big’ the new Oracle EBS 12i release really is. So, I downloaded the software from Edelivery and created a stage area.
From the stage area I started the rapidwiz to check the required disk space.

Here is the required disk space for Oracle EBS 12i :

Database ORACLE_HOME = /ebs/db/tech_st/10.2.0
required = 4234.0
Database System File Directory = /ebs/db/apps_st/data
required = 19487.0
Database Log File Directory = /ebs/db/apps_st/data
required = 3165.0
Database Transaction File Directory = /ebs/db/apps_st/data
required = 66244.0
Database Archive File Directory = /ebs/db/apps_st/data
required = 45061.0
APPL_TOP = /ebs/apps/apps_st/appl
required = 5078.0
APPL_TOP mount 2 = /ebs/apps/apps_st/appl
required = 2639.0
APPL_TOP mount 3 = /ebs/apps/apps_st/appl
required = 4167.0
APPL_TOP mount 4 = /ebs/apps/apps_st/appl
required = 2882.0
COMMON_TOP = /ebs/apps/apps_st/comn
required = 2743.0
Apps ORACLE_HOME = /ebs/apps/tech_st/10.1.3
required = 1553.0
Tools ORACLE_HOME = /ebs/apps/tech_st/10.1.2
required = 1138.0
So, you see the database is much bigger then release 11.5.10.2 (10g vs 9i). Also the new APPL_TOP is a little bigger then the old one.

To check for yourself, here is the required disk space for Oracle EBS 11.5.10.2

Database ORACLE_HOME
required = 3091.0
Database System File Directory
required = 12023.0
Database Log File Directory
required = 160.0
Database Transaction File Directory
required = 28870.0
Database Archive File Directory
required = 21206.0
APPL_TOP
required = 5707.0
APPL_TOP mount 2
required = 3212.0
APPL_TOP mount 3
required = 5852.0
APPL_TOP mount 4
required = 2524.0
COMMON_TOP
required = 2963.0
Apps ORACLE_HOME (iAS)
required = 2600.0
Tools ORACLE_HOME (8.0.6)
required = 3027.0

Applications unlimited

Yesterday morning I visited a seminair about Oracle Fusion middleware. It was an overview, so a lot of info in a few hours. The overview helped me understanding SOA, BPELL, FUSION APPS, ESB a little bit.

In the afternoon I visited Oracle again, this time in Amsterdam for the Applications Unlimited session. In a beautifull environment, the board room in the Rembrandt Tower, I saw the Applications unlimited presentation from New York. Saw some old colleques and enjoyed the whole scene....

Now back to work , trying to complete the staging area for the EBS 12i....

Before installing the new Oracle EBS 12i release, these are the disk space requirements

Rapid Install installs the file system and database files for all products, regardless of
their licensed status. The approximate file system requirements in a standard
installation are:

Applications node file system (includes AS
10.1.2 ORACLE_HOME, AS 10.1.3
ORACLE_HOME, COMMON_TOP,
APPL_TOP, and INST_TOP)
28 GB
Database node file system (Fresh install) 45 GB
Database node file system (Vision Demo database)
133 GB
The total space required for a standard system,
not including the stage area, is 73 GB for a fresh install with
a production database, and 161 GB for a fresh install with a Vision
Demo database.

Note: The database node disk space requirements for both the
production database and the Vision Demo database include database
1-10 Oracle Applications Installation Guide: Using Rapid Install
files (.dbf) and the 10g R2 database ORACLE_HOME.
Stage area

For a production database install, running Rapid Install from a stage area requires at
least 33 GB to accommodate the file system and database files in the stage area.

EBS 12i

This weekend i used to download the new EBS 12i release. More then 100 gb of EBS cd's I had to download. Now I'm looking for a place to install the EBS 12i...

This week is also the launch of the new EBS 12i in Amsterdam. This launch is on wednesday. In the morning I'm visiting Oracle for a fusion middleware overview, and later on I'm going to Amsterdam for the global launch...


http://www.oracle.com/webapps/events/EventsDetail.jsp?p_eventId=61972&src=4852728&src=4852728&Act=26

Installing WEB ADI in EBS 11.5.10.2

Install Webadi (BNE.D) on EBS 11.5.10.2

1. note 287080.1 on Metalink

2. check if WEB ADI is spliced into your environment

[oracle@ebs2 prod_ebs2]$ sqlplus apps/apps
SQL*Plus: Release 10.2.0.2.0 - Production on Tue Jan 16 08:23:42 2007
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL> Select * from APPLSYS.FND_APPLICATION where APPLICATION_SHORT_NAME = 'BNE';

APPLICATION_ID APPLICATION_SHORT_NAME
-------------- --------------------------------------------------
LAST_UPDATE_DAT LAST_UPDATED_BY CREATION_DATE CREATED_BY LAST_UPDATE_LOGIN
--------------- --------------- --------------- ---------- -----------------
BASEPATH PRODUCT_CODE
-------------------- --------------------------------------------------
231 BNE01-SEP-04 2 01-JAN-80 1 0BNE_TOP BNE

SQL>

If a record returns, then WEB ADI has been spliced.

2. install patch 4291053
3. install patch 3218526 (BNE.D)
4. from the adadmin menu compile apps schema (1)
Compile/Reload Applications Database Entities
---------------------------------------------------
1. Compile APPS schema
2. Compile menu information
3. Compile flexfields
4. Reload JAR files to database
5. Return to Main Menu

5. in the same adadmin session compile flexfield data in AOL tables (option 3)

6. restart application middle tier
7. Now you have a responsability 'Desktop Integraton' (see screen print on top..)