Oracle Enterprise Manager (OEM) is a vital tool for any Oracle DBA to properly manage and monitor the operational environment. OEM helps identify issues and measure database performance in an accessible format providing an enterprise-wide view of their databases with the ability to zoom in for detailed information, as needed.
Like any piece of software, it is important to keep OEM up to date through patches and version releases. In this blog post, we will tackle one of the more challenging scenarios when it comes to upgrading OEM – Upgrading from 13.2 to 13.4 using the silent upgrade option.
In this scenario, we have an existing single instance deployment of OEM 13.2 on Linux with a repository 19c database using multi-tenant architecture (CDB with a repository PDB) located on a separate server.
Due to security requirements, we are unable to use the GUI functionality; the upgrade will need to be executed using the silent option. This deployment of OEM also uses custom third-party certificates that require additional configuration during the upgrade.
Step 1: Download Software and Update Response Files
The first step is to download the software and stage it on the server where OEM is located. The software consists of seven files found here.
Once the software has been uploaded to the server, we will create and update the response files required for the silent install:
There are quite a few sections that need to be updated in this file, and a few will require customized values based on your environment, but the below is a good starting point:
Step 2: Repository Database Pre-Requisites
To upgrade to 13.4, you must meet several prerequisites on the repository database side, including query checks, parameter updates, and adding more space to tablespaces. Not doing so could cause the upgrade to fail and force you to restore your database from a backup.
The first set of prerequisites are query checks, they should come back with no rows returned:
If the first command returns any results, you will need to remove or upgrade the agents listed in the output.
If the second query returns results, please follow Oracle MOS Note 2670368.1.
The next set of commands are parameter updates. In our scenario, there are parameters that need to be updated at both the CDB and PDB levels. Once the commands have been run, be sure to restart the database before starting the upgrade:
The last set of commands are related to tablespace management and permissions. The sysman user should have an unlimited quota on the Users tablespace, and both the “MGMT_TABLESPACE” and “MGMT_AD4J_TS” tablespaces should have their space increased. In this example, I chose to add an additional datafile for both tablespaces to provide sufficient room for growth.
Step 3: OEM/Oracle Management Server (OMS) Pre-Requisites
Next, some steps are required to run from the OMS prior to kicking off the upgrade involving the emkey and use of third-party certificates. The first command is copying the emkey to the repository database and is a relatively quick operation.
The second section of commands are required if you are using third-party certificates to secure OEM. The WebLogic certificates must be reset to the demonstration certificates for the upgrade to run successfully.
Once all of the above steps are completed, you are now ready to start the upgrade.
Step 4: Upgrade OEM
Shut down the OMS processes and stop the agent located on OEM server prior to starting the upgrade. The upgrade requires a significant amount of temporary space; in this example, I used an additional flag on the upgrade command to designate a temporary directory that is larger than /tmp.
If you are using this additional flag, please be sure that the new directory has the noexec bit not set, or else the upgrade will fail during the automatic pre-requisite checks.
The upgrade command could take some time to execute; in our example environment, it ran for about 4 hours. As the upgrade can take some time to complete, the best way to follow the progress and discover any issues with the upgrade is to tail (tail -f) the log file.
Step 5: Post-Upgrade Tasks
Once the upgrade has been successfully completed, there are a few tasks left to run. First, the allroot.sh command should be run as root:
At this point, OEM should be running, and you can verify that the upgrade was a success by logging in and clicking on your username in the top right corner, and selecting “About Enterprise Manager”, you should see the below pop-up:
Additionally, if you use third-party certificates, WebLogic needs to be secured with those certificates. For reference, I have included a blog post that goes through the whole process of securing OEM with third-party certificates, execute Step 9 to secure WebLogic.
The emkey that we copied to the repository during the pre-requisites can now be removed:
Finally, we should reset the hidden parameter updates that were made to the repository database. In this scenario, we will run these commands “as SYSDBA” in the PDB:
Step 6: Upgrade Agent Tasks
The final step of the process, upgrading the agent, is relatively straightforward but can be prone to issues. Specifically, I ran into an issue during the upgrade that could cause the upgrade to fail to complete, and the agents would not start; the below two commands helped resolve the issue. I would recommend running these steps first before proceeding with the agent upgrade. The first step is run on the OEM server using the emcli utility.
The second command should be run on the server of any agent that will be upgraded. In my case, I had to run this command five times (once on the OMS and then on each database server that has an OEM agent). This command updates the Cipher Suites that the agent can use for communication.
After running these commands, you can then upgrade the agents through the OEM console GUI. This upgrade can be done by logging in and then going through the following menus: Setup – Manage Cloud Control – Upgrade Agents. The agent upgrade process is pretty quick and can be done with multiple agents in parallel.
Conclusion
Staying up to date with software releases is very important. It is the best way to keep your environment secure while also introducing new features that can be utilized to increase productivity. Following the six steps established above can help you painlessly perform a silent upgrade of your OEM to 13.4.
AEM can help your organization implement the OEM upgrade and other best practices to keep your systems secure and optimized. Please reach out to our team to learn more about our service offerings.
