AEM Blue

If you are using Oracle Enterprise Manager (OEM) to administer, monitor, and manage your database environments...

...you clearly understand the benefits of leveraging these capabilities to centralize key tasks particularly for environments involving numerous databases. Anything that makes a Database Administrator’s life easier could be considered a significant win, allowing them to focus on more project-related work and avoid visits from outside entities complaining about issues.

One of the biggest concerns for all IT professionals is security. If a tool like OEM were to experience a Denial of Service attack, suddenly the robust capability available to manage databases across the enterprise would become inoperable. If a similar attack were performed at the same time for databases, the DBA would be left scrambling by trying to log into each server and diagnose the source of system unresponsive.

Rest-assured: if management determines the impact could have been easily avoided through standard security measures, this could represent a career-altering situation. This blog attempts to provide the key steps for properly securing OEM to help maintain employment and protect your organization’s most critical databases.

After installing OEM 13c and connecting your databases and related components to the tool, the next most important task is to secure the Oracle Management Server (OMS). While this can be done with self-signed certificates, a more secure method is to use third-party certificates through an internal or external Certificate Authority (CA). One commonly used external CA is Verisign, which charges a fee associated with each certificate requested but avoids the extra labor required to setup an environment to support certificate generation.

To take full advantage of Oracle’s latest security features and fixes, it’s strongly recommended to implement the latest version of OEM available for download. For this example, we have installed OEM 13.2 which currently represents the latest release while installing the software on a RHEL 6 server. RHEL 6 is considered a very mature, stable operating system release commonly found across IT environments.

The OEM software has successfully installed and we’ll now work with the Certificate Authority (CA) to provide them with the signed certificates and receive the associated certificate chain for properly securing the OMS.

1. Create a Directory

The first step is to create a directory where the wallet will be based and set up the environment variables.

mkdir /home/oracle/certificate
cd /home/oracle/certificate
export PATH=$ORACLE_HOME/oracle_common/bin:$PATH

2. Generate a User Certificate

Next, we will create the wallet and generate a user certificate that will be signed by the CA. The CN will usually be what shows up in your /etc/hosts file.

orapki wallet create -wallet /home/oracle/certificate -auto_login -pwd testpwd1

orapki wallet add -wallet /home/oracle/certificate -dn "CN=server.name,OU=Test,OU=Test,OU=Test,O=U.S. Government,C=US" -keysize 2048 -sign_alg sha256 -pwd testpwd1

3. Export the Certificate

Now, export the certificate request that will be sent to the CA for signing.

orapki wallet export -wallet /home/oracle/certificate -dn "CN=server.name,OU=Test,OU=Test,OU=Test,O=U.S. Government,C=US" -request CSR.txt

4. Load Certificates into Oracle Wallet

At this point, you send the CSR off to your CA for signing. When you receive the files back you should have a signed server certificate as well as the signing certificates that form the chain. In our case, we have a root and an intermediate certificate. Place these on a location on the server and then load them into the Oracle wallet.  Any certificate that is a signing authority will use the flag -trusted_cert while the signed server certificate will always be the -user_cert.

orapki wallet add -wallet /home/oracle/certificate -trusted_cert -cert /home/oracle/certificate/root.cer -pwd testpwd1
orapki wallet add -wallet /home/oracle/certificate -trusted_cert -cert /home/oracle/certificate/intermediate.cer -pwd testpwd1
orapki wallet add -wallet /home/oracle/certificate -user_cert -cert /home/oracle/certificate/OMS.crt -pwd testpwd1

5. Verify with Orapki Display

With all the certificates loaded, you can run an orapki display command to verify. You will see the request for certificate (blue) that we first made and then the three certificates (server cert – red, trusted certificates – orange) we loaded into the wallet. I’ve highlighted the different types of certificates so that you can see them more clearly.

orapki wallet display -wallet ./
Oracle PKI Tool : Version 12.1.3.0.0
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
Subject:        CN=server.name,OU=Test,OU=Test,OU=Test,O=U.S. Government,C=US
User Certificates:
Subject:        CN=server.name,OU=Test,OU=Test,OU=Test,O=U.S. Government,C=US
Trusted Certificates:
Subject:        CN=Entrust Certification Authority - L1R,OU=(c) 2014 Entrust\, Inc. - for authorized use only,OU=See www.entrust.net/legal-terms,O=Entrust\, Inc.,C=US
Subject:        CN=Entrust Root Certification Authority - G3,OU=(c) 2012 Entrust\, Inc. - for authorized use only,OU=See www.entrust.net/legal-terms,O=Entrust\, Inc.,C=US

6. Combine Root and Intermediate Certificates

One final step in the wallet directory is to combine the Root and Intermediate certificates into a text file named trusted_certs.txt, after following the steps below just check the file real quick to verify that there are no spaces.

cp L1Rchain.txt trusted_certs.txt
cat L1Rroot.txt >> trusted_certs.txt

The final product should look similar to the below.

cat trusted_certs.txt
-----BEGIN CERTIFICATE-----
MIIFNTCCBB2gAwIBAgINAMvIuaMAAAAAUdNQvjANBgkqhkiG9w0BAQsFADCBvjEL
MAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1Nl
ZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAyMDEy
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----MIIERzCCAy+gAwIBAgINAMK7Y+oAAAAAUNC1oTANBgkqhkiG9w0BAQUFADCBvjEL
MAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1Nl
ZSB3d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAyMDEy
IEVudHJ1c3QsIEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEyMDAGA1UE
-----END CERTIFICATE-----

The wallet setup is complete. The rest of this post will be focused on securing OMS and the various components that OMS uses. You will need the SYSMAN and agent registration password close by as they will be used throughout this process.

7. Secure the OMS to Use the Wallet and Agent

We will first secure the OMS to use the wallet we just created, then restart the OMS to make sure that everything comes up correctly. As you will see throughout, a restart of OMS is needed after just about every step in this process.

cd /home/oracle/product/mw/bin
emctl secure console -wallet /home/oracle/certificate
emctl stop oms -all
emctl start oms

This is then to use secure the agent.

cd $AGENT_HOME/bin
./emctl secure agent <registration password>

8. Use Trusted_Certs.txt

We will now use the trusted_certs.txt file, which we created earlier, and secure the agents and the OMS using this file. You will also need to run this same process for the agent on any remote agents you have deployed. This will be followed up by a restart of OMS.

cd $AGENT_HOME/bin
./emctl secure add_trust_cert -trust_certs_loc /home/oracle/certificate/trusted_certs.txt

cd $OMS_HOME/bin
./emctl secure oms -wallet /home/oracle/certificate -trust_certs_loc /home/oracle/certificate/trusted_certs.txt

./emctl stop oms -all
./emctl start oms

9. Test the Agents and Secure the Weblogic Component

Once OMS is back up, I usually will then test all of the agents by running a status and then upload command to make sure that all of the pieces are communicating like they should.

cd $AGENT_HOME/bin
./emctl status agent
./emctl upload

Now, we will secure the Weblogic component.

$<OMS_HOME>/bin>
./emctl secure wls -wallet /home/oracle/certificate
./emctl stop oms -all
./emctl start oms

10. Allow OMS Host and Weblogic to Communicate

The last step in this process is to secure the agent that is on the OMS host so that it can communicate with the Weblogic component. We will upload the root certificate and the intermediate certificate to the agent.  It’s a good idea to run an upload agent command after this step to verify.

cd $AGENT_HOME/bin
./emctl stop agent
./emctl secure add_trust_cert_to_jks -trust_certs_loc /home/oracle/certificate/root.cer -alias rootcacert -password welcome
./emctl secure add_trust_cert_to_jks -trust_certs_loc /home/oracle/certificate/intermediate.cer -alias intercacert -password welcome
./emctl start agent
./emctl upload agent

11. Verify Your Work

Now that we have completed this step, you can verify everything by running a emctl status oms -details command. The consoles should be locked and the HTTPS ports should now be in use. I’ve highlighted these differences in the screenshot below.

/opt/oracle/product/mw/bin/emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host     : server.name
HTTP Console Port       : 7788
HTTPS Console Port      : 7802
HTTP Upload Port        : 4889
HTTPS Upload Port       : 4903
EM Instance Home        : /opt/oracle/product/gc_inst132/em/EMGC_OMS1
OMS Log Directory Location : /opt/oracle/product/gc_inst132/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://server.name:7802/em
Upload URL: https://server.name:4903/empbs/upload

WLS Domain Information
Domain Name         : GCDomain
Admin Server Host   : server.name
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: server.name
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up

BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up

BI Publisher HTTP Managed Server Port   : 9701
BI Publisher HTTPS Managed Server Port  : 9803
BI Publisher HTTP OHS Port           : 9788
BI Publisher HTTPS OHS Port         : 9851
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://server.name:9851/xmlpserver
BI Publisher Server Logs: /opt/oracle/product/gc_inst132/user_projects/domains/GCDomain/servers/BIP/logs/

 

BI Publisher Log     : /opt/oracle/product/gc_inst132/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log

12. Troubleshoot if Necessary

If you run into any errors, it is good to start with the emctl.log which should point you to the cause of the issue. I also used Oracle MOS notes 1367988.1 and 1368940.1 when configuring everything.

Conclusion

AEM has gone through the process of securing the OMS for a number of customers. We view the steps outlined in the blog as a mandatory configuration change required for implementing OEM to improve the overall security posture of an organization. We have identified a number of other security practices designed to prevent Denial of Service and other such attacks from occurring.

Happy to talk more. We can help your organization implement these best practices to avoid the financial operational, and overall impact associated with security breaches—and keep all security-minded DBAs gainfully employed for years to come!

RECOMMENDED BLOG POSTS

5 Lessons for Finding the Right Test Automation Software

This is the second blog post in a two-part series examining test automation software. This blog post focuses on lessons learned for finding the right software product for your organization. We recommend you also read our first post, which is dedicated to understanding the process for moving from manual to automated testing.

5 Keys to Successful Test Automation

This blog post is the first in a two-part series on website testing automation that can help your organization better understand how to maximize the effectiveness of your tests and find the right tools to meet your needs. Below we offer insights that can help your organization improve its testing automation process. Our follow-on blog post will help your organization understand the different software tools available to begin automating your tests.

Changing Oracle Database Passwords with No Downtime

Some customers want to change all passwords every 60 days, which is a more secure security practice yet is a burden to the administrative staff needing to work off hours due to the system impact.