There are many aspects of the patching process which have been automated by Oracle to assist and reduce the complexity, but these engineered systems contain intricate components requiring system and database administrators to account for the local changes prior to beginning the patch cycle.
Here’s what you need to know to begin a patch on Oracle Exadata.
Settings
The main challenges to successful patching include maintaining and tracking the changes implemented to the base OS image. These changes range from security settings to the installation of additional packages to support other software which has been installed on the compute or database nodes.
Some of the items to be changed for security include:
- PAM settings
- SELinux settings (not supported by Oracle)
- session timeouts, and
- umask
Several of the settings are controlled by the host_access_control program. This script comes with each Exadata, allowing the organization a programmatic way to ensure the changes made are kept throughout the patch cycle.
In some instances, the system administrators have made “manual changes." Those settings will need to be reverted to factory defaults as they should have been implemented through the host_access_control program, whether this happens on a compute or storage node.
Each change should be fully documented and tracked to ensure proper change management best practices are being utilized. If you are unsure how a change will impact the system or if that change is supported by Oracle, always open an SR with support as your first line of defense. No custom users should ever be added to the storage nodes as these accounts will be wiped out during the patching process. If you have any dependencies on those accounts, they will fail.
After each patch cycle, the systems should be checked to ensure that the changes implemented have not been impacted by the patch. It should be noted that if you turn on SELinux, you will need to disable it during the patching process and re-enable after the patch completes. This additional step will require more reboots, causing your patching window to increase.
Software
Adding Linux OS packages to support third-party software installed on the compute nodes serves as a prerequisite for most deployments. Most, if not all, Exadata customers should install security software to protect their systems, such as:
- Antivirus
- Host-based intrusion and detection
- Automation engines, and
- End-point protection and encryption software
It becomes a tricky balance of meeting the needs of your organization with the realities of an engineered system. These devices are built and designed as fixed-use appliances with one function in mind: to run the Oracle database that supports your business.
Regulations, laws, and policies require businesses to maintain secure systems in addition to the ever-present threat of hackers attacking systems. Laws such as Sarbanes-Oxley and Health Information Protection and Privacy Act (HIPPA) necessitate a complete and total audit trail and system accountability all the way to the database access level. Other regulations such as Payment Card Industry (PCI) and Risk Management Framework (RMF) have overlapping and complex requirements, forcing the implementation of strict controls and complex security protections.
Compliance requirements only add to the need for properly documenting changes made and software installed. In some cases, the software packages installed to secure your system need to be removed from the compute nodes and reinstalled after the patch cycle is complete. The storage nodes do not support the addition of any software packages outside of what was delivered by Oracle. The highly specialized and custom Linux kernel deployed on these devices are “bare bones” and contain only what’s necessary to serve the ASM disks to the compute nodes.
If additional software is installed onto the storage nodes, you are only increasing the threat vector of these devices and will not be supported by Oracle. Proceed at your own risk. Best practices for the storage nodes call for ssh to be disabled through the exacli command, leaving only the https port open externally on the devices.
Finalizing the Patch
The system and firmware patch are only part of the equation. In some cases, these patches are not nearly as complex as the database patch. Hot fixes and “one off” patches seem to always creep their way onto every database. Tracking these patches and, in some cases, proactively requesting newer patch versions to match compatibility with the newly delivered quarterly patch is essential to ensure the system is restored to full functionality once patching is complete.
When factoring in the Grid Infrastructure, RDBMS, and OEM agent patches, which all come with the full quarterly stack download, it is clear that there is always additional work to be completed by the multiple groups.
The key points are to develop a:
- Patch plan
- Communication plan, and
- Documentation plan, encompassing all changes which are made
This is not a one-time effort! This needs to be updated with each patch. Changes should be documented on an ongoing basis through your change management process.
Hope you found this helpful. If you have any questions or would like to discuss these points further, please feel free to get in touch.
