Best Practices for Oracle Linux for Production Systems

I found this needed a blog post because most of the customers I meet, just install their Redhat/Oracle Linux environment and start using it out of the box in production. I believe this list should be included in every post-installation procedure.

  • Hostname : Make sure it’s a FQDN. Especially when you connect with NFS to other systems. If your hostname is not FQDN, locks will not be freed on the NFS server when you reboot.
  • Support : If you have support, make sure you register your system with ULN.
  • Update : Update your system with yum or up2date to the latest version.
  • Hugepages : If you are running Oracle Databases, this is a must. Metalink note. 361468.1
  • Ipmitool : This allows for control over the hardware from inside the OS. Can be very usefull for Cluster setups or automated scripts to collect information.
  • Kexec : This allows the system to dump the kernel-memory to disk whenever a kernel panic occurs. Instead of rebooting or hanging, the system boots into a separate kernel with the task of dumping the memory to disk in the form of a vmcore file. This file can then later be analysed with the crash utility. Don’t forget to test it!!
  • magic sysrq key : This enables some key-strokes in the console to force a kernel to do all sorts of things ( show locks, reboot without FS corruption, … ). It is often used to dump a kernel stacktrace to /var/log/messages and reboot a system after soft hangs ( hangs on console with numlock flashing ). This is default enabled in OL5 but in OL6 you need to enable it manually. Also, make sure you know the keystrokes for when you need them.
  • Oswatcher BB : Monitoring tool of Oracle. Can show you if there were spikes just before or leading to the crash. Metalink note.301137.1
  • vncserver : allows for X11 environment over vnc. Faster then X11 over the net and allows you to continue where you left off when you lose your connection during an installation or configuration.
  • oratop : utility for near real-time monitoring of databases, RAC and Single Instance. Metalink note. 1500864.1.
  • dstat : allows you to view all of your system resources in real-time
  • Rlwrap: Saves you time ūüėČ

If anyone is interested in how to perform some of these tasks, let me know and I’ll consider writing some blog entries about them. But most procedures can be found in the manual or official pages about it. Keep in mind that this list also applies to Oracle Engineered Systems.( ODA, EXADATA, … )


Virtualization Strategy with Oracle VM and Oracle Linux

I’m giving a presentation on this topic on thursday 25 Oct for OracleOpenXperience. Here are the slides for my presentation. Hope you find it interesting.

Oracle OpenXperience – Virtualization Strategy with Oracle VM & Oracle Linux

News from Oracle Open World 2012

I’m just back from OOW and the following is a small summary of the items that were hot and new.

Clusterware 12c announced for 2013

  • Flexasm : ASM no longer needs to be on the same node as the database. Allows for HA on ASM level.
  • Flexcluster : Clusterware now has a lightweight mode in the installer for application/middletier setups. You can combine these with the normal clusterware mode to form 1 big cluster. ( hub nodes and leaf nodes )
  • You can now create a Scan vip for each public network.
  • gsd service is removed because there is no support for 9i databases on clusterware 12c.
  • Shared GNS : You can now have a shared GNS over multiple clusters.

Database 12c announced for 2013

  • RAC Application Continuity : First ever database to preserve commit outcome and ensure application continuity during unplanned downtime.
  • Pluggable Databases : Databases can now be plugged in a container. Consolidating the background processes, redo and undo.
  • Xstream : A GUI on streams apply/capture processes is further enhanced but only available when you have Golden Gate license.
  • dbua : now has backup/restore and moving database files during upgrade built in.
  • dbua : Gathering statistics is now optional.
  • dbua : Now runs the upgrade scripts in parallel mode. Speeding up the whole process.
  • smart flash : Can use SSD’s as extention of Buffer Cache.
  • RMAN : can now do recover table
  • RMAN : Standby databases can now use RMAN to apply an incremental backup automatically ( Recover database from for standby )
  • RMAN : Now automatically converts datafiles from 1 platform to the other. ( big-little endian )
  • dbconsole is renamed to “EM Express” and now runs from inside the database.
  • Many more new features but I concentrated on virtualization and linux during OOW while my colleague did 12c.

OVM 3.2.1 in public beta

  • Support for mySQL databases as repository for the OVM Manager
  • OVM Manager now supports OVM on SPARC
  • Statistics of OVM Servers now available in OVM Manager ( Health tab)
  • Minor GUI changes and fixes
  • UEK2 kernel in dom0 for driver compatibility with Oracle Linux
  • OVM CLI now integrated and further enhanced
  • next Beta refresh will contain open Vswitch
  • OVM is now integrated in ODA and Exalogic, but each node has a local OVM cluster. No HA.

Oracle Linux

  • ksplice : allows for kernel upgrades without downtime
  • yum security updates : Now possible to apply only security updates to your system using yum
  • Dtrace : Solaris utility that was ported to Oracle Linux.
  • BTRFS : ZFS-like filesystem for Linux.
  • UEK3 is coming
  • SELinux profiles for every oracle product will be provided in the future
  • Linux Containers : Allows for isolation of applications while maintaining 1 OS.
  • support for swapfile over NFS
  • cgroups : Allows for limiting CPU,IO and memory on process level.

Acquisition of Xsigo was announced

  • Allows for virtualizing of network layer.
  • Will be integrated in OVM Manager in future releases.

    • I’ll try to focus on some of these new things in later Blogposts. As always if you have some questions, feel free to contact me or comment.

Installing rlwrap on Linux saves you time

Small Blog entry to spread the word. I hate having to manage a Linux system without this ūüėȬ†

rlwrap is an utility that allows you to use up and down arrows in the various commands like sqlplus, rman, adrci, … 
To enable it, you only have to do the following : 

  • Install readline-devel

    (root) # yum install readline-devel

  • Download latest rlwrap from and unzip it somewhere and Compile it.

    (root) # ./configure
    (root) # make
    (root) # make install

  • For OL6 you can use this RPM instead of manually making it:

    Then install it with :

    (root) # yum install rlwrap-0.37-1.el6.i686.rpm

  • Configure aliases in .bash_profile so that rlwrap gets enabled when you start the command

    alias sqlplus=”rlwrap sqlplus”
    alias dgmgrl=”rlwrap dgmgrl”
    alias rman=”rlwrap rman”
    alias lsnrctl=”rlwrap lsnrctl”
    alias asmcmd=”rlwrap asmcmd”
    alias adrci=”rlwrap adrci”
    alias impdp=”rlwrap impdp”
    alias expdp=”rlwrap expdp”

You’ll never go back again ūüėČ

How to move eth devices on Oracle Linux 6.x

When you install more than 1 Linux server at a site, it often happens that the patched interfaces get a different interface name. I’ll try to explain in this blogentry how to get them renamed correctly.

What we want

eth0      LAN 1Gb
eth1      LAN 1Gb
eth2      STORAGE 10Gb
eth3      STORAGE 10Gb

What we have

We installed 3 brand new servers and when we detect the interfaces :

# ifup eth0-9
# ethtool eth0-9

It results in the following mixed up configuration

– Server A

eth0      LAN 1Gb
eth1      LAN 1Gb
eth2      STORAGE 10Gb
eth3      STORAGE 10Gb

– Server B

eth2      LAN 1Gb
eth3      LAN 1Gb
eth8      STORAGE 10Gb
eth9      STORAGE 10Gb

– Server C

eth2      LAN 1Gb
eth3      LAN 1Gb
eth4      STORAGE 10Gb
eth5      STORAGE 10Gb

How To Correct it?

1) Adjust the /etc/udev/rules.d/70-persistent-net.rules file by interchanging the interfacenames. ( f.e. change eth2 to eth0 and viceversa )
2) Adjust the mac adress in the ifcfg-ethx config files in /etc/sysconfig/network-scripts.
3) reboot

Hope this helps somebody.