Oracle VM Disaster Recovery

A lot of my clients ask me about Disaster Recovery in a OVM setup. I hope this new event of oracle gives us some more insight. You can register here. I’ll certainly check it out. The whitepaper the event is based on can be found here.

Consistency between the Primary and DR site is not handled in this paper.According to Oracle this is a task for the application (f.e. Dataguard ) or the Storage layer (f.e. EMC Recoverpoint ). The white paper handles all the necessary tasks to make sure vm’s can be seen and started on the DR site. I hoped there was more possible with the tight integration of UEK and OVM. But offcourse, is that really needed when you can already have consistency solutions on storage/application level?

I believe VMWare has some solutions for this. Anyone care to elaborate on those?

 

 

Advertisements

Configuring Kerberos for Oracle Databases 11.2 with win2008R2 AD

In this blog entry I try to document how to enable Kerberos. This procedure was actually created and followed during a project at one of my customers.

The Infrastructure

AD
– windows 2008R2 server
– domain : milkyway.space.com
– Kerberos installed and enabled
– DES encryption default disabled

Server :
– moon.milkyway.space.com
– database : crater
– version : 11.2.0.3.4

Client
– windows 7 enterprise edition
– 11.2.0.3 client

The Procedure

  • On the AD server
    • Create a service account in Active Directory for the database server moon to validate the Kerberos tickets with. This user does not need any specific rights but enable “password never expires”. We called this account “ssoval”
    • ensure that you deselect Setup option “Use DES Encryption” and select option “Do not require Kerberos PreAuthentication” for this user
    • Make sure that the SPN is set to the correct realm
      setspn -A oracle/moon.milkyway.space.com@MILKYWAY.SPACE.COM ssoval
      (oracle is just the name of the service, we reuse this name in the kerberos config to point here. This has no connection to service_names of the database.)
    • Extract a keytab file for this user so we don’t need to enter password to create tickets
      ktpass -princ oracle/moon.milkyway.space.com@MILKYWAY.SPACE.COM -crypto all -pass ssoval -mapuser ssoval -out v5srvtab
    • Put this file on the database server. I’ve put it in /etc/v5srvtab
  • On the Database Server “moon”
    • Make sure Advanced Security Option is installed, this is a paying option on top of Enterprise Edition.
    • Generate a kerberos ticket, this will be used for connection to the kerberos server for ticket validation
      $ORACLE_HOME/jdk/bin/kinit -k -t /etc/v5srvtab oracle/moon.milkyway.space.com
      ( You might want to create a crontab job for this so that you always have a valid ticket )
    • Adjust the sqlnet.ora
      SQLNET.KERBEROS5_CONF=/etc/krb5.conf
      SQLNET.KERBEROS5_KEYTAB=/etc/v5srvtab
      SQLNET.KERBEROS5_CONF_MIT=TRUE
      SQLNET.AUTHENTICATION_KERBEROS5_SERVICE=oracle
      SQLNET.AUTHENTICATION_SERVICES=(BEQ,KERBEROS5)
    • Create the /etc/krb5.conf file
      [libdefaults]
      default_realm = MILKYWAY.SPACE.COM
      [realms]
      MILKYWAY.SPACE.COM = {
      kdc = DC1.MILKYWAY.SPACE.COM:88
      kdc = DC2.MILKYWAY.SPACE.COM:88
      }
      [domain_realm]
      .milkyway.space.com = MILKYWAY.SPACE.COM
      milkyway.space.com = MILKYWAY.SPACE.COM
  • On the Database “crater”
    • Clear OS_AUTHENT_PREFIX
      SQL> alter system set OS_AUTHENT_PREFIX=’’ scope=spfile;
    • Disable remote_os_authent
      SQL> alter system set remote_os_authent=false;
    • restart the database
  • On the Windows Clients
    • Make sure ASO is installed.
    • Adjust the sqlnet.ora
      SQLNET.AUTHENTICATION_SERVICES= (BEQ,KERBEROS5)
      SQLNET.KERBEROS5_CONF =c:\kerberos\krb5.conf
      SQLNET.KERBEROS5_CONF_MIT = true
      SQLNET.KERBEROS5_CC_NAME=OSMSFT://
      This last line is important for windows clients because this reuses the already generated tickets available on the system as a result of your AD login. Hence it enables the SSO login. Keep in mind that the Oracle tool okinit will fail with OSD error if this cache is set when you try to get manual tickets.
    • Create the c:\kerberos\krb5.conf file identical as on the server except for the port numbers
      [libdefaults]
      default_realm = MILKYWAY.SPACE.COM
      [realms]
      MILKYWAY.SPACE.COM = {
      kdc = DC1.MILKYWAY.SPACE.COM
      kdc = DC2.MILKYWAY.SPACE.COM
      }
      [domain_realm]
      .milkyway.space.com = MILKYWAY.SPACE.COM
      milkyway.space.com = MILKYWAY.SPACE.COM
    • Make sure the file services in directory c:\windows\system32\drivers\etc has “kerberos5” in the list as first entry
      kerberos 88/tcp kerberos5 krb5 kerberos-sec #Kerberos
      kerberos 88/udp kerberos5 krb5 kerberos-sec #Kerberos

Now you are ready to use Kerberos Authentication.

Example for an user “Bjorn”

  • Create an user Bjorn on the AD server in domain MILKYWAY.SPACE.COM
    Ensure that you :

    • deselect Setup option “Use DES Encryption”
    • select option “Do not require Kerberos PreAuthentication”

    The username is case sensitive, so make sure you have the correct case.

  • Create an user Bjorn on the database crater
    SQL> create user BJORN IDENTIFIED EXTERNALLY as 'Bjorn@MILKYWAY.SPACE.COM';
    SQL> grant create session to BJORN;
  • Login to the windows desktop and connect to the database over TNS for example :
    C:\> sqlplus /@crater
    CONNECTED

    SQL> show user
    USER is "BJORN"

    SQL> select sys_context('userenv ', 'session_user') from dual;
    SYS_CONTEXT('USERENV','SESSION_USER')
    -----------------------------------------
    BJORN

    SQL> select sys_context('userenv','external_name') from dual;
    SYS_CONTEXT('USERENV','EXTERNAL_NAME')
    -----------------------------------------------------------------
    Bjorn@MILKYWAY.SPACE.COM

Troubleshooting

  • KDC has no support for encryption type : pre-11gR2 only supports DES encryption. The company where I performed this setup, did not want to enable this legacy protocol ( and rightly so ), so only connections with 11.2 and higher clients to 11.2 and higher databases will work in this setup.
  • Cannot find KDC for requested realm : Make sure your services file is correctly formatted and kerberos5 is the first protocol in the list for port 88

Special thanks to antonio mata gomez from Oracle Belgium for support in this project

Best Practices for Oracle Linux for Production Systems

I found this needed a blog post because most of the customers I meet, just install their Redhat/Oracle Linux environment and start using it out of the box in production. I believe this list should be included in every post-installation procedure.

  • Hostname : Make sure it’s a FQDN. Especially when you connect with NFS to other systems. If your hostname is not FQDN, locks will not be freed on the NFS server when you reboot.
  • Support : If you have support, make sure you register your system with ULN.
  • Update : Update your system with yum or up2date to the latest version.
  • Hugepages : If you are running Oracle Databases, this is a must. Metalink note. 361468.1
  • Ipmitool : This allows for control over the hardware from inside the OS. Can be very usefull for Cluster setups or automated scripts to collect information.
  • Kexec : This allows the system to dump the kernel-memory to disk whenever a kernel panic occurs. Instead of rebooting or hanging, the system boots into a separate kernel with the task of dumping the memory to disk in the form of a vmcore file. This file can then later be analysed with the crash utility. Don’t forget to test it!!
  • magic sysrq key : This enables some key-strokes in the console to force a kernel to do all sorts of things ( show locks, reboot without FS corruption, … ). It is often used to dump a kernel stacktrace to /var/log/messages and reboot a system after soft hangs ( hangs on console with numlock flashing ). This is default enabled in OL5 but in OL6 you need to enable it manually. Also, make sure you know the keystrokes for when you need them.
  • Oswatcher BB : Monitoring tool of Oracle. Can show you if there were spikes just before or leading to the crash. Metalink note.301137.1
  • vncserver : allows for X11 environment over vnc. Faster then X11 over the net and allows you to continue where you left off when you lose your connection during an installation or configuration.
  • oratop : utility for near real-time monitoring of databases, RAC and Single Instance. Metalink note. 1500864.1.
  • dstat : allows you to view all of your system resources in real-time
  • Rlwrap: Saves you time 😉

If anyone is interested in how to perform some of these tasks, let me know and I’ll consider writing some blog entries about them. But most procedures can be found in the manual or official pages about it. Keep in mind that this list also applies to Oracle Engineered Systems.( ODA, EXADATA, … )

IOUG Virtualization SIG – Day 1

So, Day 1 of the Virtualization SIG on www.ioug.org is over. All by all an interesting day.
The schedule was :

Session 1 – Oracle on Oracle VM – Expert Panel
Session 2 – Maximizing your Virtualized Environment with Oracle VM
Session 3 – The RAC OVM Templates and the new DeployCluster tool on OVM3
Session 4 – The Latest on Oracle VM
Session 5 – Simplifying Application Deployment in Cloud Using Virtual Assemblies and EM 12c

The first session was a general session about Cloud, Virtualization and introduction to OVM.

Roger Lopez talked about OVM and it’s features in detail in the second session. I had the pleasure of seeing Roger on OOW with this session. It was very well structured and provided a look at how you deploy RAC clusters on OVM with the new templates and DeployCluster tool. A perfect introduction for the next session off course where Saar Maoz talked really enthusiastic about this tool and explained in detail how it worked and how you could go to a very low level and perform the commands yourself. It was no surprise to me that the tool used the same OVM API as I blogged about here. It was a surprise however to see how robust the application was written and how well it adjusted to reruns and failures. There was even some time for demo’s that I really appreciated.

Xsigo was something I talked about in my presentation aswell and was glad to see it in the presentation of Ronen Kofman ( Latest on OVM ). It really simplifies your whole Network topology. I have no idea on prices yet though. The rest of his presentation handled the new features in the 3.2.1 open Beta.

The last session of the day handled the Virtual Assembly Builder, A pretty powerful tool to create, manage and deploy your assemblies as fully functional interconnected Virtual Machines. It’s on my todo list to play around with it.

Tonight is Day 2, the VMWare day. I hope this time it will talk a little bit more about best practices on running Oracle Databases Virtualised, but from the titles of the sessions, I’m sure that will be the case.

You can register here

Virtualization Strategy with Oracle VM and Oracle Linux

I’m giving a presentation on this topic on thursday 25 Oct for OracleOpenXperience. Here are the slides for my presentation. Hope you find it interesting.

Oracle OpenXperience – Virtualization Strategy with Oracle VM & Oracle Linux

using ovmd and ovm_vmmessage to (re)configure your virtual machines.

What you need

  • Installation of ovm_utils on your manager ( p13602094_30_Linux-x86-64.zip available on metalink )
  • A VM with ovmd installed. Keep in mind that all OVM3 templates come with OVMD already installed. If you need to install it manually see this manual

How it works

ovmd is a listening process on your vm that allows messages to be send from and to the manager.
ovm_vmmessage is a utility included in ovm_utils that allows you to send messages to a vm in key/value pair.
the ovmd has a script called configure that can be executed to set system information. When you call it manually :

(vm1) # ovmd -s configure
Parameters:

u’com.oracle.linux.network.hostname’:
u’com.oracle.linux.network.host.0′:
u’com.oracle.linux.network.device.0′:
u’com.oracle.linux.network.hwaddr.0′:
u’com.oracle.linux.network.mtu.0′:
u’com.oracle.linux.network.onboot.0′:
u’com.oracle.linux.network.bootproto.0′:
u’com.oracle.linux.network.ipaddr.0′:
u’com.oracle.linux.network.netmask.0′:
u’com.oracle.linux.network.gateway.0′:
u’com.oracle.linux.network.dns-servers.0′:
u’com.oracle.linux.network.dns-search-domains.0′:
u’com.oracle.linux.root-password’: [required]

============================================================================

Input or edit parameter:

To use the current value, press “Enter”.
To input an empty string, input ” and press “Enter”.

Script: network
Key: com.oracle.linux.network.hostname
Description: System host name.
Required: False

Please input new value:

the parameters section will show you which keys the script accepts and recognises.
You can send these key/value pairs from the manager to the vm to skip the dialog.
f.e.

(manager) # ./ovm_vmmessage
Arguments :
-u [user name] Oracle VM Manager admin username (required)
-p [password] admin password (required)
-h [hostname] Oracle VM Manager hostname (required)
-X use secure (ssl) connection to port 54322
-v [vm name] virtual machine name
-U [vm UUID] virtual machine UUID
-k [key] key to send
-V [value] key-value to send (required when using -k)
-q [key] key to query

(manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.hostname -V test_ovmd
(manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.root-password -V rootroot

If you now check on your vm for the values, you’ll see the following :

(vm1) # ovmd -l
{“com.oracle.linux.network.hostname”:”test_ovmd”}
{“com.oracle.linux.root-password”:”rootroot”}

In order to make them active. Just run the ovmd -s configure script again. No dialog will be shown and all settings will be saved and active after a reboot.

(vm1) # ovmd -s configure
(vm1) # reboot

Configuring Templates

Now it becomes interesting!
Some of you might have recognised the questions the ‘ovmd -s configure’ command asked. These are actually the questions a newly created vm asks when it is cloned from the OVM3 templates of oracle. Wouldn’t it be interesting you don’t need to launch the console for each and every vm you create?

Well try this :

  • Create a new VM by cloning it from an OVM3 template/assembly available on edelivery ( f.e. OVM_OL6U2_x86_64_PVM.ova )
  • Boot the vm and let it sit on its dialogs
  • Run the following commands :

    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.hostname -V vm1
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.device.0 -V eth0
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.onboot.0 -V yes
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.bootproto.0 -V static
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.ipaddr.0 -V 10.100.23.36
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.netmask.0 -V 255.255.255.0
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.gateway.0 -V 10.100.23.254
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.network.dns-servers.0 -V 10.0.13.44
    (manager) # ./ovm_vmmessage -u admin -p password -h localhost -v vm1 -k com.oracle.linux.root-password -V rootroot

  • And voila. The minute your root password is set ( remember to do it last ), the information is loaded and the vm becomes active with all these settings applied. No need to launch the console. Awesome for batch creating vm’s.

It can be nice to reset all settings and make sure ovmd boots with the configure script again on next boot. You can do that by editing the /etc/sysconfig/ovmd script and set INITIAL_CONFIG=yes. Now stop the vm and clone it as vm/template, on first boot the dialog will start again and you’ll be able to send the commands or answer the questions.

I would like to point out this was tested on OracleVM 3.2.1 public beta but it should be applicable to 3.1.1 aswell.

You can find more information about this on the blog of Wim Couckaert.