TTS in Combination with RMAN backups on dataguard

At a Customer’s site we recently upgraded a database from 10.2.0.5 to 11.2.0.3 by using Transportable Tablespace ( TTS ). This worked flawlessly but we ran into an issue taking backups at the dataguard location of this database.

We followed the normal procedures for being able to take backups on Dataguard and use them for Primary

      • Use a RMAN catalog
      • Register the Primary database
        RMAN> REGISTER DATABASE;
      • Configure the DB Unique Names
        RMAN> CONFIGURE DB_UNIQUE_NAME DB CONNECT IDENTIFIER 'DB_PRIM';
        RMAN> CONFIGURE DB_UNIQUE_NAME DB_DG CONNECT IDENTIFIER 'DB_DG';
        
        RMAN> LIST DB_UNIQUE_NAME OF DATABASE;
        
        List of Databases
        DB Key  DB Name  DB ID            Database Role    Db_unique_name
        ------- ------- ----------------- ---------------  ------------------
        1       DB       336860753        PRIMARY          DB
        1       DB       336860753        STANDBY          DB_DG

At this moment, we can take backups on DB_DG and make them available to DB by changing the unique name in the catalog

RMAN> change backup for db_unique_name DB_DG reset db_unique_name to DB;

However for this one database we can’t seem to do anything with it inside RMAN.

RMAN> show all; 
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of show command at 04/26/2013 13:58:11
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 04/26/2013 13:58:11
RMAN-20999: internal error

RMAN> backup database; 
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 04/26/2013 14:05:11
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 04/26/2013 14:05:11
RMAN-20999: internal error

This turns out to be Bug 13000553 ( Metalink Id. 13000553.8 ) that occurs when you

  • take backups on dataguard 
  • use TTS
  • Add a datafile to the Tablespace you transported

At the moment of writing there is no fix for this. The only workaround is to take RMAN backups on the Primary Database.

Configuring Kerberos for Oracle Databases 11.2 with win2008R2 AD

In this blog entry I try to document how to enable Kerberos. This procedure was actually created and followed during a project at one of my customers.

The Infrastructure

AD
– windows 2008R2 server
– domain : milkyway.space.com
– Kerberos installed and enabled
– DES encryption default disabled

Server :
– moon.milkyway.space.com
– database : crater
– version : 11.2.0.3.4

Client
– windows 7 enterprise edition
– 11.2.0.3 client

The Procedure

  • On the AD server
    • Create a service account in Active Directory for the database server moon to validate the Kerberos tickets with. This user does not need any specific rights but enable “password never expires”. We called this account “ssoval”
    • ensure that you deselect Setup option “Use DES Encryption” and select option “Do not require Kerberos PreAuthentication” for this user
    • Make sure that the SPN is set to the correct realm
      setspn -A oracle/moon.milkyway.space.com@MILKYWAY.SPACE.COM ssoval
      (oracle is just the name of the service, we reuse this name in the kerberos config to point here. This has no connection to service_names of the database.)
    • Extract a keytab file for this user so we don’t need to enter password to create tickets
      ktpass -princ oracle/moon.milkyway.space.com@MILKYWAY.SPACE.COM -crypto all -pass ssoval -mapuser ssoval -out v5srvtab
    • Put this file on the database server. I’ve put it in /etc/v5srvtab
  • On the Database Server “moon”
    • Make sure Advanced Security Option is installed, this is a paying option on top of Enterprise Edition.
    • Generate a kerberos ticket, this will be used for connection to the kerberos server for ticket validation
      $ORACLE_HOME/jdk/bin/kinit -k -t /etc/v5srvtab oracle/moon.milkyway.space.com
      ( You might want to create a crontab job for this so that you always have a valid ticket )
    • Adjust the sqlnet.ora
      SQLNET.KERBEROS5_CONF=/etc/krb5.conf
      SQLNET.KERBEROS5_KEYTAB=/etc/v5srvtab
      SQLNET.KERBEROS5_CONF_MIT=TRUE
      SQLNET.AUTHENTICATION_KERBEROS5_SERVICE=oracle
      SQLNET.AUTHENTICATION_SERVICES=(BEQ,KERBEROS5)
    • Create the /etc/krb5.conf file
      [libdefaults]
      default_realm = MILKYWAY.SPACE.COM
      [realms]
      MILKYWAY.SPACE.COM = {
      kdc = DC1.MILKYWAY.SPACE.COM:88
      kdc = DC2.MILKYWAY.SPACE.COM:88
      }
      [domain_realm]
      .milkyway.space.com = MILKYWAY.SPACE.COM
      milkyway.space.com = MILKYWAY.SPACE.COM
  • On the Database “crater”
    • Clear OS_AUTHENT_PREFIX
      SQL> alter system set OS_AUTHENT_PREFIX=’’ scope=spfile;
    • Disable remote_os_authent
      SQL> alter system set remote_os_authent=false;
    • restart the database
  • On the Windows Clients
    • Make sure ASO is installed.
    • Adjust the sqlnet.ora
      SQLNET.AUTHENTICATION_SERVICES= (BEQ,KERBEROS5)
      SQLNET.KERBEROS5_CONF =c:\kerberos\krb5.conf
      SQLNET.KERBEROS5_CONF_MIT = true
      SQLNET.KERBEROS5_CC_NAME=OSMSFT://
      This last line is important for windows clients because this reuses the already generated tickets available on the system as a result of your AD login. Hence it enables the SSO login. Keep in mind that the Oracle tool okinit will fail with OSD error if this cache is set when you try to get manual tickets.
    • Create the c:\kerberos\krb5.conf file identical as on the server except for the port numbers
      [libdefaults]
      default_realm = MILKYWAY.SPACE.COM
      [realms]
      MILKYWAY.SPACE.COM = {
      kdc = DC1.MILKYWAY.SPACE.COM
      kdc = DC2.MILKYWAY.SPACE.COM
      }
      [domain_realm]
      .milkyway.space.com = MILKYWAY.SPACE.COM
      milkyway.space.com = MILKYWAY.SPACE.COM
    • Make sure the file services in directory c:\windows\system32\drivers\etc has “kerberos5” in the list as first entry
      kerberos 88/tcp kerberos5 krb5 kerberos-sec #Kerberos
      kerberos 88/udp kerberos5 krb5 kerberos-sec #Kerberos

Now you are ready to use Kerberos Authentication.

Example for an user “Bjorn”

  • Create an user Bjorn on the AD server in domain MILKYWAY.SPACE.COM
    Ensure that you :

    • deselect Setup option “Use DES Encryption”
    • select option “Do not require Kerberos PreAuthentication”

    The username is case sensitive, so make sure you have the correct case.

  • Create an user Bjorn on the database crater
    SQL> create user BJORN IDENTIFIED EXTERNALLY as 'Bjorn@MILKYWAY.SPACE.COM';
    SQL> grant create session to BJORN;
  • Login to the windows desktop and connect to the database over TNS for example :
    C:\> sqlplus /@crater
    CONNECTED

    SQL> show user
    USER is "BJORN"

    SQL> select sys_context('userenv ', 'session_user') from dual;
    SYS_CONTEXT('USERENV','SESSION_USER')
    -----------------------------------------
    BJORN

    SQL> select sys_context('userenv','external_name') from dual;
    SYS_CONTEXT('USERENV','EXTERNAL_NAME')
    -----------------------------------------------------------------
    Bjorn@MILKYWAY.SPACE.COM

Troubleshooting

  • KDC has no support for encryption type : pre-11gR2 only supports DES encryption. The company where I performed this setup, did not want to enable this legacy protocol ( and rightly so ), so only connections with 11.2 and higher clients to 11.2 and higher databases will work in this setup.
  • Cannot find KDC for requested realm : Make sure your services file is correctly formatted and kerberos5 is the first protocol in the list for port 88

Special thanks to antonio mata gomez from Oracle Belgium for support in this project

IOUG Virtualization SIG – Day 1

So, Day 1 of the Virtualization SIG on www.ioug.org is over. All by all an interesting day.
The schedule was :

Session 1 – Oracle on Oracle VM – Expert Panel
Session 2 – Maximizing your Virtualized Environment with Oracle VM
Session 3 – The RAC OVM Templates and the new DeployCluster tool on OVM3
Session 4 – The Latest on Oracle VM
Session 5 – Simplifying Application Deployment in Cloud Using Virtual Assemblies and EM 12c

The first session was a general session about Cloud, Virtualization and introduction to OVM.

Roger Lopez talked about OVM and it’s features in detail in the second session. I had the pleasure of seeing Roger on OOW with this session. It was very well structured and provided a look at how you deploy RAC clusters on OVM with the new templates and DeployCluster tool. A perfect introduction for the next session off course where Saar Maoz talked really enthusiastic about this tool and explained in detail how it worked and how you could go to a very low level and perform the commands yourself. It was no surprise to me that the tool used the same OVM API as I blogged about here. It was a surprise however to see how robust the application was written and how well it adjusted to reruns and failures. There was even some time for demo’s that I really appreciated.

Xsigo was something I talked about in my presentation aswell and was glad to see it in the presentation of Ronen Kofman ( Latest on OVM ). It really simplifies your whole Network topology. I have no idea on prices yet though. The rest of his presentation handled the new features in the 3.2.1 open Beta.

The last session of the day handled the Virtual Assembly Builder, A pretty powerful tool to create, manage and deploy your assemblies as fully functional interconnected Virtual Machines. It’s on my todo list to play around with it.

Tonight is Day 2, the VMWare day. I hope this time it will talk a little bit more about best practices on running Oracle Databases Virtualised, but from the titles of the sessions, I’m sure that will be the case.

You can register here

News from Oracle Open World 2012

I’m just back from OOW and the following is a small summary of the items that were hot and new.

Clusterware 12c announced for 2013

  • Flexasm : ASM no longer needs to be on the same node as the database. Allows for HA on ASM level.
  • Flexcluster : Clusterware now has a lightweight mode in the installer for application/middletier setups. You can combine these with the normal clusterware mode to form 1 big cluster. ( hub nodes and leaf nodes )
  • You can now create a Scan vip for each public network.
  • gsd service is removed because there is no support for 9i databases on clusterware 12c.
  • Shared GNS : You can now have a shared GNS over multiple clusters.

Database 12c announced for 2013

  • RAC Application Continuity : First ever database to preserve commit outcome and ensure application continuity during unplanned downtime.
  • Pluggable Databases : Databases can now be plugged in a container. Consolidating the background processes, redo and undo.
  • Xstream : A GUI on streams apply/capture processes is further enhanced but only available when you have Golden Gate license.
  • dbua : now has backup/restore and moving database files during upgrade built in.
  • dbua : Gathering statistics is now optional.
  • dbua : Now runs the upgrade scripts in parallel mode. Speeding up the whole process.
  • smart flash : Can use SSD’s as extention of Buffer Cache.
  • RMAN : can now do recover table
  • RMAN : Standby databases can now use RMAN to apply an incremental backup automatically ( Recover database from for standby )
  • RMAN : Now automatically converts datafiles from 1 platform to the other. ( big-little endian )
  • dbconsole is renamed to “EM Express” and now runs from inside the database.
  • Many more new features but I concentrated on virtualization and linux during OOW while my colleague did 12c.

OVM 3.2.1 in public beta

  • Support for mySQL databases as repository for the OVM Manager
  • OVM Manager now supports OVM on SPARC
  • Statistics of OVM Servers now available in OVM Manager ( Health tab)
  • Minor GUI changes and fixes
  • UEK2 kernel in dom0 for driver compatibility with Oracle Linux
  • OVM CLI now integrated and further enhanced
  • next Beta refresh will contain open Vswitch
  • OVM is now integrated in ODA and Exalogic, but each node has a local OVM cluster. No HA.

Oracle Linux

  • ksplice : allows for kernel upgrades without downtime
  • yum security updates : Now possible to apply only security updates to your system using yum
  • Dtrace : Solaris utility that was ported to Oracle Linux.
  • BTRFS : ZFS-like filesystem for Linux.
  • UEK3 is coming
  • SELinux profiles for every oracle product will be provided in the future
  • Linux Containers : Allows for isolation of applications while maintaining 1 OS.
  • support for swapfile over NFS
  • cgroups : Allows for limiting CPU,IO and memory on process level.

Acquisition of Xsigo was announced

  • Allows for virtualizing of network layer.
  • Will be integrated in OVM Manager in future releases.


    • I’ll try to focus on some of these new things in later Blogposts. As always if you have some questions, feel free to contact me or comment.

Adding and removing disks in ASM

A first for me. I had seen asm in the past but never had hands on experience with it. A client needed some disks moved from 1 diskgroup to the other and so started my journey into ASM. 🙂

The client was using grid control for management of the RAC an ASM instances, the point and click interface is really intuitif and makes these actions childplay. HOWEVER, not all clients have grid control and in my case I didn’t have the sys password to connect with the asm instance. A local connection was the only possible solution and that meant… fun 🙂

I was glad Grid control didn’t work so I could get to know the command line interface. The folowing is my procedure.

1. detect the group number from the source and Dest group :

SQL> select group_number,name from v$asm_diskgroup;

GROUP_NUMBER NAME
———— ——————————
3 DG_SRC_DF
5 DG_DEST_DF

2. Get an overview of the disks from that diskgroup :

SQL> select DISK_NUMBER, name, failgroup, group_number from v$asm_disk where group_number=3 order by name;

0 asmdiskA17                    FailgroupA                                  3
1 asmdiskA18                    FailgroupA                                  3
2 asmdiskA19                    FailgroupA                                  3
3 asmdiskA22                    FailgroupA                                  3
4 asmdiskA23                    FailgroupA                                  3
5 asmdiskA24                    FailgroupA                                  3
6 asmdiskA25                    FailgroupA                                  3
7 asmdiskA26                    FailgroupA                                  3
8 asmdiskA27                    FailgroupA                                  3
9 asmdiskA28                    FailgroupA                                  3
10 asmdiskB39                    FailgroupB                                  3
11 asmdiskB40                    FailgroupB                                  3
12 asmdiskB41                    FailgroupB                                  3
13 asmdiskB42                    FailgroupB                                  3
14 asmdiskB43                    FailgroupB                                  3
15 asmdiskB44                    FailgroupB                                  3
16 asmdiskB45                    FailgroupB                                  3
17 asmdiskB48                    FailgroupB                                  3
18 asmdiskB49                    FailgroupB                                  3
19 asmdiskB50                    FailgroupB                                  3

3. Remove 7 disks of each failgroup = 14 :

alter diskgroup DG_SRC_DF drop disk asmdiskA28;
alter diskgroup DG_SRC_DF drop disk asmdiskA27;
alter diskgroup DG_SRC_DF drop disk asmdiskA26;
alter diskgroup DG_SRC_DF drop disk asmdiskA25;
alter diskgroup DG_SRC_DF drop disk asmdiskA24;
alter diskgroup DG_SRC_DF drop disk asmdiskA23;
alter diskgroup DG_SRC_DF drop disk asmdiskA22;
alter diskgroup DG_SRC_DF drop disk asmdiskB50;
alter diskgroup DG_SRC_DF drop disk asmdiskB49;
alter diskgroup DG_SRC_DF drop disk asmdiskB48;
alter diskgroup DG_SRC_DF drop disk asmdiskB45;
alter diskgroup DG_SRC_DF drop disk asmdiskB44;
alter diskgroup DG_SRC_DF drop disk asmdiskB43;
alter diskgroup DG_SRC_DF drop disk asmdiskB42;

4. check if the rebalancing is done :

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
———— —– —- ———- ———- ———- ———- ———- ———–
3 REBAL RUN           1          1      18652      69864        722          70

5. when done , check if the disks are gone with query from point 2.
6. add the disks to the new diskgroup

alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA28’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA27’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA26’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA25’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA24’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA23’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA22’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB50’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB49’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB48’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB45’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB44’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB43’;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB42’;

7. Disks are added and ASM will rebalance the load over al the disks in background  :

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
———— —– —- ———- ———- ———- ———- ———- ———–
5 REBAL RUN           1          1      18652      69864        722          70

After  searching and trying  it was really quite a good learning case to get to know ASM a bit better.  And know i can add ASM knowledge to my resumé 😉