Snapmanager for Oracle 3.0 – First Impressions – Part 2

A short update on the many problems I have had with Snapmanager 3.0 for Oracle

protection manager

Since 3.0 SMO supports migration of snapshots to a disaster netapp. There is an extra license and software involved though namely Protection Manager. If you are thinking of using it, don’t. To take advantage of these Protection Manager features through SnapDrive, you must have a DFM server with version 3.7 or later, and your storage systems must be running Data ONTAP 7.3 or later. ( 7.3 is currently in Beta )

Scheduling

Scheduling has no pre or post script options. So if you want to manually call something before or after snapshotting ( say for example snapmirror, or a status mail ), you have to go back to full scripting. Scheduling is also NOT possible on command line. I have no idea what the devs were thinking. Every SMO action has a command line option except the scheduling, a pain in the ass if you want to automate creations of profiles, backups and backup schedules.

a third and UNACCEPTABLE “feature” is that SMO scheduling is NOT cluster aware. On creation of the schedule, the hostname gets written in the database ( SMO_30_SCHEDULEDOPERATION ), when you run your database on another host the software forgets the profiles as it is now another host/db combination. Recreating the profiles doesn’t work because the name is already in use.

I felt naughty and updated the tables manually with the virtual IP’s of the databases, but to no avail.

Netapp Name

Not a real SMO problem but rather a snapdrive problem. When taking snapshots, Snapdrive uses the hostname from the netapp. This means that when you try to mount one of the snapshots and the name you use to talk to your netapps is different then the hostname, Snapdrive gives an error :

for example :

netapp name : stor01
lan network : stor01l
storage network : stor01s

mounts are on storage network :

stor1s:/vol/vol_df
stor1s:/vol/vol_arch
stor1s:/vol/vol_redo1
stor1s:/vol/vol_redo2

When you try to run a verify, restore or mount :

/usr/sbin/snapdrive snap connect -fs /test/DB/oradata/datafiles /opt/NetApp/smo/mnt/-test-DB-oradata-datafiles-20081212141831810_0 -destfv stor1:/vol/vol_df SnapManager_20081212141831851_test_df -snapname stor1:/vol/vol_df:smo_test_20081212_141644_hourly_1_8a809e8a1e2b576f011e2b5778580001_0 -autorename -noreserve
0002-044 Command error: The following Storage System Volume(s) do not exist in the snapshot and cannot be renamed
Volumes : stor1:/vol/vol_df

When adding stor1 in /etc/hosts with ip stor1s and changing the mounts to use stor1 instead of stor1s, it works. So you have to mount your volumes with the hostname of your netapp. Something to keep in mind.

Archivelogs
Archivelogs are still not cleaned up automatically. You need a separate script to do the cleanup. ( post-script would solve this for a part )

…
More to come

Snapmanager for Oracle 3.0 – First impressions – part 1

Ah… 4 december the day SMO 3.0 was released to the public. I couldn’t help it, I had to free up a machine and start testing it. Netapp announced some major updates in this major release.

• built-in scheduler
• built-in support for snapvault/snapmirror
• new GUI

I expected a lot from this release because, face it, SMO 2.2 was lacking in professionalism. It looked ugly, felt clumsy but it did its job. Some people still are doubting the added value of a tool like SMO, we’ll see if netapp can change their mind with 3.0.

Testsystem

My tests are done on Oracle Enterprise linux 5.2 with latest patches. I’ll snapshot a 11.1.0.7 database. The SMO repository will also exist inside a 11.1.0.7 database.All mounts are done over NFS.

Installing

Right out of the box snapdrive and snapmanager install flawlessly. What could catch you off guard is that snapdrive 4.1 comes with https enabled out of the box. So if your filer doesn’t support https, you need to change the snapdrive.conf file so it uses http to connect.

GUI

Oh i like it already, the JRE is part of the startup process this time and gets downloaded and installed automatically. Still not working with java 6 though. It even works in firefox, finally

Ok, first startup… looks better but still just a sterile look. On unix though, my subwindows are just empty. Could be a problem with my machine, cause i get the same effect with dbca from oracle. so let’s give SMO the benefit of the doubt. Ok, starting my virtual machine to test in windows. All seems fine. Time to create the repository and let’s use the wizard from the gui for once ( normally i’m command line minded, but in this case the gui needs to be tested thoroughly )

ORA-17002 : Ouch, IO exception while connecting with jdbc to the database. Only 1 entry on metalink and talks about hostresolving on RAC, not the case here and hostresolving is set correctly.Seems my tests are temporary to a halt while i figure this out. I hope the 11G repository is not the problem. Tune in later for more

Back after a short intermezzo with a workaround. On command line I have no issues connecting with jdbc and creating the repository

smo repository create -repository -dbname SMO -host ls-alfresco -login -username SMO -port 1521
Enter password for database connection SMO@ls-alfresco:1521/SMO: ***
[ INFO] SMO-20019: Set password for repository “SMO@SMO/ls-alfresco:1521″ in user credentials for “root”.
[ INFO] SMO-09202: Creating new schema as SMO on jdbc:oracle:thin:@//ls-alfresco:1521/SMO.
[ INFO] SMO-09205: Schema generation complete.
[ INFO] SMO-09209: Performing repository version INSERT.
[ INFO] SMO-09210: Repository created with version: 62
[ INFO] SMO-13048: Repository Create Operation Status: SUCCESS
[ INFO] SMO-13049: Elapsed Time: 0:00:09.084

It also works with a colleague on windows, so it seems there is an issue isolated to my machine. I reran netca, tested the connection and now it worked. Later I ran into a problem with SOAP when i tried to create a profile. I should warn you to checkwhich version of java your OS uses to launch the java application. Make sure 1.6 is either not installed or not enabled in the java control panel. My problems were fixed after disabling java 1.6.

The gui now exists out of 3 parts. Monitoring, Scheduling and backups/clones. They are easy accessible using the tabs on top. The monitoring tab looks like the old GUI screen but now we see a reports section in the sidebar where we can see all operations in a certain time range ( last hour, last month, … ). The scheduling tab lets you.. well… schedule things and the backup/clone tab shows a detailled list of backups and clones.

Profiles

I’m glad to see SMO is now working with retention classes and can be configured for both duration and count. One of the major changes is that 1 profile now can have multiple retentions. The defaults are adjustable in smo.conf. No more creating 4 profiles for every database for hourly, daily, weekly, monthly.Also new is that smo checks your database for a valid restore profile and if your database can be restored using fast restore. It not only says if its possible or not, you also get a reason why. For example :

–[ WARN] SMO-05071: Database profile ALFRESCO is not eligible for fast restore:  Restore Plan:
Preview:
The following files will be restored completely via: storage side file restore
/alfresco/DB/oradata/alfresco01.dbf
/alfresco/DB/oradata/nagios01.dbf
/alfresco/DB/oradata/sysaux01.dbf
/alfresco/DB/oradata/system01.dbf
/alfresco/DB/oradata/undotbs01.dbf
/alfresco/DB/oradata/users01.dbf

Analysis:
The following reasons prevent certain files from being restored completely via: storage side file system restore
Files in file system /alfresco not part of the restore scope will be reverted.
Files not in restore scope:
/alfresco/DB/flash_recovery/ALFRESCO/archivelog/2008_12_05/o1_mf_1_548_4ml7jmjs_.arc
/alfresco/DB/oradata/control02.ctl
/alfresco/DB/oradata/redo02b.rdo
Files to restore:
/alfresco/DB/oradata/alfresco01.dbf
/alfresco/DB/oradata/nagios01.dbf
/alfresco/DB/oradata/sysaux01.dbf
/alfresco/DB/oradata/system01.dbf
/alfresco/DB/oradata/undotbs01.dbf
/alfresco/DB/oradata/users01.dbf

So let’s bring our database over to the netapp so we can test further.

This concludes part 1

8 things about myself

Ok, thanks to Geert I’m tagged and have to spread this virus. ( Gee, thanks Geert )

1. My family situation is nothing but simple but I’ll not bore you with details. Parents divorced, 1 brother that is living together with his girlfriend. They have a beautifull daughter together who is also my Godchild.

2. I love my godchild to death. She’s so smart, funny. She’s 3 years old and her name is Amber.

3. I’m a catperson, I had cats my whole life and just love the little critters. I have the scars on my hands to prove it :p ( I love teasing them and that takes its toll )

4. I studied Informatics in Kortrijk and had the opportunity to do my thesis in Spain. Valladolid to be precise, it was a great experience. We had to make a program for teachers that work with people that have Down syndrome. We did the whole thing in C++ Builder and have rewritten the whole windows API for working with pictures… It was awesome But at that time I decided I never wanted a full-time programming job.

5. I’m very passionate about my interests. So passionate that it’s sometimes a bad quality for me. Whenever I’m interested in something, I can’t be slightly interested. I have to know everything about it. The bad thing is that because of the intense research, I’m bored with it after a couple of weeks. This happens both in my professional and private live

6. For some reason, I hate PLSQL. I really hate it. I’m a perl, php man and although I’m always open for new experiences, PLSQL will never warm my heart.

7. I’m a roleplayer. Yeah I’m one of those nerds that loves going to fantasy places in his mind with a fictive character. My favourite character is Corda, a soft-hearted female healertype. ( It is not gay to play a girl, I repeat… :p ) My favourite roleplaysystem is warhammer Fantasy Roleplay. Come to think of it, it has been a long time since I played or DM’d ( dungeon mastered ) a game.

8. Wine, If you want to give me a present, give me a good white wine I don’t drink the red kind, but always willing to try. I used to be into special beers ( Duvel, Leffe, Westmalle, … ) but now it’s winetime.. ( nice word :p )

Ok, that were 8 things about myself, I hope I didn’t bore you ( too much ). My turn to tag and I tag Nobody. That’s right… Nobody. Not that I don’t want to, but I’m apparently last in the line from my friends and don’t really want to tag unknown people.

Adding and removing disks in ASM

A first for me. I had seen asm in the past but never had hands on experience with it. A client needed some disks moved from 1 diskgroup to the other and so started my journey into ASM.

The client was using grid control for management of the RAC an ASM instances, the point and click interface is really intuitif and makes these actions childplay. HOWEVER, not all clients have grid control and in my case I didn’t have the sys password to connect with the asm instance. A local connection was the only possible solution and that meant… fun

I was glad Grid control didn’t work so I could get to know the command line interface. The folowing is my procedure.

1. detect the group number from the source and Dest group :

SQL> select group_number,name from v$asm_diskgroup;

GROUP_NUMBER NAME
———— ——————————
3 DG_SRC_DF
5 DG_DEST_DF

2. Get an overview of the disks from that diskgroup :

SQL> select DISK_NUMBER, name, failgroup, group_number from v$asm_disk where group_number=3 order by name;

0 asmdiskA17                    FailgroupA                                  3
1 asmdiskA18                    FailgroupA                                  3
2 asmdiskA19                    FailgroupA                                  3
3 asmdiskA22                    FailgroupA                                  3
4 asmdiskA23                    FailgroupA                                  3
5 asmdiskA24                    FailgroupA                                  3
6 asmdiskA25                    FailgroupA                                  3
7 asmdiskA26                    FailgroupA                                  3
8 asmdiskA27                    FailgroupA                                  3
9 asmdiskA28                    FailgroupA                                  3
10 asmdiskB39                    FailgroupB                                  3
11 asmdiskB40                    FailgroupB                                  3
12 asmdiskB41                    FailgroupB                                  3
13 asmdiskB42                    FailgroupB                                  3
14 asmdiskB43                    FailgroupB                                  3
15 asmdiskB44                    FailgroupB                                  3
16 asmdiskB45                    FailgroupB                                  3
17 asmdiskB48                    FailgroupB                                  3
18 asmdiskB49                    FailgroupB                                  3
19 asmdiskB50                    FailgroupB                                  3

3. Remove 7 disks of each failgroup = 14 :

alter diskgroup DG_SRC_DF drop disk asmdiskA28;
alter diskgroup DG_SRC_DF drop disk asmdiskA27;
alter diskgroup DG_SRC_DF drop disk asmdiskA26;
alter diskgroup DG_SRC_DF drop disk asmdiskA25;
alter diskgroup DG_SRC_DF drop disk asmdiskA24;
alter diskgroup DG_SRC_DF drop disk asmdiskA23;
alter diskgroup DG_SRC_DF drop disk asmdiskA22;
alter diskgroup DG_SRC_DF drop disk asmdiskB50;
alter diskgroup DG_SRC_DF drop disk asmdiskB49;
alter diskgroup DG_SRC_DF drop disk asmdiskB48;
alter diskgroup DG_SRC_DF drop disk asmdiskB45;
alter diskgroup DG_SRC_DF drop disk asmdiskB44;
alter diskgroup DG_SRC_DF drop disk asmdiskB43;
alter diskgroup DG_SRC_DF drop disk asmdiskB42;

4. check if the rebalancing is done :

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
———— —– —- ———- ———- ———- ———- ———- ———–
3 REBAL RUN           1          1      18652      69864        722          70

5. when done , check if the disks are gone with query from point 2.
6. add the disks to the new diskgroup

alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA28′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA27′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA26′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA25′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA24′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA23′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupA disk ‘/asmdisks/asmdiskA22′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB50′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB49′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB48′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB45′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB44′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB43′;
alter diskgroup DG_DEST_DF add FAILGROUP FailgroupB disk ‘/asmdisks/asmdiskB42′;

7. Disks are added and ASM will rebalance the load over al the disks in background  :

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES
———— —– —- ———- ———- ———- ———- ———- ———–
5 REBAL RUN           1          1      18652      69864        722          70

After  searching and trying  it was really quite a good learning case to get to know ASM a bit better.  And know i can add ASM knowledge to my resumé

OCA studies ( first test exam )

I couldn’t wait, With 2 chapters still to go in the book I tried a test exam to see where i’m at. I’ll keep the score for last ( if it was good, i’m sure it would be first ) but first some things I’ve learned

Lessons Learned :

• Read the question, reread the question, look at the answers and reread the question. From all the mistakes I made 75% were made because I didn’t fully understand the question. An example?

you must retain committed data for 10 minutes without hampering any uncommitted transactions.
I was so involved with the decision of retention guarantee or not that I forgot it was mentioned as minutes and selected the retention_target to 10 and no retention guarantee option instead of the 600.

• Watch out for traps. I sometimes selected something based on a complete cycle while only 1 part was mentioned.

What happens when a user executes the merge statement?
I selected everything that Oracle does in this case but forgot that commit was not included. So the SCN change to controlfile and the writes to redolog were wrong ( they happen on commit ).

• Syntax. I thought it wasn’t important but it is. Often you get a question about which view you need to query with very close options.

Which view contains audit information about table x
dba_obj_audit_opts or dba_audit_obj_opts or … Confusing

• And then offcourse the plain old mistakes i made. After carefull review, Some mistakes were really stupid ( selecting dba_objects instead of dba_views for view information for example ) , others were because I haven’t learned them yet ( flashback table, is that even part of the oca exam? )

I’ll conclude with my score on this exam. I still passed according to oracle norms but it wasn’t too great. 72%

My Top Tens

Added a page here  where I list all my top tens, feel free to comment on them.

To my surprise it was quite easy to make a top 10 for all those lists, the problem was putting them in order. Somehow i succeeded but expect some changes here and there

OCA studies ( Continued )

Chapter 16 : Managing Undo.

Somehow I think this will be a hard chapter. I just had 2 very easy chapters about Oracle monitoring and can’t seem to find the courage to start with Managing Undo. It can’t be that hard can it?

That makes me wonder why I find it a hard subject. ( I shiver when i hear the word undo )

My relationship with undo hasn’t been a good one ( insert bad childhood joke here ), especially the ora-1555 has been especially cruel to me. At my previous job we often had the problem and each time I thought I had found the solution, it happened again. I added undo space, put the retention longer, … Nothing seemed to work.

I had several questions about the usage of the undo space and never could get clear answers. What is the ora-1555 error? How do I solve it permenantly? By reading the certification book and understanding the inner mechanics of an oracle database I already have a clearer view:

Ora-1555 is the error that is used when the database looses the ability to give other queries a consistent view of the database. The undo space keeps the “old records” so that query’s running while the data is (being) changed and is uncommitted still see the old data. When a database can’t guarantee this anymore, the running query is cancelled with ora-1555. the retention period and size all determine how long that is. Size is the most important one I guess. The new undo advisor in 10g should be able to predict the undo space needed because that was a real mess.

Now let’s study chapter 16 and see if that is correct

The Beauty of Oracle Streams

Geert De Paep a collegue and friend at Uptime has a very interesting article about using Oracle Streams for migrating your live database to a higher or even lower release ( even accross platforms ). Read for yourself at his blog :
http://geertdepaep.wordpress.com/2007/11/24/the-beauty-of-oracle-streams/

OCA studies

I’m currently studying for my Oracle DBA OCA exams. ( Oracle Certified Associate)

I was planning to do the exam in December but the schedule at work was quite hectic. But today I got good and bad news. There was a course planned about Veritas clustered filesystem and RAC installation for 8 days, this was cancelled due to financial issues. That gave me a mixed feeling, one side really hates not having that course because it was very interesting and would give me more experience with oracle RAC, the other side  of me likes the 8 free days to study and take the exam.

My dream to be OCP before I turn 30 is still alive

Forza Motorsport 2

After a day of assassins creed I decided to try out forza 2 . It was included in the console package and I was a big fan of the first game on the normal xbox. I had high expectations for this game and I’m happy to report that they have been met. I’m addicted !!

Currently fighting my way through the carreer mode and now and then playing on xbox live. My skillevel is not high enough yet to be competitive in online play but it will be in a few days Look for me on xbox live. Gamertag “doneagle”

My favourate car atm is an honda acura integra fully upgraded to U999. Anyone want to sell me a good design for it? Hint : something from BLEACH would be awesome

Cya online.