Recently I had a client where oracleasm scandisks took over 10 minutes to run. This meant a reboot took 12 minutes instead of 2 minutes and performing any kind of HA tests where disks were involved had this 10-min wait.
The Problem
This client had an enormous amount of Logical Volumes per server and this was the only difference with other systems where I had succesfully used asmlib. So I went to check how these were defined on an OS level.
It turned out the devices were created in /dev/mapper and were visible in /proc/partitions but there were no devices created for them in /dev.
(root) # cat /proc/partitions|grep dm-
253 0 2097152 dm-0
253 1 4194304 dm-1
253 2 524288 dm-2
253 3 4194304 dm-3
253 4 524288 dm-4
253 5 524288 dm-5
253 6 6291456 dm-6
253 7 4194304 dm-7
253 8 524288 dm-8
253 9 16777216 dm-9
253 10 4194304 dm-10
253 11 2097152 dm-11
253 12 16777216 dm-12
253 13 17846400 dm-13
253 14 267696000 dm-14
253 15 267696000 dm-15
253 16 267696000 dm-16
253 17 267696000 dm-17
253 18 2973120 dm-18
253 19 107078400 dm-19
253 20 107078400 dm-20
253 21 267696000 dm-21
253 22 62462400 dm-22
253 23 2973120 dm-23
253 24 267691031 dm-24
253 25 107073161 dm-25
253 26 267691031 dm-26
253 27 62462336 dm-27
253 28 267691031 dm-28
253 29 107073161 dm-29
253 30 2971993 dm-30
253 31 17840118 dm-31
253 32 2971961 dm-32
253 33 267691031 dm-33
253 34 267691031 dm-34
253 35 524288 dm-35
253 36 10485760 dm-36
253 37 4194304 dm-37
253 38 52428800 dm-38
253 39 15728640 dm-39
253 40 2097152 dm-40
253 41 18874368 dm-41
253 42 524288 dm-42
253 43 1048576000 dm-43
253 44 10485760 dm-44
253 45 4194304 dm-45
253 46 15728640 dm-46
253 47 17846400 dm-47
253 48 267696000 dm-48
253 49 267696000 dm-49
253 50 267696000 dm-50
253 51 267696000 dm-51
253 52 2973120 dm-52
253 53 107078400 dm-53
253 54 107078400 dm-54
253 55 267696000 dm-55
253 56 62462400 dm-56
253 57 2973120 dm-57
253 58 2971961 dm-58
253 59 107073161 dm-59
253 60 107073161 dm-60
253 61 62462336 dm-61
253 62 267691031 dm-62
253 63 267691031 dm-63
253 64 17840118 dm-64
253 65 267691031 dm-65
253 66 2971993 dm-66
253 67 267691031 dm-67
253 68 267691031 dm-68
253 69 2935808 dm-69
253 70 2931831 dm-70(root) # ls -l /dev/dm-*
brw-rw—- 1 root root 253, 13 Mar 3 14:17 /dev/dm-13
brw-rw—- 1 root root 253, 14 Mar 3 14:17 /dev/dm-14
brw-rw—- 1 root root 253, 15 Mar 3 14:17 /dev/dm-15
brw-rw—- 1 root root 253, 16 Mar 3 14:17 /dev/dm-16
brw-rw—- 1 root root 253, 17 Mar 3 14:17 /dev/dm-17
brw-rw—- 1 root root 253, 18 Mar 3 14:17 /dev/dm-18
brw-rw—- 1 root root 253, 19 Mar 3 14:17 /dev/dm-19
brw-rw—- 1 root root 253, 20 Mar 3 14:17 /dev/dm-20
brw-rw—- 1 root root 253, 21 Mar 3 14:17 /dev/dm-21
brw-rw—- 1 root root 253, 22 Mar 3 14:17 /dev/dm-22
brw-rw—- 1 root root 253, 23 Mar 3 14:17 /dev/dm-23
brw-rw—- 1 root root 253, 24 Mar 3 14:17 /dev/dm-24
brw-rw—- 1 root root 253, 25 Mar 3 14:17 /dev/dm-25
brw-rw—- 1 root root 253, 26 Mar 3 14:17 /dev/dm-26
brw-rw—- 1 root root 253, 27 Mar 3 14:17 /dev/dm-27
brw-rw—- 1 root root 253, 28 Mar 3 14:17 /dev/dm-28
brw-rw—- 1 root root 253, 29 Mar 3 14:17 /dev/dm-29
brw-rw—- 1 root root 253, 30 Mar 3 14:17 /dev/dm-30
brw-rw—- 1 root root 253, 31 Mar 3 14:17 /dev/dm-31
brw-rw—- 1 root root 253, 32 Mar 3 14:17 /dev/dm-32
brw-rw—- 1 root root 253, 33 Mar 3 14:17 /dev/dm-33
brw-rw—- 1 root root 253, 34 Mar 3 14:17 /dev/dm-34
brw-rw—- 1 root root 253, 47 Mar 3 14:50 /dev/dm-47
brw-rw—- 1 root root 253, 48 Mar 3 14:50 /dev/dm-48
brw-rw—- 1 root root 253, 49 Mar 3 14:50 /dev/dm-49
brw-rw—- 1 root root 253, 50 Mar 3 14:50 /dev/dm-50
brw-rw—- 1 root root 253, 51 Mar 3 14:50 /dev/dm-51
brw-rw—- 1 root root 253, 52 Mar 3 14:50 /dev/dm-52
brw-rw—- 1 root root 253, 53 Mar 3 14:50 /dev/dm-53
brw-rw—- 1 root root 253, 54 Mar 3 14:50 /dev/dm-54
brw-rw—- 1 root root 253, 55 Mar 3 14:50 /dev/dm-55
brw-rw—- 1 root root 253, 56 Mar 3 14:50 /dev/dm-56
brw-rw—- 1 root root 253, 57 Mar 3 14:50 /dev/dm-57
brw-rw—- 1 root root 253, 58 Mar 3 14:50 /dev/dm-58
brw-rw—- 1 root root 253, 59 Mar 3 14:50 /dev/dm-59
brw-rw—- 1 root root 253, 60 Mar 3 14:50 /dev/dm-60
brw-rw—- 1 root root 253, 61 Mar 3 14:50 /dev/dm-61
brw-rw—- 1 root root 253, 62 Mar 3 14:50 /dev/dm-62
brw-rw—- 1 root root 253, 63 Mar 3 14:50 /dev/dm-63
brw-rw—- 1 root root 253, 64 Mar 3 14:50 /dev/dm-64
brw-rw—- 1 root root 253, 65 Mar 3 14:50 /dev/dm-65
brw-rw—- 1 root root 253, 66 Mar 3 14:50 /dev/dm-66
brw-rw—- 1 root root 253, 67 Mar 3 14:50 /dev/dm-67
brw-rw—- 1 root root 253, 68 Mar 3 14:50 /dev/dm-68
brw-rw—- 1 root root 253, 69 Mar 3 14:58 /dev/dm-69
brw-rw—- 1 root root 253, 70 Mar 3 14:58 /dev/dm-70
I knew oracleasm used /proc/partitions as its leading table of devices to check, so I believed a timeout occured while trying to access the non-existing devices. It turns out this was correct.
The Solution
Oracle Linux 5 & Redhat 5 do not create the devices for LVM2 devices by default. It took me some time to find this, but in the udev rules there is a clear ignore rule.
(root) # cat /etc/udev/rules.d/90-dm.rules
KERNEL==”dm-[0-9]*”, ACTION==”add”, OPTIONS+=”ignore_device”
When we disable this by commenting it out and retrigger the udev rules, our devices get created.
(root) # cat /etc/udev/rules.d/90-dm.rules
#KERNEL==”dm-[0-9]*”, ACTION==”add”, OPTIONS+=”ignore_device”(root) # udevtrigger
(root) # ls -ltr /dev/dm-*
brw-rw—- 1 root root 253, 15 Mar 3 14:17 /dev/dm-15
brw-rw—- 1 root root 253, 14 Mar 3 14:17 /dev/dm-14
brw-rw—- 1 root root 253, 19 Mar 3 14:17 /dev/dm-19
brw-rw—- 1 root root 253, 20 Mar 3 14:17 /dev/dm-20
brw-rw—- 1 root root 253, 16 Mar 3 14:17 /dev/dm-16
brw-rw—- 1 root root 253, 21 Mar 3 14:17 /dev/dm-21
brw-rw—- 1 root root 253, 17 Mar 3 14:17 /dev/dm-17
brw-rw—- 1 root root 253, 22 Mar 3 14:17 /dev/dm-22
brw-rw—- 1 root root 253, 23 Mar 3 14:17 /dev/dm-23
brw-rw—- 1 root root 253, 25 Mar 3 14:17 /dev/dm-25
brw-rw—- 1 root root 253, 24 Mar 3 14:17 /dev/dm-24
brw-rw—- 1 root root 253, 30 Mar 3 14:17 /dev/dm-30
brw-rw—- 1 root root 253, 27 Mar 3 14:17 /dev/dm-27
brw-rw—- 1 root root 253, 26 Mar 3 14:17 /dev/dm-26
brw-rw—- 1 root root 253, 31 Mar 3 14:17 /dev/dm-31
brw-rw—- 1 root root 253, 33 Mar 3 14:17 /dev/dm-33
brw-rw—- 1 root root 253, 32 Mar 3 14:17 /dev/dm-32
brw-rw—- 1 root root 253, 34 Mar 3 14:17 /dev/dm-34
brw-rw—- 1 root root 253, 18 Mar 3 14:17 /dev/dm-18
brw-rw—- 1 root root 253, 28 Mar 3 14:17 /dev/dm-28
brw-rw—- 1 root root 253, 29 Mar 3 14:17 /dev/dm-29
brw-rw—- 1 root root 253, 13 Mar 3 14:17 /dev/dm-13
brw-rw—- 1 root root 253, 53 Mar 3 14:50 /dev/dm-53
brw-rw—- 1 root root 253, 47 Mar 3 14:50 /dev/dm-47
brw-rw—- 1 root root 253, 55 Mar 3 14:50 /dev/dm-55
brw-rw—- 1 root root 253, 49 Mar 3 14:50 /dev/dm-49
brw-rw—- 1 root root 253, 52 Mar 3 14:50 /dev/dm-52
brw-rw—- 1 root root 253, 54 Mar 3 14:50 /dev/dm-54
brw-rw—- 1 root root 253, 51 Mar 3 14:50 /dev/dm-51
brw-rw—- 1 root root 253, 56 Mar 3 14:50 /dev/dm-56
brw-rw—- 1 root root 253, 48 Mar 3 14:50 /dev/dm-48
brw-rw—- 1 root root 253, 57 Mar 3 14:50 /dev/dm-57
brw-rw—- 1 root root 253, 50 Mar 3 14:50 /dev/dm-50
brw-rw—- 1 root root 253, 64 Mar 3 14:50 /dev/dm-64
brw-rw—- 1 root root 253, 61 Mar 3 14:50 /dev/dm-61
brw-rw—- 1 root root 253, 66 Mar 3 14:50 /dev/dm-66
brw-rw—- 1 root root 253, 60 Mar 3 14:50 /dev/dm-60
brw-rw—- 1 root root 253, 58 Mar 3 14:50 /dev/dm-58
brw-rw—- 1 root root 253, 59 Mar 3 14:50 /dev/dm-59
brw-rw—- 1 root root 253, 68 Mar 3 14:50 /dev/dm-68
brw-rw—- 1 root root 253, 65 Mar 3 14:50 /dev/dm-65
brw-rw—- 1 root root 253, 63 Mar 3 14:50 /dev/dm-63
brw-rw—- 1 root root 253, 67 Mar 3 14:50 /dev/dm-67
brw-rw—- 1 root root 253, 62 Mar 3 14:50 /dev/dm-62
brw-rw—- 1 root root 253, 69 Mar 3 14:58 /dev/dm-69
brw-rw—- 1 root root 253, 70 Mar 3 14:58 /dev/dm-70
brw-r—– 1 root disk 253, 46 Mar 4 11:34 /dev/dm-46
brw-r—– 1 root disk 253, 45 Mar 4 11:34 /dev/dm-45
brw-r—– 1 root disk 253, 42 Mar 4 11:34 /dev/dm-42
brw-r—– 1 root disk 253, 40 Mar 4 11:34 /dev/dm-40
brw-r—– 1 root disk 253, 36 Mar 4 11:34 /dev/dm-36
brw-r—– 1 root disk 253, 44 Mar 4 11:34 /dev/dm-44
brw-r—– 1 root disk 253, 39 Mar 4 11:34 /dev/dm-39
brw-r—– 1 root disk 253, 41 Mar 4 11:34 /dev/dm-41
brw-r—– 1 root disk 253, 43 Mar 4 11:34 /dev/dm-43
brw-r—– 1 root disk 253, 37 Mar 4 11:34 /dev/dm-37
brw-r—– 1 root disk 253, 12 Mar 4 11:34 /dev/dm-12
brw-r—– 1 root disk 253, 38 Mar 4 11:34 /dev/dm-38
brw-r—– 1 root disk 253, 35 Mar 4 11:34 /dev/dm-35
brw-r—– 1 root disk 253, 4 Mar 4 11:34 /dev/dm-4
brw-r—– 1 root disk 253, 10 Mar 4 11:34 /dev/dm-10
brw-r—– 1 root disk 253, 7 Mar 4 11:34 /dev/dm-7
brw-r—– 1 root disk 253, 3 Mar 4 11:34 /dev/dm-3
brw-r—– 1 root disk 253, 6 Mar 4 11:34 /dev/dm-6
brw-r—– 1 root disk 253, 1 Mar 4 11:34 /dev/dm-1
brw-r—– 1 root disk 253, 11 Mar 4 11:34 /dev/dm-11
brw-r—– 1 root disk 253, 0 Mar 4 11:34 /dev/dm-0
brw-r—– 1 root disk 253, 9 Mar 4 11:34 /dev/dm-9
brw-r—– 1 root disk 253, 5 Mar 4 11:34 /dev/dm-5
brw-r—– 1 root disk 253, 8 Mar 4 11:34 /dev/dm-8
brw-r—– 1 root disk 253, 2 Mar 4 11:34 /dev/dm-2
After these actions, oracleasm scandisks now runs in a few seconds.
(root) # time oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks…
Scanning system for ASM disks…real 0m0.480s
user 0m0.134s
sys 0m0.259s
Oracle in the field 