Oracle on OpenSolaris, part 1

This is an annotated log of a development installation of Oracle 11g on OpenSolaris x64.

I've made a metric ton of notes on various installation, troubleshooting, and development sessions over the years. Most of them have ended up in text files and email. While there's something to be said for letting this knowledge fade as software changes and techniques become outdated, rather than allowing it to accumulate in a mountain of disorganized cruft, there's also something to be said for delegating the entire issue to Google. :)

It's been a while since I set up this box so I'll start by reviewing the network configuration.

gregory@live:~$ dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
e1000g0 Ethernet up 1000 full e1000g0
e1000g1 Ethernet unknown 0 half e1000g1

gregory@live:~$ dladm show-ether
LINK PTYPE STATE AUTO SPEED-DUPLEX PAUSE
e1000g0 current up yes 1G-f bi
e1000g1 current unknown yes 0M-h bi

gregory@live:~$ dladm show-vnic
LINK OVER SPEED MACADDRESS MACADDRTYPE VID
vnic200 e1000g0 1000 2:8:20:d5:a6:43 random 0
vnic199 e1000g0 1000 2:8:20:ea:8a:76 random 0

Ok. I'll put a new vnic on e1000g0 for the new zone.

gregory@live:~$ pfexec dladm create-vnic -l e1000g0 vnic201

gregory@live:~$ dladm show-vnic
LINK OVER SPEED MACADDRESS MACADDRTYPE VID
vnic200 e1000g0 1000 2:8:20:d5:a6:43 random 0
vnic199 e1000g0 1000 2:8:20:ea:8a:76 random 0
vnic201 e1000g0 1000 2:8:20:8b:63:ab random 0

gregory@live:~$ dladm show-link
LINK CLASS MTU STATE OVER
e1000g0 phys 1500 up --
e1000g1 phys 1500 unknown --
vnic200 vnic 1500 up e1000g0
vnic199 vnic 1500 up e1000g0
vnic201 vnic 1500 up e1000g0

Zone setup

I'll copy an the configuration of an existing zone to save a bit of typing. I don't want to clone it as it's a live zone containing a bunch of Rails instances.

gregory@live:~$ zonecfg -z rorpub export

create -b
set zonepath=/zones/rorpub
set brand=ipkg
set autoboot=true
set limitpriv=default,dtrace_proc,dtrace_user
set ip-type=exclusive
add net
set physical=vnic200
end
add dataset
set name=rpool/delegates/rorpub
end

I'll just edit that instead.

I recall reading that zones and zfs are now supposed to play nice together, and that zonecfg should automagically create a zfs filesystem for the zonepath.

At least on this (snv_111b) release, the "automatic" part must be taken with a grain of salt. What actually happens is that the zonecfg install fails because it can't find the filesystem. Then zonecfg uninstall also fails because it can't find /ROOT under the filesystem it already failed to find, leaving you with no obvious way to proceed. This bug is probably fixed in newer builds. I have a boot environment patched to snv_134 but I can't reboot right now.

So I'll make sure the filesystem is created, mounted, and has root-only permissions.

gregory@live:~# pfexec zfs create rpool/zones/oracle
gregory@live:~# pfexec chmod 700 /zones/oracle

Hold on a minute. Typing pfexec for every. single. command. is bollocks. This is just as silly as Windows UAC spam; the extra step quickly becomes rote, making it even less likely that the person at the keyboard will show due care with privileged access. I may as well become root now and drop privileges when I'm done. I'd rather use su than pfexec bash, because my shell prompt is set up to remind me I'm root.

So if you see several pages of commands run as root in the future, don't assume it's because I'm too dense to use pfexec or sudo.

gregory@live:~# su -
Password:
Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008
root@live:~# cd /

root@live:/# zfs set mountpoint=/zones/oracle rpool/zones/oracle

I want to delegate some space to this zone and make sure it has an 8k blocksize. I plan to install Oracle here, and using an 8k Oracle blocksize on top of a 128k ZFS blocksize will cause ZFS to read 16 times as much data than it needs on every random read. This is Bad.

There's more tuning to be done but this step should be done before any datafiles are created.

Oh, good. Oracle now properly supports ZFS (PDF, white paper, May 2010).  The "tuning is evil" philosophy used within ZFS used to cause serious conflicts with Oracle's philosophy that everything in the OS ought to be tuned around the RDBMS.

root@live:/# zfs create -o mountpoint=none -o recordsize=8k rpool/delegates/oracle
root@live:/# zfs list -o name,recsize,mountpoint,volsize,zoned | grep oracle
rpool/delegates/oracle 8K none - off
rpool/zones/oracle 128K none - off

Zone creation

root@live:/# zonecfg -z oracle
zonecfg:oracle> create -b
zonecfg:oracle> set zonepath=/zones/oracle
zonecfg:oracle> set brand=ipkg
zonecfg:oracle> set autoboot=true
zonecfg:oracle> set limitpriv=default,dtrace_proc,dtrace_user
zonecfg:oracle> set ip-type=exclusive
zonecfg:oracle> add net
zonecfg:oracle:net> set physical=vnic201
zonecfg:oracle:net> end
zonecfg:oracle> add dataset
zonecfg:oracle:dataset> set name=rpool/delegates/oracle
zonecfg:oracle:dataset> end
zonecfg:oracle> commit
zonecfg:oracle> verify
zonecfg:oracle> exit

root@live:/# zoneadm -z oracle install
Publisher: Using opensolaris.org (http://pkg.opensolaris.org/dev/).
Image: Preparing at /zones/oracle/root.
Cache: Using /var/pkg/download.
Sanity Check: Looking for 'entire' incorporation.
Installing: Core System (output follows)
DOWNLOAD PKGS FILES XFER (MB)
Completed 20/20 3021/3021 42.55/42.55

PHASE ACTIONS
Install Phase 5747/5747
Installing: Additional Packages (output follows)
DOWNLOAD PKGS FILES XFER (MB)
Completed 37/37 5598/5598 32.52/32.52

PHASE ACTIONS
Install Phase 7329/7329

Note: Man pages can be obtained by installing SUNWman
Postinstall: Copying SMF seed repository ... done.
Postinstall: Applying workarounds.
Done: Installation completed in 101.537 seconds.

Next Steps: Boot the zone, then log into the zone console
(zlogin -C) to complete the configuration process

root@live:/# zoneadm -z oracle boot
root@live:/# zoneadm list -cv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
1 rorpub running /zones/rorpub ipkg excl
2 oracle running /zones/oracle ipkg excl
- barebones installed /zones/barebones ipkg excl

I should have scripted the system identification to avoid all the network config ESC-2-ing. We'll just pretend I did and move on.

System identification is completed.

oracle console login: root
Password:
Aug 17 20:59:31 oracle login: ROOT LOGIN /dev/console
Sun Microsystems Inc. SunOS 5.11 snv_111b November 2008


edit: Apparently I've been doing everything the hard way. I might have started with zonemgr instead. It didn't hurt to review the quirks of zonecfg in order to clarify why wrapping it in a script is a really good idea.

Now create a couple of filesystems under our delegate filesystem to hold the installation files and the installation.

root@oracle:~# zfs create -o mountpoint=/db rpool/delegates/oracle/db
root@oracle:~# zfs create -o mountpoint=/junk rpool/delegates/oracle/junk

root@oracle:/# zfs list -o name,used,avail,mountpoint,recsize
NAME USED AVAIL MOUNTPOINT RECSIZE
rpool 94.7G 819G /rpool 128K
rpool/delegates 3.52G 819G none 128K
rpool/delegates/oracle 2.36G 29.6G none 8K
rpool/delegates/oracle/db 19K 29.6G /db 8K
rpool/delegates/oracle/junk 2.36G 29.6G /junk 8K
rpool/zones 2.48G 819G none 128K
rpool/zones/oracle 248M 819G /zones/oracle 128K
rpool/zones/oracle/ROOT 248M 819G legacy 128K
rpool/zones/oracle/ROOT/zbe 248M 819G legacy 128K

There's the filesystem with the 8k blocksize, as ordered. Time to go back to the global zone.

root@oracle:~# logout
oracle console login: ~.
Password: ~.
[Connection to zone 'oracle' console closed]

root@live:/# zfs set quota=32g rpool/delegates/oracle

root@live:/# zfs list | grep oracle
rpool/delegates/oracle 38K 32.0G 19K none
rpool/delegates/oracle/db 19K 32.0G 19K /db
rpool/zones/oracle 246M 821G 22K /zones/oracle
rpool/zones/oracle/ROOT 246M 821G 19K legacy
rpool/zones/oracle/ROOT/zbe 246M 821G 246M legacy

And dump the oracle installation files into the oracle zone from the zip files in the global zone. They were retrieved out here because the global zone is currently the only one with a window manager and a browser. If I wanted to retrieve them from the command line, I'd still have to start a download from a browser where I could accept the license agreement and then transfer a cookie file to a command line tool, which is kind of a pain.

Mounting the Downloads directory with lofs probably would have worked too, but this is easier.

root@live:/export/home/gregory/Downloads# unzip solaris.x64_11gR2_database_1of2.zip -d /zones/oracle/root/junk
root@live:/export/home/gregory/Downloads# unzip solaris.x64_11gR2_database_2of2.zip -d /zones/oracle/root/junk

I'll have to sort out a DNS issue and install additional packages before proceeding. That'll be in the next post.