yesterday’s upgrade

For those that are used to doing Solaris upgrades, you may have noticed that before when you did an upgrade instead of a fresh install, the config info you enter for hostname, name service, timezone and stuff was all ignored and it would leave it the way it was before. One of the new things with Solaris 10 is that it actually updates the system with the info you type into during the install. I can see why that would be useful, but the fact that it ignored it in the past made it easy just to breeze past those questions entering random stuff quickly and then hit upgrade knowing that the system would leave it the way it was. So you can’t do that anymore. Bummer. That also means cleaning up resolv.conf and re-enabling IPv6 because I answered some stuff wrong. Oops.

Besides that for some reason my startup for running /usr/local/sbin/named instead of /usr/sbin/in.named disappeared during this upgrade. Not really sure how, but I had to put that back too.

The terminal server is still randomly crashing, but now it is able to stay up for a few hours at a time. So still need to figure out what is going on there, but it is staying up long enough that it’s actually useful. Yay.

Today will try to upgrade the last remaining build 69 server to build 72. That’s the one that Rick uses as a router and died and set into motion all this upgrading.

Emily went off to school today without complaint. This morning they have a field trip to a candy factory, so should be fun for her.

Computer headaches

Last night just before going to bed, Rick tells me that he can’t access the net. My servers are hosted there so he routes out through one of my servers. Unfortunately the terminal server flaked out in the middle of December where it would stay up for five minutes and then crash. Needless to say, it’s highly aggravating to have to relogin every five minutes to the console to debug a system problem. That makes remote administration challenging when something really goes wrong. I also have remote power management so I can power cycle a wedged server, but that still doesn’t help with more serious problems that prevent the boot process from completing. And on top of all that the server is running an old beta of Solaris 10 (build 69) that has a few too many bugs. I had wanted to upgrade to the more stable build 72, but the terminal server issue fouled that up. Power cycled the server and was able to ping it for a while, but it never fully came up. Since Rick was at work there wasn’t much possible to do at that point, so I headed to bed after making plans with Rick to debug it when he got home.

Now normally this is one of those situations that would have me up for a couple of hours running through what could have gone wrong, and possible ways to fix those scenarios, and things to try to get the terminal server working again, etc. But amazingly enough I went to sleep almost immediately after turning off the light!

So the next morning(my time)/evening(his time) Rick helps me get the terminal server up at least part way. We were able to get it to the point where it would average 30 minutes of uptime between crashes. That ain’t pretty, but it at least gives a decent amount of time to get in and twiddle with things.

So when I finally get it up and running and connect up to the sick server, I reboot and get told there’s a problem with / and I need to run fsck -o f. Unfortunately it wedges before I can login, and even booting up single user it wedges before the single user shell comes up. Normally that’d be the boot where you do a “boot net -s”, but it turns out the sick server is the only netinstall server set up on the network! (I need to make sure at least two minimal netinstall servers are available at any one time.) Fortunately I still had the Solaris 10 build 10 ISOs on disk, but it takes a while to get from ISO to netinstall server.

Once I did, I was able to get in, fsck repair everything, unmirror root, reboot, then remirror root and swap and get back to business. Pretty straightforward at that point. But annoying that it couldn’t even handle getting into single user without resorting to an external boot device. I suspect the disks might be a problem that caused the filesystem corruption, because it has occasionally had problems before. It has IBM disks, and the IBM disks made before the Hitachi buyout have some issues. They’ve never died outright, so hard to really point definitively to a problem. I probably should upgrade them to a more reliable brand considering the frequency of problems.

So at that point I decide it’s probably a good idea to get going on upgrading my two remaining build 69 servers to build 72 to get better stability. I figured with the terminal server staying up a half hour at a time, I’d be able to get through starting the install and the rest would be pretty much on auto-pilot so it wouldn’t matter if the term server died. So I decide to upgrade the one that is not the router/dhcp/netinstall server and do the other less critical one first.

And oddly enough things went fairly smoothly and the terminal server managed not to crash at all during the upgrade. I’d like to think that the upgrade might have had something to do with it, but the terminal server was crashing before when it only had power and a connection to another Solaris 9 server, no network, no other terminal ports connected. So it’s just one of those annoying things where it doesn’t work and then it suddenly works and you have no idea what the hell caused it. As of now the terminal server has been up for 5 hours and 23 minutes which is a pretty amazing accomplishment considering it was hard pressed to do 5 minutes straight a month ago. So one upgrade down, will go for the other upgrade tomorrow.

One of the problems with upgrading from Solaris 10 build 69 to build 72 is that the dhcp-server service gets fudged up somehow and will give a dependency cycle error on boot up. The solution:

svccfg delete dhcp-server
svccfg import /var/svc/manifest/network/dhcp-server.xml
init 6

Should be just fine after that. You need to reboot, because that problem prevents other services in the multi-user-server subset from starting.

Hopefully when the final release of Solaris 10 comes out (soon I hope), these rough edges will be smoothed out.

I love egg

Eggs!
Get your eggs, here!
Fresh and white eggs are here!

Wiggle jiggle, yellow middle
That’s the best of what you are
(I love you egg)
White and tender, surround the center
Cozy, sitting in the crackling shell.

Vitamins and minerals in you
Oodles of the proteins, too.
(oodle doodle!).
Popular and perfect and so complete in every way
(I love you egg, egg!)

Come into my tummy, oh so very yummy!
Crack! Crack! Crack!
Chip a chip away your shell and come to me!
(Get your eggs!)

I love you (fresh egg!)
I love you (white eggs!)
Really really love you so (egg fresh fresh egg!)
Eggs, I really love you like the sky above (eggs are the best!).

I love you (fresh egg!)
I love you (white eggs!)
Really really love you so (egg fresh fresh egg!)
Three hundred and sixty five days,
I really love you so!
I really love you so!

Mmm, yummy!

Emily starts to enjoy school

Emily started going to preschool just over a week ago. For the most part she’s been pretty upset about it, and not really eating much or taking nap time. Today for the first day she actually took a loooong nap and ate half a bowl of noodles for lunch. So we think she is starting to at least get comfortable with it.

Maggie took some of my cookies and brownies to her shop and some of her customers asked if I would sell them. 🙂

Bookworm

Went down to Silex Books today. I signed up for a vip card back in October and they sent me email about it way back then to come pick it up, but I hadn’t gotten around to it. Silex used to be the only place in Taipei you could find a decent selection of Science Fiction books in English. Now we have Page One in the tallest building in the world which has a bit better Science Fiction selection, but they are a lot more expensive. Silex’s main specialty is children to young adult books in English, but they also have a pretty nice Science Fiction section upstairs, and their prices are much more reasonable compared to Page One. Anyways, it turns out they are having a bit of a clearance sale, so I was able to pick up a bunch of books for TWD99 and TWD149 each (that’s about $3 and $5 in US dollars). So, um, now I’m even more behind in my reading.

On the way back the bus driver had the AC on. That’s one thing I still can’t understand in Taiwan. It’s 15C out, so why the hell do you need the AC on? I wish air conditioners would come with an automatic cutoff when it is below 20C, cause there’s no reason to have it on anymore.

Third time’s the charm for brownies

Tonight I made brownies for the third time and finally got them to come out well in the new oven. This time I used both elements for the first twenty minutes, then just the bottom for the last ten minutes. And while I had the ball rolling, I whipped up a batch of chocolate chip cookies from the chocolate chips I got last week at Jason’s. The first batch was the opposite of the first brownie batch, singed on the bottom, fine on top. If it’s not one thing it is another. Fortunately with the size of the oven it worked out to four batches, so I was able to get the next three to come out fine by doing both elements for 5 minutes, then just top for 4 minutes.

The recipes I used are from Hershey’s Kitchen. They have a lot of great recipes on their website in addition to the ones they print on the label. The brownies I made are HERSHEY’S Best Brownies and for cookies I used the HERSHEY’S Classic Chocolate Chip Cookies recipe. For all their recipes, go to http://www.hersheys.com/recipes/. There’s a lot of recipes in each category but you can get right to the best ones by clicking on the “Top 10” link for each category. By the way, I made both WITHOUT NUTS. I like nuts just fine, but for some reason I don’t care for them in cookies and brownies.

scrungew00t goes home, Emily gets upset

Today was scrungew00t‘s last day in Taiwan, just dropped him off at the bus to the airport. He ended up with so much crud that he didn’t have enough luggage space, so I lent him one of my suitcases. Even with that extra one it was pretty tight. That actually works out fine for me, because my suitcases are usually nearly empty on the way to the US but I bring back two full suitcases of crud. So effectively I’m getting him to take the suitcase back for me.

Unfortunately when Emily saw me walk out the door carrying my suitcase, she burst into tears saying “Daddy, don’t leave!” No matter how much I explained to her, she thought I was going on a trip somewhere. So when I came back 20 minutes later, she was surprised and said “Daddy, you’re back!” She’s been going to daycare for the first time this week, so she’s kinda stressed out about a lot of things and she was in an especially bad mood today. Poor baby.

cold shower on a cold day

As if it is not bad enough that the weather is cold and miserable, the battery in the water heater up and died just after I started showering. And it had beautifully cruel timing too, cutting out just after I’d gotten all wet, but before I had shampooed and soaped up. The only thing worse would have been if it had cut out while I shampooed and before rinsing. So I had to dry off, get dressed again, find a battery, go outside on the back balcony to change it and poke it until it would work again, then back to take a shower again. Grr.

Americans are probably used to the kind of water heater where you have a big tank in a closet or garage somewhere, and the water heater keeps a nice 30 gallons or so of nice hot water on hand for whenever you need it. These usually have an always-on pilot light and do not need electricity to operate. In Taiwan we have small tank-less heaters that are also gas powered, but do not have an always-on pilot light. They are a small box about 2.5 feet high, 1.5 feet wide and 8 inches deep. You need to put in a D size battery to fire the pilot light and control the logic. When the battery fails, the heater stops working. And since there is no tank to hold the nice hot water, it is a very abrupt and sudden failure.

And as an update to my last entry, the other factor it is so miserable when it is cold here is that my dear wife insists on opening windows for fresh air even when it is cold. 🙁