?

Log in

No account? Create an account

Previous Entry | Next Entry

another server upgrade - and FAIL

It's been a while. (a year and a half) And I've got two servers to upgrade. It's 9:15a now (Tuesday, for the record), and I've been at this for about 5 hours. Pretty deep in it.

upgrading to FC9 and FC8 (two machines, and one I couldn't trick to make the jump all the way from FC6 to FC9... the other I could get FC7 to FC9 with some fiddling). upgrading kernels to 2.6.26.5 from 2.6.18 and 2.6.17.

I want to remember to get kernels from:
http://update.pureserver.info/linux-kernel/v2.6/

If I accidentally killed my old config, I can get one at:
/proc/config.gz

(this goes in the downloaded src (/usr/local/linux/src/linux/)'s directory as .config

make oldconfig
make all
make modules_install
make install

(here I have some worries:

WARNING: No module ehci-hcd found for kernel 2.6.26.5, continuing anyway
WARNING: No module ohci-hcd found for kernel 2.6.26.5, continuing anyway
WARNING: No module uhci-hcd found for kernel 2.6.26.5, continuing anyway

)

Meanwhile: here's a kicker: imap _virtual_ accounts could access their mail, but actual unix accounts couldn't. took a bit to find that. Woof. But I think my system's good now! WHOO! WHY COULDN'T I RECORD WHAT I DID? OF COURSE IT'S HAPPENING AGAIN. At least it remembered my MTA properly this time. Okay, this is why: pam_auth_unix is failing. Now, what to do about that?

And yes, you need to recompile eAccelerator, and YES, that's why httpd isn't starting.

And ... new kernel failed on the box going to FC8, but FC8 appears happy enough anyway?

Random nifty detail--extracting single files out of an rpm:


rpm -qlp rpmfile | grep whatyourelookingfor

this tells you full path of what you're looking for....

rpm2cpio rpmfile | cpio -ivd ./ullpathandfile

Note the ./ -- that puts it relative to the current directory instead of (possibly?) overwriting your system.


Note to self: NEVER EVER _EVER_ EV4R!!11! screw up your system _while in the middle of an upgrade_. Seriously. Wow. Still hoping I can recover from some accidental problems with libc.so.6 that got much worse very quickly. I was actually probably just forgetting to do the ... erm.

it's 12:23am on Wednesday now. I'm still going. Mistakes were made.

8:47am. I got four hours of sleep, I think, waiting for /home/ to untar. It probably didn't take 4 hours (sheesh, I hope not more than 1, maybe 1.5, but I really needed it--I'd been up since Monday).

Things are looking up, despite some lost configurations, and the fact that fedora dropped support for courier-imapd/courier-authlib a long time ago. Have compiled from source and am making progress with them. Web stuffs is up, but email's not quite right yet. And then the big-scaries, I lost all my configs for tomcat for a few legacy sites. ((also lost my named.conf, but was able to reconstruct from my secondary)).

Hey, finally learned about ld.so.conf and Fedora services (well, how to build something for init.d that chkconfig recognizes).

Mind you, I didn't really want to learn this stuff. Really. Ooh. Dizzyhungry. Hmm.

Also learned about postgrey -- grey listing support for postfix. YAY! :) But oh boy, don't forget about spamassassin. EEK. Let's get that set up quick... hopefully quick... (it's 9:50a now...)

okay... 11a. postgrey is installed, spamassassin is on hold. did some db recovery (hooboy!)

11:55a -- oh, and bind needs to accept requests from non-local servers if you're going to get anywhere with doing DNS. struggling with tomcat.

12:11p -- WHOO! tomcat is up. Next: postgresql (the problem here being it's an _old_ version of postgresql, not supported anymore.)

12:51p -- postgresql is up, tomcat is talking to it, hells yeah.

Any guesses as to what I might have forgotten (besides spamassassin?)

For future reference, this looks like the be-all end-all guide to postfix.

Comments

( 13 comments — Leave a comment )
xjenavivex
Oct. 8th, 2008 08:09 pm (UTC)
My head really hurts now.
kaolinfire
Oct. 8th, 2008 08:11 pm (UTC)
probably not as much as mine. ;)
xjenavivex
Oct. 8th, 2008 08:14 pm (UTC)
I bet!
realthog
Oct. 8th, 2008 08:18 pm (UTC)

You know, I understood about three words of all that. Unfortunately, they weren't sequential.
2nd_madison
Oct. 8th, 2008 09:33 pm (UTC)
What realthog typed.

Gah.
(Deleted comment)
kaolinfire
Oct. 8th, 2008 10:29 pm (UTC)
There was some in there, sort of, but you might have glazed over it. I certainly feel like I did ;)
blueblindbee
Oct. 8th, 2008 10:13 pm (UTC)
Despite the unfortunateness, these are really fun to read. I don't understand half of it, but it's interesting.

so is it all fixed? :)
kaolinfire
Oct. 8th, 2008 10:14 pm (UTC)
maybe? :)
girlspell
Oct. 8th, 2008 10:43 pm (UTC)
Hm...In the middle of all this, don't you ever get into a "panic"mode? You know how it is. Something is running. You're pretty sure you know what you're doing and then weird things pop up and you break into a cold sweat?

Funnty things about computers, servers, etc. All you have to do is hit them, and then they kind of fix themselves.

I work at a university. We're our own server for the internet. They have nasty, beastly looking drivers stored away from the public. Miles of cable too. Lightning fast internet service. Fastest I've ever seen. Also everything is upgraded over and over. In the computer field, new languges pop up and constanly replaces another language. As soon as you learn one thing, time to replace it.
kaolinfire
Oct. 8th, 2008 10:46 pm (UTC)
Cold sweat hit as soon as I realized I'd trashed the OS, though I still had some hope of patching it... for a little while. That eroded over the span of about 8 hours, along with my strength for emotions. ;)

If the server had been local, this would not have been a problem. I wish that were the case. Remotely upgrading a server OS scares me every time, though this is the first time I've bunged it (out of around 8 times).
zotsf
Oct. 9th, 2008 05:07 am (UTC)
ack...you probably tinkered a bit more than you needed to ;) You are such a problem solver :)
kaolinfire
Oct. 9th, 2008 05:18 am (UTC)
well, first yum broke, then I got distracted, then I was moving libraries around trying to fix yum, and obliterated libc through a symlink... and libc is the spirit of life.

And it was definitely downhill from there (I forget what forced a reboot, but I swear that wasn't intentional... or was it? Oh, yes, because breaking libc broke things so soundly the only thing I could do was try to recover: when you can't ls, rm, cp, ... you're all kinds of screwed).
(Deleted comment)
kaolinfire
Oct. 10th, 2008 11:18 pm (UTC)
Thanks :)
( 13 comments — Leave a comment )

Latest Month

February 2016
S M T W T F S
 123456
78910111213
14151617181920
21222324252627
2829     

Tags

Powered by LiveJournal.com
Designed by chasethestars