7123 - After the great LJ Crash of '05
Jan. 15th, 2005 08:53 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
scottobear
And, we're back.
Thank you, Jambi.
Obligatory Star Wars ref:
TOSHI STATION, Tatooine (Imperial Press) — Local wizard Ben Kenobi yesterday announced he had detected a "great disturbance" that could possibly affect the entire universe.
It was as if millions of voices suddenly cried out in terror and were suddenly silenced," Kenobi said. "I fear something terrible has happened."
I've had three people (jokingly) accuse me of knocking down LJ. It wasn't me!!
Our data center (Internap) lost all its power, including redundant backup power, for some unknown reason. (unknown to me, at least) We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.
Update #1, 7:35 pm PST: we're up on 'dirty' power for now (it works, but it's unreliable), and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.
Update #2, 10:11 pm: So far so good. Things are checking out, but we're being paranoid. A few annoying issues, but nothing that's not fixable. We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again. In the past we've always trusted Internap's insanely redundant power and UPS systems, but now that this has happened to us twice, we realize the first time wasn't a total freak coincidence. C'est la vie.
Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn't commit to disk when asked, so InnoDB's durability wasn't so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We'll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we're going to take a bit of a break while the binlogs replay... Again, our apologies for the downtime. This has definitely been an experience.
Update #4: 9:12 am: We're back at it. We'll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.
Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh* We're still at it. We'll be doing a full write-up when we're done, including what we'll be changing to make sure verify/restore operations don't take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn't fast enough. And also that some of our machine's storage subsystems lie. Anyway, we're still at it... it's long because we're making sure to back up even the partially out of sync databases that we're restoring, just in case we encounter any problems down the road with the restored copy, we'll be able to merge them. And unfortunately backups and networks are too slow.
slashdot reported on it, and brad was kind enough to supply some tech details
The Art of William Schmidt ... morbid and fun. [via]
Oh I agree...
"This is film feminism in the 21st century: Take a cruel, bloodthirsty, homicidal character, but instead of making the character a man, make it a woman. Then pretend you've created an exemplar of female strength. I liked this fake feminism better when it was called misogyny."
-Mick LaSalle San Francisco Chroinicle (referring to Electra.. but it could be any of a big bundle of flicks lately.)
Pentagon reveals rejected chemical weapons
THE Pentagon considered developing a host of non-lethal chemical weapons that would disrupt discipline and morale among enemy troops, newly declassified documents reveal.
Most bizarre among the plans was one for the development of an "aphrodisiac" chemical weapon that would make enemy soldiers sexually irresistible to each other. Provoking widespread homosexual behavior among troops would cause a "distasteful but completely non-lethal" blow to morale, the proposal says.
Other ideas included chemical weapons that attract swarms of enraged wasps or angry rats to troop positions, making them uninhabitable. Another was to develop a chemical that caused "severe and lasting halitosis", making it easy to identify guerrillas trying to blend in with civilians. There was also the idea of making troops' skin unbearably sensitive to sunlight.
The proposals, from the US Air Force Wright Laboratory in Dayton, Ohio, date from 1994. The lab sought Pentagon funding for research into what it called "harassing, annoying and 'bad guy'-identifying chemicals". The plans have been posted online by the Sunshine Project, an organization that exposes research into chemical and biological weapons.
Spokesman Edward Hammond says it was not known if the proposed $7.5 million, six-year research plan was ever pursued.
Man... I wish.
How'd you win the war, grampa? Oh, it was easy, once we had the enemy buggering themselves while angry wasps and rats closed in on them, causing them to die severely sunburned and uttering their last words on really stinky breath.
Brief sojourn out with Danny today... sushi, a little mage knight (at the library, because twistyland had a yu-gi-oh tourney going on), bad traffic and pleasant conversation. we listened to Doug's old cassette.. Danny's been listening for about 2 weeks now. I have to find Doug's number, so Danny can call him up to say howdy.
I loaned him the pirate game and a few figures for him to teach his sci-fi club how to play next Friday. We had a nice time.
Current music's Lyrics - Now You're A Man - DVDA
Hey
What makes a man?
Is it the power in his hands?
Is it his quest for glory?
Give it all you've got,
to fight to the top
So we can know your story
Now you're a man
A man, man man
Now you're a man
A manny manny man
A man man man
You are now a man
You're a man
Now you're a man
Live it, Live it
What make a man?
Is it the woman in his arms?
Just cause she has big titties?
Or is it the way he fights every day?
....No, it's probably the titties
Now you're a man
A man, man man
Now you're a man-man
Man-man man-man man
Now you're a man
M-A-N, man
Man, man-man man
Now you're a man
1 year ago - DVD cereal, Mars, cluck cluck cluck
2 years ago - smokies, palm synch, empenadas, Malawi Vampires
3 years ago - Not currently available - 6 entries [update - nice walk, moody, gave sappho a lj code, midnight pooper, newt patrols]
4 years ago - pictures of me with
![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif){kind=small}
tarpo at No-nose maldanado's, company on the rocks, ramble on humanity, temptation island, super-soldiers, link to 100 scotto-things
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
And Me, Without a User Pic for this Account!
Date: 2005-01-16 03:53 am (UTC)Well, I guess it's worth waiting thirty hours or more for the site to come back, if that wait guarantees that three and a half years of posts are still there when it's all over.
One of the comments on that /. post said that Parsident Boosh just named the head of Internap to some council that's going to advise him on technological infrastructure. Oh, dear.
I wonder how much it would cost to create a mirror of a site as big as LJ? Someday, Seattle is bound to get wiped out by a volcano. I'd feel more secure if the servers were in one of those geologically stable salt domes in Missouri.
But, wow, some of those /. users are <i>snotty!</i>
Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 03:56 am (UTC)I'd like to think that a lot of LJ is backed up on web.archive.org
look! some of your stuff is there !
http://web.archive.org/web/*/http://livejournal.com/users/flying_blind/
Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:30 am (UTC)Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:32 am (UTC)http://web.archive.org/web/*/http://livejournal.com/~flying_blind/
http://web.archive.org/web/*/http://flying_blind.livejournal.com/
Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:51 am (UTC)Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:52 am (UTC)does it feel odd to have the library of congress remembering your journal?
Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:55 am (UTC)Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:58 am (UTC)Re: And Me, Without a User Pic for this Account!
Date: 2005-01-16 04:59 am (UTC)no subject
Date: 2005-01-16 12:02 pm (UTC)Lemme know.. if not we may be trying it again later this or next week.. I'll keep in touch
no subject
Date: 2005-01-16 12:03 pm (UTC)Next week would probably work better!
no subject
Date: 2005-01-16 04:58 pm (UTC)I enjoyed reading the comments over at Sdot :0) Back when I worked at an ISP, I can think of only one situation over at Internap that gave us pause. My boss would pull on his hair when he was really bent..and there was quite a bit of hairpulling that day. Don't ask me what that situation was though..my mind is prone to dumping out details from places I couldn't wait to get away from. :0)
no subject
Date: 2005-01-16 05:02 pm (UTC)Was she intrigued or did she yawn at the mention? Keep in mind, that I'm beginning to come to the thought that we're nuts for keeping these things, not the other way around. So neither answer will cause her to look...un-shiny. ;) If she wanted to shoot us or beat us up or something, that would be another story.
no subject
Date: 2005-01-16 05:43 pm (UTC)no subject
Date: 2005-01-16 05:55 pm (UTC)no subject
Date: 2005-01-17 01:04 am (UTC)She thinks it's interesting, but lj isn't her thing... personal journalling has more appeal to her. Since she doesn't have internet at home, she doesn't get much of a chance to read my journal... but she doesn't mind, because we talk about things over the phone and in person.. no need to play computer catch-up.
I don't think that she realises my devotion to LJ yet... she was rather surprised that I have about five years of daily reference stored online.
She doesn't want to shoot or beat up anyone. ;)
no subject
Date: 2005-01-17 01:07 am (UTC)8 hours of being lumps in bed, searching stuff on the laptop and watching tv with italian food right at arm's reach, with my two favorite local life-forms snugggled up close.
Who could ask for more?
Feel free to talk about the hair-puilling guy.. was he going bald??
no subject
Date: 2005-01-17 02:11 pm (UTC)Tell Mel not to get internet! It's sort of like being de-flowered..once you do it, you can't ever go back! :P
"I don't think that she realises my devotion to LJ yet... she was rather surprised that I have about five years of daily reference stored online."
If she moves in...do you think we'll lose you? :/ I see that happen all the time..it's odd sometimes, to think that we'll all lose each other at some point.
no subject
Date: 2005-01-17 02:12 pm (UTC)good mel. :0)
no subject
Date: 2005-01-17 02:16 pm (UTC)of course you don't know the answer to that..I was just talking about loud. :0)
no subject
Date: 2005-01-17 02:42 pm (UTC)no subject
Date: 2005-01-17 02:43 pm (UTC)no subject
Date: 2005-01-17 02:55 pm (UTC)Who knows what the future will bring? I don't like the idea of losing the sense of community and fondness for folks here. I suspect I'll lose LJ before LJ "loses me".
If she and I cohabitate, there will be internet, and it will be ubiquitous. :)
no subject
Date: 2005-01-17 03:33 pm (UTC)Ha...where do I start with the hair pulling guy. What a character. I thought I hated him for a long time, because he stole about $1000 of my commissions. One of the few people I've ever felt that way about. I left two weeks later. Over it now - but he has a prominent spot in my brain. :P
Hmm..I seem to remember it was thinning in some places, but not in front. He was only 38 at the time? Looked much older because he had a thick beard and mustache. He would pull on his hair, and it would stick up all over. We would all try not to make a peep. If he did it directly in front of me while talking to me, I would try soooo hard to keep a straight, professional face.
He was the president - and Oh my, Uber-intelligent on a tech level. We always had smart techie people - but this guy took the cake. Also, very quirky, *STRESSED-OUT*, and his social skills were a mess.
His photo shows up weekly in our alternative newspaper. The first time I saw it (after leaving that place), I gasped - said OMG and slammed the paper shut. Opened it - and slowly peeked again. :D If I remember correctly, one of my old friends there said he purposely pulled on his hair to make it stick up, before having the photo taken. I have one in my car..so I should go look.
The boss hated to be interrupted, so when you went to his office to talk to him, he would say "WHAT?!?!" in this voice that just *crushed* people. They looked so dejected afterward. Lower your voice to it's deepest level, make it sound angry and draw it out to "WHATTT?!?" and you've got it. A good friend of mine at work found the boss man's driver's license...xeroxed the photo and blew it up to make a sort-of mask. He would hold that mask up to my window when I was on important calls to clients, and say "WHAT?!" and I would try so hard not to laugh in the client's ear.
To this day, if I run into one of the old peeps, they can say that word and we laugh. I also still have a copy of that xeroxed mask & probably always will. I can't really explain it. ;)
I can say so much about him...he even had some issues that came up with the big gov. guys (don't want to print out which guys here), and that was a mess. He would wear the same clothes over and over each day...once I was the only one that noticed he didn't wash a pair of his pants for a week.
I just googled his full name, to see what he's been up to. Very interesting...
"WHAT?!"
Date: 2005-01-17 04:00 pm (UTC)mmm... I'm way overdue for some good thai food. Or anything with curry.
That office sounds like quite a collection of characters! I think that I would have trouble not saying anything to hair-tugger man.
It's so strange how many techies can be socially confused. I guess that a lot of people allocate "skill units" in only a few places, instead of distributing them more evenly.
One of my employees suffered from "same clothes syndrome" too.. back when I was managing a movie theater. His socks actually smelled so bad that we had to send him home to chage, with orders to wash both himself and his clothes with hot water and soap before returning...another employee was a clarion programmer who was just filthy... his thighs were smeared with filth rubbed off from his forarms whe he rested 'em... e had to send him along to his house to bather and launder, too.
The dirty pants man also kept uncovered food in the back seat of his car... in the sun. Stuff made with mayo, like tuna cassarole... he'd eat off of it for three or four days until it was all gone.. never refridgerated.
I still shudder when I think of it. Bleah.
no subject
Date: 2005-01-17 05:11 pm (UTC)no subject
Date: 2005-01-17 05:41 pm (UTC)I agree... I don't get jealous unless flirting is going on, rather than just convos. (although sometimes folks can get confused as to one or the other.) I think that you and I understand one another reasonably well, all told. It seems that most girls can get along better with boys than with thier own gender... a different dynamic, I suspect.
I'm glad we can babble!