Hello Wednesday. Today we intend to be:
Several people have reported problems with PHP - this is because the new configuration no longer supports register_globals because it can allow whacking great security flaws if you're sloppy with it. It's recommended that you use the superglobals instead.
Several people have also been shy about mailing problems or questions to us - don't be! While we're polishing stuff off, this is the best time to tell us, whatever it is. We now recommend you mail us through the admin list.
Outgoing mail seems to have been blocked for some reason - what this means is that if you try and send mail using SquirrelMail or Pine, it probably won't have been delivered yet. However, Dan's just kicked it and it seems to be going okay now, but it'll take a while to clear.
Okay, I'm sending out passwords. NOTE: Some people may be texting my old mobile number. Don't bother, I won't get them. However, there's a recorded message on my old number that gives you my new one. (I changed my number about a month ago. If you have a number older than that, it's wrong)
Mail is now flowing happily into butter. I've turned off redirects, so have a last check of your external box in an hour or so and it shouldn't get any more. However, it'll take a while for all the queued mail to reach butter - up to a day for everything that's queued up at various parts of the internet to filter through. Anything sent now should go straight through, though.
Dan is working on getting POP3 and IMAP access working happily.
Current status: Box is up, ssh is up, Apache is up, Squirrelmail is up (which means IMAP should be up too) but POP3 is down - we'll fix that.
I'm now going to start texting out passwords. If you haven't got yours within the next hour, tell me, supplying your username and a mobile number. Once I've conferred with the others I'll also turn off mail redirection (though some mail will be going straight to butter already)
NOTE: If you haven't done so yet, please read the blog entry below this one - it's important.
After a day when I thought I'd never be allowed out of my car, things are looking up! Thanks to Mr Cronin and a few very generous donations, we have a new butter, and it rocks. It's a 2.4GHz Pentium 4 (HyperThreading, noch) with a gig of RAM in a bright orange 1U rackmount case with serious ventilation - more than capable of handling twice J-Colo's current load. Dan and Richard spent the evening giving it a new Linux install and moving our data over. Everything is currently on track for putting the machine back on the net at around 10 AM.
However, things you should know:
Please keep an eye on this blog, just in case; we're not quite done yet...
Butter is now in the back of my car and the three of us (Dan, Richard and I) will be up most of tonight rebuilding it. What we're hoping to do is replace it with an entirely new machine - we're still finalising that. Once that's done, Richard should be taking it back to Mailbox tomorrow morning, and it should be on the net again by 10AM. That's the plan, anyway.
Also want to say a big THANK YOU to everyone who's been in touch with offers of help or just sympathy - it means a lot to us. We're hoping that once this is over we'll have a much stronger J-Colo for everyone. (A big shout, in particular, to James Cronin of The Flirble Organisation, who stayed up well past his bedtime helping us sort out the mail situation.)
At time of writing, butter.j-colo.net (the main J-Colo box) is offline for maintenance. Earlier today a major security breach was detected that seriously compromised the machine. (Basically, we've been hacked.) In order to resume service, we need to do some serious offline maintenance work.
As far as we can tell, all our data is fine, plus we have backups of all mail and websites. We are hoping to have butter operational again, to some degree, around noon on Tuesday (this may change depending on circumstances). However, service will probably be spotty for the next week - we hope to have everything back to normal soon after that. Our top priority is giving everyone access to their email. Websites may have to wait for a little while; however, we may be able to arrange emergency hosting for vital sites.
Please be assured that we're doing everything we can to fix this as quickly as possible - after all, the admins are among those who have vital services running on butter, so it's just as important to us as it is to you.
As of now, all your J-colo-hosted mail is being collected on cheese.j-colo.net, which is the secondary mail exchanger. Any mail it has will be delivered to butter as normal when it comes back up. However, it may not have all the mail - some may have arrived on butter before it went down, and some may have bounced back to the sender. (However, none should have been lost in the ether)
If you need to get at email that's currently being collected by cheese, we can set it to redirect to an external email address during the outage. Just let me know.
Firstly, we're still trying to work out how this happened. If you've ever given out your J-Colo login details, if you're running any interesting extra programs on your J-colo account, if you're running complex CGI/PHP code that you downloaded from somewhere, please tell us immediately.
Secondly, the admins are trying to fix this while juggling their day jobs and are, as such, incredibly short of spare time. If you are able to get between North London and Fulham (where J-Colo is hosted) during working hours, we want to hear from you. Also, we may need help from anyone with a decent amount of UNIX/Linux experience.
(Also, the maintenance on butter is basically going to include a complete rebuild - so if you already know of something on butter that hasn't been working like it should have been, and we may not know about it, now's a good time to alert us. We hope that, once this is over, J-colo will be running much more smoothly than it has been in a while.)
Regular news updates will be posted to the blog at this address: http://status.j-colo.net/
Best way to get in touch is via me: yoz@yoz.com