If you have any remaining issues - even if you have already mailed us with them - please log them in our new Helpdesk -
http://www.j-colo.net/otrs/customer.pl - you will need to sign on for an account.
Please give as much detail as possible in the helpdesk ticket.
All the old virtual hosts have been recreated, but not all of your
website contents have been moved over yet. If you try going to your
website address, it's likely to be working already (assuming you host
DNS with us too). However, it's possible that we haven't moved your
stuff over either because there's too much of it to move quickly or
because some of your data has been corrupted. We'll be doing our best
to notify those who have lost data, but if you have your own backups
of your website then you can speed up the process by connecting to
marmite.j-colo.net and uploading data yourself.
In other news, MySQL is fully back, and it appears there has been
no data loss.
Mail should be working!
If you point your mail client at marmite.j-colo.net, you should be picking up new mail. Your old mail is not lost, it is backed up and we will (try to) restore it for you tomorrow.
If you have mail filters and forwards they may not work. These will also be done tomorrow.
You can use the webmail again by going to https://mail.j-colo.net - this is secure although we don't have a certificate yet so you should get a security warning when you log-in. This is much more secure than the previous mail server.
Websites will take longer, we may start on them tomorrow.
Our main problem over the next few days is going to be copying over large amounts of data from a very slow ADSL line to our super fast new server.
Butter is dead and is not coming back, so we are migrating everyone over to Marmite which lives in the US.
This means we need to restore you on a one by one basis. Please email myself, ted or Richard (contacts in previous posts) to get in the queue.
We can get you running with e-mail fairly quickly, web will take a few more days and finally getting you old data off butter will take a little longer.
Please let us know your priority and urgency so we can plan this better.
Mail is being moved over right now but things are slow as the pipe we're using to transfer over is really narrow. If anyone can offer us a high-bandwidth upstream line (better than ADSL) for the box, please leave a note in the comments below or mail us. Also, nobody has SSH access to the new box yet by default, but it's easy to - just ask.
If you want to get in touch, it's now best to mail dan(at)danjacobs.com and richard(at)vmlinuz.org - I (Yoz) am off to America so I'll be incommunicado for the next couple of days. Dan's phone number is 07956 246659 - it's better to text rather than call, he's really busy.
If you really need to talk to someone, you can call Richard in Hong Kong on +85261718423 during Hong Kong day hours - Hong Kong is on GMT+8 (BST+7) so do the maths.
At time of writing, Dan has been able to salvage the vast majority of data from the stricken machine. It looks like everyone's home directories (including mail archives) should be just fine. We're less certain about all the websites and MySQL databases until we've had time to examine them properly.
We're taking advantage of this downtime to do something long overdue, which is move accounts over to the new server in the US, which is a far faster and more reliable serving platform, with 24/7 support, rapid hardware fixes, decent backup space etc. However, getting everyone over may take a little time. Our first priority is mail: both the existing mail archives, and the mail that's been queued up waiting to be delivered while the box is down. So we're going to be spending most of today on that. If you need your mail as a matter of urgency, please leave a note in the comments of this blog entry and we'll prioritise you.
Once your mail is moved over (which it may already have been - check!), your mail server will be marmite.j-colo.net, running both POP and IMAP. (Marmite is, as you may have guessed, the replacement for butter.) Your existing username and password should work fine - let us know if it isn't. If you're using IMAP, please don't go creating any new mail folders until all your mail archive is moved over - we don't want to overwrite any new stuff by accident.
Once again, we apologise wholeheartedly for yet another extended bout of downtime. Be assured that we're all working flat out to fix things, and make them much better for the future.
(Sorry to all those who were looking for this blog earlier - the DNS record for it just disappeared. Very odd.)
Looks like we've had another major problem with butter (our main server), and we're trying to diagnose. There is a definite probability of data loss, though we don't know how high that probability is right now. We have butter back from Fulham for examination. In the meantime, we're trying to migrate users over to our new spare server in the USA, which can be used for picking up new mail. We're also going to move websites onto it. If you want to move to a different hosting service (which is entirely understandable) we may be able to help with that too.
There'll be more info posted here as I get it. In the meantime, if you want to get in touch with me, either email yoz@yoz.com or call 07977 536208.