Temporary Outage – October 09, 2016
We lost a second drive in clementine today, which will affect some users and all phone service. We are working on migrating the information to a new server and are hoping to have everything back up shortly.
Update: All users and phone service was moved to mandarin, and everything is back up functioning normally.
Notice: Physical Server Move – February 20, 2016
All of the servers need to be moved this weekend (the move has been in the works for the last three or four years...) and so there will be some downtime for each of the servers, around 15 minutes starting at 5:30AM, and finishing up by 7:30AM on Sunday morning.
Hopefully, this time frame will be as least intrusive as possible for you and your visitors/customers.
UPDATE: 2/21/2016 7:48AM: Everything went as well as it could have, and most of the servers had even less downtime than expected. We took advantage of the downtime and ran all of the pending upgrades at the same time. Let us know if you have any problems, but we've looked over log files and websites, and everything looks like it is working well.
New SSH keys for Orange – December 15, 2015
We're increasing the security on orange to get it up to (mostly) modern standards. The fingerprint for the RSA key is:
Sorry for the inconvenience, but is better to be inconvenienced than insecure...
Hardware failure – October 21, 2014
Lime Daley experienced an unspecified hardware failure in a hard drive this morning, and caused some sites to be down for a number of hours. The hardware is being worked on at the moment, and most customers are fully back up. Some customers are still missing data, and that data will hopefully be automatically restored over the next couple of hours.
Heartbleed SSL Bug: Resolved – April 09, 2014
Lime Daley has updated all of our SSL certificates due to the recently discovered bug in the widely used OpenSSL libraries.
The vulnerability affected our email and web services. Unfortunately, there isn't any way to detect whether an attacker has made any use of this vulnerability or not. To be safe, you should update any passwords you have.
This vulnerability does not apply to those customers on "orange", as that server was using a version of the software that was not affected by this bug.
Additionally, access via sftp and ssh were also not affected.
Any services not protected by SSL (like web pages that don't have an https:// at the front, but just http://) are also not affected, because non-ssl services are eavesdroppable by default any way.
Hardware Maintenance: Clementine – March 01, 2014
A hard drive failed in clementine, and the plan is to replace the drive today, provided that USPS delivered it to the right place. I'm hoping to replace it without taking the machine down, but we might need to power down the machine for a little while to avoid losing any data during the replacement.
I'll update this page as any new information happens.
10:30AM Update: The drive has been replaced, and the machine is back up and running. Let me know if you see any problems, but everything looks good on my end. The server might be a little slow today as it copies the data to the new drive.
Storm Status – October 29, 2012
Since the servers are located on the east coast, we are potentially affected by Hurricane Sandy. But, the backup power systems were tested this morning, generators have their fuel tanks topped up, and we are not expecting any problems with the service. Stay safe at home checking your email... :)
Domains Made Easy / GoDaddy DNS outage – September 10, 2012
For those customers who use domainsmadeeasy as their DNS provider, you are likely seeing sporadic outages. A hacker has claimed responsibility, but hasn't said why. DomainsMadeEasy's phone is busy/down, so I haven't been able to talk to them to get an estimated uptime.
I suppose there is one reason to use dnsmadeeasy instead - the domainsmadeeasy DNS is free with a domain registration, and I have to pay for dnsmadeeasy, but DNS Made Easy has handled pretty large attacks successfully in the past.
Server outage – August 22, 2012
It's been a while since I've had anything interesting to report, and that's usually considered a good thing...
We're having some trouble with one of the servers and replacement parts are on order and will be installed tomorrow. In the meantime, if you are on "tangerine", your web page will be inaccessible. Email will still be delivered relatively normally in most cases, and will be saved for later delivery in all other cases.
Sorry for the inconvenience. Please call if you have any questions.
UPDATE: 08/23: 12:30PM. The power supply is going bad and has been temporarily replaced. Most customers (e.g. all but one) are now up and running.
UPDATE: 08/23: 4:00PM. A permanent power supply has been installed and everything is back to normal.
Internet Routing - Level3 – August 22, 2011
Level3 (an upstream provider for Lime Daley) appears to be having some issues today, and that is causing some flakiness in our network. Staff is onsite monitoring the issue, and may end up moving traffic off of Level3 until they fix their issue.
[Update: it turned out to be a Sprint issue, and it took them a while to figure it out. It only affected the connections between some people and some websites.]