Tonight between the hours of 21:00 and 22:50 PST, our co-location provider experienced a massive network failure of some kind. There was a brief period of time (between 21:33 and 22:05 PST) where service was restored, but then went down again.

During these time frames, Parodius services (web, FTP, mail, etc.) would have been unreachable. Any Email sent to accounts hosted at Parodius during the outage times will eventually arrive to you due to the retry nature of SMTP.

At this time we do not know the root cause or details of the outage, but our provider has stated that all customers will receive a full write-up on Monday, January 9th.

We apologise for the outage, especially given the extreme duration. We have strict SLA guidelines with our provider and thus will be credited for the downtime.

On 2011/09/10 at approximately 22:00 PDT, we will be performing a major server upgrade on our primary web server. This will consist of both a hardware and software upgrade. We estimate the maintenance to take approximately 4 hours, but may run longer than that if any issues are encountered.

During this time you will experience timeouts when visiting any of our hosted sites, as well as possible DNS lookup failures. Mail services (both inbound and outbound), FTP/SFTP services, and SSH services will also will also be unavailable during this window.

We apologise in advance for any inconvenience this may cause.

Update (01:05 PDT) -- The upgrade was completed successfully. Those who use SSH or SFTP may receive warnings about host key mismatches; this is expected (please update your host key entries, or remove the old and let your software add the new host key).

On 2011/08/28 from approximately 00:00 to 00:15 PDT, our co-location provider will be performing maintenance (IOS upgrades) on their core switching fabric.

The Parodius network will be unreachable during that time.

We'll be doing a full server rebuild, including upgrades of all software on the machine, in about an hour (starting at 00:00 PDT). The procedure should take a couple of hours, barring any complexities.

Update (01:52 PDT) -- All upgrades were completed successfully.

We've made some (long-needed) changes to our mail infrastructure. Until today, we used the hostname mx01.sc1.parodius.com for all of our SMTP and POP3/IMAP infrastructure. This was an intermediary legacy change (when we were in the process of moving datacenters) which never got reverted.

Going forward, users should use mail.parodius.com for both their inbound POP3/IMAP server name, as well as their outbound mail server name (for those who use the feature). Our FAQ has been updated to reflect this change.

Please update your Email client settings!

The old mx01.sc1.parodius.com hostname has been removed from DNS; users who continue to use it will (eventually, based on DNS caching) begin seeing an error stating the hostname doesn't exist. This is expected.

Our mail server SSL certificates have been renewed and updated to reflect this change as well; use of the old name (while still resolvable) will result in your mail client complaining about a "name mismatch" (mx01.sc1.parodius.com doesn't equal mail.parodius.com) or may fail with the error "bad certificate". This is also expected.

For those using SSL fingerprints (e.g. fetchmail users), you will need to update your fingerprint to reflect the new SSL certificate.

At approximately 22:00 PST, our primary MySQL / secondary DNS server went offline. At this time we don't know the root cause. There is strong evidence the power supply has failed, however.

Users may be seeing timeouts or connection failures when their web applications attempt to connect to the MySQL server. DNS resolution for things under the parodius.com domain, as well as domains we host, may take slightly longer than usual (but should be working).

We will be going to the datacenter later tonight to investigate the problem, and/or move services off to another machine. We'll provide further updates as we have them.

Update (23:43 PST) — we're preparing to leave for the datacenter, and will be replacing the box with a brand new 4-core, 4-disk system. It will take us some time to migrate the previous servers' data to the new system. There is no ETR at this time.

Update (04:20 PST) — we've returned from the datacenter. The new server has been installed, but has not been configured at this time.

Regarding the bad hardware: analysis shows the power supply is in excellent condition (all voltages are perfect); only when hooked up to the motherboard does it fail to power on. The same happens with an alternate PSU. We believe the issue to be a bad motherboard or a bad CPU, as either of these will cause a system not to power up. Further investigation will be done in days to come.

All data which was on the system safe/intact. We're now beginning a combination of data restoration + system reconfiguration (for the new server). This will take some time, so please be patient.

Update (06:47 PST) — we have finished the restoration. MySQL services are again up and running. We apologise for the unexpected interruption in service.

Post-outage follow-up — hardware analysis initially determined the failed component to be the motherboard. This conclusion was based on a testing the PSU with a voltage tester (it passed), and replacing the CPU (the motherboard would still not power on). We then replaced the motherboard, but experienced the same problem. We then hooked up a spare PSU and the system powered up normally, concluding the faulty part was the PSU. Upon disassembling the PSU, we found a single bulging capacitor which also showed signs of having leaked. We will be replacing the PSU and eventually repurposing the system.

On 11/11 starting at approximately 23:36 PST and lasting until 23:59 PST, our co-lo provider witnessed what appeared to be some form of network interruption event. The Parodius network and related services (including the co-lo providers' own web site) were all inaccessible.

We do not have technical details of the outage at this time, but a ticket was opened with their NOC to have the issue investigated, with provided network logs (both ingress and egress).

All Parodius systems/servers were functioning normally during this outage.