Emergency Maintenance 5PM PDT Bend, OR Datace (Resolved)
  • Priority - Medium
  • Affecting Other - Cascade Divide Datacenter
  • Dear valued clients,

    Thank you for your continued business! This is an emergency service impacting maintenance notification:

    Services affected: Cascade Divide, Bend OR datacenter
    Date: September 11th, 2019
    Start time: 5PM PDT (GMT -8)
    Duration: Approximately 15 minutes, though 30 minutes is being allocated
    Scope: Replacing a faulty line card that appears to have unexpectedly rebooted this morning, out of an abundance of caution

    We appreciate your cooperation and continued business.

  • Date - 09/11/2019 17:00 - 09/12/2019 03:40
  • Last Updated - 09/11/2019 19:33
June 25th/26th 2019 Segra Datacenters Mainten (Resolved)
  • Priority - Medium
  • Affecting Other - Segra Datacenters (DC74, Charlotte NC)
  • Dear Valued Customer,

    The SEGRA Data Center network team has been planning network upgrades at our Charlotte, NC data centers -  CLT1, CLT2 and CLT4 for the past few months.  During each night of the maintenance window, we will be performing router reboots to complete these upgrades.  Each reboot is only expected to last 5-10 minutes. 

    These reboots will allow us to provide additional services at the data center and we do not anticipate any extended period of outages beyond the standard reboot time of each router themselves. 

    If you have any questions or concerns, please contact H4Y Customer Care by ticket at my.h4y.us and we will be happy to assist you further.

  • Date - 06/26/2019 00:01 - 09/11/2019 19:35
  • Last Updated - 06/19/2019 00:06
Los Angeles (Psychz) Power Issues (Resolved)
  • Priority - Critical
  • Affecting Other - Los Angeles Psychz
  • Dear Valued Customers,

    Over the course of the past week, we have faced 3 power outages (including today) as a result of a faulty PDU unit at the datacenter.   Attempted repairs to said PDU at the facility failed and replacement  unit is on order.  The replacement ETA is 1-2 weeks.   Meanwhile, the existing unit has been bypassed so it cannot malfunction again.   Servers that are fed by A/B redundant circuits have not been affected.   Today, at approximately 1:21PM PST, the Los Angeles (Psychz) Datacenter lost utility power entirely.   This is a common Los Angeles phenomenon during the hot summer months and typically occurs,  without any sign of disruption or service impact, on average once or twice over the course of a year.   Upon the loss of utility power today,  one of two diesel backup generators was activated at the facility in Los Angeles and failover occurred within 30 sec, but it still caused downtime for all branch circuits up the line.   Additional downtime may have been observed as individual machines, switches, and routers boot back up.   Those with A/B redundancy were unaffected as the healthy battery backups took over.   We are working with the datacenter to ensure that full resolution to this issue occurs immediately and does not cause any more power interruptions.   As of 3:40pm PST, the utility power has still not been restored and generator power is in use.   The generator is capable of running indefinitely as fuel is available.    We will provide updates at https://my.h4y.us/serverstatus.php as they become available.

    Thank you for your patience and understanding!

  • Date - 06/12/2019 22:49 - 06/19/2019 00:00
  • Last Updated - 06/12/2019 22:56
Peer1 LA Network Outage 3/12/2019 (Resolved)
  • Priority - Critical
  • Affecting System - Peer1 Los Angeles
  • Beginning 3/12/2019 at approximately 07:45AM PST (GMT -8) our Peer1 Los Angeles datacenter began to experience various network interruptions intermittently, with services flapping.

    At approximately 14:49 PST (GMT -8) the Peer1 network started to suffer a prolonged outage. The outage is still under investigation at this time, however it is believed to be sourced from an aggregation switch that may have become faulty.  A previous distribution switch was already replaced, as part of the diagnosis. The Peer1 Los Angeles datacenter team has all resources working on the incident, however there is no expected ETA at this time for complete resolution. We will be posting updates as we receive them at this status page, and you may also monitor https://status.cogecopeer1.com/ directly if desired.

    At approximate 16:14PM PST (GMT -8) services were restored. An aggregation switch was determined to be faulty, as well as the previously replaced distribution switch. JTAC has been involved to determine the root cause of both switches failing. We will continue to monitor the situation and provide a complete RFO as soon as a root cause analysis is available.

    Once the incident has been resolved, clients can open a ticket in the billing department for SLA credit as applicable.

    Thank you for your continued patience and understanding.

  • Date - 03/12/2019 14:49 - 03/13/2019 04:06
  • Last Updated - 03/12/2019 23:22
(Los Angeles Psychz) - Cooling system issue (Resolved)
  • Priority - Critical
  • Affecting Other - Los Angeles Psychz Only
  • URGENT (Los Angeles Psychz ONLY) - Cooling system issue 1/9/2018

    **RESOLVED**

    A problem with the A/C / cooling system at the Los Angeles (Psychz) datacenter has been discovered and some equipment has been taken offline while the issue is resolved.   We do not have an exact ETA for resolution but we will post updates as we have them.   We have a great history of uptime and strive for 100%.   Billing credits will be available per our SLA and we sincerely apologize for any problems this is causing you.

    UPDATE:

    -----------------------------

    At approximately 5am PST, an overheat situation developed on a faulty A/C unit powering core routers.   It did not failover properly and caused automated shut downs of several systems and damage to fiber optic cabling.  Cabinet Z7 in Los Angeles Psychz was first affected by loss of connectivity and then other cabinets were affected as well.   As of 7:15am PST, We've replaced the faulty equipment and cabling as needed and ensured proper cooling.   BGP routes may continue to converge and cause some packetloss as operations return to normal.

    Update:

    ----------------------------

    Service is restored and servers are back online now. Sorry for any inconvenience this has caused.

  • Date - 01/09/2018 14:40
  • Last Updated - 01/09/2018 16:44
Oregon network (Resolved)
  • Priority - Critical
  • Affecting Other - Bend, OR - Cascade Divide Network
  • We are working on an issue with LSN in Bend, OR.   Traffic will be re-routed shortly.

  • Date - 01/23/2017 22:46 - 01/23/2017 22:48
  • Last Updated - 01/23/2017 22:47
Service Affecting Maintenance Sat, November 5 (Resolved)
  • Priority - Medium
  • Affecting System - Cascade Divide
  • UPDATE 7:34AM PST - All servers up.   If you are still experiencing any issues.   Please file a ticket.   Welcome to Bend, OR!

    UPDATE 4:32AM PST - 50% of servers are networked, cabled, and powered up.   We continue to work on the remainder.   All networking and routing has been tested and given the OK.  

    UPDATE 1:33AM PST - All equipment has arrived at the Bend facility.   Staff is unpacking and racking the equipment into our cabinets as planned.  Switches will be powered up first followed by servers. 

    UPDATE 10:03PM PST - IP routing has been tested and confirmed except for a few stragglers.   Equipment is still enroute and more updates will follow shortly.

    UPDATE 7:38PM PST - All servers have been cleanly shut down.   We are now safely transporting them to the Bend facility.   So far so good!   Please check back for updates.

     

    Dear Valued Clients,
    This notice is to inform you that we will be relocating equipment at our Cascade Divide Roseburg, OR location to the Cascade Divide Bend, OR facility overnight Saturday, November 5th at 6pm PST. This relocation is necessary due to our requirements for a larger and more redundant facility. The new facility features more transit providers, additional redundancy, more space, and larger capacity in general. This relocation will indeed cause a service interruption on the night of November 5th, but it should be as minimal as humanly possible. There will be NO IP space changes, rack-level networking changes, and the new facility is in the same geographic region and state. We chose the most off-peak time while allowing for unforeseen conditions to complete the maintenance with plenty of time to spare before the following morning. Your data, IPs, and configurations are NOT at risk. There will be no functional difference to your service, though you can look forward to enhanced redundancy and reliability, plus pricing benefits for bandwidth and much more in the future. Updates as we go will be posted at https://my.h4y.us/serverstatus.php 

    Details:

    If you have a dedicated or colo server, we suggest shutting it down (halting) it before November 5th at 6PM PST. If that is not possible, our staff will also begin powering down equipment at that time. We will attempt to shut down all servers cleanly in all cases (using CTRL+ALT+DEL where possible, or by logging in and halting). We will also cleanly shut down all VPS accounts and shared servers. Servers will all be booted up and checked once they arrive at the Bend, OR facility.

    The physical relocation will begin by 7pm PST. We have at least one staff member assisting PER CABINET so the equipment will be loaded and relocated as quickly as possible. We expect that equipment will be at the Bend facility and powering up within 4 hours. The maintenance period will extend to 3AM PST on November 6th to account for any unforeseen issues. Please check our network status page for updates and be aware that we will be busy with phonecalls and support requests the entire night. If possible, refrain from contacting us for status updates until we announce that servers are racked/cabled and should be powered back up.

    Your business is appreciated! Please contact us if you have any questions, concerns, or special instructions for us during this relocation. Our goal is to ensure clients waking up Sunday morning will simply return with ease to business as usual. We will post status updates as they become available on the night of November 5th. We will also post the Bend, OR datasheet and info to our site within the coming days. Thank you for choosing us!

  • Date - 11/05/2016 18:00
  • Last Updated - 11/06/2016 16:19
**EMERGENCY** Scheduled Network Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - Peer1 LA
  • Dear Customer,

    Please be advised that we will be performing scheduled network maintenance in our Los Angeles (West 7th Street) facility during the following date and time:

    From: July 22, 2016 - 00:00 PDT (July 22, 07:00 UTC)

    To:   July 22, 2016 - 02:00 PDT (July 22, 09:00 UTC)

    The window will occur on Friday July 22nd from 00:00 - 02:00. During this timeframe, Network Engineers will reboot a virtual switch chassis, due to the nature of the maintenance downtime of around 10 minutes is expected during this window and services will be affected.

    This work will be SERVICE IMPACTING. The appropriate staff will be present for the entire duration of this maintenance window.

  • Date - 07/22/2016 00:00 - 08/09/2016 11:12
  • Last Updated - 07/21/2016 18:56
Cascade Divide Datacenter Outage 7/18 (Resolved)
  • Priority - Critical
  • Affecting Other - Cascade Divide Datacenter
  • 07/18/2016 15:01 PST: We are currently investigating an outage in our Cascade Divide, Roseburg OR datacenter. Datacenter staff is aware of the outage. Updates to follow.

    7/18/2016 15:49 PST: We have been advised by the datacenter that their primary router has failed, and failover to the backup router failed. A manual failover is currently taking place.

  • Date - 07/18/2016 15:01 - 07/18/2016 20:38
  • Last Updated - 07/18/2016 15:52
Cascade Divide Datacenter Outage (Resolved)
  • Priority - Critical
  • Affecting System - Cascade Divide Datacenter
  • At 10:57 AM PST (GMT-8) our NOC team noticed an outage at our Cascade Divide, Roseburg OR datacenter. The NOC made contact with the datacenter and they are aware of the issue, and investigating.

    UPDATE: 11:12 AM PST -- There is a confirmed fiber cut which is affecting all circuits (including redundant and protected circuits) in and out of the datacenter.

    UPDATE: 11:23 AM PST -- Response teams are en route from all major fiber providers to the facility (LSN, HE, Level3, etc). We do not yet have any ETA.

    UPDATE: 12:04 PM PST -- Teams are on site from all carriers and repairs are underway. Substantial damage has been done to the multiple fiber paths and utility poles in the area, including full fiber sheath cuts. Crews are giving a rough estimate of 4-5 hours.

    UPDATE: 1:55PM PST -- Crews have updated their original estimate. The new expected resolution time is 7:00PM PST. Our on-scene technicians have shared pictures, available at: http://imgur.com/a/023V2

    UPDATE: 4:06PM PST -- Aerial lines have been pulled across roadways, and work is underway on terminating and splicing the fiber bundles. The ETA remains the same.

    UPDATE: 5:47PM PST -- Service has been restored to the first fiber bundle. At this time if you have any remaining issues please open a ticket so we can address them.

  • Date - 04/13/2016 10:57 - 04/18/2016 10:16
  • Last Updated - 04/13/2016 17:50
SERVICE IMPACTING Maintenance: 9/24/15 9PM-11 (Resolved)
  • Priority - Medium
  • Affecting System - Peer1 LA
  • Dear valued clients,

    Thank you for your continued business! This is a maintenance notification regarding our Los Angeles (Peer1) location.

    On Thursday, September 24th starting at 9PM PST (Pacific Standard Time), we will be relocating some of our equipment in the Los Angeles datacenter into new cabinet space that we already setup only a few yards away.

    The goal of this maintenance period is to provide you a more reliable infrastructure. Brand new switches and PDUs are already installed, and ready to go!

    SHARED CUSTOMERS:

    This is a notification for the following servers:
    CA-Alpha
    CA-Bravo
    CA-Foxtrot
    CA-Kilo
    CA-Lima
    CA-Zeta

    DEDICATED CUSTOMERS:

    The new switches will increase our capacity and port limits for all clients. Most dedicated clients will receive free port upgrades.

    We will take incredible care in cleanly shutting down all machines where possible (including self-managed servers where possible by using CTRL+ALT+DEL). If you have a Windows self-managed server or otherwise would like to ensure the shutdown is clean, please take it down before 9AM PST. We will ensure that all dedicated servers are powered back up and are online once the migration is complete.

    We anticipate this maintenance window to last 1.5-2 hours.

    Please let us know how we can make this as painless as possible for you and your clients. We will be available for any questions you have before, during, and after the move.

  • Date - 09/24/2015 21:00 - 07/21/2016 18:56
  • Last Updated - 09/16/2015 13:04
DC74 Datacenter Maintenance April 11th (Resolved)
  • Priority - Medium
  • Affecting Other - DC74 Datacenter
  • Thank you for your continued business!

    Please be advised of the following service impacting maintenance notification.

    Reason for Notification: Router Maintenance
    Location: DC74 Data Centers – CLT4 facility, 1612 Cross Beam Drive, Charlotte NC
    Start Date & Time: Saturday, April 11th, 2015 starting at 2300 EST


    Expected End Date & Time: Saturday, April 11th, 2015 ending at 2330 EST

    PLEASE NOTE: This maintenance does not affect shared, reseller, and VPS clients as you are in a seperate datacenter. This affects DC74 datacenter ONLY.

    Description: We will be performing router maintenance potentially requiring a reboot of the routers. This will impact BGP routing and could potentially cause up to a 15-30 minute loss of network connectivity. The date and time chosen is the lowest traffic routing period for the entirety of the datacenter. Please be aware of this maintenance window and potential service disruption.

    Please let me us know if you have any questions.

  • Date - 04/11/2015 23:00 - 09/16/2015 13:04
  • Last Updated - 04/09/2015 18:29

Powered by WHMCompleteSolution