SEAS Computer and Network Services Outages
This page will catalog planned and recent interruptions in SEAS computer and network services.
| Start Time | End Time | Planned? | Additional Notes |
| 5/23/13 01:45AM | 5/23/13 04:15AM | No | Due to a problem with electrical power in the SEAS data center, most CETS services - including email and web service - were offline for a couple of hours overnight. Power was lost about 1:45am. We were onsite at about 3:15am and, by using an alternate power feed, all services were back up around 4:15am. |
| 5/04/13 06:00AM | 5/04/13 09:00AM | Yes | We will be physically moving servers within the GRW data center because of the construction work that will be going on in GRW this summer. Both email and central file services will be down. The email servers will not be accessible during the downtime. |
| 4/01/13 10:30PM | 4/02/13 08:30AM | No | Several Speclab machines lost power overnight. The situation was resolved the next morning. |
| 3/18/13 1:30AM | 3/18/13 2:00AM | No | Our file server stopped responding, which disrupted most SEAS services. No email messages were lost. |
| 1/12/13 9:00AM | 1/12/13 11:45AM | No | Email was not available for 3/8ths of the SEAS accounts. No email messages were lost. |
| 1/12/13 8:00AM | 1/12/13 6:00PM | Yes | There will be a full building electrical shutdown in GRW this Saturday, 1/12/13. Email and web services will not be affected by the electrical shutdown. You will be able to access your email during this time, and email sent to SEAS will be delivered normally. However, some CETS services will be down from approximately 8am to 6pm Saturday. Eniac will not be available for remote SSH logins. Speclab, biglab, and the Windows virtual lab will also be down. |
| 1/7/13 6:00AM | 1/7/13 8:00AM | Yes | We need to upgrade the software on a few SEAS switches. These switches connect the computers that host many of our production services, such as email, websites, licenses, and others. There are a few different switches that need to be upgraded and rebooted. Each one should take less than 15 minutes, so the downtime for any individual service should not be longer than that. We are reserving a two-hour maintenance window to complete the upgrades and troubleshoot any unexpected problems. |
| 12/1/12 7:15AM | 12/1/12 8:30AM | No | Some SEAS services were unavailable due to an unscheduled network outage in the SEAS machine room. All services should be working now. Any email received during the outage should have been queued and delivered once the servers were back online. |
| 10/4/12 7:00AM | 10/4/12 8:00AM | Yes | We will be performing maintenance on the primary SEAS licensing server. During this time you may be unable to obtain licenses for newly launched instances of Matlab, Abaqus, and Maple. Existing jobs should continue uninterrupted. |
| 8/20/12 6:00AM | 8/20/12 8:00AM | Yes | During this time frame parts of the Mezzanine and the 2nd floor of Towne may experience short periods of network loss, so please prepare accordingly. |
| 8/9/12 6:00AM | 8/9/12 8:00AM | Yes | During this time frame the first two floors of GRW will not have network access, so please prepare accordingly. I apologize for the inconvenience this scheduled maintenance may cause. We need this window to upgrade networking equipment. If you have any concerns regarding this time-frame please get back to me as soon as possible. |
| 8/7/12 6:00AM | 8/7/12 8:00AM | Yes | We need to perform some upgrades to the core SEAS network router. The router will need to be rebooted as part of this maintenance, which will result in a loss of Internet connectivity for all SEAS wired, wireless, and PennNet phone connections. We do not expect the actual network outage to last more than a few minutes, but we are reserving the entire two hour window for testing and troubleshooting. |
| 8/5/12 8:00AM | 8/5/12 9:00AM | Yes | The SEAS print server and mailing list server will be down for maintenance. No mail will be lost during the outage. |
| 7/31/12 7:00AM | 7/31/12 8:00AM | Yes | During this time frame portions of the building may not have network access, so please prepare accordingly. I apologize for the inconvenience this scheduled maintenance may cause. We need this window to upgrade networking equipment. If you have any concerns regarding this time-frame please get back to me as soon as possible. |
| 4/13/12 5:00PM | 4/16/12 9:45AM | No | There was a configuration problem on the SEAS DHCP server that caused some loss of wired network connectivity in the Hayden building this weekend. The problem started just after 5PM on Friday. It was isolated to Hayden and it did not affect phones or wireless networking. Unfortunately, we were not monitoring for this specific error, so we did not notice the problem until this morning. We know the root cause of the problem and we are working on adding a feature to our automated monitoring system to alert us right away if this happens again. |
| 2/15/12 7:00AM | 2/15/12 8:00AM | Yes | CETS will be performing maintenance on several SEAS servers. During this time you may be disconnected from Eniac sessions, and experience delayed or unsuccessful connections with mail or other services. We expect to finish the work before 8AM on Wednesday. Downtime should be limited to about 5 minutes for each service. No user email or other data will be affected other than the potential brief loss of connectivity during the maintenance window. |
| 1/7/12 9:00AM | 1/7/12 12:00PM | Yes | CETS will be performing upgrades to the main SEAS file servers. During this time you may experience delayed connections and slower than normal access to data stored in your home directory or in other disk space on the main servers. It is recommended that any processes requiring intense disk activity be scheduled outside this window. We expect to finish the work before noon on Saturday. No user email or other data will be affected by this upgrade other than the potential slowness during the maintenance window. |
| 12/24/11 7:00AM | 12/24/11 7:45AM | Yes | The SEAS webmail server will be down for maintenance. No incoming mail will be lost during the outage. Email will still be available via standard mail clients such as Outlook, Thunderbird, Apple Mail, etc. |
| 12/17/11 7:00AM | 12/17/11 7:30AM | Yes | The SEAS webmail server will be down for maintenance. No incoming mail will be lost during the outage. Email will still be available via standard mail clients such as Outlook, Thunderbird, Apple Mail, etc. |
| 12/12/11 8:00AM | 12/12/11 9:00AM | No | A power outage made some services unavailable. |
| 11/10/11 7:00AM | 11/10/11 8:00AM | Yes | We believe we finally found the bug causing us problems with our DHCP server upgrade. We have the new server working in a test environment, and we plan to move it to production. This should only affect wired clients in SEAS. CETS-run servers, wireless connections, and PennNet phones should not be affected. |
| 11/1/11 7:00AM | 11/1/11 8:00AM | Yes | We ran in to complications when we tried to upgrade the DHCP server a few weeks ago. We have solved some problems and are going to try again. This should only affect wired clients in SEAS. CETS-run servers, wireless connections, and PennNet phones should not be affected. |
| 10/11/11 7:00AM | 10/11/11 8:00AM | Yes | We will be upgrading the SEAS DHCP server. Some computers may be unable to obtain an IP address while the server is offline. We expect the actual outage to last only 15 minutes, but we are reserving the full hour for troubleshooting. |
| 9/26/11 9:00PM | 9/26/11 10:00PM | No | Our monitoring system noticed a problem with the network equipment that serves the top three floors of GRW. The outage was caused by a hardware failure. |
| 9/14/11 7:00PM | 9/14/11 9:00PM | No | Webmail was intermittently unvailable to some users. |
| 7/13/11 7:00AM | 7/13/11 8:00AM | Yes | We're kicking off our summer network maintenance. This will affect the first three floors of Levine. Computers in these areas may experience intermittent network connectivity during this hour. |
| 5/12/11 4:00AM | 5/12/11 5:00AM | Yes | Penn Weblogin will be unavailable for 30 to 60 minutes for an upgrade. This will not impact email or any non-web connections to SEAS computer systems. |
| 4/27/11 6:00AM | 4/27/11 6:15AM | Yes | There will be a brief wireless outage in the SEAS buildings lasting less than 15 minutes. It will affect AirPennNet, AirPennNet-Guest, and AirPennNet-Help. |
| 4/26/11 6:00AM | 4/26/11 6:15AM | Yes | There will be a brief wireless outage in the SEAS buildings lasting less than 15 minutes. It will affect AirPennNet, AirPennNet-Guest, and AirPennNet-Help. |
| 3/8/11 9:30PM | 3/8/11 10:30PM | No | At 9:39 pm CETS was alerted that GRW had lost power, and immediately started working to find out what had happened. By 10:25 things were mostly recovered. Most machines were able to stay up due to UPS power. Please email cets@seas if you notice any continuing issues. |
| 3/3/11 11:00PM | 3/4/11 1:00AM | Yes | There will be a 1-2 hour network outage in the LRSM and Edison buildings sometime between 11 PM on March 10 and 5 AM on March 11, 2011. During this maintenance window, ISC will be modifying a network closet in DRL to address problems discovered when that building flooded this summer. The fiber circuit between LRSM/Edison and the main SEAS router in GRW goes through this DRL closet. While the entire project is estimated to take 6 hours, they will try to minimize the downtime for any individual circuit. The expected outage for LRSM & Edison should be less than 2 hours. |
| 12/28/10 7:00PM | 12/29/10 12:00PM | Yes | We are going to have temporary emergency power which should be enough to operate the email servers, the main web server, the network router and VoIP, but everything else which is dependent on GRW will not work. Not working will be all CETS labs, several department teaching labs, virtual lab, staff desktop computers, Eniac/ssh, CIS managed linux desktop computers, Windows share drives, license servers, Meeting Maker, Calcium, and all web services except static pages. |
| 10/12/10 7:00AM | 10/12/10 8:00AM | Yes | We will be rebooting a few of our network switches. This will cause a brief network outage for a small number of clients in GRW, LRSM, Levine, Moore, and Towne. Each reboot should only take a minute. |
| 10/11/10 7:00AM | 10/11/10 8:00AM | Yes | We will be rebooting a few of our network switches. This will cause a brief network outage for a small number of clients in GRW, LRSM, Levine, Moore, and Towne. Each reboot should only take a minute. |
| 10/7/10 2:15PM | 10/7/10 2:35PM | No | There was a network outage affecting the 3rd through 5th floors of GRW. |
| 9/2/10 7:00AM | 9/2/10 8:00AM | Yes | We need to make some configuration changes to prevent another network outage like we experienced last Friday. |
| 8/27/10 11:00AM | 8/27/10 12:15PM | No | There was a widespread outage of the SEAS network. This affected wired, wireless, and VoIP networks in various parts of SEAS. |
| 8/5/10 5:00PM | 8/5/10 7:00PM | Yes | Alliance (the production CGI web server) will be down for up to 2 hours. We expect the actual downtime to be less, but reserve the full 2-hour window for testing. |
| 8/4/10 7:45AM | 8/4/10 9:00AM | No | A problem occurred on the SEAS network that affected most of the Towne building and most of the Moore building. It also affected the wireless and VoIP networks in the Skirkanich building. |
| 8/3/10 5:00PM | 8/3/10 8:00PM | Yes | Fling (the development CGI web server) will be down for up to 3 hours. We expect the actual downtime to be less, but reserve the full 3-hour window for testing. |
| 6/12/10 6:22AM | 6/12/10 7:02AM | No | The Penn Campus experienced a power outage (blink) sometime this evening. Our primary server room power needed to be manually reset after the blink. |
| 4/8/10 6:00AM | 4/8/10 9:00AM | No | One of our email server nodes crashed due to a random-access memory hardware failure. About 400 user accounts were affected; they were unable to check their email during the outage. |
| 4/1/10 7:00AM | 4/1/10 8:00AM | Yes | We will be replacing some uninterruptible power supplies in the SEAS network core. Parts of SEASNet will experience short outages of less than one minute each while we move the network switches over to their new power supplies. |
| 3/11/10 7:30AM | 3/11/10 8:30AM | Yes | We will be performing some maintenance on the AirPennNet wireless network in SEAS. AirPennNet will not be available in Moore, GRW, Levine, Towne, Skirkanich, Hayden, LRSM, or Edison buildings while this work is taking place. |
| 2/3/10 5:15AM | 2/3/10 7:45AM | No | At 5:15 this morning, our automatic monitoring scripts noticed that some of the network equipment in Skirkanich was not reachable. This affected computers on the 4th floor of the Skirkanich building. |
| 1/5/10 7:00AM | 1/5/10 8:00AM | Yes | We will be upgrading a network link to the third floor of Towne. Computers in that area may experience intermittent network outages during this maintenance window. This work may affect some VoIP phones on the 3rd floor of Towne, but should not affect the wireless network. |
| 11/16/09 1:00PM | 11/16/09 1:30PM | Yes | The SEAS email server SSL certificate will be updated. Some older mail programs may require that you accept the new certificate. |
| 10/7/09 7:00AM | 10/7/09 8:00AM | Yes | We will be upgrading some network hardware in the Towne building, which may cause brief network outages in Towne. |
| 9/19/09 9:15PM | 9/19/09 11:30PM | No | A power outage caused many of our servers to go down for several hours. Service was restored by 11:30PM. |
| 8/20/09 7:00AM | 8/20/09 8:00AM | Yes | We will be upgrading some network switches in the GRW building, which may cause brief network outages in GRW. |
| 8/18/09 7:00AM | 8/18/09 8:00AM | Yes | We will be upgrading some network switches in the Levine building, which may cause brief network outages in Levine. |
| 7/14/09 7:00AM | 7/14/09 9:00AM | Yes | The AirSEAS/SEASNet registration site running on troll.seas.upenn.edu will be unavailable while it is upgraded to the new Penn Weblogin authentication |
| 7/11/09 7:00AM | 7/11/09 10:00AM | Yes | We will be upgrading the main SEAS file server with new hardware to improve performance and storage capacity. This file server contains all the user home directories and central "project" disk space. |
| 6/25/09 8:00PM | 6/26/09 2:00PM | No | The hardware running the WebDAV server died and had to be replaced. |
| 5/21/09 7:30AM | 5/21/09 7:45AM | Yes | There will be a short network outage to the new SIG computer graphics lab while we upgrade the connection. This outage should last less than 15 minutes. |
| 5/21/09 7:00AM | 5/21/09 7:00AM | Yes | We will be updating the configuration on the core SEAS router. We do not expect this to result in any network downtime, but are performing this work during an early-morning maintenance window in case something goes wrong. |
| 5/19/09 7:00AM | 5/19/09 7:05AM | Yes | There will be a very brief network outage to the SEAS network link into FBA while we relocate a fiber jumper. We expect this outage to last less than 5 minutes. |
| 5/5/09 7:00AM | 5/5/09 8:00AM | Yes | We plan to test the automatic failover of our primary SEAS network link over to a secondary link. If this works as expected, there will be two very brief network blinks between SEAS and the rest of campus. |
| 4/29/09 7:00AM | 4/29/09 8:00AM | Yes | We will be upgrading some hardware serving the Distributed Systems Lab (DSL) in Moore. Network connectivity will be disrupted between DSL and everywhere else during this time. |
| 4/4/09 9:00AM | 4/4/09 10:20AM | No | Power outage caused disruption to many SEAS network services. |
| 4/1/09 7:00AM | 4/1/09 8:00AM | Yes | Network hardware upgrade in Towne. There will be a brief loss of network connectivity for basement and second floor of Towne Building. |
| 3/31/09 2:20PM | 3/31/09 2:40PM | No | Network outage affected all SEAS network services, including email. No mail was lost. |
| 2/20/09 7:40PM | 2/20/09 8:30PM | No | Mail sent in Webmail was delayed for up to one hour. No mail was lost. |
| 2/18/09 11:10AM | 2/18/09 11:25AM | No | A hardware failure caused network downtime on the 3rd-5th floors of GRW. Service was restored within 15 minutes for most users. |
| 2/11/09 7:30AM | 2/11/09 9:30AM | Yes | Webmail will be down so it can be upgraded. |
| 1/28/09 7:00AM | 1/28/09 8:00AM | Yes | Router replacement will couse brief network outages in LRSM and Towne buildings |
| 1/7/09 7:00AM | 1/7/09 11:00AM | No | Eclipse filer was down, which affected many websites and web services |
| 12/18/08 6:00AM | 12/18/08 8:00AM | Yes | Network downtime for switch upgrade |