SEAS Computer and Network Services Outages

This page will catalog planned and recent interruptions in SEAS computer and network services.

Start Time End Time Planned? Additional Notes
03/26/14 4:30AM 03/26/14 8:00AM No Hardware failure on a critical host took several web services offline overnight. Affected services included SEAS and CIS websites, software license managers, and the account management website. Staff responded in the early morning and restored service. CETS plans to replace the affected hardware, and has already improved monitoring to reduce downtime under similar circumstances in the future.
01/08/14 7:00AM 01/08/14 8:00AM Yes Planned upgrades to several servers. During this time you may experience a 5 minute period during which you cannot access email. In addition, Connections to the virtual Windows lab will be interrupted and unavailable for about 5 minutes. No user email or other data will be affected by this upgrade other than the downtime mentioned above during the maintenance window.
01/06/14 8:00AM 01/06/14 9:00AM Yes Reboot of the Eniac login server for a kernel update. During the reboot all user sessions on Eniac will be interrupted, and users may not be able to reconnect for a period of 10-20 minutes.
12/12/13 8:00AM 12/12/13 9:00AM Yes Planned maintenance on the Eniac login server. All user sessions on Eniac will be interrupted, and users may not be able to reconnect for a period of 10-20 minutes.
11/22/13 8:00AM 11/22/13 9:00AM Yes Planned maintenance to convert the Eniac login server from a virtual machine to physical hardware. Remote logins via SSH will be unavailable during this time.
10/21/13 7:00AM 10/21/13 9:00AM Yes Planned maintenance to update hardware that provides hosting for certain lab, group sites, and all student sites that use CGI, PHP, and MySQL. See here for a full list of affected hosts.
10/19/13 9:00AM 10/19/13 10:00AM No A University-wide network outage affected remote access to SEAS computing resources. The ISC resolved the issue 1 hour after it started, and access to SEAS services was restored at te same time.
7/31/13 08:30AM 7/31/13 11:00AM No Due to hardware failure, several hundred users lost access to email. Both IMAP connections and webmail logins failed for several hundred users. We migrated the affected users to another system by 11:30AM. No mail was lost during the outage; mail was held on our gateway mail servers and delivered to mailboxes once service was restored.
5/23/13 01:45AM 5/23/13 04:15AM No Due to a problem with electrical power in the SEAS data center, most CETS services - including email and web service - were offline for a couple of hours overnight. Power was lost about 1:45am. We were onsite at about 3:15am and, by using an alternate power feed, all services were back up around 4:15am.
5/04/13 06:00AM 5/04/13 09:00AM Yes We will be physically moving servers within the GRW data center because of the construction work that will be going on in GRW this summer. Both email and central file services will be down. The email servers will not be accessible during the downtime.
4/01/13 10:30PM 4/02/13 08:30AM No Several Speclab machines lost power overnight. The situation was resolved the next morning.
3/18/13 1:30AM 3/18/13 2:00AM No Our file server stopped responding, which disrupted most SEAS services. No email messages were lost.
1/12/13 9:00AM 1/12/13 11:45AM No Email was not available for 3/8ths of the SEAS accounts. No email messages were lost.
1/12/13 8:00AM 1/12/13 6:00PM Yes There will be a full building electrical shutdown in GRW this Saturday, 1/12/13. Email and web services will not be affected by the electrical shutdown. You will be able to access your email during this time, and email sent to SEAS will be delivered normally. However, some CETS services will be down from approximately 8am to 6pm Saturday. Eniac will not be available for remote SSH logins. Speclab, biglab, and the Windows virtual lab will also be down.
1/7/13 6:00AM 1/7/13 8:00AM Yes We need to upgrade the software on a few SEAS switches. These switches connect the computers that host many of our production services, such as email, websites, licenses, and others. There are a few different switches that need to be upgraded and rebooted. Each one should take less than 15 minutes, so the downtime for any individual service should not be longer than that. We are reserving a two-hour maintenance window to complete the upgrades and troubleshoot any unexpected problems.
12/1/12 7:15AM 12/1/12 8:30AM No Some SEAS services were unavailable due to an unscheduled network outage in the SEAS machine room. All services should be working now. Any email received during the outage should have been queued and delivered once the servers were back online.
10/4/12 7:00AM 10/4/12 8:00AM Yes We will be performing maintenance on the primary SEAS licensing server. During this time you may be unable to obtain licenses for newly launched instances of Matlab, Abaqus, and Maple. Existing jobs should continue uninterrupted.
8/20/12 6:00AM 8/20/12 8:00AM Yes During this time frame parts of the Mezzanine and the 2nd floor of Towne may experience short periods of network loss, so please prepare accordingly.
8/9/12 6:00AM 8/9/12 8:00AM Yes During this time frame the first two floors of GRW will not have network access, so please prepare accordingly. I apologize for the inconvenience this scheduled maintenance may cause. We need this window to upgrade networking equipment. If you have any concerns regarding this time-frame please get back to me as soon as possible.
8/7/12 6:00AM 8/7/12 8:00AM Yes We need to perform some upgrades to the core SEAS network router. The router will need to be rebooted as part of this maintenance, which will result in a loss of Internet connectivity for all SEAS wired, wireless, and PennNet phone connections. We do not expect the actual network outage to last more than a few minutes, but we are reserving the entire two hour window for testing and troubleshooting.
8/5/12 8:00AM 8/5/12 9:00AM Yes The SEAS print server and mailing list server will be down for maintenance. No mail will be lost during the outage.
7/31/12 7:00AM 7/31/12 8:00AM Yes During this time frame portions of the building may not have network access, so please prepare accordingly. I apologize for the inconvenience this scheduled maintenance may cause. We need this window to upgrade networking equipment. If you have any concerns regarding this time-frame please get back to me as soon as possible.
4/13/12 5:00PM 4/16/12 9:45AM No There was a configuration problem on the SEAS DHCP server that caused some loss of wired network connectivity in the Hayden building this weekend. The problem started just after 5PM on Friday. It was isolated to Hayden and it did not affect phones or wireless networking. Unfortunately, we were not monitoring for this specific error, so we did not notice the problem until this morning. We know the root cause of the problem and we are working on adding a feature to our automated monitoring system to alert us right away if this happens again.
2/15/12 7:00AM 2/15/12 8:00AM Yes CETS will be performing maintenance on several SEAS servers. During this time you may be disconnected from Eniac sessions, and experience delayed or unsuccessful connections with mail or other services. We expect to finish the work before 8AM on Wednesday. Downtime should be limited to about 5 minutes for each service. No user email or other data will be affected other than the potential brief loss of connectivity during the maintenance window.
1/7/12 9:00AM 1/7/12 12:00PM Yes CETS will be performing upgrades to the main SEAS file servers. During this time you may experience delayed connections and slower than normal access to data stored in your home directory or in other disk space on the main servers. It is recommended that any processes requiring intense disk activity be scheduled outside this window. We expect to finish the work before noon on Saturday. No user email or other data will be affected by this upgrade other than the potential slowness during the maintenance window.
12/24/11 7:00AM 12/24/11 7:45AM Yes The SEAS webmail server will be down for maintenance. No incoming mail will be lost during the outage. Email will still be available via standard mail clients such as Outlook, Thunderbird, Apple Mail, etc.
12/17/11 7:00AM 12/17/11 7:30AM Yes The SEAS webmail server will be down for maintenance. No incoming mail will be lost during the outage. Email will still be available via standard mail clients such as Outlook, Thunderbird, Apple Mail, etc.
12/12/11 8:00AM 12/12/11 9:00AM No A power outage made some services unavailable.
11/10/11 7:00AM 11/10/11 8:00AM Yes We believe we finally found the bug causing us problems with our DHCP server upgrade. We have the new server working in a test environment, and we plan to move it to production. This should only affect wired clients in SEAS. CETS-run servers, wireless connections, and PennNet phones should not be affected.
11/1/11 7:00AM 11/1/11 8:00AM Yes We ran in to complications when we tried to upgrade the DHCP server a few weeks ago. We have solved some problems and are going to try again. This should only affect wired clients in SEAS. CETS-run servers, wireless connections, and PennNet phones should not be affected.
10/11/11 7:00AM 10/11/11 8:00AM Yes We will be upgrading the SEAS DHCP server.  Some computers may be unable to obtain an IP address while the server is offline.  We expect the actual outage to last only 15 minutes, but we are reserving the full hour for troubleshooting.
9/26/11 9:00PM 9/26/11 10:00PM No Our monitoring system noticed a problem with the network equipment that serves the top three floors of GRW. The outage was caused by a hardware failure.
9/14/11 7:00PM 9/14/11 9:00PM No Webmail was intermittently unvailable to some users.
7/13/11 7:00AM 7/13/11 8:00AM Yes We're kicking off our summer network maintenance. This will affect the first three floors of Levine. Computers in these areas may experience intermittent network connectivity during this hour.
5/12/11 4:00AM 5/12/11 5:00AM Yes Penn Weblogin will be unavailable for 30 to 60 minutes for an upgrade. This will not impact email or any non-web connections to SEAS computer systems.
4/27/11 6:00AM 4/27/11 6:15AM Yes There will be a brief wireless outage in the SEAS buildings lasting less than 15 minutes.  It will affect AirPennNet, AirPennNet-Guest, and AirPennNet-Help.
4/26/11 6:00AM 4/26/11 6:15AM Yes There will be a brief wireless outage in the SEAS buildings lasting less than 15 minutes.  It will affect AirPennNet, AirPennNet-Guest, and AirPennNet-Help.
3/8/11 9:30PM 3/8/11 10:30PM No At 9:39 pm CETS was alerted that GRW had lost power, and immediately started working to find out what had happened. By 10:25 things were mostly recovered. Most machines were able to stay up due to UPS power. Please email cets@seas if you notice any continuing issues.
3/3/11 11:00PM 3/4/11 1:00AM Yes There will be a 1-2 hour network outage in the LRSM and Edison buildings sometime between 11 PM on March 10 and 5 AM on March 11, 2011.

During this maintenance window, ISC will be modifying a network closet in DRL to address problems discovered when that building flooded this summer.   The fiber circuit between LRSM/Edison and the main SEAS router in GRW goes through this DRL closet.  While the entire project is estimated to take 6 hours, they will try to minimize the downtime for any individual circuit.  The expected outage for LRSM & Edison should be less than 2 hours.
12/28/10 7:00PM 12/29/10 12:00PM Yes We are going to have temporary emergency power which should be enough to operate the email servers, the main web server, the network router and VoIP, but everything else which is dependent on GRW will not work. Not working will be all CETS labs, several department teaching labs, virtual lab, staff desktop computers, Eniac/ssh, CIS managed linux desktop computers, Windows share drives, license servers, Meeting Maker, Calcium, and all web services except static pages.
10/12/10 7:00AM 10/12/10 8:00AM Yes We will be rebooting a few of our network switches. This will cause a brief network outage for a small number of clients in GRW, LRSM, Levine, Moore, and Towne. Each reboot should only take a minute.
10/11/10 7:00AM 10/11/10 8:00AM Yes We will be rebooting a few of our network switches. This will cause a brief network outage for a small number of clients in GRW, LRSM, Levine, Moore, and Towne. Each reboot should only take a minute.
10/7/10 2:15PM 10/7/10 2:35PM No There was a network outage affecting the 3rd through 5th floors of GRW.
9/2/10 7:00AM 9/2/10 8:00AM Yes We need to make some configuration changes to prevent another network outage like we experienced last Friday.
8/27/10 11:00AM 8/27/10 12:15PM No There was a widespread outage of the SEAS network. This affected wired, wireless, and VoIP networks in various parts of SEAS.
8/5/10 5:00PM 8/5/10 7:00PM Yes Alliance (the production CGI web server) will be down for up to 2 hours. We expect the actual downtime to be less, but reserve the full 2-hour window for testing.
8/4/10 7:45AM 8/4/10 9:00AM No A problem occurred on the SEAS network that affected most of the Towne building and most of the Moore building. It also affected the wireless and VoIP networks in the Skirkanich building.
8/3/10 5:00PM 8/3/10 8:00PM Yes Fling (the development CGI web server) will be down for up to 3 hours. We expect the actual downtime to be less, but reserve the full 3-hour window for testing.
6/12/10 6:22AM 6/12/10 7:02AM No The Penn Campus experienced a power outage (blink) sometime this evening. Our primary server room power needed to be manually reset after the blink.
4/8/10 6:00AM 4/8/10 9:00AM No One of our email server nodes crashed due to a random-access memory hardware failure. About 400 user accounts were affected; they were unable to check their email during the outage.
4/1/10 7:00AM 4/1/10 8:00AM Yes We will be replacing some uninterruptible power supplies in the SEAS network core. Parts of SEASNet will experience short outages of less than one minute each while we move the network switches over to their new power supplies.
3/11/10 7:30AM 3/11/10 8:30AM Yes We will be performing some maintenance on the AirPennNet wireless network in SEAS. AirPennNet will not be available in Moore, GRW, Levine, Towne, Skirkanich, Hayden, LRSM, or Edison buildings while this work is taking place.
2/3/10 5:15AM 2/3/10 7:45AM No At 5:15 this morning, our automatic monitoring scripts noticed that some of the network equipment in Skirkanich was not reachable. This affected computers on the 4th floor of the Skirkanich building.
1/5/10 7:00AM 1/5/10 8:00AM Yes We will be upgrading a network link to the third floor of Towne. Computers in that area may experience intermittent network outages during this maintenance window. This work may affect some VoIP phones on the 3rd floor of Towne, but should not affect the wireless network.
11/16/09 1:00PM 11/16/09 1:30PM Yes The SEAS email server SSL certificate will be updated. Some older mail programs may require that you accept the new certificate.
10/7/09 7:00AM 10/7/09 8:00AM Yes We will be upgrading some network hardware in the Towne building, which may cause brief network outages in Towne.
9/19/09 9:15PM 9/19/09 11:30PM No A power outage caused many of our servers to go down for several hours. Service was restored by 11:30PM.
8/20/09 7:00AM 8/20/09 8:00AM Yes We will be upgrading some network switches in the GRW building, which may cause brief network outages in GRW.
8/18/09 7:00AM 8/18/09 8:00AM Yes We will be upgrading some network switches in the Levine building, which may cause brief network outages in Levine.
7/14/09 7:00AM 7/14/09 9:00AM Yes The AirSEAS/SEASNet registration site running on troll.seas.upenn.edu will be unavailable while it is upgraded to the new Penn Weblogin authentication
7/11/09 7:00AM 7/11/09 10:00AM Yes We will be upgrading the main SEAS file server with new hardware to improve performance and storage capacity. This file server contains all the user home directories and central "project" disk space.
6/25/09 8:00PM 6/26/09 2:00PM No The hardware running the WebDAV server died and had to be replaced.
5/21/09 7:30AM 5/21/09 7:45AM Yes There will be a short network outage to the new SIG computer graphics lab while we upgrade the connection. This outage should last less than 15 minutes.
5/21/09 7:00AM 5/21/09 7:00AM Yes We will be updating the configuration on the core SEAS router. We do not expect this to result in any network downtime, but are performing this work during an early-morning maintenance window in case something goes wrong.
5/19/09 7:00AM 5/19/09 7:05AM Yes There will be a very brief network outage to the SEAS network link into FBA while we relocate a fiber jumper. We expect this outage to last less than 5 minutes.
5/5/09 7:00AM 5/5/09 8:00AM Yes We plan to test the automatic failover of our primary SEAS network link over to a secondary link. If this works as expected, there will be two very brief network blinks between SEAS and the rest of campus.
4/29/09 7:00AM 4/29/09 8:00AM Yes We will be upgrading some hardware serving the Distributed Systems Lab (DSL) in Moore. Network connectivity will be disrupted between DSL and everywhere else during this time.
4/4/09 9:00AM 4/4/09 10:20AM No Power outage caused disruption to many SEAS network services.
4/1/09 7:00AM 4/1/09 8:00AM Yes Network hardware upgrade in Towne. There will be a brief loss of network connectivity for basement and second floor of Towne Building.
3/31/09 2:20PM 3/31/09 2:40PM No Network outage affected all SEAS network services, including email. No mail was lost.
2/20/09 7:40PM 2/20/09 8:30PM No Mail sent in Webmail was delayed for up to one hour. No mail was lost.
2/18/09 11:10AM 2/18/09 11:25AM No A hardware failure caused network downtime on the 3rd-5th floors of GRW. Service was restored within 15 minutes for most users.
2/11/09 7:30AM 2/11/09 9:30AM Yes Webmail will be down so it can be upgraded.
1/28/09 7:00AM 1/28/09 8:00AM Yes Router replacement will couse brief network outages in LRSM and Towne buildings
1/7/09 7:00AM 1/7/09 11:00AM No Eclipse filer was down, which affected many websites and web services
12/18/08 6:00AM 12/18/08 8:00AM Yes Network downtime for switch upgrade

 

 

 

© Computing and Educational Technology Services | Report a Problem
cets@seas.upenn.edu | 215.898.4707