Csadmin

If there is a systems emergency during non-business hours, please contact the emergency hotline (434-982-2271); there is contact information on the hotline for reaching staff at home if no information is posted there. If you aren't sure what constitutes an emergency, please click here.

ITS switches Cavalier Wireless to WPA2,

At 03:00 this morning ITS switched Cavalier Wireless to WPA2. For more information and troubleshooting information you can go to ITS Cavalier Wireless WPA2 (http://its.virginia.edu/wireless/wpa-to-wpa2.html).

Olsson Machine Room Flooded - 9/6/11, 7:15AM,

Some exposed conduit which goes through the exterior wall of Olsson connecting the 016c machine room and the courtyard between Rice and Olsson served as an overflow relief for the pond created during last night's downpour.

Most of the water simply passed through, and surprisingly little equipment damage appears to have occured. We are still assessing damage and working on the cleanup. There is one switch which was in a bad state, affecting our uplink, and we had to reset it in order to restore our connectivity to the outside world.

We will post more information and pictures as time allows.

Power blip - SAN outage - Sun, 4:30PM, Mon, 6:30AM, 7/24&7/25

Olsson Hall experienced two power failures during which the batteries in one of our key UPSes failed. This caused the storage area network (fiber channel) switches to reboot; their reboots are quite slow (>2 minutes) which is outside the window that the server OSes will tolerate. This causes the fileservers to essentially "lock up" on "scsi transport timeouts". We are having a downtime (see below) to address this.

Packet Filter Down - 5/27/11 - 6:30PM

At approximately 6:30, the packet filter appeared to go down, taking down the CS uplink. After a reboot, the filter is back up and connections are back to normal.

Emergency Downtime - Fileservers - 5PM Monday, 7/25

We must have an emergency downtime for the department's main storage systems to replace the batteries in one of our UPSes.

The batteries have failed and everytime there is a power blip, the systems are crashing (specifically, the SAN switches reboot, which takes 5 minutes, much longer than the fileserver OSes will tolerate a "scsi transport timeout). The downtime should last < 30 minutes.

Jobs currently running and systems which are dependent on NFS should remain up during the outage.

A replacement for all UPS batteries was planned for the Rice Hall move downtime, but this can no longer wait.

We'll be conducting the battery swap as quickly as possible, and hope to do this in significantly less time than the 30 minute estimate. Dead batteries occasionally swell and can be difficult (slow) to remove.

Olsson Power Outage - 6/8/11 - 5:25AM-6:00AM

We will have a downtime this coming Wednesday, June 8, 2011 while the building power is shut down. We will be taking the department systems down beginning at 5:25AM and will bring them back up as soon as the power is returned.

We will be shutting down the core systems - the mail, web and file servers, as well as the interactives located in the machine room. We will not be shutting down any systems in Small Hall; they may experience problems while the NFS filesystems are unavailable, but they generally recover if the outage is not too long. User desktops - particularly those with UPSes are the same.

Although facilities has indicated that this downtime will take half an hour, the last time they did this it was about two minutes, and we did not wind up shutting things down.

PBS Upgrades - 2/15/11

The newest version of PBS has been released and our licensing for the old version expires shortly, so an upgrade is needed. This will also resolve issues with mpiexec.

Ubuntu 10.04LTS Available! - 7/26/10

Our CS-configured distribution of Ubuntu 10.04 LTS is now available. This is our new standard distribution, and we want to encourage folks to move to this as quickly as possible. All new desktop machines we deploy will have this distribution, and we plan to have upgraded all the shared department resources - the 'interactives' and the compute clusters - by the end of August.

There is an update to GCC and glibc in this upgrade; although this is a minor release update, it's quite common for that to cause 'breakage', and a couple of users have already reported issues to us. There may be other packages which are now deprecated or no longer available, or now have substitutes which need to be installed.

In order to facilitate testing, we have already upgraded the upper power nodes - interactives - please log on to power[4..6] and test your code and environment!. Please help us find what's missing or broken before we start pushing this out across the department!

We will not force upgrades on older Ubuntu 8.04 desktop boxes until security updates are no longer available, but we will not be able to provide updated packages for those systems either; Debian/Ubuntu packages of newer software versions (eg, the latest python, GCC or emacs). However, if you hold back your box because of incompatibilities with the newest glibc, etc. we encourage you not to allow the drift to get too big.

Policy change for music/video files on webserver - 2/11/10

In the past year, our webserver has become unresponsive on a couple of occasions as a result of music/video files in users' webspace being indexed by search engine Baidu. Baidu is the most popular search engine in China; when media files on our server get listed by them, we see a huge uptick in GET requests for these files. One file, found by Baidu this week, currently accounts for 30% of the GET requests on www.cs.virginia.edu. While the file was available for download, the webserver almost completely ran out of swap space and was nearly unresponsive.

To protect against this effect in the future, we've made two policy changes to the webserver:

• Configuring Apache, by default, not to serve particular media file types nor show them in directory listings. Currently, mp3 and rmvb (a popular Asian video format) files fall into this category; more may be added in the future, as needed. If you need to serve these files for legitimate research/instructional purposes, it's easy to override that restriction by adding to your .htaccess file:
  
  <Files *.mp3>
Order allow,deny
Deny from all
</Files>

• Adding 'Disallow' statements for mp3 and rmvb files to robots.txt. Not all major search spiders respect wildcarding in 'Disallow' (i.e., *.mp3), and we're unsure how Baidu handles that, so it remains to be seen whether this will be an effective measure.

Pompey - 1/20/10

Pompey has recently been removed from the department as a shared resource (including PBS queues). It is currently being used exclusively by a research group and will be returned to the general department pool as soon as possible.

MySQL Server Upgrade - 1/15/10

We recently ugraded MySQL to V. 5.1.39, 64-bit. We made this change to bring in InnoDB table support and to pick up some security fixes. We will do a more extensive update later in this year - to bring ourselves forward to the current 5.5 release. This is essentially a maintenance update.

Solaris System Perl Upgrade - 1/15/10

In order to accomodate some newer web based packages, we have upgraded the CS Solaris PERL to 5.10. All earlier versions remain and may be called explicitly. The default is now 64-bit Perl 5.10. If you are running Perl scripts from /usr/cs/bin/perl and they are suddenly missing a needed module which we'd previously installed for you, please send mail to root so that we can add that package.

By e-mail: root@cs.virginia.edu
Emergency hotline: 982-2271 (What constitutes an emergency?)
Who we are