request.dl.ac.uk

mano

mano.dl.ac.uk is an IBM Blue Gene/L.

The BG/L hardware itself comprises 1024 nodes of dual-core 700MHz PowerPC chips with the second cpu usually dedicated i/o and communications. The frontend (or login) node is a p5-520Q with 4x1.5GHz processors, 16GB RAM and running SLES9 and this is supplemented with an identical service node for system control. GPFS is provided through two p5-505 servers each with 2x1.5GHz processors and 4GB RAM.

This configuration is changing with the advent of legion, our first frame of Blue Gene/P, which is being installed during October 2007. A good entry to /P information is at ibm.com.

You should read the information on passwords and then email bluegene@request.dl.ac.uk if you would like an account on either Bluegene/L or Bluegene/P

For up-to-date information, please log into the machine, this page will be moving soon

Message of the Day - Bluegene/L

+---------------------------------------------------------+
|                 -= Current News =-                      |
|                                                         |
|  End of Life for Bluegene/L  March 2010                 |
|                                                         |
|  The original EOL date of March has been extended to    |
|  end April and possibly longer if there is demand.      |
|  Note that the BG/L system is no longer under           |
|  maintenance and that fixes will be on reasonable       |
|  endeavours only                                        |
|                                                         |
|  2010-06-15 CPM                                         |
|    Due to the low usage levels, we have powered off the |
|  second midplane, if you wish to run a job requiring    |
|  more than 512 compute nodes (1024 cores in VN mode),   |
|  then please let me know via bluegene@request.dl.ac.uk  |
|                                                         |
|  2010-07-13 CPM                                         |
|    Upgrade to LoadLeveler 3.4.3.14 completed, cluster   |
|  submission now disabled, some issues with              |
|  over-allocation of resources are being investigated    |
|                                                         |
|                                                         |
|                -= Next Maintenance =-                   |
|  2010-07-02 CPM                                         |
|     Brief outage to attempt to resolve overallocation   |
|  of resources to jobs.                                  |
|                                                         |
|                                                         |
|                                                         |
\_________________________________________________________/

Message of the Day - Bluegene/P

+------------------------------------------------------------------+
|                      -= Current News =-                          |
|                                                                  |
|  2010-06-30 - CPM                                                |
|      Following the sucessful addition of job limits,             |
|  and to further improve the throughput of the system             |
|  we will be applying restrictions on when large jobs,            |
|  defined as 512 nodes and above, will, smaller jobs              |
|  will also be affected as we plan to shorten the                 |
|  maximum permissible run time during the day.                    |
|                                                                  |
|  ALL USERS ARE ADVISED TO READ THE FOLLOWING NOTE ABOUT          |
|  THE NEW QUEUES                                                  |
|  http://syswiki.dl.ac.uk/public/hosts/bluegene/queueing_regime/  |
|                                                                  |
|  2010-07-15 - CPM                                                |
|      An issue with the filter generating impossible queues has   |
|  now been resolved.                                              |
|                                                                  |
|  2010-07-21 - CPM                                                |
|      Current hardware issues are forcing job failures, failed    |
|  hardware has been mapped out, and will be replaced shortly      |
|                                                                  |
|                                                                  |
|                    -= Next Maintenance =-                        |
|                                                                  |
|  2010-06-31                                                      |
|    * New queueing structure will be tested and applied           |
|                                                                  |
\__________________________________________________________________/

Libraries

Some information on BG libraries from Jonathan Follows at IBM:

IBM's Engineering and Scientific Subroutine library, ESSL, is installed on the BG/L front end node. The relevant files are /opt/ibmmath/essl/4.2/lib/libesslbg.a and /opt/ibmmath/lib/libesslbg.a so linking with "-lesslbg" will pick up IBM's optimised versions of the subset of LAPACK/SCALAPACK etc.

(Any compilation using the recipe "-lessl" will pick up the libraries optimised for running on the Linux/POWER Front End Node itself, which is no use at all for code running on Blue Gene hardware.)

The ESSL "Guide and Reference" manual may be found here or here. Note that ESSL Version 4.2 is appropriate for BG/L, whereas Version 4.3 is the one for BG/P. The only "user difference" with BG/P is that "-lesslsmpbg" will be an additional option for versions of routines which use OpenMP on the BG/P nodes.

DGEMM, DCOPY, BLAS, LAPACK are provided by ESSL. Many of the ESSL routines automatically make use of the "double hummer" Blue Gene/L floating point unit. MASS is a set of acceleration libraries provided with the XL compilers themselves. Documentation may be found at IBM - see Chapter 3. Instructions for downloading and building FFTW is something to be left for the student until we have the opportunity to install the libraries "centrally" on BG.

LoadLeveler

LoadLeveler is now available on mano. Sample submit script:

#@ arguments = -np 8 -exe sanity.rts

#@ input = /dev/null
#@ output = out/$(jobid).out
#@ error = out/$(jobid).err

#@ notification = complete
#@ queue

Note that mpirun is the only supported way of starting jobs. The #@ arguments line contains the usual mpirun arguments. There are currently no class or queuing restrictions other than a 12 hour wallclock limit on jobs.

mpirun

The use of mpirun for starting jobs directly outside LoadLeveler is not longer recommended.

Further documentation

• Sysadm: Configuration notes.