Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
public:lofar_cluster [2010-03-09 12:40] – Arno Schoenmakers | public:lofar_cluster [2017-05-01 14:10] – [CEP4] Arno Schoenmakers | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== The LOFAR Cluster | + | ====== The LOFAR Clusters |
- | This page describes the LOFAR cluster | + | This page describes the LOFAR clusters |
- | We welcome authorized users on the cluster. For the time being the users are bound to only a part of the cluster. This part is called a // | + | We welcome authorized users on the LOFAR clusters. |
===== User Access ===== | ===== User Access ===== | ||
Line 13: | Line 13: | ||
> ssh -X portal.lofar.eu | > ssh -X portal.lofar.eu | ||
</ | </ | ||
- | Please send an email to //grit at astron.nl// or //h.paas | + | Please send an email to //grit at astron.nl// or //rbokhorst |
- | When you are logged in for the first time, you'll have an empty home directory. | + | When you are logged in for the first time, you'll have an empty home directory. |
+ | |||
+ | To get onto the CEP3 cluster, you first have to login at one of the two frontend nodes '' | ||
+ | |||
+ | For more sophisticated useage of '' | ||
==== LOGIN environment ==== | ==== LOGIN environment ==== | ||
Line 23: | Line 27: | ||
**DON' | **DON' | ||
- | * Store data on the '' | + | * :!: Store data in your '' |
- | * Pump around large amounts of data | + | * Pump around large amounts of data between nodes. |
* Leave unused data on the systems. Clean it up, so others can make use of the available disk space. | * Leave unused data on the systems. Clean it up, so others can make use of the available disk space. | ||
| | ||
Line 31: | Line 35: | ||
+ | ===== Contributed Software ===== | ||
- | ===== LOFAR Cluster layout (brief) ===== | + | Some groups or people would like their favourite tools or pipelines added to the cluster |
- | + | ||
- | The Lofar cluster is devided into 8 subclusters. Each subcluster is a processing cluster for a specific commsissioning group. There are 72 compute nodes (named '' | + | |
- | + | ||
- | + | ||
- | + | ||
- | ==== Frontend ==== | + | |
- | + | ||
- | A frontend has 2 Intel Xeon L5420 quad core processors, 16GB internal meomory, 2 GbE interfaces and 2TB disks in RAID5 configuration. | + | |
- | There are actually two identical frontends: '' | + | |
- | + | ||
- | ==== Processing units ==== | + | |
- | + | ||
- | The compute elements have 2 Intel Xeon L5420 quad core processors, 16GB internal meomory, 2 GbE interfaces and 1TB disks in RAID0 configuration. They can be accessed by secure shell and they are grouped. | + | |
- | + | ||
- | + | ||
- | ==== Storage Units ==== | + | |
- | + | ||
- | The storage nodes are HP DL180G5 boxes, having 2 Intel Xeon L5420 quad core processors, 16GB internal memory, 6 GbE network interfaces and 24TB disks. | + | |
- | The disks are devided into 4 partitions of 6 disks each, set up in RAID5 configuration. The XFS filesystems are called "/ | + | |
- | + | ||
- | ===== Available Subclusters ===== | + | |
- | + | ||
- | The current subcluster assignment is: | + | |
- | + | ||
- | lfe001: | + | |
- | + | ||
- | This script shows the subcluster definitions | + | |
- | + | ||
- | sub lce-nodes | + | |
- | ==== ========= | + | |
- | sub1 lce001-lce009 | + | |
- | sub2 lce010-lce018 | + | |
- | sub3 lce019-lce027 | + | |
- | sub4 lce028-lce036 | + | |
- | sub5 lce037-lce045 | + | |
- | sub6 lce046-lce054 | + | |
- | sub7 lce055-lce063 | + | |
- | sub8 lce064-lce072 | + | |
- | + | ||
- | The lce-nodes ar ethe compute/ | + | |
- | + | ||
- | ===== Old Obsoleted Stuff (only here for completeness) ===== | + | |
- | + | ||
- | ===== The lioffen processing cluster ===== | + | |
- | + | ||
- | + | ||
- | ==== Hardware setup ==== | + | |
- | + | ||
- | The 10-node post processing cluster has '' | + | |
- | + | ||
- | Each node has 2-Gbyte of physical RAM, and another 2-Gbyte swap space. Each node contains two AMD Opteron 2-GHz CPUs with 1 Mbyte cache. They are currently running Ubuntu 7.10 (Gutsy). | + | |
- | + | ||
- | ==== Usage ==== | + | |
- | + | ||
- | The offline nodes are used for processing the CS1 data products that are stored on the lifs-nodes. To this purpose, all disks of all lifs-nodes are NFS mounted on all offline nodes. | + | |
- | + | ||
- | + | ||
- | ==== Environment | + | |
- | + | ||
- | There is a login environment with some startup-scripts that make it easier to use several available tools. Please see [[public: | + | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | ==== Disks and NFS mounts | + | |
- | + | ||
- | The big disks on the off-line storage nodes ('' | + | |
- | + | ||
- | On each node is a 360 Gbyte ''/ | + | |
- | + | ||
- | Finally, there is the NFS mounted ''/ | + | |
- | + | ||
- | + | ||
- | ==== Applications | + | |
- | + | ||
- | Most applications can be found in directory ''/ | + | |
- | * See [[public: | + | |
- | + | ||
- | ===== listfen 32-bit cluster ===== | + | |
- | + | ||
- | The old 32 bit offline cluster has been removed. Only frontend " | + | |
- | + | ||
- | ==== LIFS storage nodes ==== | + | |
- | + | ||
- | There are 12 old 1TB storage nodes, lifs001 till lifs012. They are still in use for development and testing. Lifs001 till lifs008 are used by the Observatory to store recent observations. Lifs009 til lifs012 hold datasets that are a result of some kind of processing. We are already observing with the new cluster hardware. The lifs nodes will be removed the end of November. | + | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | ==== Offline processing clusters ==== | + | |
- | + | ||
- | The disks of the offline storage nodes are NFS mounted on all these offline nodes. These systems are in use for offline processing (i.e. when data has been transfered from the central storage facility). | + | |
- | + | ||
- | + | ||
- | ==== Offline storage cluster ==== | + | |
- | + | ||
- | This cluster acts as a temporary data buffer. Data is copied from the online storage systems to these systems, so that the online systems can be used to store new data. These systems are accessable from the offline processing clusters, allowing post-processing of the data. | + | |
- | More information can be found [[databuffercluster|on this page]]. | + | |
- | + | ||
- | ====== Bootleg mechanism ====== | + | |
- | + | ||
- | The new cluster is subject | + | |
- | + | ||
- | ==== Central administration ==== | + | |
- | + | ||
- | The way the CEP cluster has been set up is as follows: | + | |
- | We are using Ubuntu Linux LTS-8.04 as Operating System within the whole cluster. Bootleg takes care of all of the server management in the cluster by creating new images, administering servers, keeping track of updates and correct start up of servers by sending them new images when it finds outdated versions at boot time. | + | |
- | + | ||
- | ==== Profiles ==== | + | |
- | + | ||
- | Bootleg also administers machine profiles: Depending on the function of the server in the cluster it needs other settings and profiles. | + | |
- | This management strategy implies that we have to keep track of all programs installed in the Linux system itself, so we will be able to add them to the image. We plan to build fresh images every month (at the maintenance days). | + | |
- | + | ||
- | ==== Instantanious changes ==== | + | |
- | + | ||
- | We are also able to respond quickly on extra-program demands for system programs. Within bootleg there is a mechanism to roll-out extra programs through the cluster from a central administration point (CAP). From the CAP we order to deploy a program or a settings-change and after 1 minute the whole cluster will be updated with the change. So if you need extra system programs, simply ask, and you get it almost instantly on every server in the cluster! | + | |
- | + | ||
- | ==== Application programs ==== | + | |
- | For an application program change or addition there is another method. | ||
- | Of course you develop your programs in your own user space and/or svn repositories. So you have completely control over all of this. | ||
- | Program releases for the cluster itself reside on a central disk. | ||
- | Normally we (as system administrators) are not involved in the application program deployment. | ||
- | Only if a completely new package is added under a new directory name we | ||
- | need to make this available in the cluster by connecting it in the O.S. | ||
- | via a link to the newly created package name. | ||
- | So there is a clear interface between system programs and applications: | ||
- | The connection between them is at a well defined point, the /opt directory is intended for this. | ||
- | The same goes for database content (mysql, postgresql, ...). | ||
- | Although it might involve OS program installation and services, the database content itself will never be on the Linux system image but stored on a separate data server. | ||
- | Again: Simply ask, we can connect your program and make it available cluster-wide. | + | ===== CEP3 ====== |
+ | CEP3 is the processing cluster that is available to LOFAR users that need additional processing and storage of their data. Details on requesting access and its usage are [[cep3: | ||
+ | ===== CEP4 ====== | ||
+ | CEP4 is the main storage and processing cluster for online applications. It is not accessible by other users than LOFAR staff. Information on CEP4 can be found [[cep4: |