Role(s): Service Provider

Component(s):   Job Scheduler

License: Apache 2.0

Usage Instructions

Installation

Software Installation

Job Scheduler Toolkit API

Usage Instructions

This toolkit has the functionality of managing resources and jobs submitted in a cluster. It also allows the advance reservation of its resources. The components of this toolkit can be seen in  REF _Ref2268687893 \h Figure 15.

Job Scheduler is an open implementation of SOAP Web Service multi-user access and policy-based job control using OGF Distributed Resource Management Application API (DRMAA) routines supported by distributed resource management systems like Sun Grid Engine (SGE), LSF, PBSPro, Torque, Condor or Apple XGrid. OpenDSP acts above the Distributed Resource Management (DRM) System layer providing remote access to computational resources.

The main role of the Job Scheduler is to manage jobs within a single cluster and to provide remote access to cluster resources and jobs to other BREIN components. The Job Scheduler consists of a commodity DRM system (such as SGE, LSF etc.) and a component that provides secure remote access to this system.

DRMAA Service Provider (DSP)  is an open architecture implementation of SOAP Web Service for multi-user access and policy-based job control routines by various Distributed Resource Management systems. It uses DRMAA to communicate with the underlying DRM. DSP has been designed and implemented in such a way that it supports different plug-ins and modules for external communication. Consequently, it can be used and integrated with various authentication, authorization and accounting infrastructures as well as external services.

The following lines explain how the toolkit is integrated with end users’ systems. These are the steps needed to integrate the Job Scheduler in a system:

1.Extended the queue systems of the cluster with a DRMAA compliant module or install a compliant one if it cannot be extended.

2.Install the Job Scheduler in the front-end of the cluster.

3.Configure the Job Scheduler.

4.Choose the authentication mechanism: none (only in protected networks), SSL, GSI, User Name Token Profile.

1.Choose an authorization model, and define how to map client’s identity (e.g. X509 DN) to a local user account (each job must be submitted as some local user using for example a map-file).

2.Configure the NP module (the URL of the NP Provider is defined in this step).

3.Start the JS

5.Integration with the Provider Resource Manager.

1.Acquire the client credentials (e.g. X509 certificate) if needed.

2.Use the Job Scheduler client, or the Job Scheduler Java SDK, to submit/monitor and control jobs remotely directly from the Resource Manager component (the Resource Manager must create valid JSDL documents describing the jobs to be submitted to the Job Scheduler).

The following dependency diagram (Figure 16) presents the most typical interaction steps between DSP Client and DSP. Note that usually the DSP Client has to implement interfaces for WS-Subscriber and WS-NotificationConsumer, however in BREIN the Infrastructure Sensor component will be used to pass notification messages to agent-based components.

The Job Scheduler is dependent on an underlying DRM system. The DSP submits jobs to this system using its API (using implementation of OGF DRMAA). The SLRM submits jobs to DSP using the JSDL language to describe a job. An advance reservation mechanism is implemented by the Job Scheduler and it is integrated with the SLRM.

Additionally, the Job Scheduler can be integrated with the Notification Support toolkit, see  REF _Ref226868735 \h Figure 16  Installation Manual. In this case, the Job Scheduler would send notifications about requested events including changes of job states.

New modules are optional to enable semantic description of scheduling policies and objectives, and self-adaptation of a scheduling system in case of selected events.

Job Scheduler Installation

Installation Requirements

These are the prerequisites of every component in the toolkit:

·                              Hardware: It is aimed to be used in a cluster, although a minimal installation can be performed in a Linux computer

·                              Software: Linux (Unix), OpenSSL 0.97 or higher, libXML 2.4.0 or higher, unixODBC 2.2.0 or higher or iODBC 3.0.0 or higher, postgreSQL / mySql, Sun Grid Engine

Deployment Tips

The Job Scheduler toolkit should be deployed in a cluster, although it can be deployed in a single computer that accomplishes with the prerequisites. The fully distributed configuration is shown in  REF _Ref226868789 \h Figure 12.

All the components related with the Job Scheduler can be installed in a single computer for testing, but it would mean that no more than four jobs could be running at the same time (depending on the performance provided by the computer). The recommended deployment in a production environment is to have a external cluster of computers and the Job Scheduler in the front-end node of the cluster (this is a requirement in this configuration).

Software Installation

The installer of this toolkit can be any of the standard package managers included in a Red Hat distribution. In the Figure 7 a standard package manager is shown, where the needed packages is selected for installation. The following four packages must be held in the package server, the package manager will install the software accessing this server:

smoa-core-2.0.3-3.i386.rpm

smoa-comp-2.1.0-3.i386.rpm

libxml2-2.6.27-1.i386.rpm

libxml2-python-2.6.27-1.i386.rpm

Job Scheduler Toolkit API

see  JobScheduler API