Matlab Distributed Computing Server / Parallel Computing Toolbox

* work in progress - this documentation is subject to change *

Introduction 

The following document is in two sections, due to differences in the Distributed Computing Engine implementation in Matlab versions R2011b and R2015b.

The Matlab Distributed Computing Server (DCS) and Parallel Computing Toolbox (PCT) allow Matlab programs to be run non-interactively on the cluster compute nodes. 

This infrastructure allows a single Matlab client instance to submit Matlab tasks to the cluster batch scheduling system (SLURM).

These tasks can (currently) be independently distributed single-process tasks. The current limit is 32 tasks. (Parallel tasks, e.g. parfor loops, will be implemented later)

Importantly - given the occasional overbooking of Matlab interactive licences - it is possible to close the Matlab interactive client session while tasks are running on the cluster. Results can then be fetched from the remote tasks at a later time, thereby freeing up interactive licences for colleagues. 

Note that the following guide applies only to Matlab clients started on the cluster - local instances on client laptops or desktops aren't supported. Log in to the cluster with X-forwarding ("ssh -X") to get the Matlab GUI on your client machine.

Details about the current Matlab installation, number of licences, and how to access it are available here.

Distributed Matlab tasks - R2015b

** NB This section relates to Matlab version R2015b - please check your version! **

Loading the appropriate version:

module load matlab/R2015b

The following short example shows how to submit independent Matlab tasks as separate cluster jobs.

qos='short';
account='its';
path(strcat(getenv('MATLAB_ROOT'), '/toolbox/distcomp/examples/integration/slurm/shared'), path);
cluster = parallel.cluster.Generic('JobStorageLocation', '/home/linstead/tmp');
set(cluster, 'HasSharedFilesystem', true);
set(cluster, 'ClusterMatlabRoot', '');
set(cluster, 'OperatingSystem', 'unix');
set(cluster, 'GetJobStateFcn', @getJobStateFcn);
set(cluster, 'DeleteJobFcn', @deleteJobFcn);
set(cluster, 'IndependentSubmitFcn', {'independentSubmitFcn', qos, account});
j = createJob(cluster)
t1 = createTask(j, @rand, 1, { {3 3} {5 5} {9 9} });
submit(j);
get(j);
results=getAllOutputArguments(j);
results{:}

This example generates a set of three tasks (creating 3x3, 5x5 and 9x9 matrices of random numbers), each of which will be submitted via SLURM as cluster jobs. You should set 'qos' and 'account' as appropriate for your job type, as per the SLURM documentation. Other modifications are indicated in the bold sections. Note that JobStorageLocation must be different to that for previous versions of Matlab, since the job format has changed in later versions.

Distributed Matlab tasks - R2011b

** NB This section relates to Matlab version R2011b - please check your version! **

Load Matlab

When a Matlab task is submitted it is translated to a SLURM job. SLURM will then schedule the job according to (among other factors) the availability of worker licences. 

The worker licences are managed as SLURM resources, so if no licence is available, the job will be held in the Idle state until one becomes free, rather than failing at runtime for lack of a licence. (For more information about SLURM, see the cluster user guide [2]) 

Running "scontrol show licenses" on the cluster command-line will show the availability of floating resources.

For this version, load the appropriate module:

module load matlab/R2011b

Setup

Initial setup requires Matlab paths to be set. This can be done in one of two ways: 

1) within Matlab: 

path(strcat(getenv('MATLAB_ROOT'), '/toolbox/distcomp/slurm'), path);

2) on the command-line before starting Matlab, or in ~/.profile, to make the setting persistent: 

export MATLABPATH=/p/system/packages/matlab/R2011b/toolbox/distcomp/slurm:$MATLABPATH 

Example Use 

The following example demonstrates setup and distribution of multiple independent tasks. This code can be downloaded here.

% JOB SETUP
% This section can go into your .matlab/startup.m, but Matlab will
% report errors with license checkout. These errors don't seem to
% affect runs. (Any Matlab experts know why this happens?)

% Set the path to be able to find the SLURM interface functions
path(strcat(getenv('MATLAB_ROOT'), '/toolbox/distcomp/slurm'), path);

% We use a Matlab generic scheduler, customised for SLURM 
sched = findResource('scheduler', 'type', 'generic');
set(sched, 'ClusterMatlabRoot', getenv('MATLAB_ROOT'));

% set the location for storing job-specific data
% I'm using /p/tmp/linstead/matlabtmp. This directory needs to exist before 
% running
set(sched, 'DataLocation', strcat('/p/tmp/', getenv('USER'), '/matlabtmp'));

% set the SLURM job QOS/class (e.g. short, medium, long, io)
qos = 'short';

% set the SLURM job account, i.e. your project group. Mine is 'its', yours 
% will differ.
account = 'its';

% specify the functions which implement the interfaces between Matlab and 
% SLURM. Note that 'qos' and 'account' are passed to the submit function. 

set(sched, 'SubmitFcn', {'slurmDistributedSubmitFcn', qos, account});
set(sched, 'DestroyJobFcn', 'slurmDestroyJob');
set(sched, 'DestroyTaskFcn', 'slurmDestroyTask');

% END JOB SETUP

% EXAMPLE JOB
% Create a job attached to our scheduler
j = createJob(sched);

% create some tasks, here some random matrices: 
t1 = createTask(j, @rand, 1, { {3 3} {5 5} {9 9} });
t2 = createTask(j, @rand, 1, {2 2});
t3 = createTask(j, @rand, 1, {6 6});

% similarly, jobs can be prepared in a loop: 
for ii = 1:5
  % get lists of prime numbers 
  t(ii) = createTask(j, @primes, 1, {ii*10});
end

% ...and submit it.
submit(j)
% The tasks for this job are submitted to the cluster using 
% SLURM's sbatch command via a job wrapper:
%
% In slurmDistributedSubmitFcn
% /p/system/slurm/bin/sbatch --ntasks 1 --qos short --account its 
%   --workdir /p/tmp/linstead/matlabtmp 
%   '/p/system/packages/matlab/R2011b/toolbox/distcomp/slurm/distributedJobWrapper.sh'
% Task: 1; jobID: 41378
% /p/system/slurm/bin/sbatch --ntasks 1 --qos short --account its 
%   --workdir /p/tmp/linstead/matlabtmp 
%   '/p/system/packages/matlab/R2011b/toolbox/distcomp/slurm/distributedJobWrapper.sh'
% Task: 2; jobID: 41379
% /p/system/slurm/bin/sbatch --ntasks 1 --qos short --account its 
%   --workdir /p/tmp/linstead/matlabtmp 
%   '/p/system/packages/matlab/R2011b/toolbox/distcomp/slurm/distributedJobWrapper.sh'
% Task: 3; jobID: 41380

% check job status
get(j)

% Job status output looks like this:
%       Configuration: ''
%                Name: 'Job11'
%                  ID: 11
%            UserName: 'linstead'
%                 Tag: ''
%               State: 'finished'
%          CreateTime: 'Thu Jan 28 15:19:45 CET 2016'
%          SubmitTime: 'Thu Jan 28 15:19:55 CET 2016'
%           StartTime: 'Thu Jan 28 14:19:58 GMT 2016'
%          FinishTime: 'Thu Jan 28 14:20:03 GMT 2016'
%               Tasks: [3x1 distcomp.simpletask]
%    FileDependencies: {0x1 cell}
%    PathDependencies: {0x1 cell}
%             JobData: []
%              Parent: [1x1 distcomp.genericscheduler]
%            UserData: []

% if it's finished, get results
results=getAllOutputArguments(j);

% show results
results{:}

% At this point, you can quit the interactive Matlab session and fetch the 
% output data from your tasks at a later time. 

% To cancel a task (i.e. a SLURM job) and clean up: 
destroy(t1); 

% You can also cancel a task manually, by running 'scancel <jobID>' on the 
% command-line. 

% The following is only necessary if you need to wait for data to be 
% returned. It will block further interaction until SLURM has 
% returned and thus not recommended for jobs taking longer than a few seconds. 

waitForState(j); 

% to see the current state of the job
get(j); 

% fetch all return data
results = getAllOutputArguments(j); 

% see the results: 
results{:} 

% ans =
%
%    0.9173    0.4809    0.4626
%    0.6839    0.4612    0.8009
%    0.8661    0.1562    0.2155
%
%

% to cancel the entire job and all associated tasks, and clean up: 
destroy(j); 

% Unless you run the destroy(job) function, job state for previous
% submissions is kept on disk at the location specified by the
% 'DataLocation' parameter set above. To fetch previous jobs: 
sched = findResource('scheduler','type','generic'); 

% set the DataLocation for the previous runs: 
set(sched, 'DataLocation', strcat('/p/tmp/', getenv('USER'), '/matlabtmp'));

% get a list of jobs: 
all_jobs = get(sched, 'Jobs');

% jobs can be accessed by index... 
job9 = all_jobs(9); 

% ... and queried as before: 
results = getAllOutputArguments(job9);

OLD DOCUMENTATION BELOW HERE! NOT IMPLEMENTED FOR R2011b!!

Parallel Matlab tasks with communication.

Matlab parallel applications are split between workers known as labs. In much the same way as MPI provides MPI_Comm_rank() and MPI_Comm_size() for controlling communication between it's tasks, the PCT provides the labindex and numlabs variables. (In contrast, tasks in loops automatically parallelised with parfor may not communicate).

The configuration for parallel jobs is similar to the above, with the addition of the following:

% specify the number of parallel processes to use
np = 8;

% specify the custom parallel submit function
set(sched, 'ParallelSubmitFcn', {@loadlParallelSubmitFcn, class, group});

% create a parallel job
pjob = createParallelJob(sched);

% file dependencies are the files that will be copied to the executing nodes
% ...in this case my naive parallel Matlab function for counting prime numbers
set(pjob, 'FileDependencies', {'/home/linstead/pprime.m'});

% using the same 'np' in these two variables means you want to run on 
% exactly that many processors. You can also specify a lower and upper
% value, and Matlab will use as many workers as currently available
set(pjob, 'MaximumNumberOfWorkers', np);
set(pjob, 'MinimumNumberOfWorkers', np);

% create a task in this job. createTask takes the parameters:
% job, function, number_of_output_arguments, {inputargument1, ...}
t = createTask(pjob, @pprime, 2, {1, 100000})

% and finally, submit the job.
submit(pjob);

The function being run here (pprime.m) counts the prime numbers between two bounds. Since the input range [lower..upper] can be split into independent chunks, this code is a prime :) candidate for parallelisation. It returns a pair: the count itself and the time the loop took to execute. In the parallel enviroment, the output will be an array of pairs of size numlabs, each pair containing the total and the time taken on each lab.

pprimes.m

function [total, time] = pprime(lower, upper)

  % somewhere to store the sum for the local labindex:
  localsum = 0;

  % calculate the lower and upper bounds of the chumk this labindex
  % will count.
  % note: labindex goes from 1..n, unlike MPI rank which is 0-based
  chunksize = idivide(int32(upper-lower), numlabs);
  mylower = lower + ((labindex-1) * (chunksize+1));
  myupper = mylower + chunksize;

  % start a timer to benchmark the main loop
  ticID = tic;

  % loop over the local bounds
  for i = mylower : myupper
    isprime = 1; % initially TRUE
    if i <= 1
        isprime = 0; % FALSE
    elseif i == 2
        isprime = 1; % TRUE
    else
        for j = 2 : i-1
            if ( mod (i, j) == 0 )
                isprime = 0; %FALSE
            end
        end
    end
    if isprime == 1
        % here we increment just our local sum
        localsum = localsum + 1;
    end
  end

  % stop the timer
  time = toc(ticID);

  % update the global total based on all the localsums
  total = gplus(localsum);
end

Parallel Matlab tasks with 'parfor'

The pprime example doesn't need to take advantage of the ability of labs to communicate (eg. via labBroadcast). Each iteration of the main loop is independent of the others so the code could be simplified with the parfor loop construct.

Parfor requires a matlabpool which in turn requires a configuration on the client. Let's do that:

Setting up a matlabpool configuration

In the Matlab GUI client, select menu item "Parallel", then "Manage Configurations".

Select 'File', 'New', then 'generic' from the list of options

(A warning about configuring cluster machines may appear at this point, just click OK)

"Configuration name" and "Description" can be set to whatever values you prefer. The name will be used when starting a matlabpool job. The rest of the values can be filled in as below (changing any paths to your own scratch or home directory. FileDependencies can be set per-job rather than per-configuration, too, so you can leave that field empty.)

The Scheduler tab:

 The Jobs tab:

Running a parfor job

The code for setting up schedulers and jobs is almost identical to the pprime example, but this time replace

pjob = createParallelJob(sched);

with

pjob = createMatlabPoolJob('configuration', 'loadl')

and the location of our simplified primes function is set thus:

set(pjob, 'FileDependencies', {'/home/linstead/prime.m'});

Submitting the job then opens a matlabpool of the required size, runs the job, and shuts down the pool.

prime.m

function [total time] = prime(lower, upper)
%% prime counts the prime numbers between lower and upper bounds
  total = 0;
  % start a timer to benchmark the main loop
  ticID = tic;
  parfor i = lower : upper
    isprime = 1; % TRUE
    if i <= 1
        isprime = 0; % FALSE
    elseif i == 2
        isprime = 1; % TRUE
    else
        for j = 2 : i-1
            if ( mod (i, j) == 0 )
                isprime = 0; %FALSE
            end
        end
    end
    if isprime == 1
        total = total + 1;
    end
  end
  % stop the timer
  time = toc(ticID);
end

It's also possible to open a 'loadl' matlabpool interactively. It's recommended that you only use this for testing, since a pool opened this way blocks a client licence, plus np worker licences, plus np cluster processors until the pool is manually closed!

Known problems/bugs:

Because the Distributed Server/Parallel Computing Toolbox does not directly support LoadLeveler, this configuration has required customisation of existing generic interfaces. As such, not all of the features documented by Mathworks may be available here. Feedback on your experiences using these tools would be much appreciated.

References: 

[1] Parallel Computing Toolbox: http://www.mathworks.com/help/pdf_doc/distcomp/distcomp.pdf 

[2] PIK Cluster Computing Guide: iDataPlex Cluster User Guide - January 2012