forked from SchedMD/slurm
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathRELEASE_NOTES
235 lines (215 loc) · 12.2 KB
/
RELEASE_NOTES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
RELEASE NOTES FOR SLURM VERSION 16.05
27 April 2016
IMPORTANT NOTES:
ANY JOBS WITH A JOB ID ABOVE 2,147,463,647 WILL BE PURGED WHEN SLURM IS
UPGRADED FROM AN OLDER VERSION! Reduce your configured MaxJobID value as needed
prior to upgrading in order to eliminate these jobs.
If using the slurmdbd (Slurm DataBase Daemon) you must update this first.
The 16.05 slurmdbd will work with Slurm daemons of version 14.11 and above.
You will not need to update all clusters at the same time, but it is very
important to update slurmdbd first and having it running before updating
any other clusters making use of it. No real harm will come from updating
your systems before the slurmdbd, but they will not talk to each other
until you do. Also at least the first time running the slurmdbd you need to
make sure your my.cnf file has innodb_buffer_pool_size equal to at least 64M.
You can accomplish this by adding the line
innodb_buffer_pool_size=64M
under the [mysqld] reference in the my.cnf file and restarting the mysqld. The
buffer pool size must be smaller than the size of the MySQL tmpdir. This is
needed when converting large tables over to the new database schema.
Slurm can be upgraded from version 14.03 or 14.11 to version 15.08 without loss
of jobs or other state information. Upgrading directly from an earlier version
of Slurm will result in loss of state information.
If using SPANK plugins that use the Slurm APIs, they should be recompiled when
upgrading Slurm to a new major release.
HIGHLIGHTS
==========
-- Implemented and documented PMIX protocol which is used to bootstrap an
MPI job. PMIX is an alternative to PMI and PMI2.
-- Change default CgroupMountpoint (in cgroup.conf) from "/cgroup" to
"/sys/fs/cgroup" to match current standard.
-- Add Multi-Category Security (MCS) infrastructure to permit nodes to be bound
to specific users or groups.
-- Added --deadline option to salloc, sbatch and srun. Jobs which can not be
completed by the user specified deadline will be terminated with a state of
"Deadline" or "DL".
-- Add an "scontrol top <jobid>" command to re-order the priorities of a user's
pending jobs. May be disabled with the "disable_user_top" option in the
SchedulerParameters configuration parameter.
-- Added new job dependency type of "aftercorr" which will start a task of a
job array after the corresponding task of another job array completes.
-- Add --gres-flags=enforce-binding option to salloc, sbatch and srun commands.
If set, the only CPUs available to the job will be those bound to the
selected GRES (i.e. the CPUs identifed in the gres.conf file will be
strictly enforced rather than advisory).
-- Added wrappers for LSF/OpenLava commands
-- Added Grid Engine options to qsub command wrapper.
RPMBUILD CHANGES
================
-- Remove all *.la files from RPMs.
-- Implemented the --without=package option for configure.
CONFIGURATION FILE CHANGES (see man appropriate man page for details)
=====================================================================
-- New configuration parameter NodeFeaturesPlugins added.
-- Change default CgroupMountpoint (in cgroup.conf) from "/cgroup" to
"/sys/fs/cgroup" to match current standard.
-- Introduce a new parameter "requeue_setup_env_fail" in SchedulerParameters.
If set, a job that fails to setup the environment will be requeued and the
node drained.
-- The partition-specific SelectTypeParameters parameter can now be used to
change the memory allocation tracking specification in the global
SelectTypeParameters configuration parameter. Supported partition-specific
values are CR_Core, CR_Core_Memory, CR_Socket and CR_Socket_Memory. If the
global SelectTypeParameters value includes memory allocation management and
the partition-specific value does not, then memory allocation management for
that partition will NOT be supported (i.e. memory can be over-allocated).
Likewise the global SelectTypeParameters might not include memory management
while the partition-specific value does.
-- Split partition's "Priority" field into "PriorityTier" (used to order
partitions for scheduling and preemption) plus "PriorityJobFactor" (used by
priority/multifactor plugin in calculating job priority, which is used to
order jobs within a partition for scheduling). If only "Priority" is
specified, that value will be used for both the "PriorityTier" and
"PriorityJobFactor" value.
-- New configuration file "knl.conf" added specifically for Intel Knights
Landing processor support.
-- Make it so jobs/steps track ':' named gres/tres, before hand gres/gpu:tesla
would only track gres/gpu, now it will track both gres/gpu and
gres/gpu:tesla as separate gres if configured like
AccountingStorageTRES=gres/gpu,gres/gpu:tesla
-- Add TCPTimeout option to slurm[dbd].conf. Decouples MessageTimeout from TCP
connections.
-- Added SchedulingParameters option of "bf_min_prio_reserve". Jobs below
the specified threshold will not have resources reserved for them.
-- Add SchedulerParameter "no_env_cache", if set no environment cache will be
used when launching a job, instead the job will fail and drain the node if
the environment isn't loaded normally.
-- Remove the SchedulerParameters option of "assoc_limit_continue", making it
the default value. Add option of "assoc_limit_stop". If "assoc_limit_stop"
is set and a job cannot start due to association limits, then do not attempt
to initiate any lower priority jobs in that partition. Setting this can
decrease system throughput and utlization, but avoid potentially starving
larger jobs by preventing them from launching indefinitely.
-- Rename partition configuration from "Shared" to "OverSubscribe". Rename
salloc, sbatch, srun option from "--shared" to "--oversubscribe". The old
options will continue to function. Output field names also changed in
scontrol, sinfo, squeue and sview.
-- Add TopologyParam option of "TopoOptional" to optimize network topology
only for jobs requesting it.
-- Configuration parameter "CpuFreqDef" used to set default governor for job
step not specifying --cpu-freq (previously the parameter was unused)
-- Use TaskPluginParam for default task binding if no user specified CPU
binding. User --cpu_bind option takes precident over default. No longer
any error if user --cpu_bind option does not match TaskPluginParam.
COMMAND CHANGES (see man pages for details)
===========================================
-- sbatch to read OpenLava/LSF/#BSUB options from the batch script.
-- Add sbatch "--wait" option that waits for job completion before exiting.
Exit code will match that of spawned job.
-- Job output and error files can now contain "%" character by specifying
a file name with two consecutive "%" characters. For example,
"sbatch -o "slurm.%%.%j" for job ID 123 will generate an output file named
"slurm.%.123".
-- Increase default sbcast buffer size from 512KB to 8MB.
-- Add "ValidateTimeout" and "OtherTimeout" to "scontrol show burst" output.
-- Implemented the checking configuration functionality using the new -C
options of slurmctld. To check for configuration errors in slurm.conf
run: 'slurmctld -C'.
-- Burst buffer advanced reservation units treated as bytes (per documentation)
rather than GB.
-- Add "features_act" field (currently active features) to the node
information. Output of scontrol, sinfo, and sview changed accordingly.
The field previously displayed as "Features" is now "AvailableFeatures"
while the new field is displayed as "ActiveFeatures".
-- Enable sbcast data compression logic (compress option previously ignored).
-- Add --compress option to srun command for use with --bcast option.
-- Added "sacctmgr show lostjobs" to report any orphaned jobs in the database.
-- Add reservation flag of "purge_comp" which will purge an advanced
reservation once it has no more active (pending, suspended or running) jobs.
-- Add ARRAY_TASKS mail option to send emails to each task in a job array.
OTHER CHANGES
=============
-- Add mail wrapper script "smail" that will include job statistics in email
notification messages.
-- Removed support for authd. authd has not been developed and supported since
several years.
-- Enable the hdf5 profiling of the batch step.
-- Eliminate redundant environment and script files for job arrays. This
greatly reduces the number of files involved in managing job arrays.
-- Burst buffer/cray - Add support for multiple buffer pools including support
for different resource granularity by pool.
-- Stop searching sbatch scripts for #PBS directives after 100 lines of
non-comments. Stop parsing #PBS or #SLURM directives after 1024 characters
into a line. Required for decent perforamnce with huge scripts.
-- New node features plugin infrastructure added. Currently used for support
of Intel Knights Landing processor.
-- If NodeHealthCheckProgram configured HealthCheckInterval is non-zero, then
modify slurmd to run it before registering with slurmctld.
-- select/cray - Initiate step node health check at start of step termination
rather than after application completely ends so that NHC can capture
information about hung (non-killable) processes.
-- Display thread name along with thread id and remove process name in stderr
logging for "thread_id" LogTimeFormat.
API CHANGES
===========
Removed PARAMS macro from slurm.h.
Removed BEGIN_C_DECLS and END_C_DECLS macros from slurm.h.
Changed members of the following structs
========================================
In burst_buffer_info_t, Changed gres_cnt to pool_cnt
Changed gres_ptr to pool_ptr
In burst_buffer_pool_t: Changed avail_cnt to total_space
In job_desc_msg_t: Changed nice from 16 to 32-bits
In partition_info_t: Split priority into priority_job_factor and priority_tier
In slurm_job_info_t: Changed nice from 16 to 32-bits
Added members to the following struct definitions
=================================================
In burst_buffer_info_t: Added other_timeout and validate_timeout
In burst_buffer_resv_t: Added pool
In job_desc_msg_t: Added deadline, mcs_label
In node_info_t: Added features_act and mcs_label
In resource_allocation_response_msg_t: Added ntasks_per_board, ntasks_per_core,
ntasks_per_socket
In slurm_job_info_t: Added deadline, num_tasks, mcs_label
In slurm_ctl_conf_t: Added mcs_plugin, mcs_plugin_params, tcp_timeout, and
node_features_plugins
In slurm_step_launch_params_t: Added ntasks_per_board, ntasks_per_core,
ntasks_per_socket
In update_node_msg_t: Added features_act
In slurm_job_info_t: Added start_protocol_ver
In slurm_step_layout_t: Added start_protocol_ver
In job_step_info_t: Added start_protocol_ver
Added the following struct definitions
======================================
Added top_job_msg_t for user to reorder his jobs
Removed members from the following struct definitions
=====================================================
In burst_buffer_resv_t: Removed gres_cnt and gres_ptr
Changed the following enums and #defines
========================================
Added DEBUG_FLAG_TIME_CRAY and DEBUG_FLAG_NODE_FEATURES
Added RESERVE_FLAG_PURGE_COMP
Added JOB_DEADLINE (job state)
Added MAIL_ARRAY_TASK (mail flag)
Changed MAX_TASKS_PER_NODE from 128 to 512
Changed NICE_OFFSET from 10000 to 0x80000000
Added new job state/wait reasons: FAIL_DEADLINE, WAIT_QOS_MAX_BB_PER_ACCT,
WAIT_QOS_MAX_CPU_PER_ACCT, WAIT_QOS_MAX_ENERGY_PER_ACCT,
WAIT_QOS_MAX_GRES_PER_ACCT, WAIT_QOS_MAX_NODE_PER_ACCT,
WAIT_QOS_MAX_LIC_PER_ACCT, WAIT_QOS_MAX_MEM_PER_ACCT,
WAIT_QOS_MAX_UNK_PER_ACCT, WAIT_QOS_MAX_JOB_PER_ACCT,
WAIT_QOS_MAX_SUB_JOB_PER_ACCT
Added new partition limit enforcement flags: PARTITION_ENFORCE_NONE,
PARTITION_ENFORCE_ALL, PARTITION_ENFORCE_ANY
Added select plugin IDs: SELECT_PLUGIN_BLUEGENE, SELECT_PLUGIN_CONS_RES,
SELECT_PLUGIN_LINEAR, SELECT_PLUGIN_ALPS, SELECT_PLUGIN_SERIAL,
SELECT_PLUGIN_CRAY_LINEAR, SELECT_PLUGIN_CRAY_CONS_RES
Added job flags: GRES_ENFORCE_BIND and TEST_NOW_ONLY
Added job resource sharing flags: JOB_SHARED_NONE, JOB_SHARED_OK,
JOB_SHARED_USER, JOB_SHARED_MCS
Added the following API's
=========================
Added slurm_top_job() function to reorder a user's jobs
Changed the following API's
============================
Added use_protocol_ver parameter to slurm_job_step_stat().