forked from SchedMD/slurm
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathNEWS
13353 lines (13036 loc) · 754 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.
* Changes in Slurm 23.02.2
==========================
-- Fix regression introduced with the migration to interfaces which caused
sshare to core dump. Sshare now initialized the priority context correctly
when calculating with PriorityFlags=NO_FAIR_TREE.
-- Fix IPMI DCMI sensor initialization.
-- For the select/cons_tres plugin, improve the best effort GPU to core
binding, for requests with per job task count (-n) and GPU (--gpus)
specification.
-- scrontab - don't update the cron job tasks if the whole crontab file is
left untouched after opening it with "scrontab -e".
-- mpi/pmix - avoid crashing when running PMIx v5.0 branch with shmem support.
-- Fix building switch topology after a reconfig with the correct nodes.
-- Allow a dynamic node to register with a reason, using --conf, when the
state is DOWN or DRAIN.
-- Fix slurmd running tasks before RPC Prolog is run.
-- Fix slurmd deadlock iff the controller were to give a bad alias_list.
-- slurmrestd - correctly process job submission field "exclusive" with boolean
True or False.
-- slurmrestd - correctly process job submission field "exclusive" with strings
"true" or "false".
-- slurmctld/step_mgr - prevent non-allocatable steps from decrementing values
that weren't previously incremented when trying to allocate them.
-- auth/jwt - Fix memory leak in slurmctld with 'scontrol token'.
-- Fix shared gres (shard/mps) leak when using --tres-per-task
-- Fix sacctmgr segfault when listing accounts with coordinators.
-- slurmrestd - improve error logging when client connections experience
polling errors.
-- slurmrestd - improve handling of sockets in different states of shutdown to
avoid infinite poll() loop causing a thread to max CPU usage until process
is killed.
-- slurmrestd - avoid possible segfault caused by race condition of already
completed connections.
-- mpi/cray_shasta - Fix PMI shared secret for hetjobs.
-- gpu/oneapi - Fix CPU affinity handling.
-- Fix dynamic nodes powering up when already up after adding/deleting nodes
when using power_save logic.
-- slurmrestd - Add support for setting max connections.
-- data_parser/v0.0.39 - fix sacct --json matching associations from a
different cluster.
-- Fix segfault when clearing reqnodelist of a pending job.
-- Fix memory leak of argv when submitting jobs via slurmrestd or CLI commands.
-- slurmrestd - correct miscalculation of job argument count that could cause
memory leak when job submission fails.
-- slurmdbd - add warning on startup if max_allowed_packet is too small.
-- gpu/nvml - Remove E-cores from NVML's cpu affinity bitmap when
"allow_ecores" is not set in SlurmdParameters.
-- Fix regression from 23.02.0rc1 causing a FrontEnd slurmd to assert fail on
startup and don't be configured with the appropriate port.
-- Fix dynamic nodes not being sorted and not being included in topology,
which resulted in suboptimal dynamic node selection for jobs.
-- Fix slurmstepd crash due to potential division by zero (SIGFPE) in certain
edge-cases using the PMIx plugin.
-- Fix issue with PMIx HetJob requests where certain use-cases would end up
with communication errors due to incorrect PMIx hostname info setup.
-- openapi/v0.0.39 - revert regression in job update requests to accept job
description for changes instead of requiring job description in "job" field.
-- Fix regression in 23.02.0rc1 that caused a step to crash with a bad
--gpu-bind=single request.
-- job_container/tmpfs - skip more in-depth attempt to clean up the base path
when not required. This prevents unhelpful, and possibly misleading, debug2
messages when not using the new "shared" mode.
-- gpu/nvml - Fix gpu usage when graphics processes are running on the gpu.
-- slurmrestd - fix regression where "exclusive" field was removed from job
descriptions and submissions.
-- Fix issue where requeued jobs had bad gres allocations leading to gres not
being deallocated at the end of the job, preventing other jobs from using
those resources.
-- Fix regression in 23.02.0rc1 which caused incorrect values for
SLURM_TASKS_PER_NODE when the job requests --ntasks-per-node and --exclusive
or --ntasks-per-core=1 (or CR_ONE_TASK_PER_CORE) and without requesting
--ntasks. SLURM_TASKS_PER_NODE is used by mpirun, so this regression
caused mpirun to launch the wrong number of tasks and to sometimes fail to
launch tasks.
-- Prevent jobs running on shards from being canceled on slurmctld restart.
-- Fix SPANK prolog and epilog hooks that rely on slurm_init() for access to
internal Slurm API calls.
-- oci.conf - Populate %m pattern with ContainerPath or SlurmdSpoolDir if
ContainerPath is not configured.
-- Removed zero padding for numeric values in container spool directory names.
-- Avoid creating an unused task-4294967295 directory in container spooldir.
-- Cleanup container step directories at step completion.
-- sacctmgr - Fix segfault when printing empty tres.
-- srun - fix communication issue that prevented slurmctld from connecting to
an srun running outside of a compute node.
* Changes in Slurm 23.02.1
==========================
-- job_container/tmpfs - cleanup job container even if namespace mount is
already unmounted.
-- When cluster specific tables are be removed also remove the job_env_table
and job_script_table.
-- Fix the way bf_max_job_test is applied to job arrays in backfill.
-- data_parser/v0.0.39 - Avoid dumping -1 value or NULL when step's
consumed_energy is unset.
-- scontrol - Fix showing Array Job Steps.
-- scontrol - Fix showing Job HetStep.
-- openapi/dbv0.0.38 - Fix not displaying an error when updating QOS or
associations fails.
-- data_parser/v0.0.39 - Avoid crash while parsing composite structures.
-- sched/backfill - fix deleted planned node staying in planned node bitmap.
-- Fix nodes remaining as PLANNED after slurmctld save state recovery.
-- Fix parsing of cgroup.controllers file with a blank line at the end.
-- Add cgroup.conf EnableControllers option for cgroup/v2.
-- Get correct cgroup root to allow slurmd to run in containers like Docker.
-- Fix "(null)" cluster name in SLURM_WORKING_CLUSTER env.
-- slurmctld - add missing PrivateData=jobs check to step ContainerID lookup
requests originated from 'scontrol show step container-id=<id>' or certain
scrun operations when container state can't be directly queried.
-- Automatically sort the TaskPlugin list reverse-alphabetically. This
addresses an issue where cpu masks were reset if task/affinity was listed
before task/cgroup on cgroup/v2 systems with Linux kernel < 6.2.
-- Fix some failed terminate job requests from a 23.02 slurmctld to a 22.05 or
21.08 slurmd.
-- Fix compile issues on 32-bit systems.
-- Fix nodes un-draining after being drained due to unkillable step.
-- Fix remote licenses allowed percentages reset to 0 during upgrade.
-- sacct - Avoid truncating time strings when using SLURM_TIME_FORMAT with
the --parsable option.
-- data_parser/v0.0.39 - fix segfault when default qos is not set.
-- Fix regression in 22.05.0rc1 that broke Nodes=ALL in a NodeSet.
-- openapi/v0.0.39 - fix jobs submitted via slurmrestd being allocated fewer
CPUs than tasks when requesting multiple tasks.
-- Fix job not being scheduled on valid nodes and potentially being rejected
when using parentheses at the beginning of square brackets in a feature
request, for example: "feat1&[(feat2|feat3)]".
-- Fix a job being scheduled on nodes that do not match a feature request that
uses parentheses inside of brackets and requests additional features outside
of brackets, for example: "feat1&[feat2|(feat3|feat4)]".
-- Fix regression in 23.02.0rc1 which made --gres-flags=enforce-binding no
longer enforce optimal core-gpu job placement.
-- switch/hpe_slingshot - add option to disable VNI allocation per-job.
-- switch/hpe_slingshot - restrict CXI services to the requesting user.
-- switch/hpe_slingshot - Only output tcs once in SLINGSHOT_TCS env.
-- switch/hpe_slingshot - Fix updating LEs and ACs limits.
-- switch/hpe_slingshot - Use correct Max for EQs and CTs.
-- switch/hpe_slingshot - support configuring network options per-job.
-- switch/hpe_slingshot - retry destroying CXI service if necessary.
-- Fix memory leak caused by job preemption when licenses are configured.
-- mpi/pmix - Fix v5 to load correctly when libpmix.so isn't in the normal
lib path.
-- data_parser/v0.0.39 - fix regression where "memory_per_node" would be
rejected for job submission.
-- data_parser/v0.0.39 - fix regression where "memory_per_cpu" would be
rejected for job submission.
-- slurmctld - add an assert to check for magic number presence before deleting
a partition record and clear the magic afterwards to better diagnose
potential memory problems.
-- Clean up OCI containers task directories correctly.
-- slurm.spec - add "--with jwt" option.
-- scrun - Run under existing job when SLURM_JOB_ID is present.
-- Prevent a slurmstepd crash when the I/O subsystem has hung.
-- common/conmgr - fix memory leak of complete connection list.
-- data_parser/v0.0.39 - fix memory leak when parsing every field in a struct.
-- job_container/tmpfs - avoid printing extraneous error messages when running
a spank plugin that implements slurm_spank_job_prolog() or
slurm_spank_job_epilog().
-- Fix srun < 23.02 always getting an "exact" core allocation.
-- Prevent scontrol < 23.02 from setting MaxCPUsPerSocket to 0.
-- Add ScronParameters=explicit_scancel and corresponding scancel --cron
option.
* Changes in Slurm 23.02.0
==========================
-- scrun - Install into /bin instead of /sbin, which also ensures it is
included in the RPM packages.
-- data_parser/v0.0.39 - avoid erroring on unknown association id.
-- data_parser/v0.0.39 - avoid fatal while dumping TRES values.
-- Improve error message for using --cpus-per-gpus without any GPUs.
-- switch/hpe_slingshot - fix rolling upgrades from 22.05 to 23.02
-- Change cgroup/v1 behavior when waiting for a pid to be moved by waiting some
time between retries. This helps with stray cgroup dirs on slow kernels.
-- Workaround a bug in kernels < 3.18 with cpuset which randomly leaves stray
cgroups after jobs have ended.
-- Rebuild the prepacked buf going to the slurmstepd if TRES changed.
-- Add new 'make clean-contrib' target to build system to clean contribs dir.
-- Treat newlines as delimeters for hostlist_create(). (This is used internally
by "scontrol show hostlist" and for handling SLURM_JOB_NODELIST.)
-- scontrol - Print "Invalid job id specified" for 'scontrol show step ...'
when the job does not exist, rather than "Unexpected error".
-- Handle mismatched TRES count from the slurmstepd's instead of fatal()'ing.
-- Fix GPU setup on CRAY systems when using the CRAY_CUDA_MPS environment
variable. GPUs are now correctly detected in such scenarios.
-- data_parser/v0.0.39 - improve flag handling to avoid flags having bits left
unset after parsing.
-- topology/tree - Add new TopologyParam=SwitchAsNodeRank option to reorder
nodes based on switch layout. This can be useful if the naming convention
for the nodes does not natually map to the network topology.
-- openapi/v0.0.38 - avoid signed math errors while dumping node resources.
-- openapi/dbv0.0.39 - avoid error while dumping account coordinators.
-- data_parser/v0.0.39 - correct dump of stats backfill mean table size.
-- openapi/dbv0.0.39 - avoid error while adding account coordinators.
-- scrun - catch invalid values of environment SCRUN_FILE_DEBUG.
-- scrun - avoid memory leak due to invalid annotation.
-- scrun - avoid false error when pidfile not requested.
-- Do not constrain memory in task/cgroup unless CR_Memory is set.
-- Fix the job prolog not running for jobs with the interactive step
(salloc jobs with LaunchParameters=use_interactive_step set in slurm.conf)
that were scheduled on powered down nodes. The prolog not running also
broke job_container/tmpfs, pam_slurm_adopt, and x11 forwarding.
-- swich/hpe_slingshot - fix issues using 22.05 commands with 23.02 slurmd.
-- swith/hpe_slingshot - avoid slurmd segfault if there is a protocol error.
-- task/affinity - fix slurmd segfault when request launch task requests of
type "--cpu-bind=[map,mask]_cpu:<list>" have no <list> provided.
-- salloc/sbatch/srun - error out if "--cpu-bind=[map,mask]_cpu:<list>" fails
to extract a list of cpus.
-- job_container/tmpfs - cleanup job_mount when container creation fails.
-- job_container/tmpfs - don't attempt to remove job_mount directory when it is
still mounted.
-- Fix regression in rc1 causing sacctmgr to segfault when printing unrelated
fields.
-- jobcomp/kafka - don't use the purge API if librdkafka < v1.0.0.
-- Change shown start time of pending array job to be start time of earliest
pending array task.
-- job_container/tmpfs - ensure that step_ns_fd is closed before cleaning up
the namespace.
-- Fix assert when suspending or requeueing array tasks with scontrol.
-- Fix federated job submissions.
-- Fix sinfo returning non-zero exit code when querying specific clusters.
-- openapi/v0.0.39 - correct OpenAPI schema for job update requests.
-- openapi/v0.0.39 - prevent assertion on jobs endpoint by altering how the
"memory_per_cpu" and "memory_per_node" fields are managed.
-- Fix regression in 23.02.0rc1 which caused slurmctld to segfault when
gres is added or removed in the configuration.
-- Removed the default setting for GpuFreqDef. If unset, no attempt to change
the GPU frequency will be made if --gpu-freq is not set for the step.
-- Fixed GpuFreqDef option. When set in slurm.conf, it will be used if
--gpu-freq was not explicitly set by the job step.
-- Update database index for usage tables used for archive and purge.
-- Fix configure script on FreeBSD.
* Changes in Slurm 23.02.0rc1
=============================
-- Make scontrol reconfigure and sending a SIGHUP to the slurmctld behave the
same. If you were using SIGHUP as a 'lighter' scontrol reconfigure to rotate
logs please update your scripts to use SIGUSR2 instead.
-- Add Account and QOS name type specifications for sprio output formatting.
-- openapi/[db]v0.0.36 - plugins have been removed.
-- openapi/[db]v0.0.37 - tagged as deprecated.
-- openapi/[db]v0.0.39 - forked plugin openapi/[db]v0.0.38.
-- Add SRUN_{ERROR,INPUT,OUTPUT} input environment variables for --error,
--input and --output options respectively.
-- Add MaxCPUsPerSocket to partition configuration, similar to MaxCPUsPerNode.
-- openapi/v0.0.39 - change nice request field from string to integer.
-- sacctmgr - no longer force updates to the AdminComment, Comment, or
SystemComment to lower-case.
-- openapi/dbv0.0.39 - more graceful handling of POSTs when expected field
lists are missing or unparsable.
-- burst_buffer/lua - pass the job's UID and GID to slurm_bb_pre_run,
slurm_bb_data_in, slurm_bb_post_run, and slurm_bb_data_out in
burst_buffer.lua.
-- openapi/v0.0.39 - add new field default_memory_per_node for partitions.
-- Change cloud nodes to show by default. PrivateData=cloud is no longer
needed.
-- Add -F/--future option to sinfo to display future nodes.
-- openapi/dbv0.0.39 - resolve job user from uid when user is not provided.
-- sacct/sinfo/squeue - use openapi/[db]v0.0.39 for --json and --yaml modes.
-- sdiag - Add --json and --yaml arguments.
-- Cgroupv2 plugin will not show "Controller is not enabled" error when started
from cmd.
-- sreport - Count planned (FKA reserved) time for jobs running in IGNORE_JOBS
reservations. Previously was lumped into IDLE time.
-- job_container/tmpfs - support running with an arbitrary list of private
mount points (/tmp and /dev/shm are nolonger required, but are the default).
-- sacct - Rename 'Reserved' field to 'Planned' to match sreport and the
nomenclature of the 'Planned' node.
-- oci.conf: Add %U as OCI pattern replacement to numeric user id.
-- cli_filter/lua, jobcomp/lua, job_submit/lua - expose all Slurm error codes.
-- jobcomp/elasticsearch - expose features field.
-- jobcomp/elasticsearch - post non-NULL but empty string fields.
-- Make advanced reservation flag MAINT not replace nodes, similar to
STATIC_ALLOC
-- NVML - Add usage gathering for Nvidia gpus.
-- node_features plugins - invalid users specified for AllowUserBoot will now
result in fatal() rather than just an error.
-- job_container/tmpfs - Set more environment variables in InitScript.
-- Make all cgroup directories created by Slurm owned by root.
-- sbatch - add parsing of #PBS -d and #PBS -w.
-- Avoid network receive error on heavily loaded machines where the network
operation can be restarted.
-- Deprecate AllowedKmemSpace, ConstrainKmemSpace, MaxKmemPercent, and
MinKmemSpace.
-- accounting_storage/mysql - change purge/archive to calculate record ages
based on end time, rather than start or submission times.
-- Add strigger --draining and -R/--resume options.
-- Allow updating SLURM_NTASKS environment variable (from scontrol update job).
-- Limit het job license requests to the leader job only.
-- Change --oversubscribe and --exclusive to be mutually exclusive for job
submission. Job submission commands will now fatal if both are set.
Previously, these options would override each other, with the last one in
the job submission command taking effect.
-- job_submit/lua - add support for log_user() from slurm_job_modify().
-- openapi/v0.0.39 - add submission of job->prefer value.
-- Fix tasks binding to GPUs when using --ntasks-per-gpu and GPUs have
different core/socket affinity.
-- Add support for padding StdOut/StdErr/StdIn format specifiers to scontrol
show job.
-- Fix sacctmgr qos filter to properly select assocs by qos inheritance rules.
-- Prevent slurmctld from starting with invalid jobcomp plugin.
-- Fix `sinfo -i` and interactive `scontrol> show node` replies for nodes
going from powering down to powered down state.
-- auth/jwt - support Azure (among other) JWKS files by preferring x5c field
over locally reconstructing an RSA256 key file.
-- auth/jwt - add optional AuthAltParameters field "userclaimfield=" to allow
overriding "sun" claim to site specific field.
-- scontrol - Requested TRES and allocated TRES will now always be printed when
showing jobs, instead of one TRES output that was either the requested or
allocated.
-- Improve scheduling performance for dynamic nodes and nodes with features by
consolidating like config records.
-- Expose argc and entire argv to job_submit and cli_filter plugins for jobs
submitted via salloc/srun or sbatch <script> (not via --wrap).
-- Job command/script name is now packed with no arguments, thus clients
displaying job information will only show the command/script name.
-- Make acct_gather_energy plugins handle slurmd reconfiguration and support
restart for gpu and xcc implementations.
-- Fix --slurmd-debug to treat its argument as log level (as documented)
instead of previous approach with it being an offset to configured log
level.
-- srun --slurmd-debug option is now only allowed for root and SlurmUser
-- Change 'scontrol requeue' behavior for scron jobs to use cronspec to
determine the next start time.
-- serializer/url-encoded - interpet query keys without a value (flags) to be
true instead of NULL.
-- Add --autocomplete= option to all client commands.
-- Allow for concurrent processing of job_submit_g_submit() and
job_submit_g_modify() calls.
-- Allow jobs to queue even if the user is not in AllowGroups when
EnforcePartLimits=no is set. This ensures consistency for all the Partition
access controls, and matches the documented behavior for EnforcePartLimits.
-- Fix srun tasks binding/allocation when using --ntasks-per-core option. Now,
--ntasks-per-core=1 implies --cpu-bind=cores and --ntasks-per-core>1 implies
--cpu-bind=threads.
-- salloc/sbatch/srun check and abort if ntasks-per-core > threads-per-core.
-- burst_buffer/lua - fix building heterogeneous job scripts where the
directive is not the default "BB_LUA".
-- If a node is in a reservation add the reservation name to it for
scontrol/sinfo/sview.
-- common/cgroup - improve possible TOCTOU situation when reading cgroup files.
-- slurmctld will fatal() when reconfiguring the job_submit plugin fails.
-- Add ResumeAfter=<secs> option to "scontrol update nodename=".
-- slurmrestd - switch to returning UNPROCESSABLE_CONTENT when parsing
succeeds but an error about the actual contents causes a request to fail.
-- slurmrestd - queries that result in empty contents will now be treated as
SUCCESS. While the query worked, nothing was returned, which allows callers
to determine how to then proceed.
-- Add PowerDownOnIdle partition option.
-- Make it so the [Allow|Deny]Accounts parameter for a partition is
hierarchical.
-- Add the ability for a add (+=) or remove (-=) nodes from a reservation
-- Fix several problems when creating/editing partitions in sview.
-- Enhanced burst_buffer.lua - pass UID and GID to most hooks; pass a table
containing detailed job information to many hooks. See
etc/burst_buffer.lua.example for a complete list of changes.
-- Add a new "nodes=" argument to scontrol setdebug to allow the debug level
on the slurmd processes to be temporarily altered.
-- Add InfluxDBTimeout parameter to acct_gather.conf.
-- job_container/tmpfs - add support for expanding %h and %n in BasePath.
-- Prevent job submission/update with afterok/afternotok dependency set to
an unknown jobid.
-- Change to a new FAIL_SIGNAL job state reason (instead of FAIL_LAUNCH) for
jobs that have raised a signal on setup.
-- Add "[jobid.stepid]" prefix from slurmstepd and "slurmscriptd" prefix from
slurmcriptd to Syslog logging. Previously was only happening when logging
to a file.
-- Make it so scrontab prints client-side the job_submit() err_msg (which can
be set i.e. by using the log_user() function for the lua plugin).
-- Remove ability for reservation to have STATIC_ALLOC or MAINT flags and
REPLACE[_DOWN] flags simultaneously.
-- Accept only one reoccurring flag when creating/updating any reservation.
-- Remove ability to update a reservation to have a reoccurring flag if it
is a floating reservation.
-- squeue - removed unused '%s' and 'SelectJobInfo' formats.
-- Add purge and archive functionality for job environment and job batch script
records.
-- srun - return an error if step suffers a node failure.
-- squeue - align print format for exit and derived codes with that of other
components (<exit_status>:<signal_number>).
-- Add new SlurmctldParameters=validate_nodeaddr_threads=<number> option to
allow concurrent hostname resolution at slurmctld startup.
-- Have 'scontrol show hostlist -' read from stdin.
-- Extend support for Include files to all "configless" client commands.
-- Make node weight usable for powered down and rebooting nodes.
-- node_features plugins - node_features_p_reboot_weight() function removed.
-- Removed 'launch' plugin.
-- Add "Extra" field to job to store extra information other than a comment.
-- Add new AccountingStoreFlags=job_extra option to store a job's extra field
in the database.
-- slurmrestd - add option to allow rest_auth/local to switch user id on first
user connection.
-- oci.conf - add DisableHooks option.
-- oci.conf - add SrunPath and SrunArgs options.
-- oci.conf - add StdIODebug, SyslogDebug, FileDebug and DebugFlags options to
allow controlling container specific logging.
-- oci.conf - remove requirement of RunTimeQuery when RunTimeRun is set.
-- slurmstepd - cleanup generated environment file for containers when step
completes instead of deferring cleanup to OCI runtime.
-- oci.conf - add configuration option "IgnoreFileConfigJson=" to allow a site
to choose to ignore config.json.
-- oci.conf - Add PID (%p) pattern replacement.
-- slurmrestd - correct ordering issue of saving changes to a container's
modified config.json where config changes by job modifing plugins would be
silently lost.
-- slurmstepd - merge container config.json environment into job environment
and then overwrite config.json environment. Avoids instance where processes in
container will not have access to job environment variables.
-- slurmstepd - defer creating container environment file until after all
plugin based modifications to the job environment are complete.
-- oci.conf - modify configuration option "CreateEnvFile=" to support creating
-- oci.conf - add RunTimeEnvExclude and EnvExclude.
-- slurmstepd - fix container pattern bundle path replacement (%b) with
bundle path instead of spool dir path.
-- oci.conf - add MountSpoolDir.
-- slurmstepd - create per task spool directory per container to avoid
conflicts when there are more than 1 container tasks on any given node.
-- mpi/pmix - set PMIX_SERVER_TMPDIR to container spool directory instead of
slurmd spool directory which caused PMIx to fail to load in containers.
-- Add new "defer_batch" option to SchedulerParameters to only defer
scheduling for batch jobs.
-- Reject jobs submitted with impossible --time, --deadline, and --begin
combinations at submission time
-- Add new DebugFlags option 'JobComp' to replace 'Elasticsearch'.
-- sacct - Add --array option to expand job arrays and display array tasks on
separate lines.
-- Fix jobs submitted with a deadline never being cleared after the
reservation is done.
-- Change srun/salloc to wait indefinitely for their allocations to be ready.
-- Do not attempt to "fix" malformed input to 'scontrol show hostlist'.
-- RSMI - Add usage gathering for AMD gpus (requires ROCM 5.5+).
-- Add job's allocated nodes, features, oversubscribe, partition, and
reservation to SLURM_RESUME_FILE output for power saving.
-- Automatically create directories for stdout/stderr output files. Paths may
use %j and related substitution characters as well.
-- Add --tres-per-task to salloc/sbatch/srun.
-- Add configurable job requeue limit parameter - MaxBatchRequeue - in
slurm.conf to permit changes from the old hard-coded value of 5.
-- Add a new "nodes=" argument to "scontrol setdebugflags" as well.
-- slurmd - add --authinfo option to allow AuthInfo options to be changed
during the configless startup.
-- helpers.conf - Allow specification of node specific features.
-- helpers.conf - Allow many features to one helper script.
-- Allow nodefeatures plugin features to work with cloud nodes.
-- squeue - removed --array-unique option.
-- openapi/v0.0.39 - expand job stderr, stdin, and stdout path replacement
symbols when dumping job information via dump_job_info().
-- Make slurmstepd cgroups constrained by total configured memory from
slurm.conf (NodeName=<> RealMemory=#) instead of total physical memory.
-- node_features/helpers - add support for the OR and parentheses operators
in a --constraint expression.
-- Fix race condition between PMIx authentication and slurmstepd task launches
that can lead to job launch failures.
-- fatal() when [Prolog|Epilog]Slurmctld are defined but not executable.
-- slurmctld - Add new RPC rate limiting feature. This is enabled through
SlurmctldParameters=rl_enable, otherwise disabled by default.
-- Validate node registered active features are a super set of node's
currently active changeable features.
-- On clusters without any PrologFlags options, batch jobs with failed prologs
nolonger generate an output file.
-- sreport - print "Top ALL Users" for Top User report when topcount = -1.
-- Add SLURM_JOB_START_TIME and SLURM_JOB_END_TIME environment variables.
-- Add SuspendExcStates option to slurm.conf to avoid suspending/powering down
specific node states.
-- job_container/tmpfs - Add "Shared" option to support shared namespaces.
This allows autofs to work with the job_container/tmpfs plugin when enabled.
-- Fix incorrect min node estimation for pending jobs as displayed by client
commands (e.g. squeue, scontrol show job).
-- Add support for DCMI power readings in IPMI plugin.
-- Prevent the backfill counter of tested jobs being reset by sdiag -r.
-- sacctmgr - Add --json and --yaml arguments.
-- openapi/v0.0.39 - Conversion of parsing and dumping of data to
data_parser/v0.0.39 plugins. Significant restructuring of OpenAPI
specification compared to previous plugin version.
-- openapi/dbv0.0.39 - Conversion of parsing and dumping of data to
data_parser/v0.0.39 plugins. Significant restructuring of OpenAPI
specification compared to previous plugin version.
-- Validate --gpu-bind options more strictly.
-- Fix slurmd segfault on bad EnergyIPMIPowerSensors= parameter.
-- Allow for --nodelist to contain more nodes than required by --nodes.
-- Add missing logic for JobCompParams configuration key pair to be available
for consumers of the get/print/write configuration API functions.
-- mpi/cray_shasta - Add new PALS/PMI environment variables.
-- Add cache around calls to getnameinfo().
-- Add CommunicationParameters=getnameinfo_cache_timeout to tune or disable the
getnameinfo cache.
-- Limit the number of successive times a batch job may be requeued due to a
prolog failure. This limit is defined by the value of the max_batch_requeue
parameter (default 5). Jobs over this limit will be put on admin hold.
-- Rename "nodes" to "nodes_resume" in SLURM_RESUME_FILE job output.
-- Rename "all_nodes" to "all_nodes_resume" in SLURM_RESUME_FILE output.
-- Add new PrologFlags=ForceRequeueOnFail option to automatically requeue
batch jobs on Prolog failures regardless of the job --requeue setting.
-- Add HealthCheckNodeState=NONDRAINED_IDLE option.
-- Add 'explicit' to Flags in gres.conf
-- Improve validation of --constraint expression syntax.
-- srun - support job array ids as an argument to --jobid.
-- Add jobcomp/kafka plugin.
-- Automatically filter out E-Cores on Intel processors to avoid issues from
differing threads-per-core values within a single socket.
-- Add LaunchParameters=ulimit_pam_adopt, which enables setting RLIMIT_RSS in
adopted processes.
-- Add new PreemptParameters=reclaim_licenses option which will allow higher
priority jobs to preempt jobs to free up used licenses. (This is only
enabled for with PreemptModes of CANCEL and REQUEUE, as Slurm cannot
guarantee suspended jobs will release licenses correctly.)
-- Make it so that the REQUEST_STEP_STAT handler only polls jobacct_gather
information once per RPC. This improves sstat results performance time.
-- Fix sbcast --force option to work with --send-libs option. Now when --force
is set, library directores transmitted will overwrite pre-existing libraries
that were previously transmitted.
-- hpe/slingshot - Add support for user requestible job VNIs.
-- hpe/slingshot - Add support for user requestible single node vnis.
-- hpe/slingshot - Add support for the instant-on feature.
-- Add ability to update SuspendExc* parameters with scontrol.
-- Add ability to preserve SuspendExc* parameters on reconfig with
ReconfigFlags=KeepPowerSaveSettings.
-- Add ability to restore SuspendExc* parameters on restart with slurmctld -R
option.
-- Fix leaving allocated gres on a non-cloud node after being allocated
from a powered down state after a slurmctld restart.
-- 'scontrol update job' can now set a GRES specification to "0" to effectively
clear a GRES speficiation from a job.
-- Jobs with old GRES that are no longer configured in any configuration files
can now be succesfully updated to use currently configured GRES.
-- Add tres_per_node to resource_allocation_response_msg to fix srun jobs
running with outdated GRES. This is necessary for srun jobs whose GRES are
updated while they are pending. By having tres_per_node in the
resourece_allocation_response_msg, slurmctld can send back the correct and
updated GRES, whereas without it, the srun job would use the old GRES.
-- Cleaned up various "warning", "Warning", and "WARNING" messages with a new
warning() logging call. All will now print as "warning: ...".
-- Add SLURM_JOB_OVERSUBSCRIBE environment variable for Epilog, Prolog,
EpilogSlurmctld, PrologSlurmctld, and mail ouput.
-- Fix node reason time skew when setting a reboot message.
-- Append ResumeTimeout message to an existing node reason instead of always
clearing it first.
-- slurmrestd - added support for Bearer Authentication using auth/jwt.
-- Add check to detect invalid HetJob component and abort job if identified.
-- sched/backfill - fix job test against advanced reservations if job uses a
preemptable QOS with Flags=NoReserve.
-- New command scrun has been added. scrun acts as an Open Container Initiative
(OCI) runtime proxy to run containers seamlessly via Slurm.
* Changes in Slurm 22.05.10
===========================
* Changes in Slurm 22.05.9
==========================
-- Allocate correct number of sockets when requesting gres and running with
CR_SOCKET*.
-- Fix handling of --prefer for job arrays.
-- Fix regression in 22.05.5 that causes some jobs that request
--ntasks-per-node to be incorrectly rejected.
-- Fix slurmctld crash when a step requests fewer tasks than nodes.
-- Fix incorrect task count in steps that request --ntasks-per-node and a node
count with a range (e.g. -N1-2).
-- Fix some valid step requests hanging instead of running.
-- slurmrestd - avoid possible race condition which would cause slurmrestd to
silently no longer accept new client connections.
-- Fix GPU setup on CRAY systems when using the CRAY_CUDA_MPS environment
variable. GPUs are now correctly detected in such scenarios.
-- Fix the job prolog not running for jobs with the interactive step
(salloc jobs with LaunchParameters=use_interactive_step set in slurm.conf)
that were scheduled on powered down nodes. The prolog not running also
broke job_container/tmpfs, pam_slurm_adopt, and x11 forwarding.
-- task/affinity - fix slurmd segfault when request launch task requests of
type "--cpu-bind=[map,mask]_cpu:<list>" have no <list> provided.
-- sched/backfill - fix segfault when removing a PLANNED node from system.
-- sched/backfill - fix deleted planned node staying in planned node bitmap.
-- Fix nodes remaining as PLANNED after slurmctld save state recovery.
-- Fix regression in 22.05.0rc1 that broke Nodes=ALL in a NodeSet.
-- Fix incorrect memory constraint when receiving a job from 20.11 that uses
cpu count for memory calculation.
-- openapi/v0.0.[36-38] - avoid possible crash from jobs submitted with argv.
-- openapi/v0.0.[36-38] - avoid possible crash from rejected jobs submitted
with batch_features.
-- srun - fix regression in 22.05.7 that prevented slurmctld from connecting
to an srun running outside of a compute node.
* Changes in Slurm 22.05.8
==========================
-- Fix potential deadlock at slurmctld startup when job has invalid qos.
-- Avoid unnecessary call to clusteracct_storage_g_cluster_tres() when pinging
dynamic nodes. This avoids significant slowdowns for slurmctld when
ping_nodes() calls all nodes to re-register.
-- openapi/dbv0.0.3[6-8] - fix segfault that could arrise from Slurm database
connnection failing.
-- Fix regression introduced in 22.05.0rc1 when updating a NodeName=<nodelist>
with NodeAddr and/or NodeHostname if the specified nodelist wasn't sorted.
-- openapi/v0.0.38 - change type of nice field for job submissions to integer
from string.
-- openapi/v0.0.38 - add oversubscribe option to job submission properties.
-- openapi/v0.0.38 - fix incorrect data used to populate "time/start" field
for jobs.
-- openapi/dbv0.0.38 - avoid dumping failure if preempt qos with level 0 is
provided by slurmdbd query.
-- Avoid an fd leak when lib dir for sbcast fails to be created.
-- common/slurmdbd_pack - fix env and script hash when unpacking a 21.08
dbd_job_start_msg_t.
-- Fix a race between job_container/tmpfs, cncu, x11 setup and adoption of pids
in cray with pam_slurm_adopt.
* Changes in Slurm 22.05.7
==========================
-- Fix slurmctlds in a federation not recovering a lost connection until a
slurmctld restart or a federation update.
-- Fix incorrectly rejecting jobs that request GRES on multiple sockets and
--gres-flags=enforce-binding.
-- Fix job core selection with --gres-flags=enforce-binding: this fixes cases
where enforce-binding was not respected.
-- openapi/v0.0.38 - set assoc_id resolution failure as a debug log instead of
an error log.
-- Fix issues when running in FrontEnd mode.
-- Fix srun and salloc I/O code to set keepaliveinterval/keepalivetime on
connections.
-- Fix overwrite race of cont_id in cgroup/v2 when using task/cgroup with
proctrack/cray_aries.
-- Fix node remaining allocated after a reconfig with a completing job that
has an EpilogSlurmctld instance still running.
-- openapi/dbv0.0.38 - fix a cast to a wrong type
-- openapi/dbv0.0.38 - correct issues where modifying association fields were
ignored.
-- openapi/dbv0.0.38 - correct issue where 16 bit integers were not able to be
unset.
-- openapi/dbv0.0.38 - correct issue where 16 & 32 bit integers were not able
to be unset.
-- Fix 'scontrol show hostlist' to read off a pipe correctly.
-- Fix 'scontrol reconfigure' with configless enabled to send updated configs
to 21.08 nodes correctly. (22.05 nodes are unaffected.)
-- Fix assertion and locks for dynamic node registrations with rpc queueing.
-- rest_auth/local - avoid logging root access as an error but instead as a
normal info log.
-- slurmrestd - switch users earlier on startup to avoid sockets being made as
root.
-- slurmrestd - disable umask during socket creation as the authentication is
handled by Slurm instead of file ACLs.
-- Fix sbcast, srun --bcast, sstat not working with cloud/dynamic nodes.
-- sacctmgr - Allow removal of user's default account when AllowNoDefAcct=yes.
-- Fix segfault caused by race condition when a reconfiguration is requested
while slurmd initializes.
-- openapi/v0.0.38 - avoid divison by zero causing slurmrestd to crash during
diag queries.
-- Update default values for text/blob columns in the database if upgrading
to MariaDB >= 10.2.1 from a pre-10.2.1 version or from MySQL to avoid
sacctmgr failures.
-- Fix several minor memory leaks in the cgroup/v2 plugin.
-- Fix possible issue when enabling controllers with long names in cgroup/v2.
-- Fix potential segfault in slurmd when using acct_gather_energy/ipmi.
-- Fix controller being able to contact non-addressable step hosts.
-- Allow setting SuspendTime to -1 when used on a partition.
-- openapi/v0.0.38 - fix maximum_memory_per_node description typo.
-- openapi/v0.0.38 - add maximum_memory_per_cpu to GET /partitions
-- openapi/v0.0.38 - add default_memory_per_node to GET /partitions
-- Fix dynamic node losing dynamic node status when changing state to FUTURE.
-- Fix adoption into most recent job in pam_slurm_adopt and cgroup/v2.
-- gres/shard - Fix plugin_name definition to print correctly in debug lines.
* Changes in Slurm 22.05.6
==========================
-- Fix a partition's DisableRootJobs=no from preventing root jobs from working.
-- Fix the number of allocated cpus for an auto-adjustment case in which the
job requests --ntasks-per-node and --mem (per-node) but the limit is
MaxMemPerCPU.
-- Fix POWER_DOWN_FORCE request leaving node in completing state.
-- Do not count magnetic reservation queue records towards backfill limits.
-- Clarify error message when --send-libs=yes or BcastParameters=send_libs
fails to identify shared library files, and avoid creating an empty
"<filename>_libs" directory on the target filesystem.
-- Fix missing CoreSpec on dynamic nodes upon slurmctld restart.
-- Fix node state reporting when using specialized cores.
-- Fix number of CPUs allocated if --cpus-per-gpu used.
-- Add flag ignore_prefer_validation to not validate --prefer on a job.
-- Fix salloc/sbatch SLURM_TASKS_PER_NODE output environment variable when the
number of tasks is not requested.
-- Permit using wildcard magic cookies with X11 forwarding.
-- cgroup/v2 - Add check for swap when running OOM check after task
termination.
-- Fix deadlock caused by race condition when disabling power save with a
reconfigure.
-- Fix memory leak in the dbd when container is sent to the database.
-- openapi/dbv0.0.38 - correct dbv0.0.38_tres_info.
-- Fix node SuspendTime, SuspendTimeout, ResumeTimeout being updated after
altering partition node lists with scontrol.
-- jobcomp/elasticsearch - fix data_t memory leak after serialization.
-- Fix issue where '*' wasn't accepted in gpu/cpu bind.
-- Fix SLURM_GPUS_ON_NODE for shared GPU gres (MPS, shards).
-- Add SLURM_SHARDS_ON_NODE environment variable for shards.
-- Fix srun error with overcommit.
-- Fix bug in core selection for the default cyclic distribution of tasks
across sockets, that resulted in random task launch failures.
-- Fix core selection for steps requesting multiple tasks per core when
allocation contains more cores than required for step.
-- gpu/nvml - Fix MIG minor number generation when GPU minor number
(/dev/nvidia[minor_number]) and index (as seen in nvidia-smi) do not match.
-- Fix accrue time underflow errors after slurmctld reconfig or restart.
-- Surpress errant errors from prolog_complete about being unable to locate
"node:(null)".
-- Fix issue where shards were selected from multiple gpus and failed to
allocate.
-- Fix step cpu count calculation when using --ntasks-per-gpu=.
-- Fix overflow problems when validating array index parameters in slurmctld
and prevent a potential condition causing slurmctld to crash.
-- Remove dependency on json-c in slurmctld when running with power saving.
Only the new "SLURM_RESUME_FILE" support relies on this, and it will be
disabled if json-c support is unavailable instead.
* Changes in Slurm 22.05.5
==========================
-- Fix node becoming IDLE while in an invalid registration state.
-- When a job is completing avoid potential dereference.
-- Avoid setting preempt_time for a job erroneously.
-- Fix situation where we don't requeue correctly when a job is finishing.
-- job_container/tmpfs - Avoid leaking namespace file descriptor.
-- common/slurm_opt - fix memory leak in client commands or slurmrestd when the
--chdir option is set after option reset.
-- openapi/dbv0.0.38 - gracefully handle unknown associations assigned to jobs.
-- openapi/dbv0.0.38 - query all associations to avoid errors while dumping
jobs.
-- Load hash plugin at slurmstepd launch time to prevent issues loading the
plugin at step completion if the Slurm installation is upgraded.
-- Fix gcc 12.2.1 compile errors.
-- Fix future magnetic reservations preventing heterogeneous jobs from
starting.
-- Prevent incorrect error message from being generated for operator/admins
using the 'scontrol top' command.
-- slurmrestd - correct issue where larger requests could result in a single
byte getting removed from inside of the POST request.
-- Fix regression in task count calculation for --ntasks-per-gpu with multiple
nodes.
-- Update nvml plugin to match the unique id format for MIG devices in new
Nvidia drivers.
-- Fix segfault on backup slurmdbd if no QoS is present in DB.
-- Fix clang 11 compile errors.
-- Fix task distribution calculations across sockets with
--distribution=cyclic.
-- Fix task distribution calculations with --ntasks-per-gpu specified without
an explicit --ntasks value.
-- Fix job arrays not showing correct features.
-- Fix job having wrong features used when using preferred features.
-- Fix task/cray_aries error finishing an interactive step, avoiding correct
cleanup.
-- Correctly set max_nodes when --ntasks=1.
-- Fix configure script on FreeBSD.
* Changes in Slurm 22.05.4
==========================
-- Fix return code from salloc when the job is revoked prior to executing user
command.
-- Fix minor memory leak when dealing with gres with multiple files.
-- Fix printing for no_consume gres in scontrol show job.
-- sinfo - Fix truncation of very large values when outputting memory.
-- Fix multi-node step launch failure when nodes in the controller aren't in
natural order. This can happen with inconsistent node naming (such as
node15 and node052) or with dynamic nodes which can register in any order.
-- job_container/tmpfs - Prevent reading the plugin config multiple times per
step.
-- Fix wrong attempt of gres binding for gres w/out cores defined.
-- Fix build to work with '--without-shared-libslurm' configure flag.
-- Fix power_save mode when repeatedly configuring too fast.
-- Fix sacct -I option.
-- Prevent jobs from being scheduled on future nodes.
-- Fix memory leak in slurmd happening on reconfigure when CPUSpecList used.
-- Fix sacctmgr show event [min|max]cpus.
-- Fix regression in 22.05.0rc1 where a prolog or epilog that redirected stdout
to a file could get erroneously killed, resulting in job launch failure
(for the prolog) and the node being drained.
-- cgroup/v1 - Make a static variable to remove potential redundant checking
for if the system has swap or not.
-- cgroup/v1 - Add check for swap when running OOM check after task
termination.
-- job_submit/lua - add --prefer support
-- cgroup/v1 - fix issue where sibling steps could incorrectly be accounted as
OOM when step memory limit was the same as the job allocation. Detect OOM
events via memory.oom_control oom_kill when exposed by the kernel instead of
subscribing notifications with eventfd.
-- Fix accounting of oom_kill events in cgroup/v2 and task/cgroup.
-- Fix segfault when slurmd reports less than configured gres with links after
a slurmctld restart.
-- Fix TRES counts after node is deleted using scontrol.
-- sched/backfill - properly handle multi-reservation HetJobs.
-- sched/backfill - don't try to start HetJobs after system state change.
-- openapi/v0.0.38 - add submission of job->prefer value.
-- slurmdbd - become SlurmUser at the same point in logic as slurmctld to match
plugins initialization behavior. This avoids a fatal error when starting
slurmdbd as root and root cannot start the auth or accounting_storage
plugins (for example, if root cannot read the jwt key).
-- Fix memory leak when attempting to update a job's features with invalid
features.
-- Fix occasional slurmctld crash or hang in backfill due to invalid pointers.
-- Fix segfault on Cray machines if cgroup cpuset is used in cgroup/v1.
* Changes in Slurm 22.05.3
==========================
-- job_container/tmpfs - cleanup containers even when the .ns file isn't
mounted anymore.
-- Ignore the bf_licenses option if using sched/builtin.
-- Do not clear the job's requested QOS (qos_id) when ineligible due to QOS.
-- Emit error and add fail-safe when job's qos_id changes unexpectedly.
-- Fix timeout value in log.
-- openapi/v0.0.38 - fix setting of DefaultTime when dumping a partition.
-- openapi/dbv0.0.38 - correct parsing association QOS field.
-- Fix LaunchParameters=mpir_use_nodeaddr.
-- Fix various edge cases where accrue limits could be exceeded or cause
underflow error messages.
-- Fix issue where a job requesting --ntasks and --nodes could be wrongly
rejected when spanning heterogeneous nodes.
-- openapi/v0.0.38 - detect when partition PreemptMode is disabled
-- openapi/v0.0.38 - add QOS flag to handle partition PreemptMode=within
-- Add total_cpus and total_nodes values to the partition list in
the job_submit/lua plugin.
-- openapi/dbv0.0.38 - reject and error on invalid flag values in well defined
flag fields.
-- openapi/dbv0.0.38 - correct QOS preempt_mode flag requests being silently
ignored.
-- accounting_storage/mysql - allow QOS preempt_mode flag updates when GANG
mode is requested.
-- openapi/dbv0.0.38 - correct QOS flag modifications request being silently
ignored.
-- sacct/sinfo/squeue - use openapi/[db]v0.0.38 for --json and --yaml modes.
-- Improve error messages when using configless and fetching the config fails.
-- Fix segfault when reboot_from_controller is configured and scontrol reboot
is used.
-- Fix regression which prevented a cons_tres gpu job to be submitted to a
cons_tres cluster from a non-con_tres cluster.
-- openapi/dbv0.0.38 - correct association QOS list parsing for updates.
-- Fix rollup incorrectly divying up unused reservation time between
associations.
-- slurmrestd - add SLURMRESTD_SECURITY=disable_unshare_files environment
variable.
-- Update rsmi detection to handle new default library location.
-- Fix header inclusion from slurmstepd manager code leading to multiple
definition errors when linking --without-shared-libslurm.
-- slurm.spec - explicitly disable Link Time Optimization (LTO) to avoid
linking errors on systems where LTO-related RPM macros are enabled by
default and the binutils version has a bug.
-- Fix issue in the api/step_io message writing logic leading to incorrect
behavior in API consuming clients like srun or sattach, including a segfault
when freeing IO buffers holding traffic from the tasks to the client.
-- openapi/dbv0.0.38 - avoid job queries getting rejected when cluster is not
provided by client.
-- openapi/dbv0.0.38 - accept job state filter as verbose names instead of
only numeric state ids.
-- Fix regression in 22.05.0rc1: if slurmd shuts down while a prolog is
running, the job is cancelled and the node is drained.
-- Wait up to PrologEpilogTimeout before shutting down slurmd to allow prolog
and epilog scripts to complete or timeout. Previously, slurmd waited 120
seconds before timing out and killing prolog and epilog scripts.
-- GPU - Fix checking frequencies to check them all and not skip the last one.
-- GPU - Fix logic to set frequencies properly when handling multiple GPUs.
-- cgroup/v2 - Fix typo in error message.
-- cgroup/v2 - More robust pattern search for events.
-- Fix slurm_spank_job_[prolog|epilog] failures being masked if a Prolog or
Epilog script is defined (regression in 22.05.0rc1).
-- When a job requested nodes and can't immediately start, only report to
the user (squeue/scontrol et al) if nodes are down in the requested list.
-- openapi/dbv0.0.38 - Fix qos list/preempt not being parsed correctly.
-- Fix dynamic nodes registrations mapping previously assigned nodes.
-- Remove unnecessarily limit on count of 'shared' gres.
-- Fix shared gres on CLOUD nodes not properly initializing.
* Changes in Slurm 22.05.2
==========================
-- Fix a segfault in slurmctld when requesting gres in job arrays.
-- Prevent jobs from launching on newly powered up nodes that register with
invalid config.
-- Fix a segfault when there's no memory.swap.current interface in cgroup/v2.
-- Fix memleak in cgroup/v2.
* Changes in Slurm 22.05.1
==========================
-- Flush the list of Include config files on SIGHUP.
-- Fix and update Slurm completion script.
-- jobacct_gather/cgroup - Add VMem support both for cgroup v1 and v2.
-- Allow subset of node state transitions when node is in INVAL state.
-- Remove INVAL state from cloud node after being powered down.
-- When showing reason UID in scontrol show node, use the authenticated UID
instead of the login UID.
-- Fix calculation of reservation's NodeCnt when using dynamic nodes.
-- Add SBATCH_{ERROR,INPUT,OUTPUT} input environment variables for --error,
--input and --output options respectively.
-- Prevent oversubscription of licenses by the backfill scheduler when not
using the new "bf_licenses" option.
-- Jobs with multiple nodes in a heterogeneous cluster now have access to all
the memory on each node by using --mem=0. Previously the memory limit was
set by the node with the least amount of memory.
-- Don't limit the size of TaskProlog output (previously TaskProlog output was
limited to 4094 characters per line, which limited the size of exported
environment variables or logging to the task).
-- Fix usage of possibly uninitialized buffer in proctrack/cgroup.
-- Fix memleak in proctrack/cgroup proctrack_p_wait.
-- Fix cloud/remote het srun jobs.
-- Fix a segfault that may happen on gpu configured as no_consume.
* Changes in Slurm 22.05.0
==========================
-- openapi/v0.0.38 - add group name to job info
-- openapi/v0.0.38 - add container field to job description.
-- openapi/v0.0.38 - add container to job submission.
-- openapi/[db]v0.0.38 - add container field to job description.
-- openapi/dbv0.0.38 - fix bug where QOS update set various limits to 0.
-- openapi/dbv0.0.38 - properly initialize wckey record.
-- openapi/dbv0.0.38 - properly initialize cluster record.
-- openapi/dbv0.0.38 - gracefully update existing QOSs.
-- Fix x11 forwarding with job_container/tmpfs and without home_xauthority.
-- Fix reconfig of dynamic nodes with gres.
-- Fix corner case issues when removing assocs/qos' while jobs are completing
and a reconfigure happens.
-- Add SlurmdParameters=numa_node_as_socket to use the numa node as a socket.
-- Avoid creating and referencing NULL script/env hash entries when not storing
the job script or job environment.
-- slurmd - If a het component job fails to launch due to a node prolog or
other failure, properly handle the whole het job.
-- Fix possible race condition while handling forwarded messages.
-- Add new SchedulerParameters option "bf_licenses" to track licenses as
within the backfill scheduler.
-- Fix resuming nodes not part of an allocation.
-- Minor memory leak fixes.
-- Use accept4() and pipe2() to ensure new file descriptors are always set
with close-on-exec flag.
-- slurmctld - allow users to request tokens for themselves by name.
-- srun - adjust output of "--mpi=list".
-- Fix "--gpu-bind=single:" having wrong env variables.
-- Sync missing pieces in slurmctld SIGHUP handler with respect to
REQUEST_RECONFIGURE RPC handler.
-- Fixed the behavior of the --states=all option in squeue to actually show
all of the states.
-- job_container - Don't constrain a job using --no-allocate.
-- Fix slurmctld segfault when a step requests a gres without a file.
-- Fix slurmctld segfault when cleaning up an overlapping step.
-- Dynamic node fixes and updates.
-- Added error logging when a node goes into an invalid state so that the user
can still see the reason even if the user set a custom reason beforehand.
-- Remove redundant message when libdbus is found during configure.
-- Fix pmix to honor SrunPortRange.
-- Fix issue with ntasks_per_socket and ntasks_per_core with --exclusive
-- openapi/v0.0.38 - change job parsing error message to reflect reality.
-- Avoid segfault/assert when --thread-spec used with config_overrides
-- Don't send SIGTERM to prematurely in run_command.
-- Add logging to inform where networking errors are occurring on slurmd end.
-- openapi/dbv0.0.38 - Enforce GET /association/ to dump only a single
association.
-- openapi/dbv0.0.38 - Enforce DELETE /association/ to delete only a single
association.
-- openapi/dbv0.0.38 - Add parameters to filter to GET /associations/.
-- openapi/dbv0.0.38 - Add DELETE /associations using parameters to set filters.
-- openapi/dbv0.0.38 - Fix plural form of delete association response schema.
* Changes in Slurm 22.05.0rc1
=============================
-- gres/gpu - Avoid stripping GRES type field during normalization if any
other GRES have a defined type field.
-- burst_buffer/datawarp - free bb_job after stage-out or teardown are done.
-- acct_gather_energy_rsmi has been renamed acct_gather_energy_gpu.
-- Remove support for (non-functional) --cpu-bind=boards.
-- accounting_storage/mysql - ensure new non-HetJobs have het_job_offset NO_VAL
in the database and fix the same field when retrieving older bad records.
-- PreemptMode now works as a condition for qos in sacctmgr.
-- scancel - add "--me" option.
-- gres/gpu - Fix configured/system-detected GRES match for some combinations
for which the sorting affected the expected selection result.
-- openapi/v0.0.38 - Fork existing openapi/v0.0.37 plugin.
-- openapi/dbv0.0.38 - Fork existing openapi/dbv0.0.37 plugin.
-- openapi/v0.0.35 - Plugin has been removed.
-- scrontab - Don't accept extra letters after a '@' repeating pattern.
-- openapi/dbv0.0.38 - Add missing method POST for /associations/.
-- Make DefMemPerCPU/MaxMemPerCPU and DefMemPerNode/MaxMemPerNode precedence
at the global level the same as in partition level, and print an error
if both of a pair are set.
-- openapi/v0.0.38 - Allow strings for JobIds instead of only numerical JobIds
for GET, DELETE, and POST job methods.
-- openapi/v0.0.38 - enable job priority field for job submissions and updates.
-- openapi/v0.0.38 - request node states query includes MIXED state instead of
only allocated.
-- openapi/dbv0.0.38 - Correct tree position of dbv0.0.38_job_step.
-- Enforce all Slurm plugin requirements for plugins specified with absolute
path.
-- Added --prefer option at job submission to allow for 'soft' constraints.
-- Remove support for PMIx 1.x
-- slurmrestd - Unlink unix sockets before binding to avoid errors where
previous run of slurmrestd did not properly cleanup the sockets.
-- Add extra 'EnvironmentFile=-/etc/default/$service' setting to service files.
-- slurmctld/agent - spawn more agent threads in a single _agent_retry().
-- openapi/v0.0.38 - change job response types to more specific types than
generic string.
-- srun - refuse to run on malformed SPANK environment variable.
-- Allow jobs to pack onto nodes already rebooting with the desired features.
-- Reset job start time after nodes are rebooted.
-- Node features (if any) are passed to RebootProgram if run from slurmctld.
-- Fix sending multiple features to RebootProgram.
-- If a task/srun prolog fails don't allow the step to continue running.
-- Fail srun when using invalid --cpu-bind options.
-- cgroup/v1 - Set swappiness at job level instead of at root level.
-- Fix issues where a step's GRES request could never be satisfied but the step
remained pending forever instead of being rejected.
-- openapi/v0.0.38 - Remove errant extra space after JOB_CPUS_SET flag.
-- slurmrestd - refuse to run with gid 0, or under SlurmUser's gid.
-- Storing batch scripts and env vars are now in indexed tables using
substantially less disk space. Those storing scripts in 21.08 will all