- The Job Router can now create an IDTOKEN for use by the job
- Fix bug where a self-checkpointing job may erroneously be held
- Fix bug where the Job Router could erroneously substitute a default value
- Fix bug where a file transfer error may identify the wrong file
- Fix bug where condor_ssh_to_job may fail to connect

- Fixes for security issues
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0001
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0002
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2022-0003

- Added Debian 11 (bullseye) as a supported platform
- Since CentOS 8 has reached end of life, we build and test on Rocky Linux 8
- The OAUTH credmon is now packaged for Enterprise Linux 8

- Fix bug where huge values of ImageSize and others would end up negative
- Fix bug in how MAX_JOBS_PER_OWNER applied to late materialization jobs
- Fix bug where the schedd could choose a slot with insufficient disk space
- Fix crash in ClassAd substr() function when the offset is out of range
- Fix bug in Kerberos code that can crash on macOS and could leak memory
- Fix bug where a job is ignored for 20 minutes if the startd claim fails

- Fix bug where condor_gpu_discovery could crash with older CUDA libraries
- Fix bug where condor_watch_q would fail on machines with older kernels
- condor_watch_q no longer has a limit on the number of job event log files
- Fix bug where a startd could crash claiming a slot with p-slot preemption
- Fix bug where a job start would not be recorded when a shadow reconnects

- CUDA_VISIBLE_DEVICES can now contain GPU-<uuid> formatted values
- Fixed a bug that caused jobs to fail when using newer Singularity versions
- Fixed a bug in the Windows MSI installer for the latest Windows 10 version
- Fixed bugs relating to the transfer of standard out and error logs
- MacOS 11.x now reports as 10.16.x (which is better than reporting x.0)

- Other authentication methods are tried if mapping fails using SciTokens
- Fix rare crashes from successful condor_submit, which caused DAGMan issues
- Fix bug where ExitCode attribute would be suppressed when OnExitHold fired
- condor_who now suppresses spurious warnings coming from netstat
- The online manual now has detailed instructions for installing on MacOS
- Fix bug where misconfigured MIG devices confused condor_gpu_discovery
- The transfer_checkpoint_file list may now include input files

- Fixes for security issues
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0003.html
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0004.html

- HTCondor can be set up to use only FIPS 140-2 approved security functions
- If the Singularity test fails, the job goes idle rather than getting held
- Can divide GPU memory, when making multiple GPU entries for a single GPU
- Startd and Schedd cron job maximum line length increased to 64k bytes
- Added first class submit keywords for SciTokens
- Fixed MUNGE authentication
- Fixed Windows installer to work when the install location isn't C:\Condor

- Don't clear GridJobId for completed grid batch jobs (HTCONDOR-534)

- Remove JSON from local issuer (HTCONDOR-367)

- Add SOFTWARE-4525-condor_watch_q_crash.patch to fix a condor_watch_q crash when watching DAGs (SOFTWARE-4525)
- Add HTCONDOR-315-Always-handle-exceptions-when-decoding-.patch to fix a schedd crash caused by malformed SciTokens (SOFTWARE-4533)

- This release of HTCondor fixes security-related bugs described at
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0001.html
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2021-0002.html

- Fix bug where negotiator stopped making matches when group quotas are used
- Support OAuth, SciTokens, and Kerberos credentials in local universe jobs
- The Python schedd.submit method now takes a Submit object
- DAGMan can now optionally run a script when a job goes on hold
- DAGMan now provides a method for inline jobs to share submit descriptions
- Can now add arbitrary tags to condor annex instances
- Runs the "singularity test" before running the a singularity job

- The RPM packages requires globus, munge, scitokens, and voms from EPEL
- Improved cgroup memory policy settings that set both hard and soft limit
- Cgroup memory usage reporting no longer includes the kernel buffer cache
- Numerous Python binding improvements, see version history
- Can create a manifest of files on the execute node at job start and finish
- Added provisioner nodes to DAGMan, allowing users to provision resources
- DAGMan can now produce .dot graphs without running the workflow

- Add libcgroup-devel requirement for EL8
- Drop bosco_findplaform patch
- Update permissions on /var/{lock,run}/condor so rpm verify works on EL8
- Don't try to build contrib code (fails on EL8)

- Add patch to use bosco 1.3 in bosco_findplatform (SOFTWARE-4080)

- Fixes addressing CVE-2019-18823
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2020-0001.html
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2020-0002.html
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2020-0003.html
- https://research.cs.wisc.edu/htcondor/security/vulnerabilities/HTCONDOR-2020-0004.html

- Added a new mode that skips jobs whose outputs are newer than their inputs
- Added command line tool to help debug ClassAd expressions
- Added port forwarding to Docker containers
- You may now change some DAGMan throttles while the DAG is running
- Added support for session tokens for pre-signed S3 URLs
- Improved the speed of the negotiator when custom resources are defined
- Fixed interactive submission of Docker jobs
- Fixed a bug where jobs wouldn't be killed when getting an OOM notification

- Amazon S3 file transfers using pre-signed URLs
- Further reductions in DAGMan memory usage
- Added -idle option to condor_q to display information about idle jobs
- Support for SciTokens authentication
- A tool, condor_evicted_files, to examine the SPOOL of an idle job

- TOKEN and SSL authentication methods are now enabled by default
- The job and global event logs use ISO 8601 formatted dates by default
- Added Google Drive multifile transfer plugin
- Added upload capability to Box multifile transfer plugin
- Added Python bindings to submit a DAG
- Python 'JobEventLog' can be pickled to facilitate intermittent readers
- 2x matchmaking speed for partitionable slots with simple START expressions
- Improved the performance of the condor_schedd under heavy load
- Reduced the memory footprint of condor_dagman
- Initial implementation to record the circumstances of a job's termination

- Obsolete condor-cream-gahp < 8.9.2 to fix upgrades from versions built with CREAM support