# changelog

author: Tim Theisen <tim@cs.wisc.edu> - 23.8.1-1
changelog:
  - Add new condor-ap package to facilitate Access Point installation
  - HTCondor Docker images are now based on Alma Linux 9
  - HTCondor Docker images are now available for the arm64 CPU architecture
  - The user can now choose which submit method DAGMan will use
  - Can add custom attributes to the User ClassAd with condor_qusers -edit
  - Add use-projection option to condor_gangliad to reduce memory footprint
  - Fix bug where interactive submit does not work on cgroup v2 systems (EL9)


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.14-1
changelog:
  - Docker and Container jobs run on EPs that match AP's CPU architecture
  - Fixed premature cleanup of credentials by the condor_credd
  - Fixed bug where a malformed SciToken could cause a condor_schedd crash
  - Fixed crash in condor_annex script
  - Fixed daemon crash after IDTOKEN request is approved by the collector


author: Tim Theisen <tim@cs.wisc.edu> - 23.9.6-1
changelog:
  - Add config knob to not have cgroups count kernel memory for jobs on EL9
  - Remove support for numeric unit suffixes (k,M,G) in ClassAd expressions
  - In submit files, request_disk & request_memory still accept unit suffixes
  - Hide GPUs not allocated to the job on cgroup v2 systems such as EL9
  - DAGMan can now produce credentials when using direct submission
  - Singularity jobs have a contained home directory when file transfer is on
  - Avoid using IPv6 link local addresses when resolving hostname to IP addr
  - New 'htcondor credential' command to aid in debugging


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.15-1
changelog:
  - Fix bug where Docker universe jobs reported zero memory usage on EL9
  - Fix bug where Docker universe images would not be removed from EP cache
  - Fix bug where condor_watch_q could crash
  - Fix bug that could cause the file transfer hold reason to be truncated
  - Fix bug where a Windows job with a bad executable would not go on hold


author: Tim Theisen <tim@cs.wisc.edu> - 23.10.1-1
changelog:
  - Improvements to disk usage enforcement when using LVM
    - Can encrypt job sandboxes when using LVM
    - More precise tracking of disk usage when using LVM
    - Reduced disk usage tracking overhead
  - Improvements tracking CPU and memory usage with cgroup v2 (on EL9)
    - Don't count kernel cache pages against job's memory usage
    - Avoid rare inclusion of previous job's CPU and peak memory usage
  - HTCondor now re-checks DNS before re-connecting to a collector
  - HTCondor now writes out per job epoch history
  - HTCondor can encrypt network connections without requiring authentication
  - htcondor CLI can now show status for local server, AP, and CM
  - htcondor CLI can now display OAUTH2 credentials
  - Uses job's sandbox to convert image format for Singularity/Apptainer
  - Bug fix to not lose GPUs in Docker job on systemd reconfig
  - Bug fix for PID namespaces and condor_ssh_to_job on EL9


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.17-1
changelog:
  - Bug fix for PID namespaces and condor_ssh_to_job on EL9
  - Augment condor_upgrade_check to find unit suffixes in ClassAd expressions


author: Tim Theisen <tim@cs.wisc.edu> - 24.0.1-1
changelog:
  - Improved tracking and enforcement of disk usage by using LVM
  - Enhancements to the htcondor CLI tool
  - cgroup v2 support for tracking and enforcement of CPU and memory usage
  - Leverage cgroups to hide GPUs not allocated to the job
  - DAGMan can now produce job credentials when using direct submit
  - New submit commands to aid in matching specific GPU requirements
  - New implementation of the Python bindings, htcondor2 and classad2
  - Improved default security configuration
  - Significant reduction in memory and CPU usage on the Central Manager
  - Support for GPUs using AMD's HIP 6 library
  - Fix bugs when -divide or -repeat was used in GPU detection
  - Proper error message and hold when Docker emits multi-line error message
  - Fix issue where an unresponsive libvirtd blocked an EP from starting up


author: Tim Theisen <tim@cs.wisc.edu> - 24.1.1-1
changelog:
  - Can print contents of stored OAuth2 credential with htcondor CLI tool
  - In DAGMan, inline submit descriptions work when not submitting directly
  - By default, put Docker jobs on hold when CPU architecture doesn't match
  - Detects and deletes invalid checkpoint and reschedules job


author: Tim Theisen <tim@cs.wisc.edu> - 24.0.2-1
changelog:
  - Add STARTER_ALWAYS_HOLD_ON_OOM to minimize confusion about memory usage
  - Fix bug that caused condor_ssh_to_job sftp and scp modes to fail
  - Fix KeyboardIdle attribute in dynamic slots that could prevent job start
  - No longer signals the OAuth credmon when there is no work to do
  - Fix rare condor_schedd crash when a $$() macro could not be expanded
  - By default, put Docker jobs on hold when CPU architecture doesn't match


author: Tim Theisen <tim@cs.wisc.edu> - 24.2.1-1
changelog:
  - Fixed DAGMan's direct submission of late materialization jobs
  - New primary_unix_group submit command that sets the job's primary group
  - Initial implementation of broken slot detection and reporting
  - New job attributes FirstJobMatchDate and InitialWaitDuration
  - condor_ssh_to_job now sets the supplemental groups in Apptainer
  - MASTER_NEW_BINARY_RESTART now accepts the FAST parameter
  - Avoid blocking on dead collectors at shutdown
  - IPv6 networking is now fully supported on Windows


author: Tim Theisen <tim@cs.wisc.edu> - 24.2.2-1
changelog:
  - Prevent the startd from removing all files if EXECUTE is an empty string
    - This problem first appeared in the withdrawn HTCondor 24.2.1 version


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.18-1
changelog:
  - Proper error message and hold when Docker emits multi-line error message
  - The htcondor CLI now works on Windows


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.19-1
changelog:
  - Numerous updates in memory tracking with cgroups
    - Fix bug in reporting peak memory
    - Made cgroup v1 and v2 memory tracking consistent with each other
    - Fix bug where cgroup v1 usage included disk cache pages
    - Fix bug where cgroup v1 jobs killed by OOM were not held
    - Polls cgroups for memory usage more often
    - Can configure to always hold jobs killed by OOM
  - Make condor_adstash work with OpenSearch Python Client v2.x
  - Avoid OAUTH credmon errors by only signaling it when necessary
  - Restore case insensitivity to 'condor_status -subsystem'
  - Fix rare condor_schedd crash when a $$() macro could not be expanded


author: Tim Theisen <tim@cs.wisc.edu> - 23.10.19-1
changelog:
  - Fix bug where jobs would match but not start when using KeyboardIdle
  - Fix bug when trying to avoid IPv6 link local addresses


author: Tim Theisen <tim@cs.wisc.edu> - 24.3.0-1
changelog:
  - Allow local issuer credmon and Vault credmon to coexist
  - Add Singularity launcher to distinguish runtime failure from job failure
  - Advertises when the EP is enforcing disk usage via LVM
  - By default, LVM disk enforcement hides mounts when possible
  - Container Universe jobs can now mount a writable directory under scratch
  - Pass PELICAN_* job environment variables to pelican file transfer plugin
  - Fix HTCondor startup when network interface has no IPv6 address
  - VacateReason is set in the job ad under more circumstances
  - 'htcondor job submit' now issues credentials like 'condor_submit' does


author: Tim Theisen <tim@cs.wisc.edu> - 23.10.20-1
changelog:
  - Fix bug where STARTD_ENFORCE_DISK_LIMITS would excessively save metadata
  - Fix bug where container_service_names did not work
  - Fix rare startd crash when collector queries time out and DNS is slow


author: Tim Theisen <tim@cs.wisc.edu> - 24.0.4-1
changelog:
  - New arc_data_staging submit command to add DataStaging block to ARC ADL
  - Fix bug where the negotiator could crash when matching offline ads
  - Fix memory leak in SCHEDD_CRON script that produce standard output
  - Fix bug where the schedd could crash if cron script runs during shutdown


author: Tim Theisen <tim@cs.wisc.edu> - 24.4.0-1
changelog:
  - Improved validation and cleanup of EXECUTE directories
  - For batch grid universe, the PATH comes from the job ad and worker node
  - Improved 'condor_q -better-analyze' for pools with partionable slots
  - The EP advertizes if Singularity is using user namespaces
  - The EP advertizes average and total bytes transferred to and from jobs
  - The condor_credmon now utilizes the shared port daemon


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.21-1
changelog:
  - Fix memory leak caused by periodic evaluation of bad ClassAd expressions
  - Fixes for bugs affecting grid jobs


author: Tim Theisen <tim@cs.wisc.edu> - 24.0.5-1
changelog:
  * Thu Feb 27 2025 Tim Theisen <tim@cs.wisc.edu> - 23.10.21-1
  - Fix bug where chirp would not work in container jobs using Docker
  - HTCondor tarballs now contain Pelican 7.13.0


author: Tim Theisen <tim@cs.wisc.edu> - 24.5.1-1
changelog:
  - Can now configure APs to acquire credentials for jobs in multiple ways
  - HTCondor marks slots as broken when the slot resources cannot be released
  - HTCondor now reliably cleans up LVM volumes used by jobs
  - HTCondor now advertises NVIDIA driver version
  - To detect stuck jobs, last write to stdout and stderr are in the job ad


author: Tim Theisen <tim@cs.wisc.edu> - 24.7.3-1
changelog:
  - condor_who now works for Glideins
  - Can add arbitrary credentials to be used by the file transfer plugins
  - Fixed WLCG token generation in the local credmon
  - Can limit the number of times that a job can be released
  - EP administrators can enforce no outbound networking for jobs
  - Add ability to use authentication when fetching Docker images
  - condor_watch_q now displays when file transfer is happening
  - To provide more consistency, using swap for jobs is disabled by default


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.25-1
changelog:
  - Fix problems where parallel universe jobs could crash the condor_schedd
  - Prevent condor_starter crash when evicting job during input file transfer
  - condor_watch_q now properly displays job id ranges by using numeric sort


author: Tim Theisen <tim@cs.wisc.edu> - 23.10.25-1
changelog:
  - Fix bug where DAGMAN_MAX_JOBS_IDLE was being ignored
  - HTCondor tarballs now contain Pelican 7.16.5 and Apptainer 1.4.1


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.8-1
changelog:
  - Fix 24.0.7 bug where cgroup v1 out-of-memory was not properly handled
  - HTCondor tarballs now contain Pelican 7.16.5 and Apptainer 1.4.1
    - Pelican 7.16.5 now includes end-to-end integrity checks for clients
  - Add Python wheel for Python 3.13, drop Python wheel for Python 3.7
  - Fix bug where DAGMAN_MAX_JOBS_IDLE was being ignored
  - Fix problems where parallel universe jobs could crash the condor_schedd
  - Prevent condor_starter crash when evicting job during input file transfer
  - condor_watch_q now properly displays job id ranges by using numeric sort


author: Tim Theisen <tim@cs.wisc.edu> - 24.8.1-1
changelog:
  - Fix claim re-use, which was broken in HTCondor version 24.5.1
  - Add support for hierarchic and delegatable v2 cgroups
  - Add the ability to put each HTCondor daemon in its own cgroup
  - Always sets the execute bit on the executable regardless of its origin
  - The EP sets the HOME environment variable to match the /etc/passwd entry
  - Add new 'halt' and 'resume' verbs to "htcondor dag"
  - Add htcondor2.DAGMan class to send commands to a running DAG
  - htcondor ap status now reports the AP's RecentDaemonCoreDutyCycle
  - Can configure condor_adstash to fetch a custom projection of attributes


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.26-1
changelog:
  - Fix ingestion of ads into Elasticsearch under very rare circumstances
  - DAGMan better handles being unable to write to a full filesystem
  - 'kill_sig' submit commands are now ignored on the Windows platform


author: Tim Theisen <tim@cs.wisc.edu> - 23.10.26-1
changelog:
  - Fix memory leak in the condor_schedd when using late materialization
  - Fix condor_master start up when file descriptor ulimit was huge
  - HTCondor tarballs now contain Pelican 7.17.0


author: Tim Theisen <tim@cs.wisc.edu> - 24.9.2-1
changelog:
  - Initial Support for Enterprise Linux 10, including the x86_64_v2 platform
  - New job attribute to report number of input files transferred by protocol
  - Optional condor_schedd history log file
  - condor_watch_q can now track DAGMan jobs when using the -clusters option
  - Fix bug that caused claim failure when previous output transfer failed
  - Fix bug where access tokens were not generated from Vault tokens


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.6-1
changelog:
  - Fix CE job route transform for job environment
  - Fix CERequirements when the default_CERequirements is not set
  - Add condor_ce_test_token tool to generate short lived SciToken for tests
  - Remove GSI from security method list to eliminate annoying warnings


author: Tim Theisen <tim@cs.wisc.edu> - 23.9.1-1
changelog:
  - Use new Job Router syntax by default
  - Update configuration files to work with HTCondor 23.9.1 and later
  - Use BatchQueue job attribute in CE routes


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.18-1
changelog:
  - Does not pass WholeNode request expressions to non-HTCondor batch systems
  - Fix certificate subject parsing in condor_ce_host_network_check


