v4.0.0

Summary

This release delivers comprehensive logging improvements with correlation ID support, enhanced Globus integration, repository management enhancements, and critical security updates. The release includes significant infrastructure improvements, dependency updates, and numerous bug fixes that improve system reliability and user experience.

Images

Images can be pulled using the overal application version number

docker pull savannah.ornl.gov/datafed/core:v4.0.0
docker pull savannah.ornl.gov/datafed/web:v4.0.0
docker pull savannah.ornl.gov/datafed/foxx:v4.0.0
docker pull savannah.ornl.gov/datafed/gcs:v4.0.0
docker pull savannah.ornl.gov/datafed/repo:v4.0.0

They can be pulled with overal application version number and component version.

docker pull savannah.ornl.gov/datafed/core:v4.0.0-2.0.0
docker pull savannah.ornl.gov/datafed/foxx:v4.0.0-1.2.0
docker pull savannah.ornl.gov/datafed/gcs:v4.0.0-2.0.0
docker pull savannah.ornl.gov/datafed/repo:v4.0.0-2.0.0
docker pull savannah.ornl.gov/datafed/web:v4.0.0-2.0.0

They can be pulled by date of release.

docker pull savannah.ornl.gov/datafed/core:v2025.12.09.10.00
docker pull savannah.ornl.gov/datafed/web:v2025.12.09.10.00
docker pull savannah.ornl.gov/datafed/foxx:v2025.12.09.10.00
docker pull savannah.ornl.gov/datafed/gcs:v2025.12.09.10.00
docker pull savannah.ornl.gov/datafed/repo:v2025.12.09.10.00

They can be pulled by commit sha.

docker pull savannah.ornl.gov/datafed/core:7b76aa37a95b46b0501d09e090e9fa316743bf55
docker pull savannah.ornl.gov/datafed/web:7b76aa37a95b46b0501d09e090e9fa316743bf55
docker pull savannah.ornl.gov/datafed/foxx:7b76aa37a95b46b0501d09e090e9fa316743bf55
docker pull savannah.ornl.gov/datafed/gcs:7b76aa37a95b46b0501d09e090e9fa316743bf55
docker pull savannah.ornl.gov/datafed/repo:7b76aa37a95b46b0501d09e090e9fa316743bf55

MAJOR - Breaking Changes

Changes that require user action and may break existing integrations

• DAPS-1688 #1689 refactor: core/repo/authz/gcs - Migrate from OpenSSL 1.x to OpenSSL 3 - Applications linking against DataFed libraries may need compatibility updates
• DAPS-1725 #1726 refactor: tests - Relocate mock core service from repository/gridftp/globus5/authz/mock_core/ to tests/mock_core/ - Tools referencing old location must be updated
• DAPS-1659 #1660 refactor: scripts - Remove dependency install scripts and externalize to Git submodule external/DataFedDependencies - All dependency script paths must be updated
• refactor: build - Require Git submodule initialization - Build process now requires git submodule update --init --recursive

MINOR - Features & Enhancements

New functionality and improvements that maintain backward compatibility

• DAPS-1663 #1704 feature: core - Add Correlation ID to Core Service logging for improved request tracing
• DAPS-1522 #1758 feature: foxx - Add comprehensive logging improvements to Version Router
• DAPS-1522 #1734 feature: foxx - Add comprehensive logging improvements to Tag Router
• DAPS-1522 #1627 feature: foxx - Add comprehensive logging improvements to Query Router
• DAPS-1522 #1648 feature: foxx - Add comprehensive logging improvements to Task Router
• DAPS-1522 #1629 feature: foxx - User router logging improvements, remove non-helpful logs from tasks.js
• DAPS-1675 #1675 feature: foxx - Add logger functions for improved debugging
• DAPS-1731 #1731 feature: scripts/compose - Add scripts to generate Globus credentials for web service
• DAPS-1741 #1741 feature: scripts - Update native client ID in initialize_globus_endpoint and globus_clean scripts
• DAPS-1676 #1697 feature: foxx/arango - Add factory pattern for repositories supporting metadata and Globus
• DAPS-1705 #1706 feature: foxx - Integrate metadata globus factory in repo router create
• DAPS-1711 #1712 feature: foxx - Standardize repo response schema
• DAPS-1713 #1714 feature: core/web/python - Make fields optional when creating repo to support metadata repositories
• DAPS-1715 #1716 feature: core - Make path, pub_key, address, endpoint optional in RepoCreateRequest
• DAPS-1718 #1719 feature: protobuf - Add ExecutionMethod enum and RepoAllocationCreateResponse
• DAPS-1670 #1670 feature: common/core/repo/python/web - Allow passing repo types in protobuf messages
• DAPS-1671 #1672 feature: foxx - Add repository and execution strategy types
• DAPS-1685 #1687 feature: compose - Enable ArangoDB SSL/TLS support in Docker Compose deployment
• DAPS-1585 #1646 update: dependencies - Upgrade SSL dependency to version 3.2.5
• DAPS-1707 #1709 feature: web - Update web dependencies before installation
• feature: ci - Add base-image-check stage to GitLab CI pipeline
• feature: ci - Enable recursive submodule strategy with GIT_SUBMODULE_STRATEGY: recursive
• feature: cmake - Add DENABLE_INTEGRATION_TESTS flag for conditional test building

PATCH - Bug Fixes, Technical Debt & Improvements

Bug fixes, refactoring, documentation, and other improvements

• DAPS-1378 #1753 fix: web - Fix mapping of multiple Globus accounts to DataFed users
• DAPS-1777 #1778 Foxx, fix: user_router fix regression in missing response.
• DAPS-1774 #1779 Core, Python, Database, Foxx, Test add query end to end tests
• DAPS-1766 fix: ci - Fix Python client provisioning job failures
• DAPS-1735 #1736 fix: foxx - Remove duplicate user_router test
• DAPS-1725-2 #1747 fix: tests - Fix centralized mock core service libraries (part 2)
• DAPS-1692 #1693 fix: ci - Fix flaky end-to-end web test reliability
• DAPS-1729 #1732 fix: ci - Fix downstream DataFed dependencies pipelines building from incorrect SHA
• DAPS-1756 #1757 fix: scripts/foxx - Add retries and connection checks to install_foxx.sh script
• DAPS-1605 #1623 fix: scripts - Fix install_foxx.sh by splitting ssl_args
• DAPS-1688 #1689 refactor: dependencies - Pin Globus SDK version for stability
• DAPS-1737 #1765 refactor: compose - Clean up ArangoDB SSL environment variables
• DAPS-1749 #1750 refactor: cmake - Set CMake policy to silence noisy warnings
• DAPS-1700 #1701 refactor: ci - Limit ArangoDB job output to last 3 hours
• DAPS-1742 #1744 refactor: scripts - Replace os.path.join with urllib.parse.urljoin for proper URL handling
• DAPS-1745 #1746 refactor: scripts - Update scripts to account for nested client credentials
• DAPS-1694 #1695 refactor: foxx - Move permissions functions from support.js to lib/permissions
• DAPS-1691 #1691 refactor: foxx - Remove circular dependency from validation.js
• DAPS-1725 #1728 refactor: config - Remove confusing Apache configuration file
• DAPS-1651 #1656/#1658 refactor: scripts/compose - Unify treatment of environment variables in compose env generator
• DAPS-1661 #1661 refactor: compose/scripts - Remove remaining occurrences of ZeroMQ system secret
• refactor: build - Standardize YAML indentation across all configuration files
• refactor: gitignore - Add patterns for *.tar.gz, *.tgz, and tests/mock_core/Version.hpp
• refactor: docker - Improve Dockerfile layer caching and dependency management
• refactor: ci - Implement job output limiting for long-running job
• refactor: only print subset of user properties.

Contributors

Joshua S Brown [email protected]
JoshuaSBrown [email protected]
Blake Nedved [email protected]
Austin Hampton [email protected]
Blake Nedved [email protected]
Polina Shpilker [email protected]

Full Changelog: v3.14.01...v4.0.0

v3.14.01

What's Changed

• [DAPS-1625] - user id map for session key by @JoshuaSBrown, @AronPerez in #1647

Full Changelog: v3.14.0...v3.14.01

v3.14.0

Merged

• Individual docker scripts #1329
• bug: Fixed permission handling within dockerfile.foxx #1621
• [TASK] Update JS and Shell CI's to Format and Commit #1620
• fix: make secret optional before removing completely. #1617
• [DAPS-1605] feature openshift support foxx #1606
• Formatting remaining scripts within DataFed #1613
• style:Formatted dependency scripts #1589
• [DAPS-1568] Formatting install scripts #1592
• [DAPS-1568] Formatted uninstall and util scripts #1594
• [DAPS-1522] Arango Removing Unneccessary Logs #1601
• [DAPS-1531] refactor useless secret user router part 3 #1611
• style: Formatted containter scripts #1588
• style:properly formatted admin_send_email_notification.sh #1581
• style: Formatted admin_toggle_user_acccount.sh #1582
• style: Formatted compose scripts #1586
• [DAPS-1568] Formatted run scripts #1593
• [DAPS-1568] Formatted globus scripts #1595
• style: Formatted export_dependency_version #1590
• [TASK] Update script for GCS #1607
• [DAPS-1572] Centralize timeout values using env variable within end-to-end testing #1598
• [DAPS-1568] Formatted generate scripts #1591
• refactor: remove system secret from http params. #1597
• style: Formatted admin_update_config_message.sh #1583
• 1596 daps feat support core service openshift #1600
• style: Formatted clear_db.sh #1587
• style:formatted the admin_get_user_emails.sh with shtmf #1571
• style: Formatted ci scripts #1584
• refactor: remove useless secret from user_router. #1577
• fix: memory bug fixed in Condition.hpp and AuthenticationManager #1574
• [DAPS-1551] ci job fail correctly on failed curl #1556
• fix:Fixed the double free issue within the GlobusAPI constructor and move operator #1550
• [TASK] Format yaml #1569
• refactor - fix exception messages in resetKey, setAccessCount, getAccessCount in AuthMap #1554
• fix: promotion logic in promotion core service condition #1552
• [DAPS-1529] Rework generate env script #1545
• fix: Added appropriate move to AuthenticationManager #1548
• Austin remove dead code #1526
• fix: add unauthorized error check in ci infrastructure job check. #1535
• fix: make protobuf file generation and compile order explicit to prevent race condition. #1528
• fix:Changed the base image from focal:ubuntu to debian:bookworm-slim #1521
• Security updates to containers #1336
• feat(yaml formatter): added yaml formatter check action file. #1511
• freature(github action): Add shell-formatter check #1509
• -Dependency_install_functions.sh #1507
• formatting(foxx) ran prettier fix #1506
• [Release] v3.13.01 Resolve Devel Branch Diff #1486
• [DAPS-1331] Collection ID Updated on restart #1468

v3.13.01

MAJOR Breaking changes

MINOR Feature

• #1392 [DAPS-1388] Add GCP Support
• #1400 Set reasonable default that will help with filling in repo form for core t…
• #1427 Improve serialization randomization
• #1431 [DAPS-1408] Provenance Visual Management Lint (2/2)
• #1447 1445 docker logs multiple containers
• #1446 Allow looping over containers if there is more than one.
• #1468 [DAPS-1331] Collection ID Updated on restart

PATCH Bug fixes/Technical Debt/Documentation

• #1335 Add .venv/ to gitignore
• #1404 Add missing python module
• #1393 Add documentation describing how to test the foxx microservices unit …
• #1390 Fix ci build
• #1426 Fix docker logs output in end-to-end tests
• #1428 Fix regression
• #1440 Upgrade formatter
• #1438 1437 fix zmq inproc bind order
• #1444 Fix conflicting dependency install flags
• #1424 [DAPS-1395] Address Memory Leaks in Core Server
• #1454 [TASK] Update tests and fixture
• #1489 [TASK] Remove icons from labels in Provenance Visuals

Full Changelog: v2025.3.31.15.03...v3.13.01-rc.4

v2025.3.31.15.03

MAJOR Breaking changes

MINOR Feature

PATCH Bug fixes/Technical Debt/Documentation

1350 - Provenance Capture Update Bug

v2025.3.31.15.01

DataFed-v2025.3.31.15.01

This is to capture the changes during release that were added. We are now going to standardize versions as follows

-patch
-hotfix
every 3 months
.1 → .2 → .3

v2025.3.12.15.01 →v2025.3.31.15.02
         ^ ^ ^   
        / / /
          12 -> major
          15 -> minor
          01 -> patch

MAJOR Breaking changes

• Merge hotfixes into devel by @t-ramz in #1326
• JSON parse error fix by @t-ramz in #1328

MINOR Feature

• Feature DAPS 1215 foxx UserGetAccessTokenRequest mapped collection support by @t-ramz in #1284
• Add small fix for storeCollectionId by @t-ramz in #1341
• [DAPS-1330] Address Download Extension Transfer by @AronPerez in #1340
• Feature - Mapped Collection Token Support by @t-ramz in #1334
• [DLT-1110] Implement Consent Required Action (3/4 & 4/4) by @AronPerez in #1242

PATCH Bug fixes/Technical Debt/Documentation

• Release february 2025 by @JoshuaSBrown in #1301
• Fix bug with playwright by @JoshuaSBrown in #1343
• Parse md json strings by @t-ramz in #1347

Full Changelog: v2025.2.12.14.00...v2025.3.31.15.01

v2025.3.12.15.01

DataFed-v2025.3.12.15.01 Pre-release

Pre-release

MAJOR Breaking changes

MINOR Feature

[987] - This implements serialized GitLab CI pipelines
[918] - Add admin script for making a user an admin
[1009] - Add collections to database for holding Globus tokens.
[1002] - Add backoff for task worker when database fails to prevent log overflow.
[1109] - Add support for Globus token association to Globus collection and user.
[1215] - Add support for fetching Globus tokens associated with Globus collection and user.
[1214] - Add support for mapped collections on transfer tasks

PATCH Bug fixes/Technical Debt/Documentation

[984] - Fixes {server_default} from showing up in path.
[990] - Will stop running containers so that we can prune them.
[988] - Removed non-working subscribe function.
[958] - Addresses issues raised by static code analyzer.
[962] - Adds script that will check that docker image is in registry
[1015] - Uses abs path to ci pipeline script for gcs build jobs
[1013] - add set +e and set -e around is active check for arango service
[1011] - Add python script and ci job to clean up GCS node keys.
[1023] - Address unbound variables in harbor ci script.
[1027] - Fix clear_db.sh script by replacing console.log with print
[1012] - Allow customized base build image for Docker dependencies and runtime Dockerfiles
[986] - Design improvement to upload and download transfer box.
[985] - Handles longer than needed timeouts on ui pages.
[1053] - Set CMake to enable foxx tests when built in the CI.
[1086] - Address XML-RPC bug from deprecated client
[1149] - Docker container GCS Collection Mount Bug Fix
[1168] - Add authz unit testing to the CI
[1200] - Add JavaScript linter (eslint) and (prettier) formatter for JavaScript
[1180] - Refactor of authz foxx module, split into objects and added unit tests
[1223] - Fix deprecated method usage for Protobuf in Python client library
[1257] - Bug in authz library parsing of authz.conf file, globus paths incorrectly sanitized when using just '/'
[1255] - Fixes bug, in libauthz with log_path var not being read in from config file.
[1268] - Bug fixes unbound variable in foxx entrypoint file.
[1269] - Update Arangodb to 3.12.4
[1288] - Bug Jupyter Notebook in documentation were fixed.
[1321] - Refactor, allow core config threads via env vars

DataFed-v2025.2.16.21.15

PATCH Bug fixes/Technical Debt/Documentation

#1322 Fix configuration setting for allowing a custom number of threads.

DataFed-v2025.2.12.14.03

Release

MAJOR Breaking changes

MINOR Feature

#1240 [DLT-1110] Mapped Collection Endpoint Browse (1/4)
#1231 Revert "[DLT-1110] Mapped Collection Endpoint Browse (1/4)"
#1228 Feature DLT 1120 retrieve transfer token
#1224 [DAPS, Foxx] - 1180 refactor Part 2 authz
#1218 1216 daps feature add improved ci scanning
#1217 Feature DLT 1127 relay new token data
#1210 1209 feature add jsdoc linter
#1196 Add support for authz lookup
#1194 Feature DLT 1109 foxx new token data
#1192 Add DataFed dependency paths to curl so SSL will be found
#1160 1106 updated proto messages for setting token
#1141 DLT 1119 Provide method for Globus Authorize URL

PATCH Bug fixes/Technical Debt/Documentation

#1298 Fix bug in CI scripts associated with repo and gcs image build
#1239 Applied fix to deprecation warning
#1289 Add github action for linting jupyter
#1286 Update the schema documentation to improve the usability.
#1291 Update python notebooks
#1270 Add env var
#1271 Update Arangodb packaged version to using 3.12.4
#1274 Fix confusion in key uses in end to end tests
#1260 1255 feature mock core server initial integration testing authz
#1262 Address JSDoc complaints
#1258 Fix edge case in GridFTP Authz where '/' is used
#1251 Bug daps 1243 datafed web logging
#1238 Throw error if OpenStack error code returned from API
#1234 Fix bug on task_router.js abort function
#1227 Update nlhoman json version
#1221 1219 daps bug gitlab ci gcs log
#1212 1211 fix jsdoc errors
#1206 1205 refactor formatting across javascript
#1201 1200 feature GitHub action javascript linter formatter
#1197 Refactor foxx unit tests so they will run correctly
#1203 Address eslint errors
#1188 Remove --dump flag
#1187 1184 bug add set e foxx install
#1186 Changes to allow for and auto generate debug log level
#1181 1180 refactor foxx authz
#1175 Add -e back to core/database/test/scripts
#1169 1168 feature add authz unit tests to ci
#1167 Add missing comment on env variable
#1161 1159 refactor authz cpp file
#1158 1154 feature add unit test authz
#1157 Add scripts for locating globus and gss api libraries
#1134 1114 add web unit testing to ci
#1163 Remove line split
#1153 Upgrade dependency of curl to 8.11
#1151 Throw error if variable is not defined properly
#1148 Provide default results for calls to gcs that allow proper evaluation later.
#1131 1130 bug entrypoint error ignoring errors
#1139 1128 Fix GCS Container json permission issues
#1136 Fix parallel runs, by adding random sleep
#1133 Add check for running container
#1129 1121 fix undefined build var in dockerfile
#1125 Remove overwrite of PATH
#1072 1071 add unit testing to web
#1112 Removed --no-cache flag
#1108 Update web version, with path due to change from CommonJS to ESM
#1079 1077 refactor support esm deprecate commonjs syntax
#1068 1065 fix ci cleanup script
#1064 1056 foxx test script fix
#1104 Fix parallel builds of retag and container build
#1092 Cleanup echo statements and make image tags explicit
#1094 make harbor script more robust to error response returned by api
#1101 Fix formatting of MessageLib and VersionUtils
#1098 Switch from using latest to CI_COMMIT_SHA
#1089 1086 bug python client pypi version
#1062 1055 fix foxx setup script
#1096 1057 trigger running foxx tests as part of ci
#1059 1057 trigger running foxx tests as part of ci
#1088 Add documentation about running DataFed Python package from source repo
#1081 Simplify cmake command
#1078 move flag to the correct position after ps
#1076 grab log output even if container fails
#1074 1073 add missing deployment key to node setup job
#1061 Add log output from ci end-to-end arango-setup job
#1054 1053 enable foxx tests in ci pipeline

DataFed-v2025.3.12.15.01

Pre-release

MAJOR Breaking changes

MINOR Feature

1. [987] - This implements serialized GitLab CI pipelines
2. [918] - Add admin script for making a user an admin
3. [1009] - Add collections to database for holding Globus tokens.
4. [1002] - Add backoff for task worker when database fails to prevent log overflow.
5. [1109] - Add support for Globus token association to Globus collection and user.
6. [1215] - Add support for fetching Globus tokens associated with Globus collection and user.
7. [1214] - Add support for mapped collections on transfer tasks

PATCH Bug fixes/Technical Debt/Documentation

1. [984] - Fixes {server_default} from showing up in path.
2. [990] - Will stop running containers so that we can prune them.
3. [988] - Removed non-working subscribe function.
4. [958] - Addresses issues raised by static code analyzer.
5. [962] - Adds script that will check that docker image is in registry
6. [1015] - Uses abs path to ci pipeline script for gcs build jobs
7. [1013] - add set +e and set -e around is active check for arango service
8. [1011] - Add python script and ci job to clean up GCS node keys.
9. [1023] - Address unbound variables in harbor ci script.
10. [1027] - Fix clear_db.sh script by replacing console.log with print
11. [1012] - Allow customized base build image for Docker dependencies and runtime Dockerfiles
12. [986] - Design improvement to upload and download transfer box.
13. [985] - Handles longer than needed timeouts on ui pages.
14. [1053] - Set CMake to enable foxx tests when built in the CI.
15. [1086] - Address XML-RPC bug from deprecated client
16. [1149] - Docker container GCS Collection Mount Bug Fix
17. [1168] - Add authz unit testing to the CI
18. [1200] - Add JavaScript linter (eslint) and (prettier) formatter for JavaScript
19. [1180] - Refactor of authz foxx module, split into objects and added unit tests
20. [1223] - Fix deprecated method usage for Protobuf in Python client library
21. [1257] - Bug in authz library parsing of authz.conf file, globus paths incorrectly sanitized when using just '/'
22. [1255] - Fixes bug, in libauthz with log_path var not being read in from config file.
23. [1268] - Bug fixes unbound variable in foxx entrypoint file.
24. [1269] - Update Arangodb to 3.12.4
25. [1288] - Bug Jupyter Notebook in documentation were fixed.
26. [1321] - Refactor, allow core config threads via env vars