Feat/drs 1.5 by k-burt-uch · Pull Request #441 · uc-cdis/indexd

k-burt-uch · 2026-04-30T21:13:53Z

Link to JIRA ticket if there is one:

New Features

New endpoint: OPTIONS /objects/{object_id} - returns a list of Authorizations that can be used to determine how to authorize requests to get object or post object.
New endpoint: OPTIONS /objects - Returns authorization information for multiple DrsObjects at once.
New endpoint: POST /objects - Get info about multiple DrsObjects
New endpoint: POST /objects/access (Bulk Access URLs) - Returns an array of URL objects that can be used to download of multiple DrsObjects.
Update to DrsObject's access_methods: region, cloud, available and authorization are added to fields.

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

CLOUD_PROVIDER_MAP and DRS_AUTHORIZATION_METADATA environment variables should be provided to support the cloud field for access_methods and the authorizations field for access_methods. See docs for more information.

github-actions · 2026-04-30T21:14:31Z

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

github-actions · 2026-04-30T22:02:14Z

filepath	passed	skipped	SUBTOTAL
tests/test_presigned_url.py	7	0	7
tests/test_centralized_auth.py	16	0	16
tests/test_graph_submit_and_query.py	7	0	7
tests/test_data_upload.py	8	1	9
tests/test_dbgap.py	4	1	5
tests/test_audit_service.py	1	0	1
tests/test_gen3client.py	1	0	1
TOTAL	44	2	46

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2026-04-30T23:46:52Z

Test summary after running integration tests

filepath	passed	failed	skipped	SUBTOTAL
tests/test_presigned_url.py	7	1	0	8
tests/test_centralized_auth.py	15	1	0	16
tests/test_graph_submit_and_query.py	7	0	0	7
tests/test_data_upload.py	8	0	1	9
tests/test_audit_service.py	1	0	0	1
tests/test_gen3client.py	1	0	0	1
tests/test_dbgap.py	4	0	1	5
TOTAL	43	2	2	47

Test summary after rerunning failed integration tests

filepath	passed	failed	SUBTOTAL
tests/test_presigned_url.py	0	1	1
tests/test_centralized_auth.py	1	0	1
TOTAL	1	1	2

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-01T20:28:05Z

Test summary after running integration tests

filepath	passed	failed	error	skipped	SUBTOTAL
tests/test_drs_endpoint.py	0	0	10	0	10
tests/test_presigned_url.py	7	1	0	0	8
tests/test_centralized_auth.py	16	0	0	0	16
tests/test_graph_submit_and_query.py	7	0	0	0	7
tests/test_data_upload.py	8	0	0	1	9
tests/test_dbgap.py	4	0	0	1	5
tests/test_audit_service.py	1	0	0	0	1
tests/test_gen3client.py	1	0	0	0	1
TOTAL	44	1	10	2	57

Test summary after rerunning failed integration tests

filepath	failed	error	SUBTOTAL
tests/test_drs_endpoint.py	0	10	10
tests/test_presigned_url.py	1	0	1
TOTAL	1	10	11

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-04T00:14:43Z

Test summary after running integration tests

filepath	passed	failed	error	SUBTOTAL
tests/test_drs_endpoint.py	0	0	10	10
tests/test_presigned_url.py	7	1	0	8
TOTAL	7	1	10	18

Test summary after rerunning failed integration tests

filepath	failed	error	SUBTOTAL
tests/test_drs_endpoint.py	0	10	10
tests/test_presigned_url.py	1	0	1
TOTAL	1	10	11

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-04T02:16:13Z

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	1	9	10
tests/test_presigned_url.py	7	1	8
TOTAL	8	10	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	9	9
tests/test_presigned_url.py	1	1
TOTAL	10	10

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-04T05:24:44Z

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	1	9	10
tests/test_presigned_url.py	7	1	8
TOTAL	8	10	18

Test summary after rerunning failed integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	1	8	9
tests/test_presigned_url.py	0	1	1
TOTAL	1	9	10

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-04T06:25:18Z

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	1	9	10
tests/test_presigned_url.py	7	1	8
TOTAL	8	10	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	9	9
tests/test_presigned_url.py	1	1
TOTAL	10	10

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-04T07:20:52Z

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	3	7	10
tests/test_presigned_url.py	7	1	8
TOTAL	10	8	18

Test summary after rerunning failed integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	1	6	7
tests/test_presigned_url.py	0	1	1
TOTAL	1	7	8

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-05-04T17:36:21Z

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5	10
tests/test_presigned_url.py	7	1	8
TOTAL	12	6	18

Test summary after rerunning failed integration tests

filepath	error	SUBTOTAL
tests/test_aggregate_mds.py	1	1
tests/test_discoverypage.py	1	1
tests/test_etl.py	1	1
tests/test_pfb_export.py	1	1
tests/test_ras_authn.py	1	1
tests/test_study_viewer.py	1	1
TOTAL	6	6

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

SpencerAxelrod · 2026-05-04T19:57:09Z

            location_type = location.split(":")[
                0
            ]  # (s3, gs, ftp, gsiftp, globus, htsget, https, file)
+            cloud = blueprint.cloud_provider_map.get(location_type)


Per the feature doc we need to support http mappings e.g.

CLOUD_PROVIDER_MAP = { "s3": "aws", "gs": "gcp", "az": "azure", "https": { "m3.aicommons.com/ai": "aws", "storage.googleapis.com": "gcp", }, }

For an HTTPS URL, location_type is "https" and if CLOUD_PROVIDER_MAP contains a nested dict under "https", cloud becomes that nested dict object. We probably need a helper function here and in the non drs indexd called like derive_cloud(url) that handles both simple scheme mappings and nested HTTPS prefix mappings

Do we have cases for this in gen3-code-vigil?

no it doesn't. And tbh it may be out of scope in the very short term

Let me add the cases here. We can backlog a ticket on GQE.

http mappings added.

SpencerAxelrod · 2026-05-04T20:10:46Z

are we adding the bulk drs object endpoint in this PR? POST /ga4gh/drs/v1/objects

k-burt-uch · 2026-05-05T19:35:20Z

are we adding the bulk drs object endpoint in this PR? POST /ga4gh/drs/v1/objects

Yes - this fell through the tracks on tasking 🤦

I'll push an update later today

github-actions · 2026-05-05T20:17:48Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	4	6	10
tests/test_presigned_url.py	7	1	8
TOTAL	11	7	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	6	6
tests/test_presigned_url.py	1	1
TOTAL	7	7

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-06-08T19:03:25Z

Integration Tests

Failed to Prepare CI environment.

The following pods are not in healthy state:
indexd-deployment-dc9bf6767-b2rjf

Please find the Github Action logs here

github-actions · 2026-06-08T19:57:49Z

Integration Tests

Failed to Prepare CI environment

Please find the Github Action logs here

github-actions · 2026-06-08T21:39:42Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	4	6	10
tests/test_presigned_url.py	7	1	8
TOTAL	11	7	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	6	6
tests/test_presigned_url.py	1	1
TOTAL	7	7

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-06-08T23:17:31Z

Integration Tests

Failed to Prepare CI environment.

The following pods are not in healthy state:
revproxy-deployment-cfb64594d-njm4m

Please find the Github Action logs here

github-actions · 2026-06-09T00:20:02Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5	10
tests/test_presigned_url.py	7	1	8
TOTAL	12	6	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5
tests/test_presigned_url.py	1	1
TOTAL	6	6

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-06-09T02:23:21Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5	10
tests/test_presigned_url.py	7	1	8
TOTAL	12	6	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5
tests/test_presigned_url.py	1	1
TOTAL	6	6

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-06-09T14:00:18Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5	10
tests/test_presigned_url.py	7	1	8
TOTAL	12	6	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	5	5
tests/test_presigned_url.py	1	1
TOTAL	6	6

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-06-09T16:43:29Z

Integration Tests

Test summary after running integration tests

filepath	passed	failed	SUBTOTAL
tests/test_drs_endpoint.py	9	1	10
tests/test_presigned_url.py	7	1	8
TOTAL	16	2	18

Test summary after rerunning failed integration tests

filepath	failed	SUBTOTAL
tests/test_drs_endpoint.py	1	1
tests/test_presigned_url.py	1	1
TOTAL	2	2

Please find the detailed integration test report here

Please find the detailed integration test report after rerunning failed tests here

Please find the Github Action logs here

github-actions · 2026-06-09T17:14:33Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_drs_endpoint.py	9	1	10
TOTAL	9	1	10

Please find the detailed integration test report here

Please find the Github Action logs here

New endpoint: OPTIONS /objects/{object_id} - returns a list of Authorizations that can be used to determine how to authorize requests to get object or post object. New endpoint: OPTIONS /objects - Returns authorization information for multiple DrsObjects at once. New endpoint: POST /objects - Get info about multiple DrsObjects New endpoint: POST /objects/access (Bulk Access URLs) - Returns an array of URL objects that can be used to download of multiple DrsObjects. Update to DrsObject's access_methods: region, cloud, available and authorization are added to fields. CLOUD_PROVIDER_MAP and DRS_AUTHORIZATION_METADATA environment variables should be provided to support the cloud field for access_methods and the authorizations field for access_methods. See docs for more information. port changes for single option port bulk changes add bulk abstract pytest fixes test test2 test whether open PR re-runs with a change Work in progess: PR feedback updates add more bulk auth option tests fix flaky tests update docstring test commit revert change code changes for cloud field fix broken tests add test for cloud field Initial changes for region field remove debugging print statement from test more changes for region field and available field fix for unit tests fix test_update_urls_metadata more test fixes switch to string 'true' more test fixes fix test_cloud_field swap fullmatch for search remove regexp vulnerability and add test for region field Fix new unit tests use get_doc instead of manual creation remove extra imports from bottom try this fix clear cache too local unit test fixes Update settings files to try pulling in environment variables first drs 1.5 service info support Run drs 1.5 integration tests Fix access_method overrides Rename access_method test More available fixes drs 1.5 helm branch for integration_tests Add cloud provider domain linking Add get_drs_objects endpoint WIP Work in progress - saving updates WIP - saving updates Work in progress - refactored single resolver/tests Clean up bulk tests/helpers pytest fix Updates for PR review ( passport handling + POST) auth nesting updated within access_methods add single post, make names more consistent improve readability Fix logging Fix merge issues and update test_drs authorization tests Fix Exceptions in DRS to use IndexdUnexpectedError Safely access default_passport_issuer Add default issuers to authz, fix cloud drs unit tests Fix supported_types for open access Debugging logs Move logger init to after blueprint registration Fix defaults for authz, removing debug logs Fix test_single_path_not_found test to use defaults Co-authored-by: jacob50231 <jacobdaugherty@uchicago.edu> Co-authored-by: nrandorf <randorf@uchicago.edu> Co-authored-by: SpencerAxelrod <saxelrod@uchicago.edu>

github-actions · 2026-06-12T15:48:36Z

Integration Tests

Failed to Prepare CI environment.

The following pods are not in healthy state:
orthanc-deployment-669b6b5576-wxhsw

Please find the Github Action logs here

github-actions · 2026-06-15T15:22:59Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_drs_endpoint.py	9	1	10
TOTAL	9	1	10

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2026-06-15T21:27:56Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_drs_endpoint.py	9	1	10
TOTAL	9	1	10

Please find the detailed integration test report here

Please find the Github Action logs here

jbarno

lgtm

jbarno · 2026-06-16T14:50:34Z

 def get_doc(
    has_version=True,
-    urls=list(),
+    urls=None,


oooh, good catch.

jbarno · 2026-06-16T14:57:03Z


    assert res.status_code == 200
-    assert res.json == expected_info
+    data = res.json


I would have kept the dictionary equality assertion the same and just changed expected_info dictionary. Truly a nothing burger though. Same test excluding the maxBulkRequestLength being a < instead of ==

MichaelLukowski

Looks good!

One small nitpick but it should be fine

MichaelLukowski · 2026-06-22T18:04:52Z

+    if "urls" in record and record["urls"]:
+        for url in record["urls"]:
+            if url.startswith("s3://") and url not in region:
+                bucket_name = url.split("/")[2]


should we urlparse for this instead of split and index? Just in case a s3 url is malformed.

Good point about malformed handling. urlparse won't provide validation, but we can add in some conditional handling for malformed scenario (i.e. if the scheme is unexpected, we can skip the split and index entirely).

On further review, I think we can leave this as is because the existing code already avoids appending url data if the input is malformed. A match in the subsequent lookup_bucket_region() helper is required and if the match is not found, then region information for the malformed url defaults to an empty string.

github-actions · 2026-06-25T14:55:46Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_drs_endpoint.py	9	1	10
TOTAL	9	1	10

Please find the detailed integration test report here

Please find the Github Action logs here

k-burt-uch force-pushed the feat/drs-1.5 branch 5 times, most recently from c21469d to af41355 Compare April 30, 2026 22:43

k-burt-uch added test-apis-drsEndpointsTest only run drs test TestPresignedURL labels May 3, 2026

PlanXCyborg added TestDrsMetadata TestPresignedURL and removed test-apis-drsEndpointsTest only run drs test TestPresignedURL labels May 3, 2026

SpencerAxelrod reviewed May 4, 2026

View reviewed changes

k-burt-uch closed this May 5, 2026

k-burt-uch reopened this May 5, 2026

SpencerAxelrod reviewed May 6, 2026

View reviewed changes

Comment thread indexd/drs/blueprint.py Outdated

SpencerAxelrod reviewed May 7, 2026

View reviewed changes

Comment thread indexd/drs/blueprint.py Outdated

SpencerAxelrod reviewed May 7, 2026

View reviewed changes

Comment thread indexd/drs/blueprint.py

k-burt-uch force-pushed the feat/drs-1.5 branch from 8c71c95 to 3462be6 Compare June 8, 2026 19:23

k-burt-uch removed the TestPresignedURL label Jun 9, 2026

k-burt-uch force-pushed the feat/drs-1.5 branch from 386bd3e to 3de0951 Compare June 12, 2026 15:37

Updated docs, fixed single table test

886428e

Remove custom helm branch for integration tests

2673d0f

jbarno previously approved these changes Jun 16, 2026

View reviewed changes

MichaelLukowski previously approved these changes Jun 22, 2026

View reviewed changes

george42-ctds and others added 3 commits June 23, 2026 17:01

Relock to update deps

78be578

Update unit test and bump version

25d8950

Relock poetry, bump pyprojects

24e7acb

k-burt-uch dismissed stale reviews from MichaelLukowski and jbarno via 24e7acb June 23, 2026 22:02

Merge branch 'master' into feat/drs-1.5

fbe33be

jacob50231 approved these changes Jun 25, 2026

View reviewed changes

jacob50231 merged commit 8c5913a into master Jun 25, 2026
9 checks passed

jacob50231 deleted the feat/drs-1.5 branch June 25, 2026 15:36

Uh oh!

Conversation

k-burt-uch commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

github-actions Bot commented May 1, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SpencerAxelrod commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k-burt-uch commented May 5, 2026

Uh oh!

github-actions Bot commented May 5, 2026

Integration Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 8, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 8, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 8, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 8, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 9, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 9, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 9, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 9, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 9, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jun 12, 2026

k-burt-uch commented Apr 30, 2026 •

edited

Loading

SpencerAxelrod commented May 4, 2026 •

edited

Loading