Skip to content

feat(cli): add competitions data push command#1085

Merged
bovard merged 1 commit into
mainfrom
bovard/competition-data-push
Jul 7, 2026
Merged

feat(cli): add competitions data push command#1085
bovard merged 1 commit into
mainfrom
bovard/competition-data-push

Conversation

@bovard

@bovard bovard commented Jun 30, 2026

Copy link
Copy Markdown
Member

Wrap kagglesdk 0.1.32's new create_competition_data RPC behind kaggle competitions data push <competition> -p folder -m "notes" [--rerun] [-r {skip,zip,tar}]. Walks the folder, uploads each file via the existing ResumableUploadContext + blob plumbing (using ApiBlobType.INBOX — see TODO; may need adjustment based on backend feedback), then sends a single CreateCompetitionData request bundling the resulting tokens. Each push replaces the prior version's file set in full.

Bumps the kagglesdk floor to 0.1.32.

@bovard bovard force-pushed the bovard/competition-data-push branch 2 times, most recently from 547f5cb to a96e895 Compare June 30, 2026 19:37

@SohierDane SohierDane left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a safe upload default but otherwise LGTM.

Comment thread docs/competition_creation.md Outdated
@bovard bovard force-pushed the bovard/competition-data-push branch 2 times, most recently from b2cd08a to 998257e Compare July 1, 2026 03:17

@stevemessick stevemessick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

@bovard bovard force-pushed the bovard/competition-data-push branch 3 times, most recently from abc6be4 to c813bd1 Compare July 1, 2026 19:43
Comment thread docs/competition_creation.md Outdated
@bovard bovard requested a review from SohierDane July 1, 2026 21:58
@bovard bovard force-pushed the bovard/competition-data-push branch from c813bd1 to b89207c Compare July 1, 2026 22:02
@bovard bovard requested a review from stevemessick July 1, 2026 22:03
@bovard bovard force-pushed the bovard/competition-data-push branch from b89207c to 7148252 Compare July 1, 2026 22:10

@stevemessick stevemessick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending Erdal's confirmation.

Comment thread src/kaggle/api/kaggle_api_extended.py Outdated
with ResumableUploadContext() as upload_context:
for rel_name, full_path in uploads:
upload_file = self._upload_file(
rel_name, full_path, ApiBlobType.INBOX, upload_context, quiet, resources=None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erdalsivri Just to confirm, reusing INBOX here won't cause any problems?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a temporary place that's passed to the databundle worker, which copies (or moves ideally) to the corresponding competitions data bucket, right? I guess it won't cause problems but It seems more consistent to use a separate bucket because datasets and models have separate buckets. I don't have full context on why this is the case though. @kaggle-agent ?

@bovard bovard requested a review from erdalsivri July 3, 2026 00:41
Wrap the new create_competition_data RPC behind
`kaggle competitions data update <competition> -p <path> -m "<notes>"
[--rerun] [--include-hidden]`. The verb is `update` per AIP-130
(standard CRUD verb for modifying an existing resource); the RPC
itself creates a new databundle version, and each call fully
replaces the prior version's file set.

- `-p` accepts either a directory (walked recursively — sub-directory
  paths are preserved in each file's name field) or a single archive
  file (e.g. a pre-packed .zip / .tar), uploaded as-is.
- Hidden files and directories (names starting with `.`) are skipped
  by default; `--include-hidden` opts them in.
- `--rerun` targets the private RERUN databundle (host admin only).

Wraps blob uploads through the existing ResumableUploadContext +
`_upload_file` plumbing (ApiBlobType.INBOX for now — see TODO;
may need a dedicated COMPETITION_DATA blob type once we test
against the backend).
@bovard bovard force-pushed the bovard/competition-data-push branch from 7148252 to 21ff490 Compare July 6, 2026 19:34
@bovard

bovard commented Jul 6, 2026

Copy link
Copy Markdown
Member Author
  What was wrong: the CLI was calling ApiStartBlobUpload with ApiBlobType.INBOX. ApiStartBlobUploadHandler.cs:24 maps INBOX and DATASET to different ClientBlobBucket values — DATASET → DATASET_VERSION_FILES_V2, INBOX → the inbox bucket. CreateCompetitionData reads from
  DATASET_VERSION_FILES_V2, so blob tokens minted against INBOX would either 404 at bundle-creation time or, worse, quietly land bytes in the wrong bucket.

  What the wizard does: typescript/projects/components/src/datasets/CompetitionDataUploaderPanel/CompetitionDataUploaderPanel.tsx:98 uploads to ClientBlobBucket.DATASET_VERSION_FILES_V2 — i.e., the public-API equivalent is ApiBlobType.DATASET.

  Fix: switched the CLI to ApiBlobType.DATASET so it hits the same bucket. Added a unit-test assertion (test_update_directory_recurses_and_preserves_paths) that pins the value, so a future edit can't silently regress to INBOX. The stale TODO comment about maybe needing a dedicated
  COMPETITION_DATA blob type is now gone, replaced by a why-comment pointing at the wizard.

  Note on why BlobFileInboxTokenEncryptor.DecryptAndVerifyAsync didn't reject INBOX-typed tokens outright: the token proto (blob_service.proto:101-137) doesn't carry the blob type — validation only checks timestamp / user_id — so the mismatch would show up as a missing-file failure
   downstream, not a token error. That's why this needed a code trace to catch.

@bovard bovard requested a review from lipovetz July 6, 2026 19:37
@bovard bovard merged commit 61fb369 into main Jul 7, 2026
15 checks passed
@bovard bovard deleted the bovard/competition-data-push branch July 7, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants