Merge Changes from Develop to Master#74
Open
aimansharief wants to merge 62 commits into
Open
Conversation
Issue KN-953 fix: vulnerabilities fixes
fix: GCP signed url and getUri fix
fix : Update POM configuration to enable publishing to Maven Central
Include "chunked" -> "true" in additionalParams to generate the URL to support resumable (chunked) upload of a file.
feat: GCP Enhancements to the PreSigned URL generation
feat : GCP chunk upload support
feat: update project dependencies and Scala version to 2.13
feat: update version to 1.4.8.1 and upgrade dependencies in pom.xml
Deprecating Access Key usage and using Service Account or IAM roles.
…lved before deployment
feat: Added the flatten-maven-plugin to ensure all variables are resolved before deployment
…tations Replace the single Scala/jclouds-based module with a multi-module pure Java 11 Maven project. Each CSP (AWS, Azure, GCP, OCI) gets its own module using the official cloud SDK, with OIDC-based auth as the primary approach and Access Key as a configurable fallback. Modules: - cloud-storage-sdk-api: Interface, models, factory (ServiceLoader), abstract base - cloud-storage-sdk-aws: AWS S3 + Ceph S3 support (AWS SDK v2) - cloud-storage-sdk-azure: Azure Blob Storage (azure-storage-blob + azure-identity) - cloud-storage-sdk-gcp: Google Cloud Storage (google-cloud-storage) - cloud-storage-sdk-oci: OCI Object Storage (oci-java-sdk-objectstorage) Key changes: - Drop jclouds, Scala, and Joda-Time dependencies entirely - Pure Java 11 with java.time for date operations - ServiceLoader-based CSP discovery via StorageServiceProvider SPI - StorageConfig uses Builder pattern with AuthType and StorageType enums - AbstractStorageService provides shared orchestration (retry, search, copy, etc.) - IStorageService extends AutoCloseable for try-with-resources support https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
- API module: unit tests for DateRangeUtil, FileUtil, and AbstractStorageService (in-memory implementation, no credentials) - CSP modules: integration tests extending BaseStorageServiceIntegrationTest that auto-skip via JUnit 5 assumeTrue() when env vars are absent - API POM: publish test-jar so CSP modules can extend the base test class - CSP POMs: depend on api test-jar for shared test infrastructure - TESTING.md: comprehensive guide covering env vars, auth types, and examples for running tests from cloud-permissioned VMs https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
Captures module layout, key source files, StorageConfig/AuthType enums, the 12 AbstractStorageService primitives, per-CSP auth patterns, service discovery via ServiceLoader, test strategy, build commands, package structure, and common modification patterns. https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
CircleCI is no longer used. Removes .circleci/config.yml and the corresponding reference in PLAN.md Phase 6. https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
- Add managed dependency for cloud-storage-sdk-api test-jar classifier in parent POM so CSP modules resolve the version correctly - Pin maven-jar-plugin to 3.3.0 in pluginManagement - Add GitHub Actions CI workflow (ci.yml) for build + test on push/PR - Add GitHub Actions SonarCloud workflow (sonar.yml) for static analysis https://claude.ai/code/session_01KrRXhCQDoUCac99315Fnh6
Compute a relativePath for files under a directory and join it to objectKey with an explicit '/' separator before calling upload. This prevents malformed object keys when objectKey doesn't end with a separator and preserves the directory structure during recursive (sync and async) uploads. The change extracts the relative path calculation and applies it in both upload loops.
Problem: Used .toList() method which is only available in Java 16+, but project targets Java 11. Solution: Replaced .toList() with .collect(Collectors.toList()) and added Collectors import.
…ent-encodes the blob name
…ploads and avoid 5 null arguments code
fix: Decoding blob url since azure sdk percent-encodes the blob name
fix: Update test cases to cover the optional slash in objectKey
Multiple improvements across storage SDKs: add try-with-resources for object streams and enforce a 100MB in-memory payload limit; make archive extraction use a configurable local_extract_path and clean up extraction directory after use. Fix zip slip vulnerability in FileUtil. Harden Blob by defensively copying payload and lastModified and add equals/hashCode; add equals/hashCode for DeleteTarget. AWS: apply S3 service configuration to presigner and detect content type using tika. Azure: decode source blob URL when copying and clarify client close behavior. GCP: switch to streaming upload/download channels, cache signing Storage when created from additional params, and add a warning when using inline service account JSON. OCI: initialize client more safely with cleanup on failure, read file bytes to compute MD5 and avoid raw FileInputStream streaming, and use Instant for expiry computation. Miscellaneous logging and minor robustness fixes.
…tion Azure SDK percent-encodes '/' in blob names as '%2F' in the request line. Reactor-netty's HttpUtil.validateRequestLineTokens rejects '%2F' in DefaultFullHttpRequest, breaking single-shot REST ops (e.g. copyFromUrl) on nested blob paths. Switching the Azure BlobServiceClient to the OkHttp HTTP transport avoids that validation while preserving all other SDK behaviour. Bump version to 2.0.2-beta. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
fix: Use OkHttp transport for Azure to bypass reactor-netty %2F rejection
Exclude the transitive azure-core-http-netty from azure-storage-blob and azure-identity so the SDK ships only the OkHttp HTTP transport. This removes reactor-netty entirely from the SDK classpath, eliminating any future netty validation regressions (e.g. the 4.1.129 bitmask bug where 1L << c wraps modulo 64 and rejects uppercase ASCII chars J/M/backtick as LF/CR/SPACE). Downgrade azure-core-http-okhttp 1.11.20 -> 1.11.0 to match the azure-core 1.45.0 bundled by downstream consumers (asset-enrichment fat jar in knowledge-platform-jobs). 1.11.20 calls BinaryData.writeTo(WritableByteChannel) which was added in azure-core 1.49.0 and triggers NoSuchMethodError at runtime. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
fix: Exclude azure-core-http-netty and align okhttp to azure-core 1.45.0
Switch AzureStorageService to use the azure-core OkHttp async HTTP client (OkHttpAsyncHttpClientBuilder) to avoid reactor-netty's request-line validation bug that can reject percent-encoded/uppercase characters in blob names. Add and pin azure.core.http.okhttp.version to 1.11.0 in pom.xml with a comment noting newer okhttp bindings require azure-core >=1.49.0 while the build currently bundles azure-core 1.45.0.
fix: Use OkHttp transport for Azure SDK
fix(azure): switch to OkHttp transport to avoid Netty %2F/uppercase request-line rejection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.