v0.53.0

[Xuanwo]

Upgrade to Rust Core v0.53

Public API

Supabase service is now an S3-compatible servcice

Supabase Storage is now an S3-compatible service instead: https://github.com/supabase/storage.

We removed the supabase native service support in OpenDAL v0.53. Users who want to access Supabase Storage can use the S3 service instead.

All metrics related layers have been refactored

All metrics layers have been refactored:

• PrometheusLayer
• PrometheusClientLayer
• MetricsLayer

They are now provides more metrics and more detailed information. All their public API have been redesigned.

For more details, please refer to opendal::layers::observe's module documentation.

Operator::default_executor has been replaced by Operator::executor

In opendal v0.53, we introduced a new concept of Context which is used to store the context of the current operator. Thanks to this design, we can now get and set the executor and http_client for given Operator instead.

All services http_client API has been deprecated and replaced by Operator::update_http_client API.

OpenDAL MSRV bumped to 1.80

Since v0.53, OpenDAL will require Rust 1.80.0 or later to build.

Raw API

Operation enum merge

To reduce the complexity of the Operation, we have merged the duplicated Operation.

For example:

• Operation::ReaderRead has been merged into Operation::Read
• Operation::BlockingRead has been merged into Operation::Read

Upgrade to Nodejs Binding v0.48

Breaking change

Public API

Now, nodejs binding op.is_exist changed to op.exists to align with nodejs API style.

What's Changed

Added

• feat(bindings/python) detailed error message by @asukaminato0721 in feat(bindings/python) detailed error message #5646
• feat(bindings/python): Add user metadata support for write by @Xuanwo in feat(bindings/python): Add user metadata support for write #5654
• feat(services/s3): add append support by @Frank-III in feat(services/s3): add append support #5428
• feat(core): Add presign delete support by @asukaminato0721 in feat(core): Add presign delete support #5647
• feat(bindings/python): add python presign_delete by @asukaminato0721 in feat(bindings/python): add python presign_delete #5661
• feat(core): Sharing context between layers and service by @Xuanwo in feat(core): Sharing context between layers and service #5662
• feat(!): Bump arrow version of parquet_opendal to 54.x by @erickguan in feat(!): Bump arrow version of parquet_opendal to 54.x #5665
• feat: Polish context related APIs by @Xuanwo in feat: Polish context related APIs #5673
• feat(services/onedrive): List dir shows metadata by @erickguan in feat(services/onedrive): List dir shows metadata #5632
• feat(java): add WriteOptions for write methods by @geruh in feat(java): add WriteOptions for write methods #5664
• feat(bindings/ruby): add operator info by @erickguan in feat(bindings/ruby): add operator info #5584
• feat(services/oss): Implement Write Returns Metadata for oss by @meteorgan in feat(services/oss): Implement Write Returns Metadata for oss #5688
• feat(core): Expose Reader::into_stream and Writer::into_sink by @Xuanwo in feat(core): Expose Reader::into_stream and Writer::into_sink #5698
• feat: adopt uv for python binding by @kemingy in feat: adopt uv for python binding #5711
• feat(services/gcs): Implement multipart/related and use it with Gcs by @wlinna in feat(services/gcs): Implement multipart/related and use it with Gcs #5691
• feat(bindings/dart): Add dart binding by @asukaminato0721 in feat(bindings/dart): Add dart binding #5591
• fix(binding/dart): pin version to avoid generator/runtime version mismatch by @asukaminato0721 in fix(binding/dart): pin version to avoid generator/runtime version mismatch #5734
• feat(binding/dart): add examples, tests by @asukaminato0721 in feat(binding/dart): add examples, tests #5740
• feat(services/onedrive): add signer to utilize the refresh token by @erickguan in feat(services/onedrive): add signer to utilize the refresh token #5733
• feat(binding/dart): hide init && change to init by @asukaminato0721 in feat(binding/dart): hide init && change to init #5742
• feat(core): Implement write returns metadata for b2 by @hoslo in feat(core): Implement write returns metadata for b2 #5750
• feat(services/hdfs_native): implement write/read/list methods by @zhaohaidao in feat(services/hdfs_native): implement write/read/list methods #5617
• feat(website): Adding a WIP to the list of bingdings by @shaonianche in feat(website): Adding a WIP to the list of bingdings #5769
• feat(services/onedrive): implement read_with_if_none_match by @erickguan in feat(services/onedrive): implement read_with_if_none_match #5763
• feat(binding/dart): add pubspec info by @asukaminato0721 in feat(binding/dart): add pubspec info #5751
• feat(services/s3): Add operation in http context by @Xuanwo in feat(services/s3): Add operation in http context #5791
• feat(bindings/python): Add repr for metadata by @yihong0618 in feat(bindings/python): Add repr for metadata #5783
• feat: Add origin private file system scaffold by @Eason0729 in feat: Add origin private file system scaffold #5758
• feat(core): Implement http related metrics support for prom client by @Xuanwo in feat(core): Implement http related metrics support for prom client #5798
• feat(core): Implement http related metrics support for otel by @leiysky in feat(core): Implement http related metrics support for otel #5800
• feat(services/lakefs): Add operation in http context by @liugddx in feat(services/lakefs): Add operation in http context #5809
• feat(github): Auto generate weekly summary by @Xuanwo in feat(github): Auto generate weekly summary #5818
• feat(binding/java): Add list with recursive support by @cuichenli in feat(binding/java): Add list with recursive support  #5718
• feat(services/huggingface): Add operation in http context by @liugddx in feat(services/huggingface): Add operation in http context #5810
• feat(core/layers): implement http related metrics support for prometheus by @koushiro in feat(core/layers): implement http related metrics support for prometheus #5847
• feat(core/layers): implement http related metrics support for metrics by @koushiro in feat(core/layers): implement http related metrics support for metrics #5848
• feat(services/onedrive): implement additional OneDrive features by @erickguan in feat(services/onedrive): implement additional OneDrive features #5784
• feat(bindings/c): add opendal_operator_check by @asukaminato0721 in feat(bindings/c): add opendal_operator_check #5851
• feat(services/aliyun_drive): Add operation in http context by @sunheyi6 in feat(services/aliyun_drive): Add operation in http context #5880

Changed

• refactor(!): Supabase is now an S3-compatible servcices by @Xuanwo in refactor(!): Supabase is now an S3-compatible servcices #5663
• refactor: Migrate s3 services to context based http client by @Xuanwo in refactor: Migrate s3 services to context based http client #5676
• refactor: Migrate oss services to context based http client by @Ziy1-Tan in refactor: Migrate oss services to context based http client #5681
• refactor: Migrate obs services to context based http client by @Ziy1-Tan in refactor: Migrate obs services to context based http client #5682
• refactor: Migrate cos services to context based http client by @Ziy1-Tan in refactor: Migrate cos services to context based http client #5683
• refactor(bindings/node)!: Change is_exist to exists to align with nodejs API style by @yihong0618 in refactor(bindings/node)!: Change is_exist to exists to align with nodejs API style #5731
• refactor(services/redis): Implement ConnectionLike for RedisConnection by @Xuanwo in refactor(services/redis): Implement ConnectionLike for RedisConnection #5748
• refactor!: Remove opendal-compat which is not maintained by @Xuanwo in refactor!: Remove opendal-compat which is not maintained #5754
• refactor: Migrate github services to context based http client by @miroim in refactor: Migrate github services to context based http client #5764
• refactor(gcs): Migrate to context based http client by @leiysky in refactor(gcs): Migrate to context based http client #5778
• refactor(core/raw)!: Merge blocking and async operations by @Xuanwo in refactor(core/raw)!: Merge blocking and async operations #5789
• refactor(core/raw)!: Use AccessorInfo instead of seperate fields by @Xuanwo in refactor(core/raw)!: Use AccessorInfo instead of seperate fields #5796
• refactor: Migrate aliyun_drive services to context based http client by @miroim in refactor: Migrate aliyun_drive services to context based http client #5815
• refactor: Migrate azfile services to context based http client by @miroim in refactor: Migrate azfile services to context based http client #5816
• refactor: Migrate dropbox services to context based http client by @miroim in refactor: Migrate dropbox services to context based http client #5827
• refactor: Migrate upyun services to context based http client by @miroim in refactor: Migrate upyun services to context based http client #5829
• refactor(core): Migrate BlockWriter to use executor from context by @Xuanwo in refactor(core): Migrate BlockWriter to use executor from context #5834
• refactor(core): Migrate MultipartWrite Executor to context based by @Xuanwo in refactor(core): Migrate MultipartWrite Executor to context based #5835
• refactor(core): Migrate PositionWrite Executor to context based by @Xuanwo in refactor(core): Migrate PositionWrite Executor to context based #5836
• refactor(core)!: Migrate to context based executor by @Xuanwo in refactor(core)!: Migrate to context based executor #5838
• refactor: tidy binding java code by @tisonkun in refactor: tidy binding java code #5840
• refactor: Migrate yandex_disk services to context based http client by @miroim in refactor: Migrate yandex_disk services to context based http client #5841
• refactor: Migrate alluxio services to context based http client by @miroim in refactor: Migrate alluxio services to context based http client #5842
• refactor: Migrate b2 service to context based http client by @miroim in refactor: Migrate b2 service to context based http client #5843
• refactor: migrate azblob services to context based http client by @Ziy1-Tan in refactor: migrate azblob services to context based http client #5845
• refactor: Migrate gdrive service to context based http client by @miroim in refactor: Migrate gdrive service to context based http client #5861
• refactor: Migrate pcloud service to context based http client by @miroim in refactor: Migrate pcloud service to context based http client #5866
• refactor!: Bump OpenDAL MSRV to 1.80 by @Xuanwo in refactor!: Bump OpenDAL MSRV to 1.80 #5868
• refactor: Introduce VercelArtifactsCore for improved service structure by @miroim in refactor: Introduce VercelArtifactsCore for improved service structure #5873
• refactor: Migrate vercel_artifacts service to context based http client by @miroim in refactor: Migrate vercel_artifacts service to context based http client #5877
• refactor: Migrate http service to context based http client by @miroim in refactor: Migrate http service to context based http client #5879
• refactor: Migrate services to context based http client by @miroim in refactor: Migrate services to context based http client #5882
• refactor: Polishing IO metrics by adding more useful metrics by @Xuanwo in refactor: Polishing IO metrics by adding more useful metrics #5883
• refactor(core): Add good default histogram buckets for metrics by @Xuanwo in refactor(core): Add good default histogram buckets for metrics #5886
• refactor: Migrate icloud service to context based http client by @miroim in refactor: Migrate icloud service to context based http client #5891
• refactor(bindings/dart)!: support macos and use exists api by @yihong0618 in refactor(bindings/dart)!: support macos and use exists api #5884
• refactor: Migrate PrometheusClientLayer to support IO metrics by @Xuanwo in refactor: Migrate PrometheusClientLayer to support IO metrics #5887
• refactor(core)!: Merge operations to build more clean metrics by @Xuanwo in refactor(core)!: Merge operations to build more clean metrics #5892
• refactor: Migrate WebHDFS service to context based http client by @miroim in refactor: Migrate WebHDFS service to context based http client #5893
• refactor(layer/metrics): Migrate MetricsLayer to support IO metrics by @koushiro in refactor(layer/metrics): Migrate MetricsLayer to support IO metrics #5900
• refactor(layer/prometheus): Migrate PrometheusLayer to support IO metrics by @koushiro in refactor(layer/prometheus): Migrate PrometheusLayer to support IO metrics #5899
• refactor(layer/otelmetrics): Migrate OtelMetricsLayer to support IO metrics by @koushiro in refactor(layer/otelmetrics): Migrate OtelMetricsLayer to support IO metrics #5901
• refactor: Introduce SftpCore for improved service structure by @miroim in refactor: Introduce SftpCore for improved service structure #5902
• refactor(observe): remove duplicated observation by @koushiro in refactor(observe): remove duplicated observation #5894
• refactor: refine RetryLayer type signature by @tisonkun in refactor: refine RetryLayer type signature #5905
• refactor(core): Remove all write_has_xxx capabilities by @Ziy1-Tan in refactor(core): Remove all write_has_xxx capabilities #5908
• refactor(services/obs): Implement Write Returns Metadata for obs by @Ziy1-Tan in refactor(services/obs): Implement Write Returns Metadata for obs #5912

Fixed

• fix(integrations/dav-server): handle encoded path correctly. by @rick-200 in fix(integrations/dav-server): handle encoded path correctly. #5650
• fix(services/onedrive): fix OneDrive behavior tests by @erickguan in fix(services/onedrive): fix OneDrive behavior tests #5652
• fix(services/gcs): Advertise write_with_cache_control in Gcs by @wlinna in fix(services/gcs): Advertise write_with_cache_control in Gcs #5658
• fix(services/gcs): Fix cache control not present in object meta by @Xuanwo in fix(services/gcs): Fix cache control not present in object meta #5660
• fix(services/s3): fix batch delete with version by @meteorgan in fix(services/s3): fix batch delete with version #5684
• fix(services/oss): fix batch delete with version for oss by @meteorgan in fix(services/oss): fix batch delete with version for oss #5687
• fix(services/onedrive): remove @odata.type field by @erickguan in fix(services/onedrive): remove @odata.type field #5696
• fix: drop useless clone for py binding and make clippy happy by @yihong0618 in fix: drop useless clone for py binding and make clippy happy #5708
• fix: drop useless file by @yihong0618 in fix: drop useless file #5714
• fix: make build_abs_path to build_rooted_abs_path for chainsafe by @yihong0618 in fix: make build_abs_path to build_rooted_abs_path for chainsafe #5715
• fix: mac can not build lua binding by @yihong0618 in fix: mac can not build lua binding #5719
• fix(website): Handling svg image correctly by @Xuanwo in fix(website): Handling svg image correctly #5725
• fix(website): Handling badges click behavior correctly by @miroim in fix(website): Handling badges click behavior correctly #5741
• fix(core)!: chainsafe services has been sunset by @yihong0618 in fix(core)!: chainsafe services has been sunset #5744
• fix: sqlite may dead lock in ci by @yihong0618 in fix: sqlite may dead lock in ci #5738
• fix: drop useless import and clippy happy for java binding by @yihong0618 in fix: drop useless import and clippy happy for java binding #5746
• fix(bindings/java): bring back false delete code by @yihong0618 in fix(bindings/java): bring back false delete code #5752
• fix(core): fix list with recursive when the object doesn't exist by @meteorgan in fix(core): fix list with recursive when the object doesn't exist #5732
• fix(website): docusaurus build on windows by @shaonianche in fix(website): docusaurus build on windows #5770
• fix: make php binding happy again by @yihong0618 in fix: make php binding happy again #5761
• fix(services/onedrive): add scope for the refresh token by @erickguan in fix(services/onedrive): add scope for the refresh token #5776
• fix: py binding benchmark can not run by @yihong0618 in fix: py binding benchmark can not run #5786
• fix: add missing last_modified for python binding by @yihong0618 in fix: add missing last_modified for python binding #5767
• doc: fix README root wrong for example close issue RFC-5871: Read Returns Metadata #5871 by correct README by @yihong0618 in doc: fix README root wrong for example close issue #5871 by correct README #5782
• fix(services/compfs): behavior async write by @Berrysoft in fix(services/compfs): behavior async write #5803
• fix(services/onedrive): chunk PUT upload remove auth header by @emliunix in fix(services/onedrive): chunk PUT upload remove auth header #5812
• fix(services/compfs): read, delete, copy, rename by @Berrysoft in fix(services/compfs): read, delete, copy, rename #5807
• fix(github): Fix discussion create by @Xuanwo in fix(github): Fix discussion create #5821
• fix(bindings/python): open python3.13t release for windows since upstream had fixed by @yihong0618 in fix(bindings/python): open python3.13t release for windows since upstream had fixed #5826
• fix: better ocaml binding and happy clippy by @yihong0618 in fix: better ocaml binding and happy clippy #5839
• fix(bindings/python): sort imports, fix template, fix bench by @kemingy in fix(bindings/python): sort imports, fix template, fix bench #5844
• fix: make haskell binding happier by @yihong0618 in fix: make haskell binding happier #5849
• doc: fix all fs env tmp file dir in doc by @yihong0618 in doc: fix all fs env tmp file dir in doc #5855
• fix: drop fixme in py binding since upstream fixed by @yihong0618 in fix: drop fixme in py binding since upstream fixed #5862
• fix: drop fixme in etcd tls since upstream fixed by @yihong0618 in fix: drop fixme in etcd tls since upstream fixed #5863
• fix: todo list since MSRV is 1.80 by @yihong0618 in fix: todo list since MSRV is 1.80 #5870
• fix: close issue let some actions not run on fork #5910 by @yihong0618 in fix: close issue #5910 #5911

Docs

• doc: add the most famous Python binding by @yihong0618 in doc: add the most famous Python binding #5707
• doc: py binding version is wrong for now by @yihong0618 in doc: py binding version is wrong for now #5709
• docs: Better ways to users showcase by @Xuanwo in docs: Better ways to users showcase #5720
• docs: Add useful links by @Xuanwo in docs: Add useful links #5735
• docs: QuestDB is not using opendal java by @Xuanwo in docs: QuestDB is not using opendal java #5736
• doc: more users for python binding by @yihong0618 in doc: more users for python binding #5756
• docs: Update nominate-committer.md by @tisonkun in docs: Update nominate-committer.md #5765
• docs: Fix rustic users entry by @QazCetelic in docs: Fix rustic users entry #5790
• docs(services/onedrive): add how to get tokens by @emliunix in docs(services/onedrive): add how to get tokens #5792
• doc: fix typo in RFC md by @yihong0618 in doc: fix typo in RFC md #5813
• doc: dotnet binding wip doc by @yihong0618 in doc: dotnet binding wip doc #5850
• doc: add oli bench command doc by @yihong0618 in doc: add oli bench command doc #5854
• doc: docker installation guide for behaviour test by @zhaohaidao in doc: docker installation guide for behaviour test #5876
• docs: Add docs on how to upgrade to opendal rust core 0.53 by @Xuanwo in docs: Add docs on how to upgrade to opendal rust core 0.53 #5918
• docs: Update changelogs for v0.53.0 by @Xuanwo in docs: Update changelogs for v0.53.0 #5934

CI

• ci(bindings/zig): update to zig version 0.14.0 by @kassane in ci(bindings/zig): update to zig version 0.14.0 #5700
• ci: add clippy to some of the binding check ci by @yihong0618 in ci: add clippy to some of the binding check ci #5717
• build: bump opentelemetry dependency to 0.29.0 by @tisonkun in build: bump opentelemetry dependency to 0.29.0 #5856
• build(deps): bump actions/setup-python from 4 to 5 by @dependabot in build(deps): bump actions/setup-python from 4 to 5 #5926
• build(deps): update datafusion requirement from 45.0.0 to 46.0.1 in /integrations/object_store by @dependabot in build(deps): update datafusion requirement from 45.0.0 to 46.0.1 in /integrations/object_store #5928
• build(deps): bump governor from 0.6.3 to 0.10.0 in /core by @dependabot in build(deps): bump governor from 0.6.3 to 0.10.0 in /core #5923
• build(deps): bump log from 0.4.25 to 0.4.27 in /bin/oay by @dependabot in build(deps): bump log from 0.4.25 to 0.4.27 in /bin/oay #5927
• build(deps): bump humantime from 2.1.0 to 2.2.0 in /bin/oli by @dependabot in build(deps): bump humantime from 2.1.0 to 2.2.0 in /bin/oli #5925
• build(deps): bump test-context from 0.3.0 to 0.4.1 in /bin/ofs by @dependabot in build(deps): bump test-context from 0.3.0 to 0.4.1 in /bin/ofs #5924
• build(deps): update pyo3 requirement from 0.23.3 to 0.24.1 in /bindings/python by @dependabot in build(deps): update pyo3 requirement from 0.23.3 to 0.24.1 in /bindings/python #5922
• ci: Disable zigbuild to workaround undefined symbol by @Xuanwo in ci: Disable zigbuild to workaround undefined symbol #5936

Chore

• chore: Fix clippy for rust edition 2024 by @Xuanwo in chore: Fix clippy for rust edition 2024 #5648
• chore: update mongodb version drop the fixme comments by @yihong0618 in chore: update mongodb version drop the fixme comments #5706
• chore: revert "services/cos: Implement Write Returns Metadata for cos" by @Xuanwo in chore: revert "services/cos: Implement Write Returns Metadata for cos" #5713
• chore(core): Fix BufferStream not exported by @Xuanwo in chore(core): Fix BufferStream not exported #5730
• chore: update flate2 version by @yihong0618 in chore: update flate2 version #5759
• chore(services/hdfs_native): enable create_dir capability by @meteorgan in chore(services/hdfs_native): enable create_dir capability #5817
• chore(github): Fix uv not installed in weekly-update by @Xuanwo in chore(github): Fix uv not installed in weekly-update #5819
• chore(github): Fix dateutil not installed by @Xuanwo in chore(github): Fix dateutil not installed #5820
• chore(github): Avoid ping all contributors by @Xuanwo in chore(github): Avoid ping all contributors #5823
• chore: make ruby clippy happy by @yihong0618 in chore: make ruby clippy happy #5830
• chore: update py binding api just by @yihong0618 in chore: update py binding api just #5898
• chore: Polish IO metric buckets by @Xuanwo in chore: Polish IO metric buckets #5903
• chore(layer/observe): cleanup useless metric metadata by @koushiro in chore(layer/observe): cleanup useless metric metadata #5904
• chore: update cpp example by @yihong0618 in chore: update cpp example #5907
• chore: chore some version like logforth by @yihong0618 in chore: chore some version like logforth #5921

New Contributors

• @rick-200 made their first contribution in fix(integrations/dav-server): handle encoded path correctly. #5650
• @Ziy1-Tan made their first contribution in refactor: Migrate oss services to context based http client #5681
• @geruh made their first contribution in feat(java): add WriteOptions for write methods #5664
• @shaonianche made their first contribution in feat(website): Adding a WIP to the list of bingdings #5769
• @leiysky made their first contribution in refactor(gcs): Migrate to context based http client #5778
• @QazCetelic made their first contribution in docs: Fix rustic users entry #5790
• @emliunix made their first contribution in docs(services/onedrive): add how to get tokens #5792
• @Eason0729 made their first contribution in feat: Add origin private file system scaffold #5758
• @Berrysoft made their first contribution in fix(services/compfs): behavior async write #5803

Full Changelog: v0.52.0...v0.53.0