Bug/local kerchunk engine

[chiaweh2]

Change Summary

This pull request improves support for opening kerchunk datasets with the kerchunk engine, particularly distinguishing between local and remote reference files. It also adds new tests to ensure the correct behavior for both remote HTTPS URLs and local files when using the kerchunk engine.

Related issue number

Related to the PR #758. This fix bug of local access on the PR and also add unittest on both local and remote reference file access using Kerchunk engine.

Checklist

• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable

Enhancements to dataset opening logic:

• Updated the _open_dataset function in source.py to avoid using fsspec when loading local reference files while engine set to kerchunk, ensuring correct handling by checking that xarray_open_kwargs['engine'] != 'kerchunk' before using fsspec.open_local.

Expanded test coverage:

• Added test_open_dataset_kerchunk_engine to verify that remote kerchunk reference files accessed via HTTPS are properly opened with the kerchunk engine.
• Added test_open_dataset_kerchunk_engine_local to verify that local kerchunk reference files are handled correctly, including testing with specific storage_options for remote S3 access.

[mgrover1]

Great! Thanks for adding a test here too

[mgrover1]

Oh wait - we need to make sure kerchunk is installed in the testing environment too

[chiaweh2]

Thanks for the quick review! I did add the kerchunk package in the previous PR #758 in requirement.txt. Or is the test environment set somewhere else? Thanks!

[mgrover1]

Thanks for the quick review! I did add the kerchunk package in the previous PR #758 in requirement.txt. Or is the test environment set somewhere else? Thanks!

It needs to included here
https://github.com/intake/intake-esm/blob/main/ci/environment-upstream-dev.yml

As in the testing infrastructure, it does not install all the dependencies in the requirements.txt file

[chiaweh2]

Looks like python3.10 is in conflict with the kerchunk version for the python3.10 CI test.

[andersy005]

Looks like python3.10 is in conflict with the kerchunk version for the python3.10 CI test.

can we drop python3.10 in a separate PR and close this issue

• bump minimum python version to 3.11 #762

?

[chiaweh2]

@mgrover1 @andersy005 just checking to see do I need to do anything for the failed testing? Many thanks!

[mgrover1]

@mgrover1 @andersy005 just checking to see do I need to do anything for the failed testing? Many thanks!

correct! I am planning on submitting a PR tonight with the fix; then we can just merge the main branch in here