Skip to content

Update python - #1465

Open
renovate-bot wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
renovate-bot:renovate/python
Open

Update python#1465
renovate-bot wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
renovate-bot:renovate/python

Conversation

@renovate-bot

@renovate-bot renovate-bot commented Sep 23, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package Change Age Confidence
accelerate ==v1.0.1 → ==1.15.0 age confidence
aiohappyeyeballs (changelog) ==2.6.1 → ==2.7.1 age confidence
arxiv ==2.1.3 → ==2.4.1 age confidence
bitsandbytes (changelog) ==0.44.1 → ==0.50.2 age confidence
cachetools (changelog) ==5.5.0 → ==5.5.2 age confidence
certifi ==2024.8.30 → ==2024.12.14 age confidence
charset-normalizer (changelog) ==3.4.0 → ==3.5.2 age confidence
click (changelog) ==8.1.7 → ==8.5.0 age confidence
cramjam ==2.11.0 → ==2.14.0 age confidence
einops ==0.8.0 → ==0.8.2 age confidence
elasticsearch ==8.15.0 → ==8.19.3 age confidence
emoji ==2.14.1 → ==2.16.0 age confidence
fast-dash ==0.2.13 → ==0.7.3 age confidence
fastapi (changelog) ==0.114.0 → ==0.143.1 age confidence
fastapi (changelog) ==0.115.0 → ==0.143.1 age confidence
fastapi (changelog) ==0.115.13 → ==0.143.1 age confidence
fastparquet ==2024.5.0 → ==2024.11.0 age confidence
gcsfs ==2024.9.0 → ==2024.12.0 age confidence
google-adk (changelog) ==1.16.0 → ==1.39.1 age confidence
google-api-core ==2.22.0 → ==2.42.0 age confidence
google-api-core ~=2.28.1 → ~=2.42.0 age confidence
google-auth (source) ~=2.34.0 → ~=2.62.0 age confidence
google-auth (source) ==2.27.0 → ==2.62.0 age confidence
google-cloud-access-context-manager (source) ==0.2.1 → ==0.6.1 age confidence
google-cloud-asset (source) ==3.27.0 → ==3.30.1 age confidence
google-cloud-bigquery (source) ==3.29.0 → ==3.46.1 age confidence
google-cloud-core (source) ==2.5.1 → ==2.8.0 age confidence
google-cloud-monitoring (source) ==2.23.0 → ==2.32.0 age confidence
google-cloud-org-policy (source) ==1.11.1 → ==1.19.0 age confidence
google-cloud-os-config (source) ==1.18.0 → ==1.26.0 age confidence
google-cloud-pubsub (source) ~=2.23.0 → ~=2.42.0 age confidence
google-cloud-storage (source) ==2.18.2 → ==2.19.0 age confidence
google-crc32c ==1.8.0 → ==1.9.0 age confidence
google-resumable-media (source) ==2.8.2 → ==2.11.0 age confidence
google_adk (changelog) ==1.2.1 → ==1.39.1 age confidence
google_cloud_bigquery (source) ==3.29.0 → ==3.46.1 age confidence
googleapis-common-protos (source) ==1.65.0 → ==1.75.5 age confidence
grpc-google-iam-v1 ==0.13.1 → ==0.14.5 age confidence
grpcio ==1.62.3 → ==1.84.0 age confidence
grpcio-status ==1.62.3 → ==1.84.0 age confidence
huggingface_hub ==0.24.7 → ==0.36.2 age confidence
joblib ==1.5.3 → ==1.6.0 age confidence
k8s-agent-sandbox ==0.4.6 → ==0.5.6 age confidence
langchain-weaviate ==0.0.2 → ==0.0.8 age confidence
llmcompressor ==0.8.1 → ==0.14.0 age confidence
numpy (changelog) ==2.4.4 → ==2.5.4 age confidence
packaging ==26.2 → ==26.3 age confidence
pandas ==2.2.2 → ==2.3.3 age confidence
peft ==v0.13.2 → ==0.21.2 age confidence
pgvector ==0.3.2 → ==0.5.1 age confidence
propcache ==0.4.1 → ==0.5.4 age confidence
proto-plus ==1.25.0 → ==1.29.0 age confidence
psutil ==6.1.0 → ==6.1.1 age confidence
psycopg2-binary (source, changelog) ==2.9.9 → ==2.9.13 age confidence
pyasn1-modules (changelog) ==0.4.1 → ==0.4.2 age confidence
pyasn1_modules (changelog) ==0.4.1 → ==0.4.2 age confidence
pycparser ==2.22 → ==2.23 age confidence
pydantic (changelog) ==2.11.7 → ==2.14.1 age confidence
pymupdf (changelog) ==1.24.10 → ==1.28.2 age confidence
pytz (source) ==2026.1.post1 → ==2026.5 age confidence
qdrant-client ==1.11.1 → ==1.19.1 age confidence
redis (changelog) ==5.0.8 → ==5.3.1 age confidence
requests (changelog) ==2.33.0 → ==2.34.2 age confidence
requests (changelog) ==2.33.1 → ==2.34.2 age confidence
rsa ==4.9 → ==4.9.1 age confidence
ruff (source, changelog) >=0.6,<=0.6.4 → >=0.17,<=0.17.0 age confidence
scikit-learn (changelog) ==1.5.1 → ==1.9.1 age confidence
scipy ==1.17.1 → ==1.18.1 age confidence
six ==1.16.0 → ==1.17.0 age confidence
streamlit (changelog) ==1.54.0 → ==1.66.0 age confidence
tensorflow ==2.17.0 → ==2.21.0 age confidence
thejsonlogger ==0.0.3 → ==0.0.5 age confidence
threadpoolctl ==3.6.0 → ==3.7.0 age confidence
trl ==v0.11.4 → ==0.29.1 age confidence
typing-extensions (changelog) ==4.15.0 → ==4.16.0 age confidence
tzdata ==2026.2 → ==2026.5 age confidence
uvicorn (changelog) ==0.30.6 → ==0.54.0 age confidence
uvicorn (changelog) ==0.34.0 → ==0.54.0 age confidence
weaviate-client ==4.7.1 → ==4.23.1 age confidence
yarl ==1.23.0 → ==1.25.1 age confidence

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

huggingface/accelerate (accelerate)

v1.15.0: : FSDP2 activation memory, dtensor improvements

Compare Source

v1.15.0: FSDP2 activation memory, dtensor improvements

FSDP2

A large batch of FSDP2 work this release: two fixes that cut activation memory at long sequence lengths, tied-embedding support on torch >= 2.13, and a round of checkpointing correctness and scale fixes.

Activation checkpointing was wrapping each child of the matched layer (self_attn, mlp, the norms) instead of the layer itself, so every inter-child activation stayed saved for backward. It now wraps the layer.

There's also a new FSDP2-only activation_checkpointing_offload, which moves the remaining per-layer checkpoint inputs to pinned CPU memory. Gradients are exactly those of plain activation checkpointing:

# fsdp2.yaml
fsdp_config:
  fsdp_version: 2
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_activation_checkpointing: true
  fsdp_activation_checkpointing_offload: true
accelerate launch --config_file fsdp2.yaml train.py
  • FSDP2 activation checkpointing: wrap the matched transformer layer itself, not each of its children by @​qgallouedec in #​4172
  • Add FSDP2 activation_checkpointing_offload: offload checkpointed layer inputs to pinned CPU memory by @​qgallouedec in #​4175
  • Fix FSDP2 tied-embedding models on torch >= 2.13: put the output embedding in the same fully_shard group by @​qgallouedec in #​4171
  • Fix FSDP2/PEFT/FULL_STATE_DICT dropping every rank's adapter shard except rank 0 by @​AmineDiro in #​4206
  • FSDP2: per-rank torch.save/load for SHARDED_STATE_DICT to fix 2800+ NPU checkpoint timeout by @​gygdh-001 in #​4105
  • Fix FSDP sharded checkpoint path resolution by @​HaomingSong in #​4119
  • Raise a clear error when FSDP is enabled on a mesh with no shard dimension by @​qgallouedec in #​4180

DTensor

Two fixes for DTensor-sharded models, which you hit with FSDP2, tensor parallelism, or any N-D parallelism setup: gradient clipping no longer fails on the foreach op when plain tensors and DTensors are mixed, and prepare_model leaves an already-sharded model where it is:

Offloading & Quantization

An entire model can now be dispatched to disk, including tied weights — useful for tools like llm-compressor that compress large models on machines that can't hold them:

Trackers

Custom trackers can be registered by name and then selected from log_with= like any built-in one:

from accelerate import Accelerator
from accelerate.tracking import register_tracker_class

register_tracker_class(MyTracker)  # MyTracker.name == "my_tracker"
accelerator = Accelerator(log_with="my_tracker")

Device support

Neuron gains a torch dynamo backend (so --torch-compile works with the Transformers Trainer) and MPS is now reported and handled properly by accelerate env and find_executable_batch_size.

CLI

Minor fixes

New Contributors

Full Changelog: huggingface/accelerate@v1.14.0...v1.15.0

v1.14.0: : AMD ROCm support, FSDP2 hardening

Compare Source

FSDP2 Improvements

This release brings a large batch of FSDP2 fixes and quality-of-life improvements: correct dtype handling on load, sharding of embeddings/norms, QLoRA crash prevention, and a more robust auto-wrap policy.

AMD ROCm support

Accelerate now works end-to-end on AMD ROCm devices. Thanks @​Abdennacer-Badaoui!

Neuron

Further Neuron improvements to reduce recompilation and cover missing device cases.

Quantization & Offloading

We improved offloading support for quantized models, including Torchao, int8, and tied-weight handling.

Data Loading

  • Feat: Support dynamic batch size in BatchSamplerShard with even_batches by @​yuxinyuan in #​3969
  • Fix iterable dataset sharding condition when n_shards == num_processes by @​SunMarc in #​3958
  • Fix implicit padding in split_between_processes when apply_padding=False and num_samples < num_processes by @​3manifold in #​4052

Minor fixes

Full Changelog: huggingface/accelerate@v1.13.0...v1.14.0

v1.13.0: : Neuron support, IPEX removal, and distributed training fixes

Compare Source

AWS Neuron support

We now have support for AWS Neuron (Trainium/Inferentia) devices. Thanks @​michaelbenayoun for adding this.

XPU Improvements

We've removed IPEX dependency and improved device-agnostic code for XPU.

FSDP2 Improvements

We've added a bunch of important fixes for FSDP2 users: upcasting only grad-requiring params, better tied embedding errors, DCP optimizer loading, bf16 optimizer step crash fix, and torch < 2.7.0 compatibility.

  • Upcast FSDP2 parameters only if requires_grad by @​ojh31 in #​3848
  • Fix FSDP2 tied embedding errors with targeted ValueError guidance by @​amanzoni1 in #​3878
  • bug: fsdp cannot load optimizer state using dcp by @​flymin in #​3904
  • fix crash in optimizer.step when fsdp2 is enabled and model is bfloat16 by [@​sywangyi](https://r

[!IMPORTANT]
✂ PR body was truncated to here.

@renovate-bot
renovate-bot requested a review from a team as a code owner September 23, 2024 00:47
@forking-renovate forking-renovate Bot added the lang: python Issues specific to Python. label Sep 23, 2024
@olivi-eh

olivi-eh commented Sep 23, 2024

Copy link
Copy Markdown
Collaborator

Looks like most of the issues look like this:

6.059 The conflict is caused by:
6.059     langchain 0.3.0 depends on langchain-core<0.4.0 and >=0.3.0
6.059     langchain-community 0.3.0 depends on langchain-core<0.4.0 and >=0.3.0
6.059     langchain-google-vertexai 1.0.10 depends on langchain-core<0.3 and >=0.2.33

Might need to manually bump langchain-google-vertexai, latest is 2.0.1

@renovate-bot
renovate-bot force-pushed the renovate/python branch 7 times, most recently from 2e4015a to 9815aef Compare September 26, 2024 18:40
@renovate-bot
renovate-bot force-pushed the renovate/python branch 15 times, most recently from 89c302d to b8a5279 Compare October 4, 2024 03:20
@renovate-bot
renovate-bot force-pushed the renovate/python branch 12 times, most recently from 096a5d6 to 9beb11e Compare October 17, 2024 00:55
@renovate-bot
renovate-bot force-pushed the renovate/python branch 10 times, most recently from c7294de to 089f9c7 Compare October 24, 2024 17:06
@renovate-bot
renovate-bot force-pushed the renovate/python branch 6 times, most recently from 16286e7 to 5492174 Compare October 28, 2024 16:18
@forking-renovate

forking-renovate Bot commented Jul 4, 2025 •
edited
Loading

Copy link
Copy Markdown

⚠️ Artifact update problem

Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.

♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

  • any of the package files in this branch needs updating, or
  • the branch becomes conflicted, or
  • you click the rebase/retry checkbox if found above, or
  • you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:

File name: cost-optimization/hpa-config-recommender/requirements.txt
Command failed: pip-compile --generate-hashes requirements.in
  ERROR: Cannot install -r requirements.in (line 5) and protobuf==5.29.6 because these package versions have conflicting dependencies.
Discarding protobuf==5.29.6 (from -r requirements.txt (line 1243)) to proceed the resolution
  ERROR: Cannot install -r requirements.in (line 5) and protobuf==5.29.6 because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/resolution.py", line 601, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers/resolution.py", line 542, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.exceptions.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('protobuf==5.29.6'), parent=None), RequirementInformation(requirement=SpecifierRequirement('protobuf<8.0.0,>=6.33.5'), parent=LinkCandidate('https://files.pythonhosted.org/packages/28/ca/fb2a5b38366bcb12990c80384b920f04b968fd834cf98be67d306c374612/google_api_core-2.42.0-py3-none-any.whl (from https://pypi.org/simple/google-api-core/) (requires-python:>=3.10)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/bin/pip-compile", line 6, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/click/core.py", line 1631, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/click/core.py", line 1552, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/click/core.py", line 1415, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/click/core.py", line 910, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/piptools/scripts/compile.py", line 509, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/piptools/resolver.py", line 642, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/piptools/resolver.py", line 677, in _do_resolve
    resolver.resolve(
  File "/opt/containerbase/tools/pip-tools/7.6.2/3.11.17/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 112, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@olivi-eh olivi-eh olivi-eh approved these changes
@alizaidis alizaidis Awaiting requested review from alizaidis alizaidis is a code owner
@yoshi-approver yoshi-approver Awaiting requested review from yoshi-approver yoshi-approver is a code owner
@theemadnes theemadnes Awaiting requested review from theemadnes theemadnes is a code owner
@moficodes moficodes Awaiting requested review from moficodes moficodes is a code owner
@aburhan aburhan Awaiting requested review from aburhan aburhan is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

dependencies lang: python Issues specific to Python.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@renovate-bot @olivi-eh