feat: Add AlloyDB Public Preview features, Model Manager, In-Database Embeddings, ScaNN fixes, engine_args and PGVector migrator - #235
Open
madamsetty-pavan wants to merge 6 commits into
Conversation
… Embeddings, ScaNN fixes, engine_args and PGVector migrator
product-auto-label
Bot
added
the
api: alloydb
…mbeddings, engine_args, ScaNN preview features, and PGVector migrator
… and clean circular imports
… to dot_product as verified on live AlloyDB cluster
anubhav756
added
the
priority: p2
Contributor
|
/gcbrun |
Contributor
|
Hi @madamsetty-pavan , could we have separate PRs for the features? Makes the review cycle a bit easier and shorter |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR brings
llama-index-alloydb-pg-pythonto full feature parity with Google Cloud AlloyDB's latest Public Preview capabilities andlangchain-google-alloydb-pg-python.Key Changes:
model_manager.py): AddedAlloyDBModelandAlloyDBModelManagerto manage Vertex AI and custom ML model endpoints in AlloyDB via thegoogle_ml_integrationPostgreSQL extension (v1.5.3+).embeddings.py): AddedAlloyDBEmbeddingimplementing LlamaIndex'sBaseEmbeddinginterface to generate embeddings natively inside SQL queries (SELECT embedding('model_id', 'text')::vector).indexes.py):DistanceStrategy.INNER_PRODUCTto map todot_prod(resolving syntax errors withdot_product).extension_name = "alloydb_scann"andget_index_function().enable_preview_features,max_allowed_num_levels).engine.py): Addedengine_args: Mapping = {}support across engine creation methods.utils/pgvector_migrator.py): Added utility functions to migrate collections from open-sourcepgvector/ LangChain / LlamaIndex tables to AlloyDB.Architecture & Design Document: