Added heartbeat to cloud download#173
Open
rcypher-databricks wants to merge 1 commit intodatabricks:mainfrom
Open
Added heartbeat to cloud download#173rcypher-databricks wants to merge 1 commit intodatabricks:mainfrom
rcypher-databricks wants to merge 1 commit intodatabricks:mainfrom
Conversation
While downloading cloud files a heartbeat will query operation status at regular intervals to keep the connection alive. Implemented heartbeat type which takes an instance of driver.Pinger and calls it at a regular interval. Implemented statusGetter type which implements driver.Pinger by calling GetOperationStatus() Updated NewBatchLoader() to take an argument of type driver.Pinger and updated batchLoader to instantiate a heartbeat instance using the pinger Updated NewRows() to create an instance of statusGetter and updated the various constructor functions to take a diver.Pinger argument. Added interface fetcher.Overwatch. NewConcurrentFetcher() now takes an Overwatch argument. An instance of Overwatch will be started/stopped in conjunction with fetcher start/stop. Signed-off-by: Raymond Cypher <raymond.cypher@databricks.com>
rcypher-databricks
requested review from
andrefurlan-db,
jadewang-db,
susodapop and
yunbodeng-db
as code owners
Contributor
andrefurlan-db
left a comment
andrefurlan-db
left a comment
There was a problem hiding this comment.
Core review: can you be triple sure that we're not leaking go routines?
Feature review: why is this needed? how can the session timeout while we're downloading results?
Contributor
During cloud fetch, the client downloads cloud files without the involvement of DBSQL or cluster so the server may shut down due to inactivity. Heartbeat thread would keep it alive. |
Contributor
Author
|
But as I said before. The operation/session are open until the client reads the entire result set or manually closes. So the server shouldn't be shutting down due to inactivity. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While downloading cloud files a heartbeat will query operation status at regular intervals to keep the connection alive.
Implemented heartbeat type which takes an instance of driver.Pinger and calls it at a regular interval.
Implemented statusGetter type which implements driver.Pinger by calling GetOperationStatus()
Updated NewBatchLoader() to take an argument of type driver.Pinger and updated batchLoader to instantiate a heartbeat instance using the pinger.
Updated NewRows() to create an instance of statusGetter and updated the various constructor functions to take a diver.Pinger argument.
Added interface fetcher.Overwatch. NewConcurrentFetcher() now takes an Overwatch argument. An instance of Overwatch will be started/stopped in conjunction with fetcher start/stop.