The Wayback Machine - https://web.archive.org/web/20211230110914/https://github.com/ray-project/ray/issues/19854
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Serve] Warn if deployments have gcs uris and is checkpointed to external kv. #19854

Open
simon-mo opened this issue Oct 28, 2021 · 1 comment
Open

[Serve] Warn if deployments have gcs uris and is checkpointed to external kv. #19854

simon-mo opened this issue Oct 28, 2021 · 1 comment
Assignees
@shrekris-anyscale
Labels
good first issue P1 serve
Milestone
Serve backlog

Comments

@simon-mo
Copy link
Contributor

@simon-mo simon-mo commented Oct 28, 2021

If a deployment has dependency on gcs file uri packages, then the package won't survive cluster restart. Serve should be able to detect this when user also specified external kv store for checkpointing.

We can have a message like this, at .deploy() time.

serve start  --checkpoint-path="file:///..."

ray.init(runtime_env=dict(working_dir="."))
serve.start(detached=True)

@serve.deployment
def f():
   pass

f.deploy()
WARNING/ERROR: Your deployment `f` has dependency on runtime environments that is uploaded Ray memory store. However, you have specified your Serve cluster to checkpoint to `file:///`. The runtime environment will not survive after cluster failure and Serve will not be able to restart `f`. ... INSERT SOLUTION HERE: use jobs, install package globally?...

actually should this an error? @edoakes @architkulkarni
@edoakes
Copy link
Contributor

@edoakes edoakes commented Oct 29, 2021

Sounds good. I'm fine w/ it being a warning (if it's a major footgun we can make it an error in the future).
@simon-mo simon-mo added this to the Serve backlog milestone Oct 29, 2021
@edoakes edoakes added the P1 label Nov 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue P1 serve
3 participants
@edoakes @simon-mo @shrekris-anyscale