scheduler

Description

You can run ECS tasks either on EC2 instances or via AWS Fargate, and these will run in AWS. With ECS Anywhere, you are now able to run the same ECS tasks on any host that has the ECS agent - on prem, in another cloud provider, etc. The control plane resides in ECS, but the execution of the task is managed by the ECS agent.

To launch tasks on hosts that are managed by ECS Anyw

https://dagster.slack.com/archives/C01U954MEER/p1642163479382400:

Hi, I'm experiencing a bug when trying to write an ALS model from pyspark.ml.recommendation to S3 and reading it back in if this takes place within a dynamically executed graph (i.e. via dynamic mapping). I wrote a custom IO manager using the pattern f's3a://{self.s3_bucket}/{key}' as _uri_for_key similar to the one currently imp

Is there a way to make a job re-occur N milliseconds after it finished previous time?

The tests fail sometimes due to millisecond based timing on some of the tests. I over optimized for local testing and it's causing tests to fail in CI occasionally. The CI pipeline is running in docker with limited resources, so not able to handle quite as precise time as my local machine.