Upgrade from Prefect 2.0 to 3.0 and use the new Push Work Pools that greatly simplify infrastructure management and reduce cost.
Prefect is an open-source orchestration tool for data engineering. The Python-based tool allows you to define, schedule, and monitor your data pipelines. Prefect is a great tool for data engineers and scientists who want to automate their data pipelines.
In an article titled Getting Started with Prefect: Powerful Orchestration for Your Data, I wrote about deploying Prefect to AWS EC2 to run what they called an Agent, which essentially ran a job after it was kicked off.
With the upgrade to Version 3.0, Prefect is retiring the longstanding Agent model in favor of a new architecture called Workers. The upgrade from v2 to v3 instantly broke my deployments, leaving me needing to figure out an alternative.
Workers are tighter integrated with the infrastructure they run on. This is a fabulous improvement because, in the past, you needed to figure out how to host the Agent on your computer. It was like installing software on a classic hosted server and maintaining it to OS updates.