pgbg¶

Postgres-orchestrated background threads for Python

POV: you want a framework-agnostic way to reliably run a plain¹ function or method in the background, repeatedly, but not all the time.

pgbg comes to the rescue with:

• A NOTIFY dispatcher that takes one database connection per process and dispatches notifications to an arbitrary number of subscribers.

• A supervisor that runs your code as a service: in a loop, in a background thread. If your code crashes, the supervisor restarts the loop. Write crash-only code, pgbg takes care of the rest.

Your services can wake up on NOTIFYs, fixed time intervals, or both.

• Postgres-based leader election with automatic failover. Make sure only one process runs work at a time.

• Framework and platform independence.

Background tasks are not a traditional worker queue. Common use cases include:

• Periodic cleanup duties for expired caches or sessions.
• Maintenance of eventually consistent read models.
• Lightweight transactional background tasks with the outbox pattern.
• Implementing worker queues ~ominous foreshadowing~.

The core needs and supports only Psycopg 3 for database access. pgbg comes with optional support for SQLAlchemy in the pgbg.sqlalchemy module that connects everything to a SQLAlchemy Engine.

Credits¶

pgbg is written by Hynek Schlawack and distributed under the terms of the MIT license.

The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.