Scheduling

Contents

Scheduling#

Bounding concurrency for a batch of work.

For coroutines, asyncio.gather with a Semaphore bounds concurrency.

import asyncio

async def bounded_gather(coros, limit=10):
    sem = asyncio.Semaphore(limit)
    async def run(c):
        async with sem:
            return await c
    return await asyncio.gather(*(run(c) for c in coros))

For threads, concurrent.futures.ThreadPoolExecutor with max_workers does the same.

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=10) as pool:
    results = list(pool.map(fetch, urls))

For processes (CPU-bound), swap to ProcessPoolExecutor; the same structure applies.

References#

Contents