Python 3.4 introduced the
asyncio module for implementing single-threaded concurrent code and multiplexing I/O using coroutines. Developers can use
asyncio and the associated helper modules such as
aiohttp to implement highly scalable, non-blocking I/O. However, modules that make blocking I/O calls (e.g.
urllib) often do not work well with
asyncio. Firebase Python Admin SDK, which is based on
requests, is also affected by this deficiency. One could simply invoke Admin SDK methods from a coroutine as shown in listing 1, but the result is not fully asynchronous.
As soon as an I/O operation is invoked by the SDK (
ref.get() in the case of listing 1), the whole program blocks until it completes. As a result the above code takes about 19 seconds to complete, when executed on my Mac over home WiFi.
requests still maintain Python 2.7 compatibility. Therefore it will be a while before native
asyncio support is provided in them. But in the meantime there is a simple trick developers can employ to use these libraries with
asyncio, while preserving the full asynchronous nature of the applications. The key to this trick is to use the
run_in_executor() method provided by the
asyncio event loop. Listing 2 illustrates this point.
Listing 2 first initializes a thread pool executor with 20 worker threads. Then instead of directly invoking Admin SDK operations from the coroutine, it delegates them to the thread pool. This results in a
Future that can be waited on using typical
asyncio primitives like
yield from or
await. This ensures that none of the coroutines started by the code gets blocked on I/O, thus allowing multiple Firebase database interactions to run in parallel. Consequently, listing 2 completes under 2 seconds in my test environment — a near 10X speedup!
Now of course this is not perfect, with the use of “threads” being a major caveat. Therefore this trick cannot possibly scale as well as a coroutines-only solution. In fact increasing the number of workers in the thread pool is the only way to scale this, which is not ideal. But it will get the job done. It certainly enables us to use
firebase_admin and a host of other classic Python libraries in conjunction with
asyncio. All in all, developers should be able to get quite a bit of mileage from this technique until native
asyncio support becomes more prevalent in the Python ecosystem.