Firebase: Python Admin SDK with asyncio
Python 3.4 introduced the asyncio module for implementing single-threaded concurrent code and multiplexing I/O using coroutines. Developers can use asyncio and the associated helper modules such as aiohttp to implement highly scalable, non-blocking I/O. However, modules that make blocking I/O calls (e.g. requests, urllib) often do not work well with asyncio. Firebase Python Admin SDK, which is based on requests, is also affected by this deficiency. One could simply invoke Admin SDK methods from a coroutine as shown in listing 1, but the result is not fully asynchronous.
As soon as an I/O operation is invoked by the SDK (ref.get() in the case of listing 1), the whole program blocks until it completes. As a result the above code takes about 19 seconds to complete, when executed on my Mac over home WiFi.
Libraries like firebase_admin and requests still maintain Python 2.7 compatibility. Therefore it will be a while before native asyncio support is provided in them. But in the meantime there is a simple trick developers can employ to use these libraries with asyncio, while preserving the full asynchronous nature of the applications. The key to this trick is to use the run_in_executor() method provided by the asyncio event loop. Listing 2 illustrates this point.
Listing 2 first initializes a thread pool executor with 20 worker threads. Then instead of directly invoking Admin SDK operations from the coroutine, it delegates them to the thread pool. This results in a Future that can be waited on using typical asyncio primitives like yield from or await. This ensures that none of the coroutines started by the code gets blocked on I/O, thus allowing multiple Firebase database interactions to run in parallel. Consequently, listing 2 completes under 2 seconds in my test environment — a near 10X speedup!
Now of course this is not perfect, with the use of “threads” being a major caveat. Therefore this trick cannot possibly scale as well as a coroutines-only solution. In fact increasing the number of workers in the thread pool is the only way to scale this, which is not ideal. But it will get the job done. It certainly enables us to use firebase_admin and a host of other classic Python libraries in conjunction with asyncio. All in all, developers should be able to get quite a bit of mileage from this technique until native asyncio support becomes more prevalent in the Python ecosystem.