Spooky `asyncio` Errors and How to Fix Them

Spooky `asyncio` Errors and How to Fix Them

Patrick Deziel | Friday, Oct 13, 2023 |  Python AsynchronousDeveloper

You’ve heard the asyncio library unlocks concurrency for Python with minimal syntactical overhead, but the terminology makes you tremble! Don’t panic — here are 3 of the most common errors you will encounter and how to fix them.

The most common errors you’ll when you start using the Python asyncio library are:

  1. RuntimeWarning: coroutine was never awaited
  2. My coroutine doesn’t run
  3. Task exception was never retrieved

Here’s what those errors mean and how to fix your code.

RuntimeWarning: coroutine was never awaited

This is the most common problem you will encounter and the easiest to fix. Consider the following program.

async def do_something_asynchronously():
    return "Boo!"

if __name__ == "__main__":
    print(do_something_asynchronously())

do_something_asynchronously is supposed to return Boo!. However, if we try to run the code we’ll just see:

<coroutine object do_something at 0x1021f2260>
RuntimeWarning: coroutine 'do_something_asynchronously' was never awaited

To understand what’s going on, let’s compare do_something_asynchronously to its synchronous alternative.

def do_something_synchronously():
    return "Mwahahaha!"

if __name__ == "__main__":
    print(do_something_synchronously())
Mwahahaha!

Notice that do_something_synchronously() returns the expected evil laughter, but do_something_asynchronously() returns a coroutine object. Coroutines can’t be called like normal functions, they have to be scheduled in the event loop. The RuntimeWarning is trying to tell you that it was never scheduled by asyncio. Without the warning, it might be difficult to tell if the function even executed.

Fixing “RuntimeWarning: coroutine was never awaited”

If do_something_asynchronously() is the entry point to your asynchronous code, use asyncio.run().

import asyncio

if __name__ == "__main__":
    asyncio.run(do_something_asynchronously())

If it’s being called by another async function, you need to await it.

async def main():
    await do_something_asynchronously()

My coroutine doesn’t run

This can be a tricky one because there’s no error but your code doesn’t run, or does not finish properly. For example, this program just exits immediately.

import asyncio

async def hello():
    await asyncio.sleep(1)
    print("Hello")

async def world():
    await asyncio.sleep(2)
    print("World")

async def do_hello():
    asyncio.create_task(hello())
    asyncio.create_task(world())

if __name__ == "__main__":
    asyncio.run(do_hello())

Here do_hello schedules two concurrent tasks with asyncio.create_task(). The problem is that asyncio.create_task() doesn’t wait for the tasks to complete. Instead, do_hello returns after scheduling the tasks. Once the event loop exits, it doesn’t care that hello and world are not completed.

Fixing non-running coroutines

The fix is to capture the references to the tasks so we have something to await on. You can use asyncio.wait to wait for multiple tasks at once.

async def do_hello():
    hello_task = asyncio.create_task(hello())
    world_task = asyncio.create_task(world())
    await asyncio.wait([hello_task, world_task])

Task exception was never retrieved

In this example we are scheduling some concurrent tasks using asyncio.create_task() and waiting for them to complete. However, one of the tasks will raise an exception.

import asyncio

async def divide(a, b):
    return a / b

async def do_math():
    task_a = asyncio.create_task(divide(4, 2))
    task_b = asyncio.create_task(divide(4, 0))
    await asyncio.wait([task_a, task_b])

if __name__ == "__main__":
    asyncio.run(do_math())

If we run this code we see the exception but we also get a separate error:

Task exception was never retrieved
future: <Task finished name='Task-3' coro=<divide() done, defined at async.py:3> exception=ZeroDivisionError('division by zero')>
Traceback (most recent call last):
  File "async.py", line 4, in divide
    return a / b
ZeroDivisionError: division by zero

If we inspect the error we see that the task Future is finished and has an exception on it. This may be confusing because in synchronous land we expect exceptions to bubble up naturally, from divide to do_math etc. However in async land multiple tasks are running concurrently in the event loop, so task exceptions must be retrieved by reference. The cause of the problem is that wait actually returns the completed tasks but because we’re not capturing the references the exceptions go uncaught.

Fixing “Task exception was never retrieved”

The easiest way to handle running multiple tasks is to use asyncio.gather. It gathers multiple tasks into one Future that you can await on. By default it will return immediately when the first exception is raised by a task. This allows you to catch exceptions more gracefully.

async def do_math():
    tasks = asyncio.gather(divide(4, 2), divide(4, 0))
    try:
        await tasks
    except ZeroDivisionError as e:
        print("Caught exception: {}".format(e))

If you want tasks to run regardless of exceptions, you can specify return_exceptions=True to return the results or exceptions as a list.

async def do_math():
    tasks = asyncio.gather(divide(4, 2), divide(4, 0), return_exceptions=True)
    results = await tasks
    print(results)
[2.0, ZeroDivisionError('division by zero')]

If you still want to use asyncio.wait, you can capture the completed tasks and read the results or exceptions manually.

async def do_math():
    task_a = asyncio.create_task(divide(4, 2))
    task_b = asyncio.create_task(divide(4, 0))
    done, _ = await asyncio.wait([task_a, task_b])
    for task in done:
        if task.exception():
            raise task.exception()
        else:
            print(task.result())

Conclusion

Asynchronous programming tends to be out of the comfort zone of many Python programmers, but it doesn’t have to be scary! Learning how to approach data processing and modeling using concurrency and parallelism can mean the difference between doing toy analytics and deploying models to production. If you’re thinking about diving into asynchronous data science, I hope this post encourages you to push past the errors so that you can build more effective real-world solutions.

Want to learn more? Check out this webinar on getting started with async data science:

Photo by Philipp Katzenberger on Unsplash

About This Post

asyncio is a handy native Python library for coroutine-based concurrency. Here are some common errors you will encounter and how to fix them.

Written by:

Share this post:

Recent Rotations butterfly

View all

To LLM or Not to LLM (Part 2): Starting Simple

Sick of hearing about hyped up AI solutions that sound like hot air? 🧐 Let’s use boring old ML to detect hype in AI marketing text and see why starting with a simple ML approach is still your best bet 90% of the time.

Building an AI Text Detector - Lessons Learned

The LLMs boom has made differentiating text written by a person vs. generated by AI a highly desired technology. In this post, I’ll attempt to build an AI text detector from scratch!

May 15, 2024

To LLM or Not to LLM: Tips for Responsible Innovation

We’re seeing a proliferation of Large Language Models (LLMs) as companies seek to replicate OpenAI’s success. In this post, two AI engineers respond to LLM FAQs and offer tips for responsible innovation.

Enter Your Email To Subscribe