Examples · Programming

Unit tests that catch bugs

Ask Qwen3-Coder for pytest tests of an IBAN checker, run them, and find the two bugs hidden in the function.

Intermediate · 50 min

What you need

A key and the openai library, as in Your first API call in Python, and pytest: pip install pytest.

The model

Qwen3-Coder 30B (qwen3-coder-30b), the model for programming.

The function

Save this as iban.py. It checks the control digits of a bank account number (IBAN). It has two bugs: do not look for them yet.

def iban_is_valid(iban: str) -> bool:
    """True if the IBAN is well formed and its check digits are right (ISO 13616, mod 97).

    Well formed: two capital letters for the country, two digits for the check, then 11 to 30
    capital letters A-Z or digits 0-9. Spaces are allowed. Anything else, lowercase and the
    letters or digits of other alphabets included, makes it invalid: False, never an error.
    """
    iban = iban.replace(" ", "")
    if len(iban) < 15 or len(iban) > 34:
        return False
    moved = iban[4:] + iban[:4]
    digits = "".join(str(int(c, 36)) for c in moved)
    return int(digits) % 97 == 0

The program

It asks the model for tests and saves them as test_iban.py.

import os
import pathlib
import re

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ai.itsincom.org/v1",
    api_key=os.environ["API_KEY"],
)

code = pathlib.Path("iban.py").read_text()

answer = client.chat.completions.create(
    model="qwen3-coder-30b",
    temperature=0,
    messages=[
        {"role": "system", "content": "You write pytest tests. Answer with one ```python block only."},
        {"role": "user", "content": (
            "Write pytest tests for this module. It is saved as iban.py, so the tests start with "
            "`from iban import iban_is_valid`. Test what the docstring promises: real valid IBANs "
            "from several countries, wrong check digits, wrong lengths, invalid characters. "
            "Do not change the function.\n\n" + code
        )},
    ],
)

text = answer.choices[0].message.content
tests = re.search(r"```python\s*(.*?)```", text, re.S).group(1)
pathlib.Path("test_iban.py").write_text(tests)
print(tests)

Then run the tests:

pytest -q test_iban.py

What you should see

Some tests fail. Now the real work starts: for each failure, who is wrong, the function or the test?

A test can be wrong too. If the model wrote a “valid” IBAN, check it with a source you trust. These five are the standard examples of the IBAN registry and of the banks' documentation, and they are all valid:

GB82 WEST 1234 5698 7654 32
DE89 3704 0044 0532 0130 00
CH93 0076 2011 6238 5295 7
IT60 X054 2811 1010 0000 0123 456
FR14 2004 1010 0505 0001 3M02 606

Exercise

Fix the function until all the right tests pass.
Send the model the function and the output of pytest, and ask: “Which is wrong, the function or the tests?”. Does it agree with you?
A user types “gb82 west 1234 5698 7654 32”. The docstring wants capitals: should your form refuse it, or convert it before the check?

Solution

The remainder. A right IBAN gives remainder 1, not 0. So % 97 == 0 must be % 97 == 1.
The characters, and where they go. The function never checks them. A - or a / crashes it with ValueError instead of returning False: int("-", 36) fails.

The quick fix, if not iban.isalnum(): return False, is not enough. isalnum() is true for the letters and digits of every alphabet, and int(c, 36) takes only some of them:

>>> "ß".isalnum(), "²".isalnum(), "１".isalnum()
(True, True, True)
>>> int("ß", 36)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 36: 'ß'
>>> int("１", 36)
1

So “ß” and “²” still crash it, and “１”, a full-width digit from an Asian keyboard, counts as a 1. Lowercase passes too, and so do letters where the check digits go.

The fix says what goes where, in plain ASCII. Save it as iban.py:

import re

# Two capital letters (the country), two digits (the check), then 11 to 30 capital letters or digits.
# In a Python pattern, [A-Z] and [0-9] are plain ASCII. \d and isalnum() are not: they also accept
# letters and digits of other alphabets, such as "ß", "²" or "１".
SHAPE = re.compile(r"[A-Z]{2}[0-9]{2}[A-Z0-9]{11,30}")

# The length of each country, from the IBAN registry. Add the countries you need.
LENGTH = {"CH": 21, "DE": 22, "FR": 27, "GB": 22, "IT": 27}


def iban_is_valid(iban: str) -> bool:
    """True if the IBAN is well formed and its check digits are right (ISO 13616, mod 97).

    Well formed: two capital letters for the country, two digits for the check, then 11 to 30
    capital letters A-Z or digits 0-9. Spaces are allowed. Anything else, lowercase and the
    letters or digits of other alphabets included, makes it invalid: False, never an error.
    """
    iban = iban.replace(" ", "")
    if not SHAPE.fullmatch(iban):  # fullmatch: the whole string, not only its start
        return False
    if iban[:2] in LENGTH and len(iban) != LENGTH[iban[:2]]:
        return False
    moved = iban[4:] + iban[:4]
    digits = "".join(str(int(c, 36)) for c in moved)
    return int(digits) % 97 == 1

fullmatch checks the whole string. re.match with a $ at the end would let a newline through.
LENGTH is optional. With it, a Swiss IBAN of 22 characters is refused, even when its check digits are right.

The tests that check it. Save them as test_iban_rules.py:

import pytest

from iban import iban_is_valid

VALID = [
    "GB82 WEST 1234 5698 7654 32",
    "DE89 3704 0044 0532 0130 00",
    "CH93 0076 2011 6238 5295 7",
    "IT60 X054 2811 1010 0000 0123 456",
    "FR14 2004 1010 0505 0001 3M02 606",
]


@pytest.mark.parametrize("iban", VALID)
def test_valid_with_and_without_spaces(iban):
    assert iban_is_valid(iban)
    assert iban_is_valid(iban.replace(" ", ""))


@pytest.mark.parametrize("iban", [
    "GB83 WEST 1234 5698 7654 32",    # wrong check digits
    "GB82 WEST 1234 5698 7654 23",    # two digits swapped
    "CH24 0076 2011 6238 5295 70",    # right check digits, but CH has 21 characters, not 22
    "GBAK WEST 1234 5698 7654 32",    # letters where the check digits go
    "3482 WEST 1234 5698 7654 32",    # digits where the country goes
    "gb82 west 1234 5698 7654 32",    # lowercase: our choice is capitals only
    "GB82-WEST-1234-5698-7654-32",    # only spaces are allowed
    "GB82\tWEST12345698765432",       # a tab is not a space
    "GB82WEST12345698765432\n",       # nor is a newline at the end
    "DE89 3704 0044 0532 0130 0ß",    # a letter of another alphabet
    "GB82 WEST 1234 5698 7654 3²",    # a superscript two
    "GB82 WEST １２３４ 5698 7654 32",  # full-width digits, from an Asian keyboard
    "",
])
def test_invalid(iban):
    assert iban_is_valid(iban) is False   # False, never an exception

pytest -q test_iban_rules.py

All 18 pass. On the quick fix with isalnum(), 7 fail: “ß” and “²” crash it with ValueError, and the full-width digits, the lowercase IBAN, the two in the wrong places and the Swiss one that is too long all come out valid.

Lowercase is a choice. Write it in the docstring, and test it. If your form should accept “gb82 west…”, convert it before the check, and only when it is plain ASCII: if text.isascii(): text = text.upper(). Without isascii(), "ß".upper() is "SS": one character that is not allowed becomes two that are.

What “valid” does not tell you

A valid IBAN is well formed, and that is all. The check digits catch typing mistakes: a wrong digit, two digits swapped. They do not tell you that the account exists, that it is open, or whose it is: anyone can compute right check digits for any number, as we did for CH24 0076 2011 6238 5295 70. Only the bank that holds the account knows. Before money goes to a new IBAN, confirm it with the person who will receive it, on a phone number or a channel you already had: see Phishing or not?.

All examples