Skip to content
Fondazione ITS INCOM

Examples · Programming

A regex that passes its tests

A model writes a regular expression for IPv4 addresses, your tests check it, and the failures go back to the model until everything passes. Then the surprise: Qwen3-Coder is not the best one here.

Intermediate · 40 min

The idea

A regular expression looks right until it meets the case you did not think of. Here the tests are the specification: the program checks every pattern the model writes and sends the failures back, each with the reason. The model corrects itself, at most four times.

This is a small agent: a model, a tool that checks its work, and a loop.

What you need

A key and the openai library, as in Your first API call in Python.

The program

import os
import re

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ai.itsincom.org/v1",
    api_key=os.environ["API_KEY"],
)

MODEL = "gemma-4-26b"

good = ["192.168.1.1", "10.0.0.255", "255.255.255.255", "0.0.0.0", "172.16.254.3"]
bad = {
    "256.1.1.1": "256 is more than 255",
    "999.0.0.1": "999 is more than 255",
    "192.168.1": "only three numbers",
    "1.2.3.4.5": "five numbers",
    "192.168.01.1": "leading zero",
    "a.b.c.d": "letters",
    "192.168.1.1 ": "space at the end",
    "1..2.3": "a number is missing",
}

messages = [
    {"role": "system", "content": "You write Python regular expressions. Answer with the pattern only, "
                                  "on one line, without quotes or explanations."},
    {"role": "user", "content": "A pattern for a whole IPv4 address in dotted decimal: four numbers "
                                "from 0 to 255, without leading zeros."},
]

for attempt in range(1, 5):
    answer = client.chat.completions.create(model=MODEL, temperature=0, messages=messages)
    pattern = answer.choices[0].message.content.strip().strip("`")
    print(f"attempt {attempt}: {pattern}")
    try:
        regex = re.compile(pattern)
    except re.error as e:
        problem = f"Python cannot compile it: {e}. "
    else:
        missed = [s for s in good if not regex.fullmatch(s)]
        extra = [f"{s!r} ({why})" for s, why in bad.items() if regex.fullmatch(s)]
        if not missed and not extra:
            print("all tests pass")
            break
        problem = ""
        if missed:
            problem += f"These must match, and they do not: {missed!r}. "
        if extra:
            problem += f"These must NOT match, and they do: {', '.join(extra)}. "
    print("   ", problem)
    messages += [
        {"role": "assistant", "content": pattern},
        {"role": "user", "content": problem + "Fix it. The pattern only."},
    ]

What you should see

attempt 1: (?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(?:\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}
    These must NOT match, and they do: '192.168.01.1' (leading zero).
attempt 2: ^(?:(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])$
all tests pass

Your patterns can be written a little differently. The first one is the classic you find on the web, and it accepts 01. The test says why it is wrong, and the second attempt is right.

The surprise

Change MODEL and run it again. When we tried it on the school's machines:

“The model for programming” is not the best one for every programming task. Measure on your own problem: that is what the tests are for.

Exercise

All examples