Skip to content
Fondazione ITS INCOM

Examples · Programming

A code review as JSON

Qwen3-Coder reviews a file and answers in a JSON schema you choose, so your program can sort the findings, check that each one points at real code, and stop a commit. Then you read them.

Intermediate · 30 min

The idea

Text is for people; programs need data. With response_format you give the model a JSON schema, and the school's machines make the answer follow it: valid JSON, with the fields you asked for, as long as the answer is complete.

The shape can be right and the content wrong: a line number one off, a problem that is not there, a real one missing. So the program checks what it can, and you read the rest.

What you need

A key and the openai library, as in Your first API call in Python.

A file to review

Save this as login.py. It works, and it has several problems.

import hashlib
import sqlite3

SECRET_KEY = "itsincom2026"


def login(db_path, username, password):
    db = sqlite3.connect(db_path)
    row = db.execute(f"SELECT password FROM users WHERE username = '{username}'").fetchone()
    try:
        return row[0] == hashlib.md5(password.encode()).hexdigest()
    except:
        return False

The program

Save it as review.py.

import json
import os
import pathlib
import sys

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ai.itsincom.org/v1",
    api_key=os.environ["API_KEY"],
)

schema = {
    "type": "object",
    "properties": {
        "findings": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "line": {"type": "integer"},
                    "code": {"type": "string"},  # the code on that line, copied by the model
                    "severity": {"type": "string", "enum": ["high", "medium", "low"]},
                    "problem": {"type": "string"},
                    "fix": {"type": "string"},
                },
                "required": ["line", "code", "severity", "problem", "fix"],
            },
        },
    },
    "required": ["findings"],
}

path = sys.argv[1]
lines = pathlib.Path(path).read_text().splitlines()
numbered = "\n".join(f"{n:4} {line}" for n, line in enumerate(lines, 1))

answer = client.chat.completions.create(
    model="qwen3-coder-30b",
    temperature=0,
    messages=[
        {"role": "system", "content": "You review code for bugs and security problems. Report only real problems."},
        {"role": "user", "content": f"Review {path}:\n\n{numbered}"},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "review", "schema": schema}},
)

# The JSON follows the schema only if the answer is complete: "length" means it was cut off.
if answer.choices[0].finish_reason != "stop":
    sys.exit(f"The review was cut off ({answer.choices[0].finish_reason}): nothing to trust, run it again.")

findings = json.loads(answer.choices[0].message.content)["findings"]


def on_its_line(f):
    """The line exists, and the code the model copied is really on it."""
    code = f["code"].strip()
    return 1 <= f["line"] <= len(lines) and code != "" and code in lines[f["line"] - 1]


checked = [f for f in findings if on_its_line(f)]
order = ["high", "medium", "low"]
for f in sorted(checked, key=lambda f: order.index(f["severity"])):
    print(f"{path}:{f['line']} [{f['severity']}] {f['problem']}")
    print(f"    fix: {f['fix']}")
for f in findings:
    if f not in checked:
        print(f"NOT ON LINE {f['line']}: «{f['code']}» [{f['severity']}] {f['problem']}")

# Exit code 1 if a finding on a real line is serious: useful in a script or a Git hook.
sys.exit(1 if any(f["severity"] == "high" for f in checked) else 0)

Run it:

python3 review.py login.py

What the program checks, and what it cannot

What a good review finds

In our runs the model found the injection, MD5 and the bare except:, every time on the right line, and it missed the secret in the code every time. Compare its answer with this list:

A review by a model is a first pass, not a verdict.

Exercise

All examples