Skip to content
Fondazione ITS INCOM

Examples · Internet of Things

Voice commands for a device

Say “turn on the lamp”: the transcription model writes it down, a model turns it into a JSON command, and a few lines of Python switch the lamp. Tested on a PC, ready for a Raspberry Pi.

Advanced · 50 min

The idea

A smart speaker does three things: it listens, it understands, it acts. Here each one is a step:

  1. the transcription turns the recorded command into text;
  2. a chat model turns the text into a JSON command, choosing only among the devices and actions you allow;
  3. your code reads the JSON and switches the device.

What you need

A key and the openai library, as in Your first API call in Python. No microphone: a voice of the school says the commands for you.

The models

1. Record a command, without a microphone

Save it as say.py:

import os
import sys

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ai.itsincom.org/v1",
    api_key=os.environ["API_KEY"],
)

text, path = sys.argv[1], sys.argv[2]
speech = client.audio.speech.create(model="tts-1", voice="ryan", input=text, response_format="wav")
speech.write_to_file(path)
print(path, "is ready")

Make three commands:

python3 say.py "Could you turn on the lamp, please?" command1.wav
python3 say.py "It's freezing in here." command2.wav
python3 say.py "Play some music." command3.wav

2. Listen, understand, act

Save it as command.py:

import json
import os
import sys

from openai import OpenAI

client = OpenAI(
    base_url="https://api.ai.itsincom.org/v1",
    api_key=os.environ["API_KEY"],
)

devices = {"lamp": "off", "fan": "off", "heater": "off"}

schema = {
    "type": "object",
    "properties": {
        "device": {"type": "string", "enum": ["lamp", "fan", "heater", "none"]},
        "action": {"type": "string", "enum": ["on", "off", "none"]},
    },
    "required": ["device", "action"],
}

# 1. Listen
with open(sys.argv[1], "rb") as f:
    heard = client.audio.transcriptions.create(model="whisper-1", file=f, language="en").text.strip()
print("heard:  ", heard)
if not heard:
    sys.exit("nothing was said")

# 2. Understand
answer = client.chat.completions.create(
    model="gemma-4-26b",
    temperature=0,
    messages=[
        {"role": "system", "content": "You control a room with a lamp, a fan and a heater. Turn what the user "
                                      "says into a command. If it is not about these devices, or it is not "
                                      "clear, answer with device none and action none."},
        {"role": "user", "content": heard},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "command", "schema": schema}},
)
command = json.loads(answer.choices[0].message.content)
print("command:", command)

# 3. Act
if command["device"] in devices and command["action"] in ("on", "off"):
    devices[command["device"]] = command["action"]
    print(f"done:    the {command['device']} is {command['action']}")
else:
    print("done:    nothing, I did not understand")

Run it on each command:

python3 command.py command1.wav

What you should see

heard:   Could you turn on the lamp, please?
command: {'device': 'lamp', 'action': 'on'}
done:    the lamp is on

And for the other two:

heard:   It's freezing in here.
command: {'device': 'heater', 'action': 'on'}
done:    the heater is on

heard:   Play some music.
command: {'device': 'none', 'action': 'none'}
done:    nothing, I did not understand

Look at the second one: nobody said “heater”. The model understood what the person wanted, which is what makes a voice assistant pleasant, and also what makes it risky.

Why the JSON schema

The schema lists the devices and the actions, and the school's machines make the answer follow it: as long as the answer is complete, json.loads works, and the device is one of your list, never "oven" or "unlock". It can still be the wrong one of your list, and a request for something that is not on it should become none, as “Play some music” did. That is why step 3 checks the command again before it acts.

The model only proposes: your code decides. A lamp can follow a guess. A heater, a door or an oven should not: ask “Shall I turn on the heater?” and wait for a yes.

On a Raspberry Pi

The same program, with two changes.

Record with the microphone instead of say.py. arecord is part of Raspberry Pi OS; this records four seconds as a WAV:

arecord -d 4 -f S16_LE -r 16000 -c 1 command.wav

Act on a pin instead of the dictionary. With an LED or a relay module on GPIO 17, step 3 becomes:

from gpiozero import LED

lamp = LED(17)
if command == {"device": "lamp", "action": "on"}:
    lamp.on()
elif command == {"device": "lamp", "action": "off"}:
    lamp.off()

gpiozero turns the pins off when the program ends, so the lamp would go dark at once. On the Pi, keep the program running: a while True: loop that records, listens and acts, one command after another.

No Pi at hand? gpiozero has pretend pins for your PC: pip install gpiozero, then run the program with the variable GPIOZERO_PIN_FACTORY=mock.

Good to know

Exercise

All examples