Prompt-Driven Actions

The TextToAction action allows you to execute a sequence of robot actions based on natural language instructions. It leverages an LLM service to interpret user prompts and generate executable actions, which are then iteratively converted and executed via the provided robot instance.

This is useful when you want to describe what the robot should do in plain text, rather than writing explicit action code.

Action

class aiviro.actions.ai.TextToAction(prompt: str, image: ndarray | None = None, variables: dict[str, Any] | None = None)

Executes a sequence of robot actions based on instructions set via user prompt.

It relies on llm service to generate the actions and then iteratively converts and executes these actions via the provided robot instance.

Parameters:

prompt – A string providing instructions or details required to generate actions for the robot.
image – An optional numpy array representing an image input used alongside for UI recognition. If None is provided, the stable screen of the robot is used. Image is not sent directly to the llm service, but it’s used to extract UI boxes and send them instead.
variables – An optional dictionary of variables to be used in generating and executing actions.

Example:

>>> from aiviro.actions.ai import TextToAction
>>> from aiviro import create_web_robot
>>>
>>> web_r = create_web_robot()
>>> TextToAction(
...     prompt="Visit the aiviro.com website, wait for 'Download Aiviro' and click it."
... )(web_r)
>>> # this will generate and execute the following actions in this order:
... # 1. visit the aiviro.com website
... # web_r.go_to_url("https://aiviro.com")
... # 2. wait for and click on Download Aiviro
... # web_r.click(web_r.wait_for(aiviro.Text("Download Aiviro")))

>>> from aiviro.actions.ai import TextToAction
>>> from aiviro import create_web_robot
>>>
>>> web_r = create_web_robot()
>>> TextToAction(
...     prompt="Type the alarm limit value into the 'Alarm limit' input field and click 'Next'.",
...     variables={"alarm_limit": 10},
... )(web_r)