Tools

A tool is a function an agent can use to complete a task. Depending on your task, a tool can perform a web search, answer questions about a document, transcribe speech to text, and much more.

Transformers provides a default set of tools for agents. These include the tools mentioned above as well as image question answering, text-to-speech, translation, and a Python code interpreter that executes the Python code generated by a LLM in a secure environment.

Set add_base_tools=True to enable this default set of tools. The tools parameter is for adding additional tools. Leave the list empty if you aren’t planning on adding any other tools to the toolbox.

from transformers import ReactCodeAgent

agent = ReactCodeAgent(tools=[], add_base_tools=True)

You could also manually load a tool with load_tool().

from transformers import load_tool, ReactCodeAgent

tool = load_tool("text-to-speech")
audio = tool("This is a text-to-speech tool")
agent = ReactCodeAgent(tools=[audio])

This guide will help you learn how to create your own tools and manage an agents toolbox.

Create a new tool

You can create any tool you can dream of to empower an agent. The example in this section creates a tool that returns the most downloaded model for a task from the Hub, and the code for it is shown below.

from huggingface_hub import list_models

task = "text-classification"
model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
print(model.id)

There are two ways you can create a tool, using a decorator or a superclass.

Tool decorator

A fast and simple way to create a tool is to add the @tool decorator.

Convert the code above into a tool by wrapping it in a function and adding the @tool decorator. The function needs:

• A clear name that describes what the tool does, model_download_counter.
• Type hints for the input and output (str).
• A description that describes the tool in more detail and its arguments. This description is incorporated in the agents system prompt. It tells the agent how to use the tool, so try to make it as clear as possible!

from transformers import tool

@tool
def model_download_counter(task: str) -> str:
    """
    This is a tool that returns the checkpoint name of the most downloaded model for a task from the Hugging Face Hub.

    Args:
        task: The task to retrieve the most downloaded model from.
    """
    model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
    return model.id

Pass the model_download_counter tool to the agents tools parameter to use it.

from transformers import CodeAgent

agent = CodeAgent(tools=[model_download_counter], add_base_tools=True)
agent.run(
    "Can you give me the name of the model that has the most downloads on the 'text-to-video' task on the Hugging Face Hub?"
)

Tool superclass

Inheritance allows you to customize the Tool superclass or build a tool much more flexibly and comprehensively. This example will show you how to build the same model_download_counter tool as a Tool class.

The Tool class needs:

• A clear name that describes what the tool does, model_download_counter.
• A description that describes the tool in more detail and its arguments. This description is incorporated in the agents system prompt. It tells the agent how to use the tool, so try to make it as clear as possible!
• An inputs attribute that describes the input type. This is a dictionary with the keys, type and description.
• An outputs attribute that describes the output type.
• A forward method containing the code to be executed when the tool is called.

Write the following code below to a file named model_download.py.

from transformers import Tool
from huggingface_hub import list_models

class HFModelDownloadsTool(Tool):
    name = "model_download_counter"
    description = """
    This is a tool that returns the checkpoint name of the most downloaded model for a task from the Hugging Face Hub."""

    inputs = {
        "task": {
            "type": "string",
            "description": "the task category (such as text-classification, depth-estimation, etc)",
        }
    }
    output_type = "string"

    def forward(self, task: str):
        model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
        return model.id

Import the tool from model_download.py and use load_tool() to load it into the agent.

from model_download import HFModelDownloadsTool
from transformers import load_tool, CodeAgent

tool = HFModelDownloadsTool()
model_counter = load_tool(tool)
agent = CodeAgent(tools=[model_counter], add_base_tools=True)

Also consider sharing your tool to the Hub with push_to_hub() so that everyone can use it!

from model_download import HFModelDownloadsTool
from transformers import load_tool, CodeAgent

tool = HFModelDownloadsTool()
tool.push_to_hub("{your_username}/hf-model-downloads")
model_counter = load_tool("m-ric/hf-model-downloads")
agent = CodeAgent(tools=[model_counter], add_base_tools=True)

Add and replace tools

Once an agent is initialized, add or replace its available tools without reinitializing the agent from scratch.

Use add_tool to add a tool to an existing agent.

from transformers import CodeAgent

agent = CodeAgent(tools=[], add_base_tools=True)
agent.toolbox.add_tool(model_download_counter)

Now you can use the default text-to-speech tool to read aloud the most downloaded model for the text-to-video task.

agent.run(
    "Can you read out loud the name of the model that has the most downloads on the 'text-to-video' task on the Hugging Face Hub and return the audio?"
)

Use update_tool to replace an agents existing tool. This is useful if the new tool is a one-to-one replacement of the existing tool because the agent already knows how to perform the task. The new tool should follow the same API as the tool it replaced or the system prompt template should be adapted to ensure all examples using the replaced tool are updated.

agent.toolbox.update_tool(new_model_download_counter)

ToolCollection

A ToolCollection is a collection of Hugging Face Spaces that can be quickly loaded and used by an agent.

Create a ToolCollection object and specify the collection_slug of the collection you want to use, and then pass it to the agent. To speed up the starting process, tools are only loaded if they’re called by the agent.

The example loads a collection of image generation tools.

from transformers import ToolCollection, ReactCodeAgent

image_tool_collection = ToolCollection(collection_slug="")
agent = ReactCodeAgent(tools=[*image_tool_collection], add_base_tools=True)
agent.run(
    "Please draw me a picture of rivers and lakes."
)

Tool integrations

Transformers supports tools from several other libraries, such as gradio-tools and LangChain.

gradio-tools

gradio-tools is a library that enables Gradio apps to be used as tools. With the wide variety of Gradio apps available, you can enhance your agent with a range of tools like generating images and videos or transcribing audio and summarizing it.

Import and instantiate a tool from gradio-tools, for example, the StableDiffusionPromptGeneratorTool. This tool can help improve prompts to generate better images.

Use from_gradio() to load the prompt generator tool.

from gradio_tools import StableDiffusionPromptGeneratorTool
from transformers import Tool, load_tool, CodeAgent

gradio_prompt_generator_tool = StableDiffusionPromptGeneratorTool()
prompt_generator_tool = Tool.from_gradio(gradio_prompt_generator_tool)

Now pass it to the agent along with a text-to-image tool.

image_generation_tool = load_tool("huggingface-tools/text-to-image")
agent = CodeAgent(tools=[prompt_generator_tool, image_generation_tool], llm_engine=llm_engine)
agent.run(
    "Improve this prompt, then generate an image of it.", prompt="A rabbit wearing a space suit"
)

LangChain

LangChain is a library for working with LLMs which includes agents and tools. Use the from_langchain() method to load any LangChain tool into an agent.

The example below demonstrates how to use LangChains web search tool.

from langchain.agents import load_tools
from transformers import Tool, ReactCodeAgent

search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])
agent = ReactCodeAgent(tools=[search_tool])
agent.run("How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?")