A few lines of Python is all it takes to get a model to use a calculator or even automate your hypervisor
Hands on Let's say you're tasked with solving a math problem like 4,242 x 1,977. Some of you might be able to do this in your head, but most of us would probably be reaching for a calculator right about now, not only because it's faster, but also to minimize the potential for error.
As it turns out, this same logic applies to large language models (LLMs). Ask a chatbot to solve that same math problem, and in most cases, it'll generate a plausible but wrong answer. But, give that model its own calculator and, with the right programming, suddenly it can accurately solve complex equations.
This isn't limited to arithmetic, either. The right tools can give LLMs the ability to execute arbitrary code, access APIs, pass off complex requests to domain-specific models, or even search the web.
These tools are one of the building blocks to achieving what folks have taken to calling "agentic AI." The idea is that given the right tools, AI models can break down, plan, and solve complex problems with limited to no supervision.
We'll leave it up to you to decide how much control you want to give these models, what guardrails they need, and how much you want to trust them. We'll focus on the mechanics today.
And so in this hands on, we'll be exploring some of the ways tool-calling can be used to augment the capabilities and address the limitations of LLMs.
Before we go any further, it's important to discuss model compatibility. Not all models support tool or function calling just yet.
As we understand it, Mistral-7B officially added tool support in Instruct version 0.3 in May, while Meta introduced the functionality with the release of Llama 3.1 last month. Other notable models with tool support available from Ollama's repos include Mistral's NeMo, Large, and Mixtral models and Cohere's Command R+.
In this guide, we'll be looking at Mistral-NeMo, but if you're running low on resources, we can confirm Llama 3.1-8B works and can fit into as little as 6GB of memory.
Going back to our example earlier it might be a good idea to give our LLM of choice a calculator to help it cope with the occasional math problem.
To get started, head over to the "Workspace" tab in the Open WebUI sidebar, open the "Tools" section, and create a new tool.
By default, Open WebUI will populate the field with an example script that adds a variety of useful tools, including:
You can leave this script unchanged, but to make it easier to see when it's working, we can add under like so:
Finally, give it a name like "BasicTools" and a brief description and click save.
To put our tools to use, select your model and start a new chat, then press the "+" icon to the left of the message box and enable BasicTools. This will tell the model to use these tools wherever appropriate for the duration of your conversation.
Now, if we prompt the model with our math problem from earlier, we'll see that it not only responds with the correct answer, but also shows the tool it used to arrive at that figure.
Depending on the context of your conversation, multiple tools may be called to address your request.
So, what's going on here? At least in Open WebUI, tools are defined as Python scripts. When formatted correctly, they can be called by the model to solve specific problems.
To make this easier to understand, we pulled this example from the Open WebUI demo we looked at in the last step. It uses Python's libraries to give the model a sense of time.
Aside from the function that actually does the work of retrieving the date and time, there are two elements to be aware of.
The first is the class, which tells the model what functions are available for it to call on.
The second element here is actually the docstring directly under the primary function. This doesn't just tell us what the function does, but it also provides the LLM instructions on what to do with the code.
If you're struggling with your model being too conservative in its use of tools, we found it can be helpful to expand your docstrings with instructions on how, when, and in what format the model should utilize the tool.
In addition to basic functions like clocks, calendars, and calculators, these tools can tie into just about anything with an exposed API.
Beyond retrieving data from remote sources, API-calls can be used to automate all kinds of things, including hypervisors like Proxmox.
In this example, we cobbled together a tool that allows the LLM to connect to our Proxmox cluster's API using the Proxmoxer Python module and gathers information on resource utilization.
Once saved in Open-WebUI, we can simply enable it in the chat and ask it for a health report on our Proxmox cluster. In this case, everything looks healthy.
Of note in this example is the docstring at the top of the script. It's used by Open WebUI to, among other things, fetch relevant Python packages prior to running.
Meanwhile, looking a little lower, we see that the docstring for function tells the model which parameters it should watch for when processing your prompt.
You could go even further than this and define functions for starting, cloning, and managing VMs and LXC containers, with the right safeguards in place, but hopefully this gives you an idea of just how extensible these functions can be.
If you're looking for inspiration, you might want to check out Open WebUI's community page. Just remember not to go blindly pasting code into your dashboard.
Open WebUI's tool-calling implementation is one of the easiest to implement, but it's just one of several out there.
Depending on your use case, one of these may be better suited to your needs. We also expect the number of models trained to take advantage of these tools to grow over the next few months.
The Register aims to bring you more local AI content like this in the near future, so be sure to share your burning questions in the comments section and let us know what you'd like to see next. ®
