AI Support Specialist With Groq - Part I | Replit

AI Support Specialist With Groq - Part I

A guide by

The Replit Team

Guide overview

What will you learn in this guide?

You will get an introduction into setting up a free open-source model with Groq Cloud. You will then connect the application to an Airtable database that contains sample product pricing and customer order information.
You will then combine the power of an LLM and function calling to create a customer service representative that can look up pricing information and create orders in the Airtable database.

What is Groq Cloud?

Groq is an open-source, cloud-native, and scalable framework for building and deploying AI models. It is designed to simplify the process of building, training, and deploying AI models, making it more accessible to developers and data scientists.

What is function calling?

Function calling (or tool use) is the process of a Large Language Model (LLM) invoking a pre-defined function instead of generating a text response. Here are three examples:

  1. Real-time information retrieval - Access up-to-date information by querying APIs, databases, or search tools. For example, an LLM could query a Weather API to provide answers on local forecasts.
  2. Mathematical calculations - LLMs can struggle with mathematical computations. Instead, a user could define a specific calculation and have the LLM call that function.
  3. API integration for actions - Leveraging APIs to take actions like booking appointments, managing calendars, sending emails, and more.

Why should I use function calling?

LLMs are non-deterministic. This offers creativity and flexibility for applications, but it can also lead to risk of hallucinations, inconsistencies, or querying outdated data. In contrast, traditional software is deterministic, executing tasks precisely as performed but lacking adaptability.
Function calling enables the best of both worlds. Your application can leverage the flexibility of an LLM while ensuring consistent, repeatable actions with a subset of pre-defined functions.

Getting started

To get started, fork this Groq template.

Set up a Groq API key

We will be using Meta's Llama 3-70B model for this project. You will need a Groq API Key to proceed. Create an account here to generate one for free.
Once you have generated a key from Groq’s console, open the Secrets pane on Replit and add paste the key in a secret labeled “GROQ_API_KEY.”

Set up your Airtable base connection

We will use Airtable for our backend. In this case, we want a set of customer data that our AI application can query. We have created an Airtable base with sample data here.

Airtable base of ecommerce sample data
Click “copy base” in the upper banner. The flow will prompt you to create a free Airtable account and workspace.
Once this is complete, provision an Airtable Personal Access Token with this link.

Airtable scopes for the Groq e-commerce bot
Scopes define what access this token provides. For now, add the following scopes:

Under “Access”, add the workspace you created with the sample data provided.
Click “Create token”. Copy the token, and paste it in the Replit Secret labeled, “AIRTABLE_API_TOKEN.”
Finally, add your Airtable base ID to the Replit Secret labeled, “AIRTABLE_BASE_ID.” This base ID can be found in the URL, and it begins with “app.” For the example data, the base ID is appQZ9KdhmjcDVSGx. Your base ID will be different. Make sure to copy your base ID.

Set up environment and initial system prompt

Paste the following code from src/01_setup.py into the main.py file in your Repl:

# Setup
import json
import os
import random
import urllib.parse
from datetime import datetime

import requests
from groq import Groq

# Initialize Groq client and model
client = Groq(api_key=os.getenv("GROQ_API_KEY"))
MODEL = "llama3-70b-8192"

# Airtable variables
airtable_api_token = os.environ["AIRTABLE_API_TOKEN"]
airtable_base_id = os.environ["AIRTABLE_BASE_ID"]

SYSTEM_MESSAGE = """
You are a helpful customer service LLM for an ecommerce company that processes orders and retrieves information about products.
You are currently chatting with Tom Testuser, Customer ID: 10
"""

Note: If you receive an error saying, “No module named groq”, then open the Packages pane, search Groq, and click install. This code does the following:

Creating the initial functions

Next, we will create the set of functions or tools that we want the LLM to have. This is a customer support representative, so we are going to give the following tools:

First paste the code for create_order from src/02_tools.py into main.py:

# Creates an order given a product_id and customer_id
def create_order(product_id, customer_id):
    headers = {
        "Authorization": f"Bearer {airtable_api_token}",
        "Content-Type": "application/json",
    }
    url = f"https://api.airtable.com/v0/{airtable_base_id}/orders"
    order_id = random.randint(1, 100000)  # Randomly assign an order_id
    order_datetime = datetime.utcnow().strftime(
        "%Y-%m-%dT%H:%M:%SZ"
    )  # Assign order date as now
    data = {
        "fields": {
            "order_id": order_id,
            "product_id": product_id,
            "customer_id": customer_id,
            "order_date": order_datetime,
        }
    }
    response = requests.post(url, headers=headers, json=data)
    return str(response.json())

Here’s what this code block does:

Next, we will add the code for the product prices and product id tools. The tools are also available in src/02_tools.py. We will paste them at the bottom of main.py:

# Gets the price for a product, given the name of the product
def get_product_price(product_name):
    api_token = os.environ["AIRTABLE_API_TOKEN"]
    base_id = os.environ["AIRTABLE_BASE_ID"]
    headers = {"Authorization": f"Bearer {airtable_api_token}"}
    formula = f"{{name}}='{product_name}'"
    encoded_formula = urllib.parse.quote(formula)
    url = f"https://api.airtable.com/v0/{airtable_base_id}/products?filterByFormula={encoded_formula}"
    response = requests.get(url, headers=headers)
    product_price = response.json()["records"][0]["fields"]["price"]
    return "$" + str(product_price)

# Gets product ID given a product name
def get_product_id(product_name):
    api_token = os.environ["AIRTABLE_API_TOKEN"]
    base_id = os.environ["AIRTABLE_BASE_ID"]
    headers = {"Authorization": f"Bearer {airtable_api_token}"}
    formula = f"{{name}}='{product_name}'"
    encoded_formula = urllib.parse.quote(formula)
    url = f"https://api.airtable.com/v0/{airtable_base_id}/products?filterByFormula={encoded_formula}"
    response = requests.get(url, headers=headers)
    product_id = response.json()["records"][0]["fields"]["product_id"]
    return str(product_id)

Finally, we will compile all these tools into a list that can be passed to the LLM. Notice that we need to add descriptions and parameters, so they can be called properly. Paste the remainder of the src/02_tools.py into main.py:

tools = [\
    # First function: create_order\
    {\
        "type": "function",\
        "function": {\
            "name": "create_order",\
            "description": "Creates an order given a product_id and customer_id. If a product name is provided, you must get the product ID first. After placing the order indicate that it was placed successfully and output the details.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "product_id": {\
                        "type": "integer",\
                        "description": "The ID of the product",\
                    },\
                    "customer_id": {\
                        "type": "integer",\
                        "description": "The ID of the customer",\
                    },\
                },\
                "required": ["product_id", "customer_id"],\
            },\
        },\
    },\
    # Second function: get_product_price\
    {\
        "type": "function",\
        "function": {\
            "name": "get_product_price",\
            "description": "Gets the price for a product, given the name of the product. Just return the price, do not do any calculations.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "product_name": {\
                        "type": "string",\
                        "description": "The name of the product",\
                    }\
                },\
                "required": ["product_name"],\
            },\
        },\
    },\
    # Third function: get_product_id\
    {\
        "type": "function",\
        "function": {\
            "name": "get_product_id",\
            "description": "Gets product ID given a product name",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "product_name": {\
                        "type": "string",\
                        "description": "The name of the product",\
                    }\
                },\
                "required": ["product_name"],\
            },\
        },\
    },\
]```

# Simple function calling  
Now that we have some tools, let’s make our first call to a single tool.

## Calling the model with the tools  
The first step is sending the conversation and the available tools to the model. We will start by giving instructions to the user, so they know how to interact with the application. For now, let’s imagine the user’s name is Tom Testuser. Add the following to main.py:

```python
multiline_text = """
Welcome, Tom Testuser! Ask questions about product price or place an order, like "Compare the price of the Microphone versus the Laptop" or "Please place an order for a Speaker". The app matches your question to one or more of our available tools: create_order, get_product_id and get_product_price.
"""

print(multiline_text)

When the program runs, it will start by giving those instructions to the user. You can click Run, and you will see that text appear in the console. Feel free to adjust the text.

But at this point, the application still does not do anything. Here’s the code to add to main.py at the bottom:

while True:
    # Get user input from the console
    user_input = input("You: ")

messages = [\
        {"role": "system", "content": SYSTEM_MESSAGE},\
        {\
            "role": "user",\
            "content": user_input,\
        },\
    ]
    # Continue to make LLM calls until it no longer decides to use a tool
    tool_call_identified = True
    while tool_call_identified:
        response = client.chat.completions.create(
            model=MODEL, messages=messages, tools=tools, tool_choice="auto", max_tokens=4096
        )
        response_message = response.choices[0].message
        tool_calls = response_message.tool_calls
        # Step 2: check if the model wanted to call a function
        if tool_calls:
            # Step 3: call the function and append the tool call to our list of messages
            available_functions = {
                "create_order": create_order,
                "get_product_id": get_product_id,
                "get_product_price": get_product_price
            }
            messages.append(
                {
                    "role": "assistant",
                    "tool_calls": [\
                        {\
                            "id": tool_call.id,\
                            "function": {\
                                "name": tool_call.function.name,\
                                "arguments": tool_call.function.arguments,\
                            },\
                            "type": tool_call.type,\
                        }\
                        for tool_call in tool_calls\
                    ],
                }
            )

# Step 4: send the info for each function call and function response to the model
            for tool_call in tool_calls:
                function_name = tool_call.function.name
                function_to_call = available_functions[function_name]
                function_args = json.loads(tool_call.function.arguments)
                if function_name == "get_product_id":
                    function_response = function_to_call(
                        product_name=function_args.get("product_name")
                    )
                elif function_name == "create_order":
                    function_response = function_to_call(
                        customer_id=function_args.get("customer_id"),
                        product_id=function_args.get("product_id")
                    )
                elif function_name == "get_product_price":
                    function_response = function_to_call(
                        product_name=function_args.get("product_name")
                    )
                messages.append(
                    {
                        "tool_call_id": tool_call.id,
                        "role": "tool",
                        "name": function_name,
                        "content": function_response,
                    }
                )  # extend conversation with function response
        else:
            print(response.choices[0].message.content)
            tool_call_identified = False

This is a lot of code, so here’s a breakdown of what it does:

Go ahead and click “Run.” You can test the application by asking a question in the console. Ask about a specific product, and see if it returns a value that matches what’s stored in the Airtable.

Adding a frontend and deploying

Now that you have built a functioning chatbot, we should add a frontend and share it. The frontend will give a chat interface that users can interact with. Deploying it will create a permanent replit.app or custom domain to share.

Note: replit.dev urls are for testing, but they only stay up for a short period of time. To keep the project active and shareable, you need to deploy it.

In the original template you forked, we included a file called 04_final_frontend.py within the src folder. The code in this file will look mostly the same, but there are a few differences:

If you would like to learn more, I recommend checking out Streamlit documentation or the Replit Streamlit quickstart guide.

To use this code, we just need to configure the Repl to run the 04_final_frontend.py file and use Streamlit. To do this, find the .replit file under "Config files." Replace all of the code in the .replit file with:

entrypoint = "src/04_final_frontend.py"
run = ["streamlit", "run", "src/04_final_frontend.py", "--server.headless", "true"]

modules = ["python-3.10:v18-20230807-322e88b"]

hidden = [".pythonlibs"]

[nix]
channel = "stable-23_05"

[deployment]
run = ["streamlit", "run", "--server.address", "0.0.0.0", "--server.headless", "true", "--server.enableCORS=false", "--server.enableWebsocketCompression=false", "--server.runOnSave=false", "src/04_final_frontend.py"]
ignorePorts = false
departmentTarget = "gce"

[[ports]]
localPort = 8501
externalPort = 80

[[ports]]
localPort = 8502
externalPort = 3000

Then Click "Run". You should see a "Webview" tab appear with a new user interface:

The final step is to deploy. In the top-right corner of the Workspace, you will see a button called "Deploy." Click the Deploy button.

Click "Setup your deployment", and follow the steps in the pane. Once the project is deployed, you will have replit.app URL that you can share with people. (Note: Deploying an application requires the Replit Core plan. Learn more here.).

What's next

You have now built a fully-functioning chatbot that can use tools, but there is much more we can do. Go to Part II of this guide to learn more about how to call multiple tools at once.