Large Language Models and Agents: From Next Tokens to Verifiable Action Loops

This example uses a replaceable fake_llm to expose the agent structure. In a real system, the LLM API can change, but schema validation, tool execution, and verification boundaries should remain outside the model.

import math
from collections import Counter

DOCUMENTS = [
    {"id": "did", "text": "Difference-in-differences compares treated and control changes over time."},
    {"id": "psm", "text": "Propensity score matching balances observed covariates before comparing outcomes."},
    {"id": "uat", "text": "Universal approximation says a wide neural network can approximate continuous functions."},
]

def tokenize(text):
    return [word.strip(".,:;!?").lower() for word in text.split()]

def vectorize(text):
    return Counter(tokenize(text))

def cosine(a, b):
    shared = set(a) & set(b)
    dot = sum(a[key] * b[key] for key in shared)
    norm_a = math.sqrt(sum(value * value for value in a.values()))
    norm_b = math.sqrt(sum(value * value for value in b.values()))
    return 0.0 if norm_a == 0 or norm_b == 0 else dot / (norm_a * norm_b)

def retrieve(query, k=2):
    qv = vectorize(query)
    ranked = sorted(
        DOCUMENTS,
        key=lambda doc: cosine(qv, vectorize(doc["text"])),
        reverse=True,
    )
    return ranked[:k]

def validate_action(action):
    allowed = {"search": {"query"}, "draft": {"claim", "evidence_ids"}}
    if action.get("name") not in allowed:
        raise ValueError("unknown tool")
    missing = allowed[action["name"]] - set(action.get("arguments", {}))
    if missing:
        raise ValueError(f"missing arguments: {missing}")
    return action

def run_tool(action):
    args = action["arguments"]
    if action["name"] == "search":
        return retrieve(args["query"])
    if action["name"] == "draft":
        ids = ", ".join(args["evidence_ids"])
        return f"{args['claim']} Evidence: {ids}."
    raise ValueError("unreachable")

def fake_llm(state):
    # Replace this with an LLM call. Keep the action schema and verifier outside the model.
    if not state["docs"]:
        return {"name": "search", "arguments": {"query": state["task"]}}
    evidence_ids = [doc["id"] for doc in state["docs"]]
    return {
        "name": "draft",
        "arguments": {
            "claim": "Use RAG before answering empirical-method questions.",
            "evidence_ids": evidence_ids,
        },
    }

def verify(answer, docs):
    cited = {doc["id"] for doc in docs}
    return all(doc_id in answer for doc_id in cited)

state = {"task": "Explain DID and PSM for a research assistant.", "docs": [], "trace": []}

for step in range(3):
    action = validate_action(fake_llm(state))
    observation = run_tool(action)
    state["trace"].append({"action": action, "observation": observation})
    if action["name"] == "search":
        state["docs"] = observation
    else:
        if verify(observation, state["docs"]):
            state["answer"] = observation
            break
        state["task"] += " Cite retrieved evidence explicitly."

print(state["answer"])
print("trace length:", len(state["trace"]))

• A user asks: "Explain DID and PSM and give Stata/R implementation cautions." A one-shot LLM answer may be fluent but does not prove its sources, version, or execution path.
• The agent version first writes the task into state, retrieves course pages and method notes, then emits structured tool calls such as search(query), open_file(path), run_code(cmd), or render_table(model).
• Every tool call produces an observation: retrieved documents, code output, table paths, or error logs. The agent writes observations back into context before deciding whether to retrieve more, draft, run checks, or ask for human confirmation.
• The final answer must carry checkable evidence: which knowledge pages were cited, which commands ran, whether route/build/test checks passed, and which assumptions still need human judgment. That boundary separates agents from ordinary chatbots.

References

• Brown et al. (2020), Language Models are Few-Shot Learnershttps://arxiv.org/abs/2005.14165
• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskshttps://arxiv.org/abs/2005.11401
• Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Modelshttps://arxiv.org/abs/2210.03629
• Schick et al. (2023), Toolformer: Language Models Can Teach Themselves to Use Toolshttps://arxiv.org/abs/2302.04761