Getting Started with AgentHub

Learn how to use AgentHub to simulate, trace, and evaluate your agents. Build better agents faster with our comprehensive testing and evaluation platform.

Onboarding

Signing up on our web-app kickstarts the onboarding process. This entails inputting your name, team name, and adding relevant repositories to our Github app. This will allow us to connect to your agents and simulate them, collect execution traces, and evaluate their performance.

If you're joining an existing team, have a teammate send you a join link from the web-app's home page.

SDK

AgentHub's lightweight SDK is designed to integrate seamlessly with your existing agent infrastructure. Install and get started in minutes:

npm i agenthub-sdk

All you need to do is implement a simple AgentRunner method to tell us where your agent entrypoint is. Here is the interface. Note that the input and output are generic and can be any arbitrary json.

interface Agent {
  run(input: any): any;
}

class AgentRunner {
  private agent: Agent;

  constructor(agent: Agent) {
    this.agent = agent;
  }

  /**
   * Wraps the agent entry point.
   *
   * @param input - The input data to pass to the agent.
   * @returns The response generated by the agent.
   */
  run(input: any): any {
    const response = this.agent.run(input); // run the agent against the input
    return response;
  }
}

Here's an example of how to implement the AgentRunner method.

import type { AgentRunner } from 'agenthub-sdk';
          
          /**
 * Implementation of chatResponseAgent that extends AgentRunner interface
 */
export class ConversationalAgentRunner implements AgentRunner {
  private agent: Agent;

  constructor() {
    this.agent = conversationalAgent;
  }

  /**
   * Run the chat response agent with the given input
   * @param input The input for the agent (string or object with query)
   * @returns Promise containing the agent's response
   */
  async run(input: string): Promise<{
    success: boolean;
  }> {
    return {
      success: true,
    };
  }
}

// Create an instance of the ChatResponseAgentRunner
export const conversationalAgentRunner = new ConversationalAgentRunner();

Evaluation Flow

Once you have implemented the AgentRunner method, you can use it to run simulations and evaluate your agent's performance.

In order to evaluate an agent, you must define a few basic building blocks: the agent, query set, and grader. We'll simulate the agent on the query set (the set of inputs) and generate a trace from the run. Then, we'll use your grader to evaluate the agent's performance. The grader will generate a report with scores, insights, and recommendations for improving your agent's performance.

Our most streamlined flow to do this is called the Evaluation Project. Read about it in our next article.