AutomationMart
Home/Browse/Evaluate AI agent response relevance using OpenAI and cosine similarity
n8n

Evaluate AI agent response relevance using OpenAI and cosine similarity

n8nn8n17 modulesv1.0
OpenAI

This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here How it works This evaluation works best for Q&A agents. For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the origi

At a glance

Evaluate AI agent response relevance using OpenAI and cosine similarity is a ready-made n8n workflow you import as a workflow JSON file — no build required. It connects OpenAI. It's free to download. Follow the 5-step import below to go live in minutes.

Platform
n8n
Connects
OpenAI
Modules
17
Price
Free
Version
v1.0
Evaluate AI agent response relevance using OpenAI and cosine similarity workflow diagram

About this workflow

This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here How it works This evaluation works best for Q&A agents. For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the original question using cosine similarity. A high score indicates relevance and the agent's successful ability to answer the question whereas a low score means agent may have added too much irrelevant info, went off script or hallucinated. Requirements n8n version 1.94+ Check out this Google Sheet for a sample data

n8n

How to import this n8n workflow

  1. 1

    Download the workflow JSON file after purchase.

  2. 2

    Open n8n → click the menu → Import from File.

  3. 3

    Select the downloaded JSON and import.

  4. 4

    Set up credentials for each node that requires them.

  5. 5

    Click Execute Workflow to test, then activate.

Setup guide

Setup guide included

Purchase to unlock the full step-by-step guide

Related N8n workflows

Generate AI document titles for Paperless-ngx with GPT-4o Mini

What it does This template automatically generates a professional, structured title for newly-ingested Paperless-ngx documents using AI (OpenAI GPT), then removes a processing tag to mark the document as handled. How it works 1. A webhook fires when a document needs titling (e.g., from a Paperless post-consumption script) 2. The document URL is validated, then the OCR text is fetched from the Paperless API 3. All PII is scrubbed from the text using n8n's Guardrails node before sending to AI 4. G

Free

Automated resume job matching engine with Bright Data MCP & OpenAI 4o mini

Notice Community nodes can only be installed on self-hosted instances of n8n. Who this is for The Automated Resume Job Matching Engine is an intelligent workflow designed for career platforms, HR tech startups, recruiting firms, and AI developers who want to streamline job-resume matching using real-time data from LinkedIn and job boards. This workflow is tailored for: - HR Tech Founders - Building next-gen recruiting products - Recruiters & Talent Sourcers - Seeking automated candidate-job fit

Free

Trigger new articles and add rows in Google Sheets with a ChatGPT Assistant

Automatically track new articles via RSS, generate insights with a ChatGPT assistant, and log them in Google Sheets for streamlined content management and analysis.

Free

Technical stock analysis with Telegram, Airtable and a GPT-powered AI agent

Video Guide I prepared a detailed guide that demonstrates the complete process of building a trading agent automation using n8n and Telegram, seamlessly integrating various functions for stock analysis. []( Youtube Link Who is this for? This workflow is perfect for traders, financial analysts, and developers looking to automate stock analysis interactions via Telegram. It’s especially valuable for those who want to leverage AI tools for technical analysis without needing to write complex code. W

Free

Automate assignment grading with GPT-4-Turbo and multi-format reports

Introduction Automates AI-driven assignment grading with HTML and CSV output. Designed for educators evaluating submissions with consistent criteria and exportable results. How It Works Webhook receives papers, extracts text, prepares data, loads answers, AI grades submissions, generates results table, converts to HTML/CSV, returns response. Workflow Template Webhook → Extract Text → Prepare Data → Load Answer Script → AI Grade (OpenAI + Output Parser) → Generate Results Table → Convert

Free

Send data to ActionSend and convert HTML to text with OpenAI GPT-3

Automatically send data to ActionSend, convert HTML to text using RegExp, and generate text with OpenAI GPT-3 for streamlined content processing.

Free

Process & summarize PDFs from email & messaging apps with OpenAI GPT

Process PDFs from Gmail, Outlook, Telegram and WhatsApp with OpenAI Transform PDF documents into actionable insights automatically across multiple platforms. This workflow monitors emails and messaging platforms for PDF attachments, extracts content, and delivers AI-powered summaries. Who's it for Business professionals, teams, and organizations needing automated PDF analysis from emails and messaging platforms. Perfect for processing contracts, reports, invoices, and documents requiring quick r

Free

Create custom PDF documents from templates with Gemini & Google Drive

This workflow contains community nodes that are only compatible with the self-hosted version of n8n. 1 — What Does It Do / Which Problem Does It Solve? This workflow turns Google Docs-based contract & form templates into ready-to-sign PDFs in minutes—all from a single chat flow. - Automates repetitive document creation. Instead of copying a rental, sales, or NDA template and filling it by hand every time, the bot asks for the required values and fills them in. - Eliminates huma

Free

Reviews

No reviews yet

Be the first to buy and share your experience.

Leave a review

Sign in to share your experience with this workflow.

Log in to review
Free
No ratings yet

Create a free account to purchase workflows.

  • JSON blueprint — instant download
  • Setup guide PDF included
  • 5 downloads · valid 30 days
  • Works with n8n

Need help setting this up?

Book a 3-hour live setup session with an Agility consultant.

₹2,499/ session
3 hrs · video call
  • Configure live on Google Meet / Zoom
  • Free follow-up if workflow has defects
  • Platform expert assigned to you
Book installation session
Free