Relay

RelayFailover and Cache Layer for LLM APIs

Relay is a drop in SDK that adds retries, provider failover, caching and telemetry on top of third party LLM APIs like OpenAI and Anthropic.

freemium
llm middlewareapi failoverretrysemantic cachebyoktelemetrysdkcloudflare workers
Indexed
Updated
4.1 (0 Number of reviews)

Log in to rate the project

Try Now

Relay is a middleware SDK that sits between an application and third party LLM APIs from vendors such as OpenAI and Anthropic. Instead of hand rolling retry and fallback code, developers point their client at Relay and get a reliability layer built in.

Core features

  • Provider failover to a backup vendor when the primary is degraded.
  • Retry logic with decorrelated jitter to avoid retry storms.
  • Response cache: exact match on the free tier, semantic cache on paid plans.
  • Telemetry dashboard for latency, cost, cache hit rate and failover events.

Deployment and plans

The proxy runs on Cloudflare Workers at the edge. Relay uses a bring your own keys model, so provider keys are encrypted and the user keeps the billing relationship with each LLM vendor. The SDK is on npm as @relay-api/sdk, with Python support noted on the site. A free tier covers roughly 1,000 requests per month. Paid plans Hobby, Pro and Scale are listed at 19, 49 and 199 USD per month for larger quotas. Confirm the current numbers on the site.

Pros & Cons

Pros

  • Ready made reliability instead of hand rolled retry code
  • Bring your own keys so billing stays with the original LLM vendor
  • Semantic cache can cut duplicate spend on similar prompts
  • Edge deployment on Cloudflare keeps overhead low

Cons

  • Adds an extra hop between the app and the LLM vendor
  • Semantic cache and custom retry policies require a paid plan
  • Python support is smaller than the JavaScript SDK at the time of review

Frequently Asked Questions

What is Relay?

A reliability middleware SDK for LLM APIs. It adds retries, provider failover, caching and telemetry on top of third party LLM vendors.

Which providers does it work with?

Documented failover pairs include third party APIs such as OpenAI and Anthropic. Check the site for the current list.

Do I still pay the LLM vendor directly?

Yes. Relay uses a bring your own keys model, so the billing relationship with the LLM vendor stays with the user.

Is there a free plan?

Yes. The free tier covers about 1,000 requests per month with basic features.

Explore More

Similar Tools

Bodega One Code

Bodega One Code

Bodega One Code is a local-first AI coding IDE with a built-in chat and autonomous agents, supporting Ollama, OpenAI, Anthropic, and more. Personal use is permanently free, with parallel agents via Fleet, scheduled automation loops, air-gap mode, and full model freedom. If you care about keeping code on your machine, this one is worth a look.

Nadir

Nadir

Nadir introduces a verifier-gated LLM router designed to cut API costs without sacrificing quality. It routes requests to cheaper models first, then uses a calibrated verifier to score responses. If quality falls short, it escalates to a more powerful model. This approach claims up to 60% cost savings while maintaining 98% quality, offering an OpenAI-compatible, two-line integration for high-volume, varied complexity workloads.

StackBuilder

StackBuilder

StackBuilder is a free, AI-driven tool that generates professional cloud architecture diagrams from natural language descriptions. It supports major platforms like AWS, Azure, GCP, and Kubernetes, and allows exports to PNG, SVG, and PDF. No registration is required, making it ideal for system design interviews, architecture documentation, and presentations.

Nest by RAVEN

Nest by RAVEN

Nest by RAVEN (also known as NestMux) is a multi-AI terminal workbench for developers. It allows parallel execution of Claude, Gemini, Codex, Copilot, and Aider within a single window. Features include Git worktrees integration, team terminal sharing, MCP panel, and broadcast prompts. It's local-first, telemetry-free, and available for free download on macOS, Windows, and Linux.

h5i

h5i

h5i is an open-source sandbox tool designed for AI coding agents like Claude Code and Codex. It encapsulates the agent, shell, dependencies, and browser within a single, isolated boundary. This lightweight sandbox launches in under 200ms, supports exporting auditable patches and execution logs, and is local-first, SaaS-free, delivered as a single Rust binary.

Deep Work Plan

Deep Work Plan

Deep Work Plan is an open-source methodology that transforms any code repository into a structured, AI-executable environment using an `init.md` file. It breaks down long-term coding tasks into atomic steps with clear acceptance criteria, validation gates, and recoverable states, preventing AI agents from derailing. It's agent-agnostic, open-source (MIT), and prevents vendor lock-in.

Open-source Alternatives

guidellm: Open-Source Tool for Evaluating and Optimizing LLM Inference

guidellm is an open-source tool developed by the vLLM team to evaluate and optimize Large Language Model (LLM) inference performance in production environments. It offers stress testing, latency analysis, and throughput assessment to help developers identify bottlenecks and fine-tune deployment configurations. The project is primarily written in Python and licensed under Apache-2.0. At the time of collection, it had 1214 stars on GitHub.

ai-gateway: Unified AI Gateway Based on Envoy Gateway

ai-gateway is an open-source project built on Envoy Gateway, offering a unified API gateway to manage access to diverse generative AI services. It simplifies AI application integration and operations by providing features like load balancing, caching, and rate limiting for various AI providers. The project is written in Go and licensed under Apache-2.0.

go-micro: Go framework fusing AI agent harness with microservices

go-micro is an open-source Go framework that fuses an AI agent harness with microservices, supporting MCP, A2A, and multi-LLM integration. It is licensed under Apache-2.0 and primarily written in Go. As of the collection time, the project had 22,755 stars on GitHub.

Kun: Local-First AI Agent Workspace

Kun is a local-first AI agent workspace that unifies coding, writing, design, research, and automation through a shared GUI and TUI runtime. The project is primarily developed in TypeScript and has an 'Other' license. As of collection time, it has 4813 GitHub stars.

terax-ai: Lightweight Tauri-based Desktop Dev Environment

terax-ai is a Tauri-based desktop development environment with a size of only 7-8 MB. It integrates a GPU terminal, CodeMirror editor, Git tools, and multi-provider AI agents, offering an all-in-one development experience. The project is primarily written in TypeScript and licensed under Apache-2.0.

jar-analyzer: Open-Source GUI Tool for Java JAR Analysis with AI Assistant

jar-analyzer is an open-source GUI tool for Java JAR package analysis, featuring an integrated AI assistant. It offers robust capabilities like JAR DIFF, method call graph exploration, DFS call chain analysis, taint analysis, and control flow graph (CFG) program analysis. Ideal for Java developers and security researchers, it streamlines code auditing and reverse engineering tasks. The primary language is Java, licensed under GPL-3.0, with 2111 GitHub stars at the time of collection.