AI Observability — Swarnim Tiwari

01

How is each platform architected and how does it instrument applications?

LLM observability platforms divide into two fundamental architecture camps: proxy-based (intercept at the network layer) and SDK-based (instrument at the code layer). The choice determines integration friction, what data is captured, and whether the observability layer can be trusted to see what the application actually sent. These four platforms span the full spectrum.

View

Closed Source · SaaS + Self-Host (Enterprise)

LangSmith

Agent Operations Platform

Architecture

@traceable decorator

LangChain auto-capture

LangGraph Studio (agent IDE)

LangSmith Fleet (deployment)

OTel ingestion (2026)

01Built by LangChain. Closed source. Cloud-hosted with self-hosting available as an Enterprise-only option. Two integration paths: automatic capture for LangChain and LangGraph code (set one environment variable and every run is traced with zero instrumentation code) and manual capture for any other framework via the @traceable decorator or RunTree API. OpenTelemetry ingestion was added in 2026, accepting traces from any OTel-compatible source.
02LangChain 1.0 (GA October 22, 2025) and LangGraph 1.0 (GA simultaneously) reorganized the LangChain ecosystem around a core agent loop with first-class middleware. LangSmith became the integrated evaluation and tracing layer — evals are now a first-class architectural concern in the LangChain stack rather than a third-party add-on. This integration depth is LangSmith's most durable competitive advantage and its clearest limitation simultaneously.
03LangGraph Studio is the platform's agent IDE: attach it to any running LangGraph agent, set breakpoints at any graph node, inspect state at each step, modify state mid-run, and resume from the modified state. The studio uses LangSmith's tracing infrastructure as its data layer. This is the most advanced agent debugging interface available in any observability platform as of 2026.
04LangSmith Fleet (formerly Agent Builder, renamed 2026) extended the platform from observability into deployment. Fleet manages agent deployment with built-in memory, state checkpointing, and HITL review queues. The tracing layer and the deployment layer share the same data model, so a production agent failure in Fleet surfaces directly as a debuggable trace in LangSmith without any integration work. The platform is no longer purely observability; it is an agent operations stack.
05AI-powered trace summarization added in 2026. For agent runs spanning dozens of LLM calls and tool invocations, LangSmith automatically surfaces the key decision points, recurring failure patterns, and anomalous behavior instead of requiring engineers to read through hundreds of nested spans manually. This feature is specifically designed for agentic workloads where traces are too large for human inspection at scale.

LangSmith is the right choice if you are already committed to LangChain or LangGraph. The automatic instrumentation, LangGraph Studio debugger, and Fleet deployment layer are deeply co-designed with those frameworks in ways that cannot be replicated by framework-agnostic alternatives. The risk is what the research consistently shows: teams not using LangChain pay a meaningful integration tax for benefits designed primarily for teams that are.

MIT License · Cloud + Self-Host · ClickHouse (Jan 2026)

Langfuse

Framework-Agnostic, Self-Hostable

Architecture

Python + JS/TS SDKs

Native integrations (15+ frameworks)

OpenTelemetry SDK

Async background flushing

Docker Compose self-host

01MIT licensed. Acquired by ClickHouse in January 2026. The acquisition brings ClickHouse's data infrastructure expertise to Langfuse's observability layer. The MIT license is maintained and the ClickHouse team committed publicly to keeping all existing product features open source. In June 2025 — before the acquisition — Langfuse had already open-sourced all remaining commercially licensed product features including LLM-as-judge evaluation, the prompt playground, and annotation queues. The product is now fully open source.
02Framework-agnostic by design, with native integrations covering every major 2026 framework: OpenAI SDK, Anthropic SDK, LangChain, LlamaIndex, PydanticAI, Vercel AI SDK, OpenAI Agents SDK, CrewAI, Mastra, and others. For non-LangChain teams — which as of 2026 includes the majority of production AI teams using PydanticAI, the OpenAI Agents SDK directly, or the Vercel AI SDK — Langfuse is the only platform in this comparison with comprehensive native integration coverage.
03Instrumentation sends traces asynchronously in background queues without blocking the application response path. Trace events batch locally and flush at configurable intervals. This is explicitly documented: the application's user-facing latency is not affected by the observability layer. In production environments where response time SLAs are strict, this architectural guarantee matters — synchronous tracing that adds latency to every response is not acceptable regardless of the tool's feature set.
04SSO (Okta, Azure AD) is MIT-licensed and included in the self-hosted version without a commercial license requirement. Most commercial open-source platforms gate SSO behind enterprise pricing tiers. Langfuse's decision to MIT-license SSO was documented in community discussions as making self-hosted enterprise deployment significantly more accessible — enterprise IT procurement often requires SSO as a mandatory requirement, and its inclusion in the free tier removes a common procurement blocker.
05Self-hosting runs on Docker Compose with PostgreSQL for trace storage and Redis for background processing. Deployment templates exist for AWS, Azure, GCP, and Kubernetes. The local trace viewer runs Langfuse entirely on a developer's machine — full trace visualization and debugging with zero data egress, important for teams iterating on prompts that process sensitive or confidential data during development.

Langfuse is the clearest choice for teams that are not using LangChain and for teams with data residency requirements. The MIT license, comprehensive framework coverage, self-host story, and the fact that every feature including LLM-as-judge evaluation is now open source makes it the most complete freely available LLM observability stack in 2026. The ClickHouse acquisition introduces long-term roadmap uncertainty but has produced no observable product changes as of the research date.

Apache 2.0 · Acquired Mintlify March 2026 · Maintenance Mode

Helicone

Proxy-Based Gateway

Architecture

HTTP proxy (change one URL)

Under 1ms overhead (self-hosted)

Request-level caching

Rate limiting at proxy

Apache 2.0, self-hostable

01Acquired by Mintlify in March 2026 after processing 14.2 trillion tokens over three years. As of mid-2026, Helicone is in maintenance mode: no new features are being developed. Existing functionality remains operational and the Apache 2.0 open-source codebase is still accessible. Teams evaluating Helicone in 2026 are evaluating a frozen product. This is not a speculation or a forecast — it is a documented fact from the acquisition announcement and confirmed in multiple independent reviews published after March 2026.
02The proxy-based architecture is Helicone's structural differentiator and the reason it remains relevant even in maintenance mode. Route any LLM API call through Helicone's endpoint by changing one URL: from api.openai.com to oai.helicone.ai. No SDK wrapping, no decorator injection, no framework dependency. Any HTTP client that can change a base URL is instrumented in under five minutes. The overhead claim is under 1ms in self-hosted mode — the lowest integration cost of any observability tool.
03The proxy layer captures what was actually sent to and received from the LLM provider at the network level, independent of application code. An application bug that constructs the wrong prompt and logs the wrong value still sends one specific thing to the provider — Helicone captures the actual provider request, not the application's potentially incorrect log of it. SDK-based instrumentation cannot make this guarantee because it relies on the application code to provide accurate trace data.
04Built-in request caching at the proxy layer: identical prompts return cached responses without a provider API call. Published figures: 20-40% cost reduction for typical enterprise workloads where repeated or near-repeated prompts are common (FAQ-style chatbots, content generation with similar templates, batch processing jobs with overlapping inputs). Cache hits appear in the dashboard with zero token cost and zero latency, making the cost savings directly measurable.
05Despite maintenance mode, the proxy architecture addresses a specific production requirement that alternatives do not match in simplicity: teams that need comprehensive LLM request logging with cost and latency tracking, no SDK dependencies, and provider-agnostic capture across a heterogeneous stack of multiple LLM providers and custom endpoints. For that specific requirement, Helicone's self-hosted Apache 2.0 codebase remains the most operationally simple answer. The maintenance mode risk is accepting a frozen feature set.

Helicone's story in 2026 is a case study in how quickly the LLM tooling landscape moves. Three years and 14.2 trillion tokens processed — significant scale — and the product was acquired and frozen. This outcome reflects a structural dynamic in LLM observability: platforms that compete on integration simplicity alone are vulnerable to being absorbed or made redundant by providers (OpenAI, Anthropic) building similar capabilities directly into their APIs. The proxy architecture is clever; clever alone is not defensible enough.

Closed Source · Cloud · ML + LLM Platform

W&B Weave

ML-Native LLM Observability

Architecture

@weave.op() decorator

Auto-instruments OpenAI + Anthropic

W&B Artifacts (versioning)

W&B Registry (model tracking)

OpenTelemetry spans output

01Weave is W&B's LLM observability layer, now production-mature as of 2026. Integration is two lines of Python: import weave and weave.init("project-name"). From those two lines, all OpenAI and Anthropic API calls in the process are traced automatically without modifying any existing code. Additional functions are traced with the @weave.op() decorator. The resulting instrumentation cost is the lowest among SDK-based alternatives in this comparison.
02The structural advantage unique to W&B Weave: models fine-tuned in W&B Sweeps appear in the same UI as the LLM evaluation runs that test them. Datasets versioned in W&B Artifacts serve both training pipelines and evaluation pipelines without duplication. For organizations training custom models and deploying them in LLM applications — fine-tuning Llama for a domain-specific use case, then serving via a RAG pipeline — this unified platform eliminates the data handoff between separate ML experiment tracking and LLM observability tools.
03W&B Registry connects the model lifecycle end-to-end. A model in the Registry is weights linked to the training run that produced them and the evaluation runs that validated them. When a new model version is proposed, the lineage from training data to training run to evaluation results is visible in one place. This is the most complete end-to-end AI artifact lineage in this comparison — LangSmith, Langfuse, and Helicone do not touch the training side of the pipeline.
04@weave.op() creates a call tree automatically from Python's call stack. A multi-agent system with nested tool calls produces a hierarchical trace showing which function called which sub-function with what inputs and outputs, without manual parent-child span management. The developer annotates functions with one decorator; Weave infers the nesting relationships from actual execution order. For teams with complex agent codebases, this reduces instrumentation code significantly.
05OpenTelemetry span emission is available for teams routing W&B traces to an existing OTel backend. For enterprises with Jaeger, Honeycomb, or Datadog already in production monitoring, Weave data can join the unified observability pipeline rather than living in a separate dashboard. This is increasingly important as AI observability consolidates into existing enterprise monitoring stacks rather than remaining in standalone LLM-specific tools.

W&B Weave is the right choice for teams that already use W&B for ML training and want to add LLM observability without adopting a separate platform. The zero-marginal-cost pricing for existing W&B subscribers is a concrete financial advantage. For teams coming to Weave without any W&B history, the platform competes well on feature depth but not on framework-specific integration breadth — Langfuse covers more frameworks, LangSmith integrates more deeply with LangChain. Weave wins on the training-to-deployment lineage story that neither can match.

02

What is the data model for traces and spans?

The trace data model determines what questions you can answer in production. A model that captures only request-level data answers cost and latency questions. A model that captures nested span trees with inputs and outputs at every step answers debugging questions. The difference between these two models is the difference between knowing something failed and knowing why.

View

Closed Source · SaaS + Enterprise Self-Host

LangSmith

Hierarchical Run Tree

Data Model

Run (top-level unit)

Child Runs (nested)

Base traces (14-day)

Extended traces (400-day)

Auto-upgrade on feedback

01The core unit is a Run object. A Run represents one execution of any unit: a chain, an LLM call, a tool, or a retrieval operation. Runs are nested: a top-level chain Run contains child LLM Runs, each of which may contain Tool Runs, which may contain Retrieval Runs. The trace is the top-level Run in this hierarchy. Every input, output, latency, token count, and cost at every level of the hierarchy is captured and stored.
02Two retention tiers with meaningfully different storage costs. Base traces (14-day retention) are the default and cheaper. Extended traces (400-day retention) preserve data for long-term analysis, compliance, and building evaluation datasets. Any trace that receives user feedback — a thumbs down on a response, a correction annotation — is automatically upgraded to Extended retention, ensuring annotated data is never deleted after 14 days. This automatic upgrade creates an important cost implication for high-feedback applications.
03The trace billing multiplier is the most consistently misunderstood aspect of LangSmith's cost model. A single user query that triggers one LangChain chain containing one retrieval step, one reranking step, and one LLM generation step does not generate one trace — it generates four Run objects. An agent that executes five tool calls generates six or more. For production workloads, multiply the expected user request volume by the number of internal pipeline steps to estimate actual billable trace volume. This multiplier is typically 3-8x in a realistic RAG pipeline.
04Time travel replay: load any historical production trace in the LangSmith UI and replay it step-by-step. For multi-step agent failures — where the root cause is a tool call in step 3 that produced incorrect data consumed by the LLM in step 7 — this replay capability is the primary diagnostic tool. The entire state at every step is preserved in the Run tree and available for inspection independent of whether the failing run was from one hour ago or one month ago.
05Online evaluators run against production traces in real time as they arrive. An LLM-as-judge evaluator configured to check response relevance scores every production trace and attaches the score to the trace automatically. These evaluation runs themselves generate billable traces. For deployments running continuous online evaluation with complex LLM-as-judge prompts, the evaluation trace volume can match or exceed the production trace volume — potentially doubling the monthly bill.

LangSmith's Run tree model is the richest hierarchical trace structure in this comparison for LangChain workloads — every step is captured with full context automatically. The cost model consequences of this richness are real: the trace multiplier, the auto-upgrade to Extended on feedback, and the evaluation trace volume all compound in ways that produce monthly bills significantly higher than teams estimate from the base per-trace pricing. Model your specific pipeline depth and feedback rate before committing at scale.

MIT License · Cloud + Self-Host

Langfuse

Five-Object Data Model

Data Model

Trace (user interaction)

Span / Observation (step)

Generation (LLM call)

Score (evaluation result)

Session (grouped traces)

01Five composable objects. Trace: one end-to-end user interaction, contains all sub-steps. Span: a generic step within a trace (retrieval, reranking, custom logic). Generation: a specific LLM call with model name, prompt template, completion, token counts, and cost — a Span specialized for LLM calls. Score: an evaluation result attached to any trace, span, or generation. Session: multiple related traces grouped as one conversation. Each object has a clean, documented schema without framework-specific assumptions baked in.
02Billable units depend on pipeline complexity, not just user request count. A simple single-call application generates approximately 2 units per user request. A complex RAG pipeline with retrieval, reranking, generation, and a safety check generates approximately 6 units per request. An agentic workflow with tool calls and reflection loops can generate 15-20 units per request. Instrument a representative sample of your pipeline and measure the actual units-per-request ratio before estimating costs — this number varies significantly by application architecture.
03Session grouping is a first-class object, not a custom property. Multiple traces from the same conversation are grouped as a Session, enabling tracking of multi-turn conversation quality as a distinct metric from single-response quality. Session-level metrics (task completion rate across turns, conversation coherence score, session abandonment rate) are built-in Langfuse concepts, not derived metrics requiring custom instrumentation. LangSmith and Weave can achieve session grouping through tags; Langfuse models it structurally.
04Score objects are attached at any level of the trace hierarchy: the whole trace, a specific span, or a specific generation within a span. A human reviewer can score a retrieval span for relevance without scoring the full trace. An automated evaluator can score only the LLM generation for hallucination without attaching a score to the retrieval step. This granularity enables quality analysis at the pipeline component level rather than only at the final output level — identifying specifically whether the retrieval or the generation is responsible for output quality degradation.
05Local trace viewer: run Langfuse via Docker Compose on a development machine and view traces in the full visualization UI without any data leaving the local environment. This is not a stripped-down development mode — it is the full Langfuse product running locally. For teams iterating on prompts that process confidential customer data or proprietary business logic, local tracing provides complete observability with zero data egress until the team decides to send data to a cloud or self-hosted production instance.

The five-object data model is Langfuse's most carefully thought-through design decision. It covers every LLM observability concept (trace, step, LLM call, evaluation, session) without framework assumptions, without per-Run billing multipliers, and with clear semantics for what belongs at each level. Teams that design their instrumentation around this model end up with clean data that is easy to query, easy to analyze, and easy to hand off to non-engineering stakeholders.

Apache 2.0 · Maintenance Mode (March 2026)

Helicone

Request-Level, Flat Model

Data Model

Request (one LLM API call)

Custom properties (headers)

Session (header-grouped)

User tracking (header)

Cache hit / miss records

01Request-level traces, not span-level traces. Helicone captures one complete HTTP request to the LLM provider and its response as one record. A complex chain with 5 sequential LLM calls appears as 5 separate request records rather than one nested trace hierarchy with 5 child spans. There are no sub-spans, no parent-child relationships between requests, and no automatic grouping of sequential calls into one agent turn. This flat model is the most significant structural limitation for complex multi-step agent debugging.
02Custom properties via HTTP headers: add Helicone-Property-[name] headers to any request to attach arbitrary key-value metadata. A header like Helicone-Property-User-ID or Helicone-Property-Tenant-ID filters requests by that dimension in the dashboard without modifying the LLM call logic. This is how Helicone achieves tenant attribution, A/B test labeling, and environment tagging — through HTTP headers rather than SDK calls. The mechanism is elegant: any HTTP client that can add headers works without a language-specific SDK.
03Session grouping via three headers: Helicone-Session-ID (groups requests into one session), Helicone-Session-Path (tracks which step in a sequence this request is), and Helicone-Session-Name (labels the session type). The session view shows all requests for one session-ID in chronological order with cumulative token and cost totals. For multi-turn conversation monitoring, this provides reasonable visibility within the flat request model's constraints.
04User tracking via Helicone-User-ID header attributes every request to a named user. Dashboard views aggregate by user: total token spend, average latency, error rate, cost per session. For production applications where cost chargeback by user or tenant is a billing requirement, Helicone's header-based attribution is the simplest implementation: add one header to every request, read per-user cost from the dashboard, no additional analytics infrastructure required.
05Response scoring and a prompt playground were added before the Mintlify acquisition, partially closing the feature gap with Langfuse for evaluation and iteration workflows. These features exist and are functional; they will not be developed further in maintenance mode. A dataset management feature was in development at acquisition time and was not completed. Teams evaluating Helicone for evaluation workflows should note that the evaluation capabilities are limited and will not grow.

Helicone's flat request model is both its most distinctive feature and its clearest ceiling. For the specific use case of "log all LLM requests across any provider with cost attribution and per-user analytics, with zero SDK changes," the proxy architecture delivers better than anything else. For multi-step agent debugging where you need to understand the causal chain from step to step within one agent turn, the flat model cannot answer those questions. The maintenance mode makes this ceiling permanent.

Closed Source · Cloud · Bundled with W&B

W&B Weave

Call Tree + Dataset Versioning

Data Model

Weave Call (one op execution)

Call tree (nested, auto-inferred)

Weave Dataset (versioned)

Weave Model (versioned)

Evaluation results table

01The Weave Call is the core unit: one execution of one @weave.op()-decorated function. Nested function calls produce a tree of Call objects automatically — the parent-child relationship is inferred from Python's call stack rather than requiring manual parent_id passing or context management. An agent with 5 nested tool calls produces a 6-level call tree (agent call containing 5 tool calls) from decorator annotations alone.
02Automatic versioning of Models and Datasets is structurally unique in this comparison. Every unique combination of model configuration generates a versioned Model object. Every unique dataset composition generates a versioned Dataset object. Evaluation runs link specific model versions to specific dataset versions. This linkage means the evaluation result is not just a score — it is a reproducible experiment: model version X, dataset version Y, scorer Z, result 0.87. Any team member can re-run the same experiment and get the same setup.
03Results Table view: every evaluation run produces a tabular view where each row is one input-output-score triple. Columns are sortable and filterable. Comparing two model versions produces a diff view that highlights rows where the output changed. This tabular format is specifically useful for presenting AI quality results to product managers and stakeholders who need to understand where quality changed without navigating a trace tree. The table abstracts the complexity without hiding the data.
04W&B Artifacts provide dataset version control outside of the evaluation system: upload a test dataset as an Artifact, version it, and reference specific artifact versions from evaluation runs. The same dataset Artifact that the training pipeline uses for validation can be consumed by the Weave evaluation pipeline. This eliminates the common scenario where training uses one data format and evaluation uses a copy that diverges over time.
05W&B Registry end-to-end lineage: a model in the Registry is linked to the training run that produced the weights, the dataset that trained on, the evaluation runs that scored it, and the production deployment that serves it. For an organization fine-tuning a custom model and deploying it in production, this lineage answers the question "which training data produced the model I am debugging right now" — a question that cannot be answered from LLM observability data alone.

Weave's call tree model produces the same debugging capability as LangSmith's Run tree for agent workloads, with the additional dimension of version-linked model and dataset tracking that LangSmith does not offer. The limitation is coverage: Weave's native integrations lag behind Langfuse in breadth, and framework-specific debugging depth lags behind LangSmith for LangGraph. Weave is the right choice when the evaluation and comparison workflow is the primary use case rather than production trace debugging.

03

How do they evaluate LLM output quality?

Evaluation is how AI systems get better. Static unit tests do not capture LLM behavior — outputs are probabilistic, quality is subjective, and the same prompt produces different results across models and versions. Production-grade evaluation requires datasets of test cases, automated scorers that run at scale, LLM-as-judge for semantic quality assessment, and human annotation workflows for ground-truth collection.

View

Closed Source · SaaS + Enterprise Self-Host

LangSmith

Offline + Online Evaluation

Evaluation Stack

Dataset management

LLM-as-judge evaluators

Custom code evaluators

Online (production) evals

Human annotation queues

01Datasets are the foundation of LangSmith evaluation. A dataset is a collection of input-output pairs used as test cases. Datasets are built from three sources: manually crafted test cases, production traces (any traced run can be added to a dataset from the UI), and imported CSV or JSON files. Building datasets from production traces is particularly powerful: real user failures become test cases that guard against regression in future versions.
02LLM-as-judge evaluators use a configured LLM to assess response quality against custom criteria. The evaluator receives the input, the output, and optionally a reference answer, then returns a score and reasoning. Multiple criteria run as separate evaluators on the same dataset: one evaluator scores factual accuracy, another scores response clarity, a third scores whether the output followed the requested format. Scores are numeric and comparable across model versions.
03Custom code evaluators run arbitrary Python functions against outputs. For evaluation criteria that an LLM cannot reliably score — regex matching for structured output validation, JSON schema compliance checking, mathematical accuracy verification — custom evaluators provide deterministic scoring. The same evaluator infrastructure runs both LLM-as-judge and custom code evaluators; results are stored in the same dataset table for comparison.
04Online evaluators run against live production traffic in real time. Every production trace is scored by configured evaluators as it arrives, without a separate batch evaluation step. For deployed applications, this means quality metrics are continuously visible in the LangSmith dashboard without manually running evaluation jobs. The cost implication: online evaluation runs generate additional billable traces at the same per-trace pricing as production traces.
05Human annotation queues route specific traces to named reviewers for labeling. Traces that fail automated evaluators above a configured threshold are automatically added to a review queue. Reviewers score responses, add corrections, and optionally promote corrected examples to the evaluation dataset. This human-in-the-loop annotation workflow enables continuous quality improvement: production failures become labeled data that trains better automated evaluators.

LangSmith's evaluation system is the most complete in this comparison for teams running LangChain workloads. The integration between production tracing, dataset building from traces, and online evaluation creates a self-reinforcing quality loop that other platforms require custom engineering to replicate. The limitation: heavy online evaluation doubles trace costs. Teams running comprehensive continuous evaluation should model evaluation trace volume separately from production trace volume in cost projections.

MIT License · Cloud + Self-Host

Langfuse

Scores + Datasets + LLM Judge

Evaluation Stack

Score objects (any level)

LLM-as-judge (MIT, June 2025)

Dataset management

Annotation queues

Experiment runs

01LLM-as-judge evaluation was commercially licensed until June 2025, when Langfuse open-sourced it along with all other previously commercial features. As of mid-2026, LLM-as-judge is available on every tier including the free Hobby plan and the self-hosted version with no restrictions. This open-sourcing was the most significant capability change Langfuse made in 2025-2026 and directly answers the competitive gap versus LangSmith for evaluation.
02Score objects attach to any level of the trace hierarchy. A Score on the full trace measures overall response quality. A Score on a specific Span measures component quality. A Score on a specific Generation measures that LLM call's output quality. This granularity enables root-cause analysis: if overall quality scores are declining, drill down to span-level scores to identify whether the retrieval step or the generation step is the degrading component.
03Datasets in Langfuse are versioned collections of test cases. Items are added from three sources: manual creation in the UI, promotion of production traces to dataset items, and SDK upload from external test case files. Experiment runs compare model versions or prompt versions against the same dataset, producing side-by-side score comparisons. The experiment run history preserves every version comparison for audit and regression analysis.
04Annotation queues route traces to human reviewers for quality labeling. Configure scoring rubrics (thumbs up/down, numeric scale 1-5, categorical labels) and reviewer assignments in the UI. Reviewers work through a focused queue of traces that need human judgment, without seeing the full dashboard. Annotation results are stored as Score objects on the traces, feeding directly into evaluation datasets and quality dashboards.
05The evaluation system works identically whether data is on Langfuse Cloud or a self-hosted instance. For teams with compliance requirements that prevent sending prompt and completion data to a SaaS, the entire evaluation workflow — LLM-as-judge, datasets, annotation queues, experiment runs — runs on self-hosted infrastructure with no feature difference from the cloud version. This is the only platform in this comparison where the full evaluation system is available without a cloud dependency.

Langfuse's evaluation system is the most accessible in this comparison across cost and deployment dimensions. Free, fully open source, self-hostable, and covering the complete evaluation workflow. The gap versus LangSmith is depth of integration with specific frameworks: LangSmith's LangGraph Studio integration enables evaluation workflows that are not possible in Langfuse for LangGraph-specific applications. For every other framework, Langfuse's evaluation system is fully competitive and often preferred because of the self-hosting option.

Apache 2.0 · Maintenance Mode (March 2026)

Helicone

Response Scoring + Playground

Evaluation Stack

Response scoring (thumbs)

Prompt playground

Basic experiments

No LLM-as-judge

No annotation queues

01Response scoring in Helicone is binary: thumbs up or thumbs down on any logged request. Scores can be attached via the dashboard UI or via the API for automated scoring integration. The binary scoring model is significantly less expressive than LangSmith's and Langfuse's multi-dimensional numeric scoring at any trace level. For teams that need simple good/bad labeling of production responses, the binary model is sufficient. For teams running quality research or building evaluation datasets with nuanced rubrics, it is not.
02Prompt playground is available for testing prompt variants against single inputs. The interface shows the model's response alongside token counts, cost, and latency. Multiple model providers are testable from the same playground via the Helicone gateway. The playground was added before the Mintlify acquisition and remains functional; it will not be extended in maintenance mode.
03Basic A/B experiment functionality was added pre-acquisition. Two prompt variants run against the same input and their outputs are compared side by side with cost and latency metrics. This provides prompt comparison capability without requiring external infrastructure. The experiment system lacks dataset management, automated scoring, or statistical significance calculation — it is a visual comparison tool rather than a rigorous evaluation framework.
04LLM-as-judge evaluation is not available in Helicone. Dataset management for systematic test case tracking is not available. Human annotation queues with reviewer assignment are not available. These three capabilities, present in LangSmith and Langfuse, represent the core of what "production evaluation" means in 2026 for teams running agent workloads. Their absence in Helicone's feature set reflects the platform's origin as a request logging and cost tracking tool that grew incrementally toward evaluation rather than being designed for it.
05For teams that use Helicone for its proxy architecture and request logging, evaluation workflows typically run in a separate tool — Langfuse, Braintrust, or a custom script — using data exported from Helicone or collected independently. This two-tool architecture is common among Helicone's production users, who treat it as a gateway layer rather than a complete observability platform. The maintenance mode makes this two-tool architecture necessary going forward rather than optional.

Helicone's evaluation capabilities in 2026 represent what the platform was before the broader LLM observability category matured around systematic evaluation. The binary scoring and basic A/B testing were meaningful additions in 2023-2024; by 2026, the standard has moved to LLM-as-judge, dataset versioning, and continuous online evaluation. In maintenance mode, Helicone cannot close this gap. Teams with serious evaluation requirements should run Helicone for gateway and logging, then use Langfuse or LangSmith for evaluation.

Closed Source · Cloud · Bundled with W&B

W&B Weave

Dataset-First Evaluation

Evaluation Stack

weave.Evaluation API

weave.Dataset (versioned)

Custom Scorer (@weave.op)

LLM-as-judge scorers

Results Table + diff view

01The weave.Evaluation API is the central evaluation primitive. Create a weave.Dataset from test cases, define Scorer functions using @weave.op(), and run weave.Evaluation to score model outputs against that dataset. The evaluation result is stored as a W&B run, comparable to other runs in W&B's comparison UI. Running the same evaluation against two model configurations produces a side-by-side comparison in the existing W&B experiment comparison interface — no separate evaluation dashboard to learn.
02Scorer functions are @weave.op()-decorated Python functions that receive an output and optionally a reference answer, and return a score. A scorer can be a simple string matching function, a regex validator, a mathematical accuracy check, or a call to an LLM-as-judge prompt. All scorers use the same API regardless of complexity. The scoring system does not distinguish between automated and LLM-based scorers at the infrastructure level — both are functions that return a score.
03LLM-as-judge is implemented as a Scorer that calls an LLM internally. W&B provides prebuilt LLM-as-judge scorer templates for common criteria (faithfulness, relevance, hallucination detection) that can be used as-is or modified. Custom LLM-as-judge prompts are defined as scorer functions using the same @weave.op() decorator as code-based scorers. The unified API means the complexity of the scorer is entirely encapsulated in the function implementation, not in the evaluation framework configuration.
04Results Table provides a row-per-test-case view of every evaluation run. Each row shows the input, the output, and every scorer's result. Columns are sortable and filterable. A diff comparison between two evaluation runs highlights rows where the output changed or where scorer results diverged. This tabular format is specifically designed for presenting AI quality data to stakeholders who need to validate quality without navigating trace hierarchies.
05W&B Artifacts connect evaluation datasets to training datasets. The same dataset Artifact used for model fine-tuning validation can be referenced in Weave evaluation runs. When a model is retrained and the training dataset is updated, the evaluation dataset can be updated in the same versioned Artifact system. The lineage between training data, trained model, and evaluation results is maintained automatically through the Artifact versioning system — a capability that requires custom engineering in any other platform in this comparison.

Weave's evaluation system is the cleanest API design in this comparison. The @weave.op() decorator handles both instrumentation and evaluation in one concept. The weave.Evaluation API separates concerns cleanly: datasets, scorers, and evaluation runs are independent versioned objects that compose. The platform's ceiling is adoption outside of existing W&B users — the API is well-designed but the breadth of framework integrations and the ecosystem of prebuilt evaluation templates is smaller than LangSmith's for LLM-specific use cases.

04

How do they handle prompt versioning and management?

A prompt is a software artifact. It should be versioned, tested before deployment, rolled back when it regresses, and audited for changes. Most teams treat prompts as strings in environment variables or hardcoded in application code until the first production prompt regression forces them to build something better. These platforms provide that infrastructure as a managed service.

View

Closed Source · SaaS + Enterprise Self-Host

LangSmith

Prompt Hub

Prompt Stack

Prompt Hub (public + private)

Version history

A/B test between versions

Pull from Hub in code

Playground (multi-model)

01Prompt Hub is LangSmith's prompt management system: a versioned registry of prompt templates that can be public (shared across the LangChain community) or private (scoped to an organization). Every commit creates a new version with a hash. Applications pull prompts from the Hub by name at runtime rather than hardcoding template strings — a prompt change is a Hub commit, not a code deployment.
02Pulling from the Hub at runtime enables prompt updates without application redeployment. A prompt change for a production application is: edit in Hub, commit, and the next application request pulls the new version automatically. For teams that need to change prompt behavior faster than their deployment pipeline allows, this runtime pull pattern is the architectural solution. The tradeoff: the application's behavior changes without a code review cycle.
03Playground supports testing prompts against multiple models side-by-side. Edit a system prompt, send the same user message to GPT-5, Claude 4, and Gemini simultaneously, compare outputs and token costs in a table. This model comparison capability is built into the playground without requiring separate API calls or custom comparison code. Changes made in the playground commit directly to the Hub prompt version history.
04A/B testing between prompt versions routes a configured percentage of production traffic to the new prompt while keeping the rest on the old version. Metrics are compared between the two versions in real time from production traffic rather than synthetic test cases. Statistical significance indicators surface when enough traffic has run on both versions to draw conclusions. This A/B infrastructure runs inside LangSmith without requiring a separate feature flag or experimentation platform.
05Prompt version history is tied to evaluation history. When a prompt version is evaluated against a dataset, the evaluation result is linked to that specific prompt version. Browsing the version history shows not just the text changes between versions but the quality metric changes associated with each version. A team can identify the exact commit that introduced a quality regression without re-running evaluations on historical versions.

The Prompt Hub's public component has built a community of shared prompts that reduces the starting-from-zero cost for common tasks. The private Hub for organization-internal prompt management solves a real problem — prompt version drift where different engineers use different versions of "the same" prompt — in a way that hardcoded strings or environment variables cannot. The runtime pull pattern is the most significant operational convenience and the one team practice change that delivers immediate quality benefits.

MIT License · Cloud + Self-Host

Langfuse

Versioned Prompt Registry

Prompt Stack

Prompt registry (versioned)

Pull via SDK at runtime

Labels (production, staging)

Playground (multi-model)

Link prompts to traces

01Langfuse's prompt registry stores versioned prompt templates that applications pull via the SDK at runtime. langfuse.get_prompt("my-prompt") returns the current production version of the template. Every change creates a new version with a version number. Applications call get_prompt by name, not by version number — the registry resolves the "production" label to the appropriate version, decoupling the application code from the specific prompt content.
02Labels assign semantic meaning to versions: "production" (the live version), "staging" (the next candidate), or any custom label. Rolling back a bad prompt version is a label reassignment: move the "production" label from the failing version to the previous stable version. The application pulls the same prompt name; the registry returns the newly labeled production version. No code deployment required for rollback.
03Prompts are linked to traces automatically when pulled via the SDK. Every trace generated from a run that used a specific prompt version records which prompt version was active. This linkage enables quality analysis by prompt version: compare output quality scores for traces where prompt v7 was active versus traces where prompt v8 was active, using the same evaluation scores attached to those traces. Quality impact of prompt changes is measurable directly from production data.
04Playground supports testing prompt versions across multiple configured LLM providers. Edit the prompt template in the playground, send test inputs, compare responses from different models, then save the playground session as a new prompt version or as a dataset item. The playground is a prompt development environment connected to the production registry rather than a separate isolated testing tool.
05On the self-hosted version, the prompt registry works identically to the cloud version with no feature restrictions. For teams where prompts contain proprietary business logic — system prompts encoding confidential product personas, retrieval strategies, or compliance rules — storing prompts in a self-hosted Langfuse instance keeps that intellectual property on controlled infrastructure rather than in a third-party SaaS.

Langfuse's prompt management with label-based rollback and trace-linked quality analysis is the most operationally mature feature for teams running in production without a LangChain dependency. The label system maps to how teams think about deployments (production vs staging) and the trace linkage answers the most important prompt management question: did this prompt version make things better or worse, measured against real production traffic.

Apache 2.0 · Maintenance Mode (March 2026)

Helicone

Prompt Playground Only

Prompt Stack

Prompt playground

No versioned registry

No runtime pull

No A/B routing

Template parameter support

01Helicone's prompt management is the playground interface: write a prompt, send test inputs, view responses with cost and latency. There is no versioned prompt registry. There is no runtime pull via SDK. There is no A/B routing between prompt versions. The playground is a testing tool, not a deployment system. Prompts tested in the Helicone playground must be manually copied into application code to be deployed — there is no managed path from testing to production.
02Template parameter support in the playground allows defining reusable templates with variables: "Answer the following question about {topic}: {question}". Variables are filled in the playground UI and visible in the generated request. For developers testing prompt templates before committing them to code, this parameterization is useful for validating that templates handle varied inputs correctly before deployment.
03No prompt version history is available in Helicone's system. Changes made in the playground do not accumulate a history. If a developer tests and iterates on a prompt across multiple sessions, earlier versions are not recoverable from Helicone. Version control for prompts in a Helicone deployment falls to whatever external version control the team uses — typically Git — rather than the observability tool.
04The gap in prompt management capabilities reflects Helicone's original positioning as a request logger and cost tracker. Prompt management was added incrementally as the category evolved, but the full stack of versioned registry, runtime pull, and A/B deployment was not completed before the Mintlify acquisition. In maintenance mode, this capability gap is permanent.
05Teams running Helicone in production who need prompt management infrastructure are building it separately: Git for version control, feature flags for A/B testing, and manual processes for deploying prompt changes. This is not unique to Helicone — many teams build their own prompt management stack — but it means Helicone provides no value in the prompt management dimension, unlike LangSmith and Langfuse which provide complete managed prompt management systems.

Helicone's prompt management story in 2026 is simply: it does not have one. The playground is useful for testing before copy-pasting into code. The absence of a versioned registry, runtime pull, rollback, and A/B routing means Helicone contributes nothing to the prompt lifecycle management problem that is the second most common source of production LLM quality issues after retrieval failures. Teams using Helicone should source their prompt management from a different tool.

Closed Source · Cloud · Bundled with W&B

W&B Weave

Model Registry + Playground

Prompt Stack

Playground (built-in)

Prompt versioning via ops

W&B Artifacts (prompt files)

W&B Registry (model + prompt)

No dedicated prompt registry SDK

01W&B Weave does not have a dedicated prompt registry with runtime pull capability comparable to LangSmith's Hub or Langfuse's prompt registry. Prompt templates in W&B can be versioned as W&B Artifacts — text files tracked in the Artifact system — but the application-level integration (pulling the current production prompt by name at runtime) requires custom code rather than a built-in SDK function. This is a genuine gap for teams that need managed prompt deployment.
02Prompt versioning through @weave.op() is available as an indirect mechanism: a function that returns a prompt template decorated with @weave.op() generates a Weave Call for every invocation, and changes to the function body create a new versioned function in Weave's call history. This gives some version tracking but is not a purpose-built prompt management system — it is the general Weave versioning model applied to prompt-returning functions.
03Playground is available for testing prompts against multiple models and comparing outputs. The playground integrates with the evaluation system: promising prompt variations tested in the playground can be saved and compared against evaluation datasets in the same workflow. This connection between interactive exploration and systematic evaluation is more direct than in Helicone's isolated playground.
04W&B Artifacts provide file-level version control for prompts stored as text files or JSON. A prompt file committed as an Artifact has a version, a hash, and linkage to any runs that consumed it. For teams already managing model configuration files through W&B Artifacts, extending that pattern to prompt files is architecturally consistent even if it requires more manual orchestration than a dedicated prompt registry.
05For teams working on fine-tuned models where the "prompt" is less important than the model weights, Weave's emphasis on model artifact versioning and lineage is a better fit than a prompt management system. A fine-tuned model with a minimal system prompt has less prompt complexity to manage than a GPT-4 deployment with elaborate prompt engineering. Weave's architecture reflects this difference in priorities.

W&B Weave's prompt management is the weakest dimension compared to LangSmith and Langfuse. The absence of a purpose-built prompt registry with runtime pull and rollback means teams using W&B for LLM observability need a separate tool for prompt management if they need managed deployment — ironically, often Langfuse, which has the most mature open-source prompt registry. For teams fine-tuning custom models where model weights matter more than prompt engineering, this gap is acceptable. For teams doing heavy prompt engineering on frontier models, it is a meaningful limitation.

05

How do they track cost and usage?

LLM cost is a runtime property, not a fixed infrastructure line item. Token pricing varies by model, version, and input-output ratio. At production scale, an unexpected prompt that generates long completions can spike monthly bills by 2-3x overnight. Cost observability is not a nice-to-have — it is a financial control requirement for any application with variable LLM usage.

View

Closed Source · SaaS + Enterprise Self-Host

LangSmith

Pipeline-Level Cost View

Cost Stack

Cost per trace (hierarchical)

Cost per step (granular)

Token usage by model

Latency percentiles (P50, P99)

Unified agent cost view (2026)

01Cost is tracked at every level of the Run hierarchy. The top-level trace shows total cost for the full agent run. Each child Run shows the cost of that specific step. Drilling from the trace level to a specific LLM call reveals the exact token counts (input, output, cache tokens) and the cost contribution of that one call to the total. This hierarchical cost view enables cost attribution that identifies which pipeline step is most expensive — not just what the total cost was.
02Unified agent cost view added in 2026 covers not just LLM token costs but the full cost of an agent run: token costs for LLM calls, retrieval costs for vector database queries, tool execution costs for third-party API calls, and any custom cost metrics the developer instruments. For complex agentic workflows where LLM tokens are not the dominant cost driver, this unified view is the only way to understand the true economics of each agent run.
03Monitoring charts in the LangSmith dashboard show cost trends over time: daily and weekly spend, cost per user, cost per feature area (broken out by tag or metadata), and latency percentiles at P50 and P99. Custom dashboards filter by any metadata dimension attached to traces — environment, team, product feature, model version. For production cost monitoring, these dashboards provide the operational visibility layer that otherwise requires a separate analytics tool.
04Token usage breakdown by model: in deployments running multiple models (routing cheap models for simple queries and expensive models for complex ones), LangSmith shows the token distribution across models automatically. This model-level breakdown is essential for validating that routing strategies are working as intended — if 80% of tokens are going to the expensive model despite a routing configuration that should send most queries to the cheap model, something is wrong.
05Evaluation runs contribute to the cost picture. Online evaluators running LLM-as-judge prompts against every production trace incur token costs with every trace. These evaluation token costs appear in LangSmith's usage breakdown separately from production LLM costs, making the cost of the evaluation infrastructure itself visible. Teams can evaluate the return on evaluation spend: are the quality improvements discovered worth the token cost of running continuous online evaluation?

LangSmith's cost tracking is the most granular in this comparison — cost attribution at every step of the pipeline hierarchy, unified coverage of LLM tokens and non-token costs, and evaluation costs separated from production costs. The irony is that the most granular cost tracking tool is also the one with the most surprising costs of its own: the trace billing multiplier, evaluation trace overage, and Extended retention pricing can produce total platform costs that exceed the production LLM costs being tracked.

MIT License · Cloud + Self-Host

Langfuse

Model-Level Cost Attribution

Cost Stack

Cost per Generation

Cost per Trace

Custom model pricing

Cost by user / session / tag

Dashboards + exports

01Cost is tracked at the Generation level — the specific LLM call object in Langfuse's data model. Token counts (input tokens, output tokens, cache read tokens for providers that offer prompt caching) are captured per Generation. Cost is calculated from those token counts using provider pricing tables maintained in Langfuse. For prompt caching use cases (Anthropic's and OpenAI's prompt caching features), separate cache token cost tracking is available.
02Custom model pricing configuration: define cost per input token and cost per output token for any model, including self-hosted models and custom deployments not in Langfuse's default pricing table. Teams running fine-tuned models on their own infrastructure enter the actual cost per token and Langfuse applies that pricing to every Generation using that model string. This is important for accurate cost attribution in deployments mixing provider API models and self-hosted models.
03Cost attribution by dimension: filter cost dashboards by user ID, session ID, prompt version, model, tag, or any custom property attached to traces. A multi-product company running multiple applications through one Langfuse instance can separate cost attribution by tagging traces with the product name. A multi-tenant SaaS can attribute cost per tenant and export that data for billing reconciliation. This dimensional filtering requires that the relevant dimension is attached as a metadata property to traces at instrumentation time.
04Data export via API: all cost and usage data is queryable via the Langfuse API and exportable to CSV or JSON. For teams with existing BI tools (Tableau, Looker, Metabase) or data warehouses where AI cost data should be combined with other operational metrics, the export capability enables integrating LLM cost data into existing reporting workflows without building a separate pipeline from LLM provider billing APIs.
05On the self-hosted version, all cost data is in the team's own PostgreSQL database. Direct SQL queries against the Langfuse database are possible — teams with SQL-fluent analytics teams can query cost data directly rather than through the Langfuse UI. This is not possible with any cloud-only platform in this comparison: the underlying data is in the vendor's infrastructure, accessible only through the provided dashboard and API.

Langfuse's cost tracking covers what matters for most production teams: per-call cost attribution with dimensional filtering by model, user, session, and custom tag. The custom model pricing configuration is the feature most underappreciated in evaluations but most valuable in production: the moment a team deploys a fine-tuned model or routes some traffic to a self-hosted inference endpoint, the default provider pricing tables stop being accurate. Custom pricing makes cost attribution correct for heterogeneous model deployments.

Apache 2.0 · Maintenance Mode (March 2026)

Helicone

Proxy-Native Cost Tracking

Cost Stack

Cost per request (proxy-captured)

Cost by user (header)

Cache cost savings

Provider-level breakdowns

Rate limit configuration

01Cost tracking is where Helicone's proxy architecture produces the most concrete value. Every LLM request is captured at the network layer with the exact token counts from the provider's response headers. Cost is calculated from provider pricing tables applied to those token counts. Because the capture happens at the proxy, not in the application, cost data is accurate even for requests from application paths that lack instrumentation — the proxy catches everything that goes through it.
02Cache cost savings are directly measurable. Every cache hit is logged as a request with zero token cost and zero latency. The cost dashboard shows the savings from caching: the actual API cost avoided by returning a cached response instead of calling the provider. For teams using Helicone's caching to reduce costs, this direct measurement of savings is the primary ROI metric. A 30% cache hit rate translating to $X in monthly API cost avoidance is visible without any custom reporting.
03Per-user cost attribution via Helicone-User-ID header: every request tagged with a user ID appears in the per-user cost view. Daily, weekly, and monthly spend per user is visible in the dashboard. For consumer applications where user-level cost monitoring is required for sustainability analysis — identifying users whose usage patterns are economically unsustainable at current pricing — this per-user cost view is immediately useful without custom analytics infrastructure.
04Provider-level cost breakdowns: in deployments routing to multiple providers through the same Helicone gateway, cost is broken down by provider and model. OpenAI spend, Anthropic spend, and any other configured provider's spend are separated automatically. For teams running model routing that sends different query types to different providers, this breakdown confirms that routing is producing the intended cost distribution.
05Rate limiting at the proxy layer allows configuring per-user or per-team request limits that enforce cost constraints automatically. A user who exceeds their configured rate limit receives a controlled error response rather than continuing to incur costs. This is a cost control mechanism that operates independently of the application code — changing rate limits in Helicone does not require application redeployment. In maintenance mode, this capability is frozen at its current state but remains functional.

Cost tracking is Helicone's strongest genuine capability — the one area where the proxy architecture provides a concrete advantage over SDK-based alternatives. The proxy captures cost data from all requests regardless of instrumentation completeness, cache savings are directly measurable, and per-user attribution requires only one header. For teams whose primary observability need is "how much are we spending on LLM APIs, by user and provider," Helicone remains a functional and operationally simple answer even in maintenance mode.

Closed Source · Cloud · Bundled with W&B

W&B Weave

Call-Level Cost + Experiment Cost

Cost Stack

Cost per Weave Call

Token usage auto-tracked

Evaluation run cost

W&B usage dashboard

No custom model pricing

01Token usage and cost are captured automatically for all auto-instrumented providers (OpenAI, Anthropic). Every Weave Call from these providers records input tokens, output tokens, and calculated cost using provider pricing tables. For @weave.op()-decorated custom functions that call LLMs, token tracking requires the function to return metadata including token counts — not automatic, but straightforward to implement with a dictionary return.
02Evaluation run costs are tracked separately from production costs. When weave.Evaluation runs scoring against a dataset, the token costs of the LLM-as-judge evaluators are logged as evaluation run costs distinct from the costs of the model being evaluated. This separation is important for teams running large-scale evaluations where evaluation token costs can approach or exceed the costs of the model variants being compared.
03Cost trends appear in the W&B dashboard as time-series charts comparable to other W&B project metrics. For teams already monitoring training run costs in W&B, LLM inference costs appear in the same interface without a separate dashboard context switch. The unified monitoring experience is a practical time-saving benefit for ML teams who are already in W&B throughout the day for training monitoring.
04Custom model pricing configuration is not available in Weave as of mid-2026. Teams using self-hosted models or fine-tuned models deployed on custom inference infrastructure see those calls tracked with zero cost because no pricing table entry exists for the model string. For teams with heterogeneous model deployments mixing provider API models and self-hosted models, cost attribution is incomplete — only the provider API portion is accurately tracked.
05The W&B pricing model (discussed in the next chapter) includes Weave with the core W&B subscription rather than charging separately per trace or per token. This means LLM cost tracking via Weave has zero marginal cost for teams already paying for W&B for ML training. At high trace volumes where LangSmith's per-trace billing and Langfuse's per-unit billing produce significant monthly spend, Weave's bundled pricing makes it cost-free for cost monitoring.

Weave's cost tracking is functional for the common case (OpenAI and Anthropic API calls) and incomplete for heterogeneous deployments (no custom model pricing). For teams already on W&B where the primary LLM providers are the two auto-instrumented providers, Weave provides good cost visibility at no additional cost. For teams running self-hosted models or routing to non-standard endpoints, Langfuse's custom model pricing configuration is necessary to maintain accurate cost attribution.

06

What does each platform cost and how do you deploy it?

The observability tool budget is often invisible until the first invoice. Per-trace billing, per-seat pricing, retention multipliers, and evaluation trace overage interact in ways that produce monthly bills far higher than initial estimates. The figures here are sourced from published pricing pages as of July 2026 — verify directly before committing, as LLM tooling pricing changes frequently.

View

Closed Source · SaaS + Enterprise Self-Host

LangSmith

Per-Seat + Per-Trace Billing

Pricing

Developer: free (5K traces, 1 seat)

Plus: $39/seat/mo

Base trace: $2.50/1K (14-day)

Extended: $5.00/1K (400-day)

Enterprise: custom

01Developer tier is free: 5,000 traces per month, 14-day retention, one seat. In practice, a single LangChain agent with tool calls and retrieval steps can generate 3-8 traces per user query. 5,000 traces supports approximately 600-1,600 user queries per month in a realistic RAG application — fine for prototyping, immediately inadequate for any production workload with real users.
02Plus tier: $39 per seat per month. Includes 10,000 base traces monthly. Additional base traces (14-day retention): $2.50 per 1,000 traces. Extended traces (400-day retention): $5.00 per 1,000 traces. A team of 5 on Plus before any trace overage: $195/month. At 200,000 traces per month (a modest production workload), trace overage at base retention adds approximately $475. Total for a team of 5 at 200K traces: $670/month. Prices verified from published sources, May-June 2026.
03The retention multiplier is the biggest cost surprise. Teams that need 400-day retention for compliance — healthcare teams building clinical AI, financial services teams subject to 7-year record-keeping requirements — pay double the per-trace rate. At 200,000 traces per month with 400-day retention, the overage cost doubles. The retention requirement is often non-negotiable; the cost is a consequence that must be modeled explicitly.
04The pipeline depth multiplier. Pricing documents state a trace is "a single end-to-end execution of your LLM pipeline." In practice, one LangChain chain execution with retrieval, reranking, and generation creates multiple Run objects — each billable. The ratio is application-specific, typically 3-8x for RAG pipelines. A pricing model that assumes one trace per user query underestimates actual costs by this multiple. Always instrument a representative sample before projecting LangSmith costs at scale.
05Self-hosting is Enterprise-only and custom-priced. LangSmith does not publish self-hosting pricing. Enterprise adds SSO/SAML, custom data retention policies, dedicated support, and higher trace volume allocations. The minimum Enterprise commitment is not published; based on market positioning, expect $1,000-5,000/month minimum. For teams with data residency requirements that require self-hosting, Enterprise is the only option — there is no lower-cost self-hosting path.

LangSmith's pricing requires careful modeling before commitment. The per-seat plus per-trace structure with retention multipliers and pipeline depth multipliers compounds in ways that produce month-one production bills 3-10x higher than initial estimates based on user request volume. The published guidance is to instrument first, measure actual trace volume per request, and model from that number — not from user request count. Teams making LangSmith cost decisions from user request projections will be surprised.

MIT License · Cloud + Self-Host (Free)

Langfuse

No Per-Seat · Usage-Based

Pricing

Hobby: free (50K units, 30-day)

Core: $29/mo

Pro: $199/mo

Teams: $499+/mo (SSO add-on)

Self-host: free (MIT, no limits)

01No per-seat pricing at any tier. Hobby, Core, and Pro are usage-based with no user count charge. A team of 20 engineers pays the same as a team of 2 at the same usage tier. This is structurally different from LangSmith's per-seat model and directly impacts team economics: adding team members to a LangSmith organization increases cost linearly; adding them to Langfuse increases cost zero.
02Hobby tier: 50,000 units per month free, 30-day data retention, 2 user seats as the only restriction. At 50,000 units and a typical RAG pipeline generating 6 units per user request, the Hobby tier supports approximately 8,000 user queries per month — 16x the effective volume of LangSmith's free tier. For early-stage products with real but limited traffic, the Hobby tier is production-usable rather than prototype-only.
03Core ($29/month) and Pro ($199/month) scale usage volume and data retention. The jump from Core to Pro is $170/month for substantially more usage — a steep cliff for teams at the boundary. SSO is available in the MIT open-source version for free; in Langfuse Cloud, it is gated behind a $499/month Teams add-on. This creates an incentive for enterprise teams needing SSO to self-host rather than use the cloud tier, where SSO is MIT-licensed and free.
04Self-hosting is MIT-licensed with no restrictions on usage, user count, or features. The entire Langfuse product — tracing, evaluation, LLM-as-judge, prompt management, annotation queues, datasets — runs on self-hosted infrastructure at zero license cost. Infrastructure cost at production scale (1M traces/month): approximately $150/month for a cloud VM and managed PostgreSQL. At 1M traces, LangSmith Cloud costs approximately $2,500+/month, Langfuse Cloud approximately $919/month, Langfuse self-hosted approximately $150/month.
05The ClickHouse acquisition (January 2026) introduces long-term pricing uncertainty. ClickHouse has a commercial model and Langfuse is now part of a commercial entity. The MIT license and current pricing are maintained as of the research date. The concern raised in community discussions: ClickHouse's business model may eventually influence Langfuse pricing or licensing in ways that benefit ClickHouse's infrastructure business. Teams making self-hosting decisions based on the current MIT license should track this situation.

Langfuse's pricing is the most accessible in this comparison at every scale. Free tier that supports real production traffic. No per-seat cost. MIT self-hosting at infrastructure cost only. At 1M traces per month, the choice between Langfuse self-hosted ($150/month infrastructure) and LangSmith Cloud ($2,500+/month) is a $28,000/year decision, and both platforms offer comparable feature sets for non-LangChain workloads. The ClickHouse acquisition is the only genuine pricing risk to monitor.

Apache 2.0 · Maintenance Mode (March 2026)

Helicone

Request-Based, Unlimited Seats

Pricing

Free: 10K req/mo (7-day)

Pro: $79/mo (unlimited seats)

Team: $200/mo

Enterprise: custom

Self-host: free (Apache 2.0)

01Free tier: 10,000 requests per month, 7-day data retention, 1 user seat. Unlike LangSmith's trace multiplier, Helicone counts one LLM API call as one request. A five-step chain consumes 5 requests. The free tier supports approximately 2,000-3,000 user interactions per month for a multi-step application — more practical for real testing than LangSmith's trace count, but still limited for any sustained production traffic.
02Pro tier at $79/month includes unlimited seats and 1 million requests per month. Team tier at $200/month adds increased request limits and priority support. The per-request pricing is simpler than LangSmith's compounded trace-plus-seat model, but Helicone's request volume limits are per-call, so a complex agent application generating many LLM calls per user interaction hits the limit faster than the user request count suggests.
03Self-hosting is available under Apache 2.0 license with no commercial restrictions. For teams with data residency requirements, the self-hosted Helicone proxy can be deployed on internal infrastructure. The proxy's core functionality — request logging, caching, rate limiting, cost tracking — is available self-hosted without a license fee. Feature parity with the cloud version at the time of the Mintlify acquisition is maintained; post-acquisition feature development stopped.
04The maintenance mode status creates a pricing risk that is distinct from feature limitations. Mintlify acquired Helicone for its engineering team and technology assets. The long-term product direction of a company that primarily makes documentation infrastructure (Mintlify's core product) offering an LLM observability gateway is unclear. Teams making multi-year platform commitments to Helicone should factor potential sunset or significant pivot risk into that decision.
05For teams using Helicone specifically for its proxy architecture and request logging (not for evaluation or prompt management), the self-hosted Apache 2.0 codebase provides the most durable path. Self-hosted code does not sunset when a company changes direction. The proxy is open source, the architecture is documented, and teams can maintain it internally without depending on Mintlify's product roadmap. For the proxy use case specifically, self-hosting is the lowest-risk deployment model in 2026.

Helicone's pricing is straightforward and its self-hosting path is the most defensible aspect of the platform post-acquisition. For the specific use case of "zero-code LLM request logging with cost tracking and caching," the self-hosted Apache 2.0 proxy is operational, documented, and maintainable without a vendor relationship. Teams evaluating Helicone for this specific, bounded use case should consider self-hosting rather than the managed service given the maintenance mode uncertainty.

Closed Source · Cloud · Bundled Pricing

W&B Weave

Bundled with W&B Core

Pricing

Free (personal, unlimited)

Team: ~$44/seat/mo

Enterprise: custom

Weave: included at all tiers

No per-trace LLM fee

01Weave is included with the W&B core subscription without a separate per-trace or per-unit fee for LLM observability. At the free personal tier, Weave is available with no cost and no usage limit for individual developers. At the Team tier (approximately $44/seat/month), Weave is included across all seats. There is no separate Weave pricing line item on the invoice — the LLM observability capability has zero marginal cost relative to the W&B subscription.
02For teams already paying for W&B Team or Enterprise for ML experiment tracking, Weave adds zero additional cost. The decision to adopt Weave for LLM observability is a pure capability decision rather than a cost decision. This is a genuinely different financial model from every other platform in this comparison: the cost comparison is W&B (which includes Weave) versus a standalone LLM observability tool, not Weave versus those tools independently.
03At high trace volumes where LangSmith and Langfuse charge per trace or per unit, W&B Weave incurs no additional cost. A production deployment generating 1 million traces per month pays the same W&B subscription regardless of Weave usage. The unlimited trace model makes cost projections for Weave linear with team size rather than with usage volume — a significant budgeting advantage for production teams with variable or unpredictable traffic.
04No self-hosting option is available for W&B or Weave outside of the Enterprise tier, where it is custom-negotiated. For teams with data residency requirements that mandate on-premises or VPC-isolated deployment, W&B offers a hybrid deployment option at Enterprise pricing. For teams that need self-hosting at lower cost, W&B is not the right platform regardless of the included Weave capabilities.
05The practical cost comparison at 1 million monthly traces for a team of 5: LangSmith Plus (14-day retention) approximately $670/month, Langfuse Cloud approximately $919/month, Langfuse self-hosted approximately $150/month, W&B Team (includes Weave) approximately $220/month with no trace overage. For teams where W&B is already in the budget for ML training, Weave is the lowest-cost path to production LLM observability by significant margin.

W&B Weave's bundled pricing is a genuine financial advantage for teams already on the W&B platform. The absence of per-trace billing eliminates the cost forecasting complexity that makes LangSmith and Langfuse difficult to budget for production workloads. The limitation is that this advantage is only accessible to teams already paying for W&B, and for teams without ML training workflows, a W&B subscription to access Weave may be more expensive than standalone Langfuse at the same usage tier. The pricing model rewards existing W&B users specifically.

M

Methodology

What counts as fact, what counts as inference, and what you should verify independently before making platform decisions.

📄

Official Documentation

Vendor docs, pricing pages, release notes, and changelog entries verified at research time, July 2026. Highest confidence tier. Pricing in particular changes frequently — verify before committing.

📰

Published Analyses

Independent reviews and comparisons from Coverge, Metacto, TECHSY, Inference.net, and MarginDash (all published April-July 2026). Used for pricing verification and cost modeling examples.

📢

Acquisition Announcements

Mintlify acquiring Helicone (March 2026) and ClickHouse acquiring Langfuse (January 2026) are documented from official announcements. Maintenance mode status for Helicone confirmed from multiple post-acquisition reviews.

💭

Author Synthesis

Comparative tradeoff assessments and architectural interpretations appear under the insight label. These are reasoned conclusions from primary sources. Disagree with the reasoning, not the source claim.

Key events that shaped this study

Two acquisitions significantly changed the landscape between Vol. 03 and this study. Langfuse was acquired by ClickHouse in January 2026. The product, MIT license, and current pricing are unchanged. The long-term implications for the roadmap are uncertain and worth monitoring. Helicone was acquired by Mintlify in March 2026 and is in maintenance mode. This is factual and documented; no new features will be developed. Teams evaluating Helicone should factor this directly into any multi-year platform decision.

Pricing for all four platforms changed between 2025 and 2026. LangSmith raised per-trace pricing from $0.50/1K to $2.50/1K base. Weaviate (covered in Vol. 03) retired the $25/month Serverless tier. Pricing in LLM tooling is not stable. Verify every number in this study against the current vendor pricing page before making budget commitments.

What is excluded

Arize Phoenix, Braintrust, Laminar, Comet Opik, and Datadog LLM Observability were excluded from this comparison. All are legitimate options referenced in the broader LLM observability landscape. Arize Phoenix is the strongest open-source alternative to Langfuse. Braintrust has the most generous free tier (1M spans/month) and the best CI/CD eval-gated deployment workflow. These may be covered in future volumes. Performance benchmarks for trace ingestion throughput and dashboard query speed were not included — these metrics are platform-version dependent and not independently verifiable at the time of research.

Research timeline

Researched July 2026. Primary sources: official documentation for LangSmith, Langfuse, Helicone, and W&B Weave. Secondary sources: Coverge pricing analyses (April 2026), TECHSY Langfuse vs LangSmith comparison (June 2026), Latitude AI agent observability tool comparison (July 2026), Metacto LangSmith guide (June 2026), QASkills Langfuse guide (June 2026), W&B Weave LLM evals guide (May 2026), and ChatForest Helicone review (May 2026). All sources published after January 2026 to reflect post-acquisition states.

Last Updated: July 21, 2026

Swarnim
Tiwari

AI Systems Researcher

Observability is the part of the stack that tells you what your system actually does versus what you designed it to do. In most production AI applications that gap is larger than the team expects.

I almost skipped this topic because it felt more like developer tooling than AI systems research. Then Helicone got frozen with 14.2 trillion tokens processed, three months after its acquisition, and I realized the consolidation story in the tooling layer was as interesting as anything in the technology itself.

Two significant events happened during the research window for this volume. I documented both as factually as I could. The interpretations are mine.

I am a student in India. If you are evaluating these platforms right now, verify the pricing directly before committing. It changed significantly between 2025 and 2026.

LinkedIn GitHub

AI Systems Studies — Publication Series

Vol. 01Production AI Architecture — OpenAI, Anthropic, Palantir, NVIDIAPublished

Vol. 02AI Agent Frameworks — OpenAI SDK, LangGraph, CrewAI, MastraPublished

Vol. 03Vector Databases — Pinecone, Weaviate, Milvus, QdrantPublished

Vol. 04AI Observability — LangSmith, Langfuse, Helicone, W&B WeaveThis Study

Vol. 05Inference Infrastructure — vLLM, SGLang, TensorRT-LLM, TGIPlanned

Vol. 06Context EngineeringPlanned

Vol. 07Memory SystemsPlanned

Vol. 08RAG ArchitecturesPlanned