310 lines
13 KiB
Markdown
310 lines
13 KiB
Markdown
# LLM Observability MCP Server
|
|
|
|
[](https://opensource.org/licenses/MIT)
|
|
|
|
A Model Context Protocol (MCP) server that provides comprehensive LLM observability tools supporting both PostHog and OpenTelemetry backends.
|
|
|
|
## Overview
|
|
|
|
This project is an MCP server designed to track and observe Large Language Model (LLM) interactions using both [PostHog's LLM Observability](https://posthog.com/docs/llm-observability) and **OpenTelemetry** for universal observability across any backend that supports OpenTelemetry (Jaeger, New Relic, Grafana, Datadog, Honeycomb, etc.).
|
|
|
|
The server can be run as a local process communicating over `stdio` or as a remote `http` server, making it compatible with any MCP client, such as AI-powered IDEs (e.g., VS Code with an MCP extension, Cursor) or custom applications.
|
|
|
|
## Features
|
|
|
|
- **Dual Backend Support**: Choose between PostHog or OpenTelemetry (or use both)
|
|
- **Universal OpenTelemetry**: Works with any OpenTelemetry-compatible backend
|
|
- **Comprehensive Metrics**: Request counts, token usage, latency, error rates
|
|
- **Distributed Tracing**: Full request lifecycle tracking with spans
|
|
- **Flexible Transport**: Run as local `stdio` process or standalone `http` server
|
|
- **Dynamic Configuration**: Environment-based configuration for different backends
|
|
- **Zero-Code Integration**: Drop-in replacement for existing observability tools
|
|
|
|
## Installation for Development
|
|
|
|
Follow these steps to set up the server for local development.
|
|
|
|
1. **Prerequisites**:
|
|
- Node.js (>=18.x)
|
|
- A [PostHog account](https://posthog.com/) with an API Key and Host URL.
|
|
|
|
2. **Clone and Install**:
|
|
|
|
```bash
|
|
git clone https://github.com/sfiorini/llm-observability-mcp.git
|
|
cd llm-observability-mcp
|
|
npm install
|
|
```
|
|
|
|
3. **Configuration**:
|
|
Create a `.env` file in the root of the project by copying the example file:
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
```
|
|
|
|
Then, edit the `.env` file with your PostHog credentials and desired transport mode.
|
|
|
|
## Configuration
|
|
|
|
The server is configured via environment variables.
|
|
|
|
### PostHog Configuration
|
|
|
|
| Variable | Description | Default | Example |
|
|
| ----------------- | --------------------------------------------------------------------------- | --------- | ------------------------------------- |
|
|
| `POSTHOG_API_KEY` | Your PostHog Project API Key (required for PostHog tool) | - | `phc_...` |
|
|
| `POSTHOG_HOST` | The URL of your PostHog instance | - | `https://us.i.posthog.com` |
|
|
|
|
### OpenTelemetry Configuration
|
|
|
|
| Variable | Description | Default | Example |
|
|
| ------------------------------- | --------------------------------------------------------------------------- | -------------------------- | ------------------------------------- |
|
|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OpenTelemetry collector endpoint | - | `http://localhost:4318` |
|
|
| `OTEL_EXPORTER_OTLP_HEADERS` | Headers for authentication (comma-separated key=value pairs) | - | `api-key=YOUR_KEY` |
|
|
| `OTEL_SERVICE_NAME` | Service name for traces and metrics | `llm-observability-mcp` | `my-llm-app` |
|
|
| `OTEL_SERVICE_VERSION` | Service version | `1.0.0` | `2.1.0` |
|
|
| `OTEL_ENVIRONMENT` | Environment name | `development` | `production` |
|
|
| `OTEL_TRACES_SAMPLER_ARG` | Sampling ratio (0.0-1.0) | `1.0` | `0.1` |
|
|
| `OTEL_METRIC_EXPORT_INTERVAL` | Metrics export interval in milliseconds | `10000` | `30000` |
|
|
|
|
### General Configuration
|
|
|
|
| Variable | Description | Default | Example |
|
|
| ----------------- | --------------------------------------------------------------------------- | --------- | ------------------------------------- |
|
|
| `TRANSPORT_MODE` | The transport protocol to use. Can be `http` or `stdio`. | `http` | `stdio` |
|
|
| `DEBUG` | Set to `true` to enable detailed debug logging. | `false` | `true` |
|
|
|
|
## Running the Server
|
|
|
|
You can run the server in two modes:
|
|
|
|
- **HTTP Mode**: Runs a web server, typically for remote clients or IDEs like Cursor.
|
|
|
|
```bash
|
|
npm run mcp:http
|
|
```
|
|
|
|
The server will start on `http://localhost:3000`.
|
|
|
|
- **STDIO Mode**: Runs as a command-line process, ideal for local IDE integration where the IDE manages the process lifecycle.
|
|
|
|
```bash
|
|
npm run mcp:stdio
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Connecting to an IDE (VS Code, Cursor, etc.)
|
|
|
|
You can integrate this tool with any MCP-compatible IDE. Add one of the following configurations to your IDE's MCP settings (e.g., in `.vscode/settings.json` for VS Code or `.kilocode/mcp.json` for a global setup).
|
|
|
|
#### Option 1: Local Stdio Process (Recommended)
|
|
|
|
This method lets the IDE manage the server as a local background process. It's efficient and doesn't require a separate terminal.
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"llm-observability-mcp-stdio": {
|
|
"command": "node",
|
|
"args": [
|
|
"/path/to/your/projects/llm-log-mcp-server/dist/index.js"
|
|
],
|
|
"env": {
|
|
"TRANSPORT_MODE": "stdio",
|
|
"POSTHOG_API_KEY": "phc_...",
|
|
"POSTHOG_HOST": "https://us.i.posthog.com"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Note**: Replace `/path/to/your/projects/llm-log-mcp-server` with the absolute path to this project directory.
|
|
|
|
#### Option 2: Remote HTTP Server
|
|
|
|
Use this if you prefer to run the server as a standalone process.
|
|
|
|
1. Run the server in a terminal: `npm run mcp:http`
|
|
2. Add the server URL to your IDE's configuration:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"llm-observability-mcp-sse": {
|
|
"url": "http://localhost:3000/sse"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Automatic Triggering via System Prompt
|
|
|
|
For IDE extensions that support system prompts, you can instruct the AI to automatically use this MCP tool for every interaction. Add the following to your IDE's system prompt configuration:
|
|
|
|
```text
|
|
Use `capture_llm_observability` MCP.
|
|
Make sure to include all parameters and for the `userId`, send `<my_username>`:
|
|
userId - The distinct ID of the user
|
|
traceId - The trace ID to group AI events
|
|
model - The model used (e.g., gpt-4, claude-3, etc.)
|
|
provider - The LLM provider (e.g., openai, anthropic, etc.)
|
|
input - The input to the LLM (messages, prompt, etc.)
|
|
outputChoices - The output from the LLM
|
|
inputTokens - The number of tokens in the input
|
|
outputTokens - The number of tokens in the output
|
|
latency - The latency of the LLM call in seconds
|
|
httpStatus - The HTTP status code of the LLM call
|
|
baseUrl - The base URL of the LLM API
|
|
```
|
|
|
|
Replace `<my_username>` with a unique identifier for yourself. This ensures that all LLM activity is automatically logged in PostHog without needing to give the command each time.
|
|
|
|
### Programmatic Usage
|
|
|
|
You can use an MCP client library to interact with the server programmatically from your own applications.
|
|
|
|
```typescript
|
|
import { McpClient } from '@modelcontextprotocol/sdk/client';
|
|
|
|
async function main() {
|
|
// Assumes the MCP server is running in HTTP mode
|
|
const client = new McpClient({
|
|
transport: {
|
|
type: 'http',
|
|
url: 'http://localhost:3000/mcp',
|
|
},
|
|
});
|
|
|
|
await client.connect();
|
|
|
|
const result = await client.useTool('capture_llm_observability', {
|
|
userId: 'user-123',
|
|
model: 'gpt-4',
|
|
provider: 'openai',
|
|
input: 'What is the capital of France?',
|
|
outputChoices: [{ text: 'Paris.' }],
|
|
inputTokens: 8,
|
|
outputTokens: 2,
|
|
latency: 0.5,
|
|
});
|
|
|
|
console.log('Tool result:', result);
|
|
|
|
await client.disconnect();
|
|
}
|
|
|
|
main().catch(console.error);
|
|
```
|
|
|
|
## Available Tools
|
|
|
|
### PostHog Tool: `capture_llm_observability`
|
|
|
|
Captures LLM usage in PostHog for observability, including requests, responses, and performance metrics.
|
|
|
|
### OpenTelemetry Tool: `capture_llm_observability_opentelemetry`
|
|
|
|
Captures LLM usage using OpenTelemetry for universal observability across any OpenTelemetry-compatible backend.
|
|
|
|
### Parameters Comparison
|
|
|
|
| Parameter | Type | Required | Description | PostHog | OpenTelemetry |
|
|
| --------------- | ------------------- | -------- | ----------------------------------------------- | ------- | ------------- |
|
|
| `userId` | `string` | Yes | The distinct ID of the user. | ✅ | ✅ |
|
|
| `model` | `string` | Yes | The model used (e.g., `gpt-4`, `claude-3`). | ✅ | ✅ |
|
|
| `provider` | `string` | Yes | The LLM provider (e.g., `openai`, `anthropic`). | ✅ | ✅ |
|
|
| `traceId` | `string` | No | The trace ID to group related AI events. | ✅ | ✅ |
|
|
| `input` | `any` | No | The input to the LLM (e.g., messages, prompt). | ✅ | ✅ |
|
|
| `outputChoices` | `any` | No | The output choices from the LLM. | ✅ | ✅ |
|
|
| `inputTokens` | `number` | No | The number of tokens in the input. | ✅ | ✅ |
|
|
| `outputTokens` | `number` | No | The number of tokens in the output. | ✅ | ✅ |
|
|
| `latency` | `number` | No | The latency of the LLM call in seconds. | ✅ | ✅ |
|
|
| `httpStatus` | `number` | No | The HTTP status code of the LLM API call. | ✅ | ✅ |
|
|
| `baseUrl` | `string` | No | The base URL of the LLM API. | ✅ | ✅ |
|
|
| `operationName` | `string` | No | The name of the operation being performed. | ❌ | ✅ |
|
|
| `error` | `string` | No | Error message if the request failed. | ❌ | ✅ |
|
|
| `errorType` | `string` | No | Type of error (e.g., rate_limit, timeout). | ❌ | ✅ |
|
|
| `mcpToolsUsed` | `string[]` | No | List of MCP tools used during the request. | ❌ | ✅ |
|
|
|
|
## Quick Start with OpenTelemetry
|
|
|
|
### 1. Choose Your Backend
|
|
|
|
**For local testing with Jaeger:**
|
|
|
|
```bash
|
|
# Start Jaeger with OTLP support
|
|
docker run -d --name jaeger \
|
|
-e COLLECTOR_OTLP_ENABLED=true \
|
|
-p 16686:16686 \
|
|
-p 4318:4318 \
|
|
jaegertracing/all-in-one:latest
|
|
```
|
|
|
|
**For New Relic:**
|
|
|
|
```bash
|
|
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net:4318
|
|
export OTEL_EXPORTER_OTLP_HEADERS="api-key=YOUR_LICENSE_KEY"
|
|
```
|
|
|
|
### 2. Configure Environment
|
|
|
|
```bash
|
|
# Copy example configuration
|
|
cp .env.example .env
|
|
|
|
# Edit .env with your backend settings
|
|
# For Jaeger:
|
|
echo "OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318" >> .env
|
|
echo "OTEL_SERVICE_NAME=llm-observability-mcp" >> .env
|
|
```
|
|
|
|
### 3. Start the Server
|
|
|
|
```bash
|
|
npm run mcp:http
|
|
# or
|
|
npm run mcp:stdio
|
|
```
|
|
|
|
### 4. Test the Integration
|
|
|
|
```bash
|
|
# Test with curl
|
|
curl -X POST http://localhost:3000/mcp \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"tool": "capture_llm_observability_opentelemetry",
|
|
"arguments": {
|
|
"userId": "test-user",
|
|
"model": "gpt-4",
|
|
"provider": "openai",
|
|
"inputTokens": 100,
|
|
"outputTokens": 50,
|
|
"latency": 1.5,
|
|
"httpStatus": 200,
|
|
"operationName": "test-completion"
|
|
}
|
|
}'
|
|
```
|
|
|
|
## Development
|
|
|
|
- **Run in dev mode (HTTP)**: `npm run dev:http`
|
|
- **Run tests**: `npm test`
|
|
- **Lint and format**: `npm run lint` and `npm run format`
|
|
|
|
## Documentation
|
|
|
|
- [OpenTelemetry Setup Guide](OPENTELEMETRY.md) - Complete OpenTelemetry configuration
|
|
- [Usage Examples](examples/opentelemetry-usage.md) - Practical examples for different backends
|
|
- [Environment Configuration](.env.example) - All available configuration options
|
|
|
|
## License
|
|
|
|
[MIT License](https://opensource.org/licenses/MIT)
|