Health Metrics

The HealthMetrics class tracks real-time crawler performance indicators using a fixed-size ring buffer. It provides rolling-window statistics for requests per second, success rate, average and p95 response times, and can export metrics in Prometheus format for monitoring integration.

import { HealthMetrics } from 'rezo/crawler';

Creating HealthMetrics

const metrics = new HealthMetrics({
  windowSize: 60000,     // Rolling window for averages in ms (default: 60 seconds)
  sampleInterval: 1000   // Sample interval for throughput calculation (default: 1 second)
});

Recording Requests

Record each completed request with its response time and success status. This is an O(1) operation with no memory allocations:

// Record a successful 150ms request
metrics.recordRequest(150, true);

// Record a failed 500ms request
metrics.recordRequest(500, false);

// Record a successful 45ms request
metrics.recordRequest(45, true);

The ring buffer holds 10,000 samples. When full, the oldest sample is overwritten. This means the rolling window always has the most recent data without unbounded memory growth.

Health Snapshots

getSnapshot()

Get a complete health snapshot with all metrics:

const health = metrics.getSnapshot(
  queueDepth,      // Current number of URLs in the queue
  activeRequests    // Current number of in-flight requests
);

The HealthSnapshot interface:

interface HealthSnapshot {
  timestamp: number;          // Snapshot timestamp
  requestsPerSecond: number;  // RPS (rolling average over window)
  successRate: number;        // Success percentage (0-100)
  failureRate: number;        // Failure percentage (0-100)
  avgResponseTime: number;    // Average response time in ms
  p95ResponseTime: number;    // 95th percentile response time in ms
  queueDepth: number;         // Current queue depth
  activeRequests: number;     // Active concurrent requests
  totalRequests: number;      // All-time total requests
  totalSuccesses: number;     // All-time successful requests
  totalFailures: number;      // All-time failed requests
  uptimeMs: number;           // Time since metrics creation
}

Example:

const health = metrics.getSnapshot(150, 10);

console.log(`RPS: ${health.requestsPerSecond}`);        // e.g., 12.5
console.log(`Success: ${health.successRate}%`);          // e.g., 97.5
console.log(`Avg latency: ${health.avgResponseTime}ms`); // e.g., 245
console.log(`P95 latency: ${health.p95ResponseTime}ms`); // e.g., 890
console.log(`Queue: ${health.queueDepth}`);              // e.g., 150
console.log(`Active: ${health.activeRequests}`);          // e.g., 10
console.log(`Uptime: ${Math.round(health.uptimeMs / 1000)}s`);

getTotals()

Get all-time statistics (not windowed):

const totals = metrics.getTotals();
// { requests: 15000, successes: 14700, failures: 300, avgResponseTime: 230 }

Health Checks

isHealthy()

Evaluate crawler health against configurable thresholds:

const healthy = metrics.isHealthy({
  minSuccessRate: 80,          // Minimum acceptable success rate % (default: 80)
  maxAvgResponseTime: 10000,   // Max average response time ms (default: 10000)
  maxP95ResponseTime: 30000    // Max p95 response time ms (default: 30000)
});

if (!healthy) {
  console.warn('Crawler health degraded');
  // Reduce concurrency, switch proxies, pause, etc.
}

The check requires at least 10 total requests before it can determine health (returns true if fewer than 10 requests have been recorded).

Prometheus Export

toPrometheusFormat()

Export all metrics in Prometheus exposition format for scraping by a Prometheus server:

const output = metrics.toPrometheusFormat('rezo_crawler');
console.log(output);

Output:

# HELP rezo_crawler_requests_total Total requests processed
# TYPE rezo_crawler_requests_total counter
rezo_crawler_requests_total 15000

# HELP rezo_crawler_requests_success_total Total successful requests
# TYPE rezo_crawler_requests_success_total counter
rezo_crawler_requests_success_total 14700

# HELP rezo_crawler_requests_failed_total Total failed requests
# TYPE rezo_crawler_requests_failed_total counter
rezo_crawler_requests_failed_total 300

# HELP rezo_crawler_requests_per_second Current requests per second
# TYPE rezo_crawler_requests_per_second gauge
rezo_crawler_requests_per_second 12.5

# HELP rezo_crawler_success_rate_percent Success rate percentage
# TYPE rezo_crawler_success_rate_percent gauge
rezo_crawler_success_rate_percent 98.0

# HELP rezo_crawler_response_time_avg_ms Average response time
# TYPE rezo_crawler_response_time_avg_ms gauge
rezo_crawler_response_time_avg_ms 230

# HELP rezo_crawler_response_time_p95_ms P95 response time
# TYPE rezo_crawler_response_time_p95_ms gauge
rezo_crawler_response_time_p95_ms 890

# HELP rezo_crawler_uptime_seconds Crawler uptime
# TYPE rezo_crawler_uptime_seconds gauge
rezo_crawler_uptime_seconds 3600

The prefix parameter (default: 'crawler') is prepended to all metric names.

toJSON()

Export as a JSON string for logging or custom monitoring:

const json = metrics.toJSON(queueDepth, activeRequests);
// Full HealthSnapshot serialized as JSON string

Ring Buffer Implementation

The ring buffer provides O(1) recordRequest() with zero allocations during normal operation:

• Fixed capacity of 10,000 samples
• Circular writes — new samples overwrite the oldest when full
• Window queries scan the buffer once to collect samples within the rolling window
• No garbage collection pressure — no objects are created or destroyed per sample

This design ensures that metrics tracking adds negligible overhead even at high request rates.

Resetting Metrics

metrics.reset();
// Clears all samples and counters, resets to initial state

Monitoring Integration Example

import { Crawler, HealthMetrics } from 'rezo/crawler';
import http from 'node:http';

const metrics = new HealthMetrics({ windowSize: 60000 });

const crawler = new Crawler({
  baseUrl: 'https://example.com',
  concurrency: 20
});

// Record metrics on each request
crawler.onDocument(async function (doc, res) {
  metrics.recordRequest(res.responseTime, res.status < 400);
});

// Expose Prometheus metrics endpoint
http.createServer((req, res) => {
  if (req.url === '/metrics') {
    res.setHeader('Content-Type', 'text/plain');
    res.end(metrics.toPrometheusFormat('crawler'));
  }
}).listen(9090);

// Periodic health check logging
setInterval(() => {
  const snapshot = metrics.getSnapshot(0, 0);
  console.log(`[health] RPS=${snapshot.requestsPerSecond} success=${snapshot.successRate}% p95=${snapshot.p95ResponseTime}ms`);
}, 10000);

await crawler.visit('https://example.com');
await crawler.done();