Logging and Monitoring

Challenge

Build a robust logging and monitoring system that tracks application events, captures errors, monitors performance metrics, and provides actionable insights for debugging and optimization.

Explanation

Logging vs Monitoring

Logging is the act of recording events and data about your application’s execution:

• Tracks what happened and when
• Provides historical records
• Helps with debugging and auditing
• Can be stored in files, databases, or external services

Monitoring is the active observation of system behavior:

• Tracks metrics and health indicators
• Alerts on anomalies or threshold breaches
• Provides real-time visibility
• Enables proactive problem detection

Logging Frameworks

Popular logging libraries for Node.js:

• Winston: Flexible, transport-based logging
• Bunyan: JSON-based structured logging
• Pino: High-performance JSON logging
• Morgan: HTTP request logging middleware
• debug: Lightweight debugging utility

Monitoring Tools

• Prometheus: Metrics collection and alerting
• Grafana: Visualization dashboards
• DataDog: Comprehensive monitoring platform
• New Relic: APM and monitoring
• CloudWatch: AWS monitoring service
• Elasticsearch: Log aggregation and analysis

Code Examples

1. Winston Logger Setup

const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp({ format: 'YYYY-MM-DD HH:mm:ss' }),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: { service: 'user-service' },
  transports: [
    new winston.transports.File({
      filename: 'error.log',
      level: 'error'
    }),
    new winston.transports.File({ filename: 'combined.log' }),
  ],
});

if (process.env.NODE_ENV !== 'production') {
  logger.add(new winston.transports.Console({
    format: winston.format.simple(),
  }));
}

logger.info('User logged in', { userId: 123 });
logger.error('Database connection failed', { error: err.message });

module.exports = logger;

2. Pino Logger with Serializers

const pino = require('pino');

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  transport: {
    target: 'pino-pretty',
    options: {
      colorize: true,
      translateTime: 'SYS:standard'
    }
  },
  serializers: {
    req: pino.stdSerializers.req,
    res: pino.stdSerializers.res,
    err: (err) => ({
      type: err.type,
      message: err.message,
      stack: err.stack
    })
  }
});

const childLogger = logger.child({ userId: 'user-123' });
childLogger.info('Processing payment');

module.exports = logger;

3. Morgan HTTP Request Logging

const express = require('express');
const morgan = require('morgan');

const app = express();

// Development logging
app.use(morgan('dev'));

// Production logging with custom format
morgan.token('user-id', (req) => req.user?.id || 'anonymous');
app.use(morgan(':method :url :status - :response-time ms - :user-id'));

app.get('/api/users', (req, res) => {
  res.json({ users: [] });
});

app.listen(3000);

4. Structured Logging Middleware

const winston = require('winston');

class ContextualLogger {
  constructor(baseLogger) {
    this.logger = baseLogger;
    this.context = {};
  }

  setContext(context) {
    this.context = { ...this.context, ...context };
  }

  info(message, meta = {}) {
    this.logger.info(message, { ...this.context, ...meta });
  }

  error(message, error, meta = {}) {
    this.logger.error(message, {
      ...this.context,
      error: error.message,
      stack: error.stack,
      ...meta
    });
  }
}

function loggerMiddleware(req, res, next) {
  req.logger = new ContextualLogger(winston);
  req.logger.setContext({
    requestId: req.id || Date.now().toString(),
    method: req.method,
    url: req.path,
  });
  next();
}

module.exports = { ContextualLogger, loggerMiddleware };

5. Prometheus Metrics

const prometheus = require('prom-client');
const express = require('express');

const httpRequestDuration = new prometheus.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5],
});

prometheus.collectDefaultMetrics();

const app = express();

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration
      .labels(req.method, req.path, res.statusCode)
      .observe(duration);
  });
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', prometheus.register.contentType);
  res.end(await prometheus.register.metrics());
});

app.listen(3000);

Best Practices

1. Use Structured Logging

   • Log in JSON format for easier parsing
   • Include contextual information (request ID, user ID)
   • Avoid logging sensitive data

2. Implement Proper Log Levels

   • ERROR: System errors requiring attention
   • WARN: Potentially problematic situations
   • INFO: General application flow
   • DEBUG: Detailed diagnostic information

3. Centralize Logging

   • Use log aggregation services (ELK, Splunk)
   • Enable searching and filtering
   • Maintain historical logs for compliance

4. Monitor Key Metrics

   • Response times and latency
   • Error rates
   • Resource usage (CPU, memory)
   • Database query performance

5. Set Up Alerts

   • Alert on error rate thresholds
   • Monitor application uptime
   • Track resource exhaustion
   • Notify on performance degradation

Real-World Applications

• E-Commerce Platform: Track transactions, orders, payment failures
• SaaS Application: Monitor API usage, user events, resource consumption
• IoT Data Pipeline: Log sensor data, data quality, processing metrics
• Microservices: Correlate logs using request tracing
• Analytics Engine: Monitor ingestion rates, query latency