API Logs: Debugging Black Boxes

The most surprising thing about monitoring and logging API traffic is that it’s often treated as a reactive "firefighting" tool when it’s actually a proactive "architectural blueprint" for understanding your system’s behavior.

Let’s see this in action. Imagine we have a simple API service built with Node.js and Express, and we’re using Winston for logging.

const express = require('express');
const winston = require('winston');

const app = express();
const port = 3000;

// Configure Winston logger
const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({ filename: 'api.log' })
  ],
});

// Middleware to log incoming requests
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const duration = Date.now() - start;
    logger.info('API Request', {
      method: req.method,
      url: req.originalUrl,
      statusCode: res.statusCode,
      durationMs: duration,
      headers: req.headers,
      body: req.body // Note: req.body might be undefined if not parsed
    });
  });
  next();
});

// Example API endpoint
app.get('/users/:id', (req, res) => {
  const userId = req.params.id;
  // Simulate fetching user data
  setTimeout(() => {
    if (userId === '123') {
      res.json({ id: userId, name: 'Alice' });
    } else {
      res.status(404).json({ error: 'User not found' });
    }
  }, 50);
});

app.post('/users', express.json(), (req, res) => { // Use express.json() to parse body
  const newUser = req.body;
  logger.info('New User Created', { userData: newUser });
  res.status(201).json(newUser);
});

app.listen(port, () => {
  logger.info(`API server listening at http://localhost:${port}`);
});

When a request comes in, say GET /users/123, the middleware intercepts it. It records the start time, and crucially, attaches a listener to the res object’s finish event. This event fires after the response has been sent. At that point, it calculates the duration, retrieves details like the HTTP method, URL, status code, and even the request headers and body, and logs them.

The logger.info('API Request', { ... }) line is where the magic happens. Winston formats this into a structured JSON log. This isn’t just a string; it’s a machine-readable record.

Here’s a sample log entry for a successful /users/123 request:

{
  "timestamp": "2023-10-27T10:30:00.123Z",
  "level": "info",
  "message": "API Request",
  "method": "GET",
  "url": "/users/123",
  "statusCode": 200,
  "durationMs": 52,
  "headers": {
    "host": "localhost:3000",
    "user-agent": "curl/7.79.1",
    "accept": "*/*"
  },
  "body": {} // For GET requests, body is typically empty
}

And for a POST /users request:

{
  "timestamp": "2023-10-27T10:31:05.456Z",
  "level": "info",
  "message": "API Request",
  "method": "POST",
  "url": "/users",
  "statusCode": 201,
  "durationMs": 2,
  "headers": {
    "host": "localhost:3000",
    "user-agent": "curl/7.79.1",
    "accept": "*/*",
    "content-type": "application/json",
    "content-length": "35"
  },
  "body": {
    "name": "Bob",
    "email": "bob@example.com"
  }
}

This structured data is the key. It allows us to build a complete picture of API interactions. For debugging, you can filter logs by statusCode to find errors (e.g., statusCode: 500), by durationMs to identify performance bottlenecks (e.g., durationMs > 500), or by url and method to pinpoint specific failing endpoints. For compliance, you can track who accessed what, when, and from where, by including headers['user-agent'] or headers['x-forwarded-for'] in your logs.

The real power comes when you centralize these logs. Tools like Elasticsearch, Splunk, or cloud-native solutions (CloudWatch Logs, Azure Monitor, Google Cloud Logging) ingest these JSON logs. You can then query them at scale. For instance, to find all requests to /users that took longer than 100ms and returned a 404:

In Elasticsearch/Kibana: http.request.method : "GET" AND http.request.url : "/users/*" AND http.response.status_code : 404 AND http.request.duration_ms > 100

In Splunk: index=your_api_index "API Request" method=GET url=/users/* status_code=404 durationMs > 100

This ability to slice and dice traffic data transforms logs from a historical record into an interactive map of your API’s landscape.

A detail often overlooked is the importance of logging the request body for POST, PUT, and PATCH requests. While it adds verbosity, it’s invaluable for debugging data-related issues. Make sure your logging middleware correctly parses the request body using something like express.json() or express.urlencoded() before your logging middleware runs, or that the body is accessible within the middleware itself. Without it, you’re debugging blind on data submission errors.

The next logical step is to integrate distributed tracing, linking requests across multiple microservices.