Part 2: Types of Logs and Logging Frameworks

Welcome back to our series on logging! In the first part, we introduced the fundamental concepts of logging, discussing its importance in software development and the basics of setting up a logging system. In this part, we will delve deeper into the different types of logs, their importance, and various logging frameworks used in software development.

Types of Logs

Application Logs: These logs capture events that occur within your application. They provide insight into the application’s behaviour, such as user actions, system processes, and error messages. Application logs are essential for debugging and tracking the flow of operations.

2024-07-07 14:35:22,987 [INFO] User 'john_doe' logged in successfully.
2024-07-07 14:36:10,123 [ERROR] Unable to connect to the database: Connection timeout.

System Logs: System logs record events from the operating system and other system-level services. They include information about hardware events, operating system errors, and system resource usage. System logs are vital for diagnosing issues at the OS level and understanding system performance.

2024-07-07 14:35:22 localhost kernel: [12345.678901] CPU0: Temperature above threshold, cpu clock throttled.
2024-07-07 14:36:10 localhost systemd[1]: Started Apache HTTP Server.

Security Logs: Security logs focus on events related to security, such as login attempts, access control changes, and potential security breaches. These logs are critical for identifying and responding to security threats.

2024-07-07 14:35:22,987 [SECURITY] Failed login attempt for user 'admin' from IP address 192.168.1.10.
2024-07-07 14:36:10,123 [SECURITY] User 'jane_doe' changed password successfully.

Audit Logs: Error logs capture information about errors and exceptions that occur within a system. These logs are invaluable for developers when troubleshooting and fixing bugs.

2024-07-07 14:35:22,987 [AUDIT] User 'john_doe' created a new record in the 'customers' table.
2024-07-07 14:36:10,123 [AUDIT] User 'jane_doe' deleted record with ID 1234 from 'orders' table.

Transaction Logs: Transaction logs capture the details of transactions within an application, such as database operations, payment processing, and other critical business operations. These logs ensure data integrity and are useful for troubleshooting transaction-related issues.

2024-07-07 14:35:22,987 [TRANSACTION] Order ID 5678 created for user 'john_doe' with total amount $250.00.
2024-07-07 14:36:10,123 [TRANSACTION] Payment ID 7890 processed successfully for Order ID 5678.

Request Logs: Request logs record information about HTTP requests made to your web application. They typically include data such as the request URL, HTTP method, status code, response time, and client IP address. Request logs are vital for monitoring web traffic, diagnosing issues with specific endpoints, and analyzing the performance of your web services.

2024-07-07 14:35:22,987 [REQUEST] GET /api/v1/users/1234 200 OK (123 ms) - IP: 192.168.1.10
2024-07-07 14:36:10,123 [REQUEST] POST /api/v1/orders 201 Created (456 ms) - IP: 192.168.1.11

Logging Frameworks

Logging frameworks simplify the process of generating and managing logs. They provide standardized methods to log messages and offer features like log formatting, filtering, and output destinations. Below, we’ll provide configuration examples for different logging frameworks, including settings for rolling, retention, and archival policies.

Log4j (Java)

<!-- logback.xml -->
<configuration>
  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>logs/app.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>logs/app-%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>
      <totalSizeCap>1GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>

  <logger name="com.example.application" level="INFO" additivity="false">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>
  </logger>

  <root level="ERROR">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>
  </root>
</configuration>

NLog (C#/.NET)

<!-- NLog.config -->
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <targets>
    <target xsi:type="File" name="file" fileName="logs/app.log" 
            layout="${longdate} [${level}] ${message} ${exception}" 
            archiveFileName="logs/archive/app.{#}.log" 
            archiveEvery="Day" 
            archiveNumbering="DateAndSequence" 
            maxArchiveFiles="30" 
            archiveAboveSize="10MB" 
            concurrentWrites="true" 
            keepFileOpen="false" 
            encoding="utf-8" />
    <target xsi:type="Console" name="console" 
            layout="${longdate} [${level}] ${message} ${exception}" />
  </targets>
  <rules>
    <logger name="*" minlevel="Info" writeTo="file,console" />
    <logger name="*" minlevel="Error" writeTo="file,console" />
  </rules>
</nlog>

Serilog (C#/.NET)

// Program.cs
using Serilog;

class Program
{
    static void Main(string[] args)
    {
        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .WriteTo.Console(outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level}] {Message}{NewLine}{Exception}")
            .WriteTo.File("logs/app.log", rollingInterval: RollingInterval.Day, 
                          retainedFileCountLimit: 30, 
                          fileSizeLimitBytes: 10 * 1024 * 1024, // 10 MB
                          rollOnFileSizeLimit: true,
                          outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level}] {Message}{NewLine}{Exception}")
            .CreateLogger();

        Log.Information("Application started.");
        Log.Error("An error occurred.");

        Log.CloseAndFlush();
    }
}

Loguru (Python)

# main.py
from loguru import logger

logger.add(
    "logs/app.log", 
    rotation="1 day", 
    retention="30 days", 
    compression="zip", 
    format="{time:YYYY-MM-DD HH:mm:ss.SSS} [{level}] {message}"
)

logger.info("User 'john_doe' logged in successfully.")
logger.error("Unable to connect to the database: Connection timeout.")

Bunyan (JavaScript/Node.js)

// app.js
const bunyan = require('bunyan');
const RotatingFileStream = require('bunyan-rotating-file-stream');

const logger = bunyan.createLogger({
  name: 'myapp',
  streams: [
    {
      level: 'info',
      stream: process.stdout
    },
    {
      type: 'rotating-file',
      level: 'info',
      path: 'logs/app.log',
      period: '1d',          // daily rotation
      count: 30,             // keep 30 back copies
      rotateExisting: true,  // Rotate log files on start-up if they exceed count
      threshold: '10m',      // Rotate log files if they exceed 10MB
      totalSize: '100m',     // Ensure the total size of log files does not exceed 100MB
      gzip: true             // Compress rotated files
    }
  ],
  serializers: bunyan.stdSerializers
});

logger.info("User 'john_doe' logged in successfully.");
logger.error("Unable to connect to the database: Connection timeout.");

Choosing the Right Framework

Selecting the right logging framework depends on several factors, including the programming language, application requirements, and the desired features. Here are some considerations to keep in mind:

Language Compatibility: Ensure the framework supports the language your application is written in.

Performance: Consider the impact of the logging framework on your application’s performance, especially in high-throughput scenarios.

Flexibility: Look for frameworks that offer flexible configuration options and support for multiple log output destinations.

Community and Support: Choose frameworks with active communities and good documentation to ensure you can find help and resources when needed.

Conclusion

Understanding the types of logs and selecting the right logging framework are essential steps in building a robust logging system. By leveraging the appropriate logging tools, you can gain valuable insights into your application’s behavior, diagnose issues more effectively, and ensure compliance with security and regulatory requirements.

Stay tuned for the next part of our series, where we will discuss best practices for log analysis, visualization and alerting to help you get the most out of your logging infrastructure.

For those who missed it, you can catch up on Part 1: An Introduction to Logging. Happy logging!

Look out for further blogs in this series to learn more.

Part 1: An Introduction to Logging
Part 3: Log Analysis, Visualization and Alerting
Part 4: Distributed Tracing and Distributed Logging
Part 5: Security in Logging
Part 6: Performance Impact of Logging
Part 7: Logging for GenAI Apps