Centralized Logging¶

By default SynthOrg writes structured JSON logs to files inside the Docker volume (/data/logs/). For production multi-instance deployments you need those logs shipped to a centralized system for aggregation, search, and alerting.

Three approaches are available -- they can be combined.

Syslog Shipping¶

The SYSLOG sink type ships each structured log event as a JSON string to a syslog endpoint over UDP (default) or TCP.

Configuration (Runtime JSON)¶

Add a syslog sink via the custom_sinks setting in the observability namespace:

[
  {
    "sink_type": "syslog",
    "syslog_host": "syslog.internal",
    "syslog_port": 514,
    "syslog_facility": "local0",
    "syslog_protocol": "udp",
    "level": "info"
  }
]

Configuration (YAML)¶

logging:
  custom_sinks:
    - sink_type: syslog
      syslog_host: syslog.internal
      syslog_port: 514
      syslog_facility: local0
      syslog_protocol: udp
      level: info

Fields¶

Receiver Configuration (rsyslog)¶

# /etc/rsyslog.d/50-synthorg.conf
module(load="imudp")
input(type="imudp" port="514")

template(name="synthorg-json" type="string"
         string="%msg%\n")

if $syslogfacility-text == 'local0' then {
    action(type="omfile" file="/var/log/synthorg/app.log"
           template="synthorg-json")
    stop
}

Receiver Configuration (syslog-ng)¶

source s_synthorg { udp(port(514)); };
destination d_synthorg { file("/var/log/synthorg/app.log"); };
filter f_synthorg { facility(local0); };
log { source(s_synthorg); filter(f_synthorg); destination(d_synthorg); };

Tips¶

• Use TCP when log loss is unacceptable (TCP retries on connection failure)
• Use UDP for high-throughput, low-latency scenarios where occasional loss is tolerable
• Dedicated facility (local0--local7) makes receiver-side filtering straightforward
• Multiple syslog sinks with different levels (e.g. ERROR to a pager, INFO to storage) are supported

HTTP Shipping¶

The HTTP sink type batches structured log records and POSTs them as JSON arrays to an HTTP endpoint. A background thread handles batching, flushing, and retries.

Configuration (Runtime JSON)¶

[
  {
    "sink_type": "http",
    "http_url": "https://logs.example.com/ingest",
    "http_headers": [["Authorization", "Bearer <token>"]],
    "http_batch_size": 100,
    "http_flush_interval_seconds": 5.0,
    "http_timeout_seconds": 10.0,
    "http_max_retries": 3,
    "level": "info"
  }
]

Configuration (YAML)¶

logging:
  custom_sinks:
    - sink_type: http
      http_url: https://logs.example.com/ingest
      http_headers:
        - ["Authorization", "Bearer <token>"]
      http_batch_size: 100
      http_flush_interval_seconds: 5.0
      http_timeout_seconds: 10.0
      http_max_retries: 3
      level: info

Fields¶

Endpoint Examples¶

The HTTP sink ships raw JSON arrays of structured log objects. Vendor-specific endpoints (e.g., Loki push API, Elasticsearch bulk API) typically expect a different wire format. Use a generic JSON-accepting collector (Vector, Fluentd HTTP input, or a lightweight proxy) to transform the array into the vendor-specific format.

Generic JSON endpoint (any receiver accepting JSON arrays):

{
  "sink_type": "http",
  "http_url": "https://logs.example.com/ingest",
  "http_headers": [["X-API-Key", "<key>"]]
}

Tips¶

• Tune batch size based on endpoint limits and network latency
• Lower flush interval (1--2s) for near-real-time shipping
• Set retries to 0 for fire-and-forget shipping where loss is acceptable
• HTTPS is recommended for shipping over untrusted networks

Docker Logging Drivers¶

Docker's built-in logging drivers capture container stdout/stderr. By default the console sink writes colored text to stderr. For structured JSON output that Docker drivers and downstream parsers can process, override the console sink format:

{"__console__": {"json_format": true}}

Or in YAML company config:

logging:
  sink_overrides:
    __console__:
      json_format: true

When to Use Docker Drivers¶

• Your infrastructure already has a log aggregation pipeline (fluentd, Logstash)
• You want container-level log capture without app-level changes
• You need cloud-native logging (AWS CloudWatch, GCP Logging, Azure Monitor)

Configuration¶

In docker/compose.yml, replace the default json-file driver:

# Fluentd driver:
x-logging: &logging
  driver: fluentd
  options:
    fluentd-address: "fluentd:24224"
    tag: "synthorg.{{.Name}}"

# Or for syslog (replace the block above):
# x-logging: &logging
#   driver: syslog
#   options:
#     syslog-address: "udp://syslog.internal:514"
#     syslog-facility: "local0"
#     tag: "synthorg"

Combining with App-Level Shipping¶

Docker drivers capture only console (stderr) output. The 10 JSON file sinks are not visible to Docker drivers. For complete log coverage:

1. App-level syslog/HTTP for routed, structured JSON from all 11+ sinks
2. Docker driver as a safety net for console output and uncaught exceptions

Compressed Archival¶

Rotated log files can be automatically gzip-compressed to save disk space.

Configuration¶

Enable compression in the rotation config:

{
  "audit.log": {
    "rotation": {
      "strategy": "builtin",
      "max_bytes": 10485760,
      "backup_count": 10,
      "compress_rotated": true
    }
  }
}

Or via RotationConfig in YAML:

logging:
  sink_overrides:
    audit.log:
      rotation:
        backup_count: 10
        compress_rotated: true

Behavior¶

• When compress_rotated is true, rotated backups are stored as .log.N.gz instead of .log.N
• Compression happens synchronously during rotation (fast for 10 MB files)
• If compression fails, the uncompressed backup is retained
• The active log file is never compressed -- only rotated backups
• Default: false (backward compatible)

Disk Space Savings¶

Structured JSON logs compress well (typically 5--10x reduction). With 10 MB rotation and 10 backup files:

Retention Strategy¶

For production deployments, combine rotation, compression, and shipping:

1. Ship all logs to centralized storage (syslog or HTTP) for long-term retention
2. Rotate local files with builtin strategy (10 MB, 5--10 backups)
3. Compress rotated backups to reduce local disk usage
4. Centralized system handles search, alerting, and long-term retention

This keeps local disk usage bounded while centralized storage provides the full audit trail.