Terraform Enterprise Log Forwarding

Terraform Enterprise supports forwarding its logs to one or more external destinations, a process called log forwarding. Log forwarding provides increased observability, assistance complying with log retention requirements, and information during troubleshooting.

Requirements

Log forwarding requires:

Terraform Enterprise running on an instance using systemd-journald. Execute systemctl status systemd-journald to check if the systemd-journald service is started and enabled.
A version of Docker that supports the journald logging driver. Execute docker info --format '{{.Plugins.Log}}' to check if the journald plugin is listed.
Network connectivity between Terraform Enterprise and the external destination(s) where logs should be forwarded.

Enable Log Forwarding

Log forwarding is disabled by default. To enable log forwarding, set the log_forwarding_enabled Terraform Enterprise application setting to the value 1.

tfe-admin app-config -k log_forwarding_enabled -v 1

When log forwarding is enabled, the Terraform Enterprise application settings show the following for log_forwarding_enabled:

    "log_forwarding_enabled": {
        "value": "1"
    },

Note: Automated license utilization reporting (which securely sends HashiCorp the minimum data required to validate license utilization) is on by default.

Configure External Destinations

The log_forwarding_config Terraform Enterprise application setting must contain valid Fluent Bit [OUTPUT] configuration specifying supported external destination(s) where Terraform Enterprise should forward logs. The default configuration does not forward any logs.

Since the Terraform Enterprise application settings are stored as JSON strings, we recommend first creating a fluent-bit.conf file with the valid Fluent Bit [OUTPUT] configuration and then using that file to configure the log_forwarding_config application setting. This method ensures that the configuration is stored in the application settings exactly how it appears in the fluent-bit.conf file.

For a Standalone installation of Terraform Enterprise:

replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"

For an Active/Active installation of Terraform Enterprise:

tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"

Once configured, the Terraform Enterprise application settings show the log_forwarding_config setting in escaped JSON string format:

    "log_forwarding_config": {
        "value": "# Match all logs and do not forward them anywhere.\n[OUTPUT]\n    Name null\n    Match *\n"
    },

That escaped JSON string renders to the following:

# Match all logs and do not forward them anywhere.
[OUTPUT]
    Name null
    Match *

To forward logs to multiple external destinations, use multiple [OUTPUT] directives.

# Forward all logs to Datadog.
[OUTPUT]
    Name datadog
    Match *
    ...

# Forward all logs to Fluent Bit or Fluentd.
[OUTPUT]
    Name forward
    Match *
    ...

Note: Do not use an [OUTPUT] directive with the stdout Fluent Bit output plugin. Doing this creates a loop that continuously emits logs!

Restart Terraform Enterprise

Once log forwarding is enabled and configured, you need to restart Terraform Enterprise for the changes to take effect. Learn how to restart Terraform Enterprise.

Supported External Destinations

You can only forward logs to one of the supported external destinations below. Each supported external destination contains example configuration for convenience.

Amazon CloudWatch

Sending to Amazon CloudWatch is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon CloudWatch. Refer to the cloudwatch_logs Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name               cloudwatch_logs
    Match              *
    region             us-east-1
    log_group_name     example-log-group
    log_stream_name    example-log-stream
    auto_create_group  On

Note: In Terraform Enterprise installations using AWS external services, Fluent Bit will have access to the same AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables that are used for object storage.

Amazon S3

Sending to Amazon S3 is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon S3. Refer to the s3 Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name                          s3
    Match                         *
    bucket                        example-bucket
    region                        us-east-1
    total_file_size               250M
    s3_key_format                 /$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz
    s3_key_format_tag_delimiters  .-

Azure Blob Storage

This example configuration forwards all logs to Azure Blob Storage. Refer to the azure_blob Fluent Bit output plugin documentation for more information.

[OUTPUT]
    name                   azure_blob
    match                  *
    account_name           example-account-name
    shared_key             example-access-key
    path                   logs
    container_name         example-container-name
    auto_create_container  on
    tls                    on

Azure Log Analytics

This example configuration forwards all logs to Azure Log Analytics. Refer to the azure Fluent Bit output plugin documentation for more information.

[OUTPUT]
    name         azure
    match        *
    Customer_ID  example-log-analytics-workspace-id
    Shared_Key   example-access-key

Datadog

This example configuration forwards all logs to Datadog. Refer to the datadog Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name        datadog
    Match       *
    Host        http-intake.logs.datadoghq.com
    TLS         on
    compress    gzip
    apikey      example-api-key
    dd_service  terraform_enterprise
    dd_source   docker
    dd_tags     environment:development,owner:engineering

Forward

This example configuration forwards all logs to a listening Fluent Bit or Fluentd instance. Refer to the forward Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name   forward
    Match  *
    Host   fluent.example.com
    Port   24224

Google Cloud Platform Cloud Logging

Sending to Google Cloud Platform Cloud Logging is only supported when Terraform Enterprise is located within GCP due to how Fluent Bit reads GCP credentials.

This example configuration forwards all logs to Google Cloud Platform Cloud Logging (formerly known as Stackdriver). Refer to the stackdriver Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name       stackdriver
    Match      *
    location   us-east1
    namespace  terraform_enterprise
    node_id    example-hostname
    resource   generic_node

Note: In Terraform Enterprise installations using GCP external services, Fluent Bit will have access to the GOOGLE_SERVICE_CREDENTIALS environment variable that points to a file containing the same GCP Service Account JSON credentials that are used for object storage.

Splunk Enterprise HTTP Event Collector (HEC)

This example configuration forwards all logs to Splunk Enterprise via the HTTP Event Collector (HEC) interface. Refer to the splunk Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name          splunk
    Match         *
    Host          example-splunk-hec-endpoint
    Port          8088
    Splunk_Token  example-splunk-token

Syslog

This example configuration forwards all logs to a Syslog-compatible endpoint. Refer to the syslog Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name                 syslog
    Match                *
    host                 example-syslog-host
    port                 514
    mode                 tcp
    syslog_message_key   log
    syslog_severity_key  PRIORITY
    syslog_hostname_key  _HOSTNAME
    syslog_appname_key   SYSLOG_IDENTIFIER
    syslog_procid_key    _PID

Warning

The `syslog_message_key` should not be changed from `log`. If that value is changed, the application will no longer forward logs.

Audit Logs

Terraform Enterprise emits its audit logs along with its application logs. Currently, log forwarding can forward either all Terraform Enterprise logs or no logs at all. To distinguish audit logs from application logs, audit log entries contain the string [Audit Log].

Here's an example audit log entry formatted for readability:

2021-08-31 04:58:30 [INFO] [7a233ad1-c50c-4737-a925-3be901e55fcb] [Audit Log]
{
  "resource":"run",
  "action":"create",
  "resource_id":"run-nL77p69bsesoF3RK",
  "organization":"example-org",
  "organization_id":"org-pveSPvxocni226Fn",
  "actor":"example-user",
  "timestamp":"2021-08-31T04:58:30Z",
  "actor_ip":"19.115.231.192"
}

If you have a requirement to split audit logs from application logs, we recommend forwarding all Terraform Enterprise logs to a log aggregation system, filtering the audit logs based on the [Audit Log] string, and forwarding just the audit logs to the desired destination.

Log Rotation

Log forwarding uses the journald Docker logging driver to send Terraform Enterprise logs to systemd-journald. This can cause increased disk utilization for the /var/log/journal path.

To limit disk utilization, configure the SystemMaxFileSize and SystemMaxFiles settings within /etc/systemd/journald.conf.

The following configuration tells systemd-journald to use up to 7GB of disk space by limiting the size a log file to 1024MB and keeping up to 7 files at any given time:

[Journal]
SystemMaxFileSize=1024M
SystemMaxFiles=7

To apply these changes, restart systemd-journald:

sudo systemctl restart systemd-journald