Fluentd

http

The out_http Output plugin writes records via HTTP/HTTPS.

This plugin is introduced since fluentd v1.7.0.

It is included in Fluentd's core.

Example Configuration

<match pattern>
  @type http

  endpoint http://logserver.com:9000/api
  open_timeout 2

  <format>
    @type json
  </format>
  <buffer>
    flush_interval 10s
  </buffer>
</match>

Please see the Configuration File article for the basic structure and syntax of the configuration file.

For <buffer>, refer to Buffer Section Configuration.

Parameters

@type

The value must be http.

endpoint

The endpoint for HTTP request. If you want to use HTTPS, use https prefix.

# Use HTTP
endpoint http://example.com/api

# Use HTTPS. You can set additional HTTPS parameters like tls_xxx
endpoint https://example.com/api

The endpoint parameter supports placeholders, so you can embed time, tag and record fields in the endpoint. Placeholders also require the buffer section in order to work. Here is an example:

endpoint http://example.com/api/${tag}-${key}
<buffer tag,key>
  # buffer parameters
</buffer>

See Buffer Section Configurations for more details.

Since v1.19.3, if a placeholder is used in the host part of the endpoint, allowed_hosts must also be set. A placeholder used only in the path or the query string is not affected.

allowed_hosts

The list of hosts permitted as the destination of the request.

This is required when a placeholder in endpoint changes the host of the request. Since placeholders are expanded with the values in the incoming records, an unexpected record could otherwise make Fluentd send the data to an arbitrary host. This parameter restricts the destination to the hosts you explicitly list.

endpoint http://${tag}:8080/api
allowed_hosts ["known-host.example.com", "another-host.example.com"]
<buffer tag>
  # buffer parameters
</buffer>

The validation is performed for each chunk on flush, and only when the expanded host differs from the host written in endpoint. Therefore, this parameter has no effect in the following cases:

• endpoint has no placeholder at all.

• endpoint has placeholders only in the path or the query string,

  e.g. http://example.com/api/${tag}.

If the expanded host differs from the one in endpoint and it is not permitted, Fluent::UnrecoverableError is raised:

• allowed_hosts is empty.
• allowed_hosts does not include the expanded host.

http_method

The method for HTTP request.

proxy

The proxy for HTTP request.

content_type

Content-Type for HTTP request. out_http automatically sets Content-Type for built-in formatters when this parameter is not specified.

Here is a table:

• json with json_array false: application/x-ndjson
• json with json_array true: application/json
• csv: text/csv
• tsv, ltsv: text/tab-separated-values
• msgpack: application/x-msgpack
• out_file, single_value, stdout, hash: text/plain

json_array

Using the array format of JSON. This parameter is used and valid only for json format. When json_array is true, Content-Type should be application/json and be able to use JSON data for the HTTP request body.

• json_array true

[
  {"key":"value", ...},
  {"key":"value", ...},
  ...
]

• json_array false

{"key":"value", ...}
{"key":"value", ...}
...

compress

The option to compress HTTP request body.

<format> Directive

The format of the payload. The default @type is json.

Here is single_value example:

<format>
  @type single_value
</format>

See formatter article for more detail.

headers

Additional headers for HTTP request.

headers {"key1":"value1", "key2":"value2"}

headers_from_placeholders

Additional placeholder based headers for HTTP request. If you want to use tag or record field, use this parameter instead of headers.

headers_from_placeholders {"x-foo-bar":"${$.foo.bar}","x-tag":"app-${tag}"}
<buffer tag,$.foo.bar>
  # buffer parameters...
</buffer>

open_timeout

The connection open timeout in seconds.

See also Ruby document.

read_timeout

The read timeout in seconds.

See also Ruby document.

ssl_timeout

The TLS timeout in seconds.

reuse_connections

Try to reuse connections. This will improve performance.

tls_ca_cert_path

The CA certificate path for TLS.

tls_client_cert_path

The client certificate path for TLS.

tls_private_key_path

The client private key path for TLS.

tls_private_key_passphrase

The client private key passphrase for TLS.

tls_verify_mode

The verify mode of TLS.

tls_version

The default version of TLS.

tls_ciphers

The cipher suites configuration of TLS.

error_response_as_unrecoverable

Raise UnrecoverableError when the response code is not SUCCESS, 1xx/3xx/4xx/5xx. If false, out_http logs error message instead of raising UnrecoverableError.

See also Handling Unrecoverable Errors.

retryable_response_codes

The list of retryable response codes. If the response code is included in this list, out_http retries the buffer flush.

<auth> Section

Specifies HTTP authentication.

method

The method for HTTP authentication.

• basic: basic authentication
• aws_sigv4: AWS Signature Version 4

Parameters for method basic

<auth>
  method basic
  username fluentd
  password hello
</auth>

username

The username for basic authentication.

password

The password for basic authentication.

Parameters for method aws_sigv4

Parameters for AWS Signature Version 4.

<auth>
  method aws_sigv4
  aws_service osis
  aws_region us-east-1
  aws_role_arn arn:aws:iam::123456789012:role/MyRole
</auth>

aws_service

The AWS service to authenticate against.

This parameter is required when you specify aws_sigv4 for method.

aws_region

The AWS region to use when authenticating.

This parameter is required when you specify aws_sigv4 for method.

aws_role_arn

The AWS role ARN to assume when authenticating.

This parameter is optional when you specify aws_sigv4 for method. If you provide it, Fluentd will assume that AWS role and send requests signing from that role. Otherwise, Fluentd will use the credentials found by the credential provider chain as defined in the AWS documentation.

Common Output / Buffer parameters

For common output / buffer parameters, please check the following articles:

• Output Plugin Overview
• Buffer Section Configuration

The Payload Content

out_http's request body depends on <format> configuration. For example, the default setting generates newline delimited JSON like this:

# \n is added by `add_newline true` parameter in <format>
{"key1":"value1","key2":"value2",...}\n
{"key1":"value1","key2":"value2",...}\n
{"key1":"value1","key2":"value2",...}\n
{"key1":"value1","key2":"value2",...}\n
...

With @type single_value:

log line1\n
log line2\n
log line3\n
log line4\n
...

Troubleshooting

400 Bad request between out_http and in_http

When getting the following error:

#0 got unrecoverable error in primary and no secondary error_class=Fluent::UnrecoverableError error="400 Bad Request 400 Bad Request\n'json' or 'msgpack' parameter is required\n"
#0 bad chunk is moved to /tmp/fluent/backup/worker0/object_3ff8a73edae8/5a71a08ca19b1b343c8dce1b74c9a963.log

Users should specify json format with json_array as true for out_http configuration:

<match **>
  @type http
  endpoint http://some.your.http.endpoint:9811/your-awesome-path
  <format>
    @type json
  </format>
  json_array true
  <buffer>
    flush_interval 2s
  </buffer>
</match>

And receiver in_http configuration should be:

<source>
  @type http
  port 9811
  bind 0.0.0.0
  <parse>
    @type json
  </parse>
</source>

Or specify msgpack format:

<match **>
  @type http
  endpoint http://some.your.http.endpoint:9882/your-awesome-path
  <format>
    @type msgpack
  </format>
  <buffer>
    flush_interval 2s
  </buffer>
</match>

And, receiver in_http configuration should be:

<source>
  @type http
  port 9882
  bind 0.0.0.0
  <parse>
    @type msgpack
  </parse>
  <format>
    @type json
  </format>
</source>

But, we recommend using in/out forward plugin to communicate with two Fluentd instances due to at-most-once and at-least-once semantics for rigidity.