Fluentd

tcp

The in_tcp Input plugin enables Fluentd to accept TCP payload.

It is included in Fluentd's core.

Don't use this plugin for receiving logs from Fluentd client libraries. Use in_forward for such cases.

Example Configuration

<source>
  @type tcp
  tag tcp.events # required
  <parse>
    @type regexp
    expression /^(?<field1>\d+):(?<field2>\w+)$/
  </parse>
  port 20001   # optional. 5170 by default
  bind 0.0.0.0 # optional. 0.0.0.0 by default
  delimiter "\n" # optional. "\n" (newline) by default
</source>

Example input:

$ echo '123456:awesome' | netcat 0.0.0.0 5170

Parsed result:

{"field1":"123456","field2":"awesome"}

Refer to the Configuration File article for the basic structure and syntax of the configuration file.

For <parse>, see Parse Section.

We have observed drastic performance improvements on Linux, with proper kernel parameter settings. If you have high-volume TCP traffic, follow Before Installing Fluentd instructions.

Plugin Helpers

• server
• parser
• extract
• compat_parameters

Parameters

See Common Parameters.

@type

The value must be tcp.

tag

The tag of output events.

port

The port to listen to.

bind

The bind address to listen to.

source_hostname_key

The field name of the client's hostname. If set, the client's hostname will be set to its key. The default is nil (no adding hostname).

With this configuration:

source_hostname_key client_host

The client's hostname is set to client_host field:

{
    ...
    "foo": "bar",
    "client_host": "client.hostname.org"
}

source_address_key

The field name for the client's IP address. If set, Fluentd automatically adds the remote address to each data record.

For example, if you have the following configuration:

<source>
  @type tcp
  source_address_key client_addr
  # ...
</source>

You will get something like below:

{
    ...
    "client_addr": "192.168.10.10"
    ...
}

source_host_key

This parameter is deprecated. Use source_hostname_key instead.

The field name of the client's hostname.

message_length_limit

The maximum number of bytes for the message.

delimiter

The payload is read up to this character. Since a TCP stream has no message boundary, in_tcp splits the received bytes at every occurrence of this string and passes each part to the <parse> section. The delimiter itself is not included in the parsed message.

This is not a pattern but a plain string, so a delimiter of two or more characters can also be specified:

<source>
  @type tcp
  tag tcp.events
  delimiter "||"  # It will work expectedly if it splits with '||' against incoming "foo||bar", not work for 'foo|bar'.
  <parse>
    @type json
  </parse>
</source>

When message_length_limit is set and the data received before the next delimiter exceeds that limit, the buffer is cleared and the remaining bytes are discarded until the next delimiter arrives. The subsequent messages are processed as usual.

<transport> Section

This section is for setting TLS transport or some general transport configurations. See Config: Transport Section for all supported parameters.

General configuration

linger_timeout

The timeout (seconds) to set SO_LINGER.

The default value 0 is to send RST rather than FIN to avoid lots of connections sitting in TIME_WAIT on closing on non-Windows.

You can set positive value to send FIN on closing on non-Windows.

<source>
  @type tcp
  # other plugin parameters
  <transport tcp>
    linger_timeout 1
  </transport>
</source>

send_keepalive_packet

Enable SO_KEEPALIVE on the underlying TCP sockets.

This is useful when you connect to Fluentd over firewalls or proxies and want to prevent connections from being closed automatically.

TLS configuration

<transport tls>
  cert_path /path/to/fluentd.crt
  # ...
</transport>

See How to Enable TLS Encryption section for how to use and see Configuration Example for all supported parameters.

Without <transport tls>, in_tcp uses raw TCP.

<security> Section

Adds <security>/<client> section to allow access by Host/IP/Network.

<client> Section

host

The IP address or host name of the client.

This is exclusive with network.

network

Network address specification.

This is exclusive with host.

<parse> Section

in_tcp uses the parser plugin to parse the payload.

For more details:

• Parser Plugin Overview
• Parse Section Configurations

Code Example

Here is a Ruby example to send an event to in_tcp:

require 'socket'

# This example uses json payload.
# In in_tcp configuration, need to configure "@type json" in "<parse>"
TCPSocket.open('127.0.0.1', 5170) do |s|
  s.write('{"k":"v1"}' + "\n")
  s.write('{"k":"v2"}' + "\n")
end

Tips

How to Enable TLS Encryption

in_tcp supports TLS transport.

Example:

<source>
  @type tcp
  port 5140
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/pki/ca.pem
    cert_path /etc/pki/cert.pem
    private_key_path /etc/pki/key.pem
    private_key_passphrase PASSPHRASE
  </transport>
  tag tcp
</source>

How to Enable TLS Mutual Authentication

Fluentd supports TLS mutual authentication (i.e. client certificate auth). If you want to use this feature, please set the client_cert_auth and ca_path options like this:

<source>
  @type tcp
  port 20001
  <transport tls>
    # ...
    client_cert_auth true
    ca_path /path/to/ca/cert
  </transport>
</source>

When this feature is enabled, Fluentd will check all the incoming requests for a client certificate signed by the trusted CA. Requests with an invalid client certificate will fail.

To check if mutual authentication is working properly, issue these commands:

$ openssl s_client -connect localhost:20001 \
  -key path/to/client.key \
  -cert path/to/client.crt \
  -CAfile path/to/ca.crt

If the connection gets established successfully, your setup is working fine.