Fluentd

file

The file buffer plugin provides a persistent buffer implementation. It uses files to store buffer chunks on disk.

Parameters

• Common Parameters
• Buffer Section Configurations

path

The directory path where buffer chunks are stored. Don't share this directory path with other buffers. Be sure to specify a unique path for each buffer.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The file name is buffer.b{chunk_id}{path_suffix}.

/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

With multiple workers, a directory is automatically created for each worker. So there is no need to specify a unique path for each worker.

<system>
  workers 2
</system>

...

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The directory worker{worker_id} is automatically created.

/var/log/fluentd/buf/worker0/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/worker0/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

/var/log/fluentd/buf/worker1/buffer.b5e2a5aca2bcd9818ad6718845ddc456a.log
/var/log/fluentd/buf/worker1/buffer.b5e2a5aca2bcd9818ad6718845ddc456a.log.meta

If you specify root_dir in system configuration and @id of the plugin, then you can omit this parameter.

<system>
  root_dir /var/log/fluentd
</system>

...

<match pattern>
  @id test_id
  ...
  <buffer>
    @type file
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The directory {root_dir}/worker{worker_id}/{@id}/buffer is used for the path. In this case, the worker{worker_id} directory is created even for a single worker.

/var/log/fluentd/worker0/test_id/buffer/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/worker0/test_id/buffer/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

Please make sure that you have enough space in the path directory. Running out of disk space is a problem frequently reported by users.

path_suffix

Changes the suffix of the buffer file.

# default 
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

# with 'path_suffix .buf'
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta

This parameter is useful when .log is not fit for your environment. See also this issue's comment.

file_permission

The permission of the buffer chunk files, in the octal format. The same permission is applied to the .meta files.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf
    file_permission 0600
  </buffer>
</match>

If this parameter is not set, file_permission in the system configuration is used. If it is not set either, 0644 is used.

dir_permission

The permission of the directory created to store the buffer chunk files, in the octal format.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf
    dir_permission 0700
  </buffer>
</match>

If this parameter is not set, dir_permission in the system configuration is used. If it is not set either, 0755 is used.

Tips

Customize a filename of the buffer chunk

You can customize the prefix of filename (buffer by default) by adding .* to the end of the path parameter.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf/custom.*
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The prefix buffer is changed to custom.

/var/log/fluentd/buf/custom.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/custom.b58eec11d08ca8143b40e4d303510e0bb.log.meta

You can also customize the entire filename by adding .*. to the path parameter.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf/custom_prefix.*.custom_suffix
  </buffer>
</match>

This config outputs the buffer chunk files as follows. In this case, path_suffix parameter is not used.

/var/log/fluentd/buf/custom_prefix.b58eec11d08ca8143b40e4d303510e0bb.custom_suffix
/var/log/fluentd/buf/custom_prefix.b58eec11d08ca8143b40e4d303510e0bb.custom_suffix.meta

Limitation