The Wayback Machine - https://web.archive.org/web/20201217023127/https://github.com/fluent/fluentd-docs-gitbook/issues/252
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-process workers and file buffer #252

Open
msamoylov opened this issue Nov 17, 2020 · 4 comments
Open

Multi-process workers and file buffer #252

msamoylov opened this issue Nov 17, 2020 · 4 comments

Comments

@msamoylov
Copy link

@msamoylov msamoylov commented Nov 17, 2020

The documentation is outdated.

With multi-process workers, you cannot use the fixed path configuration for file buffer because it conflicts buffer file path between processes.

With the given example you get config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="buffer path is not configured. specify 'path' in <buffer>"

@repeatedly
Copy link
Member

@repeatedly repeatedly commented Nov 17, 2020

Which point? I can't reproduce your error message.

% fluentd -c multi_file_buffer.conf
2020-11-17 19:48:40 +0900 [info]: parsing config file is succeeded path="multi_file_buffer.conf"
2020-11-17 19:48:40 +0900 [info]: gem 'fluent-plugin-prometheus' version '1.8.1'
2020-11-17 19:48:40 +0900 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2020-11-17 19:48:40 +0900 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2020-11-17 19:48:40 +0900 [info]: gem 'fluentd' version '1.11.4'
2020-11-17 19:48:40 +0900 [info]: [out_fwd] adding forwarding server 'localhost:24224' host="localhost" port=24224 weight=60 plugin_id="out_fwd"
2020-11-17 19:48:40 +0900 [info]: using configuration file: <ROOT>
  <system>
    workers 2
    root_dir "/tmp/fluentd"
  </system>
  <match pattern>
    @type forward
    @id out_fwd
    <service>
      host "localhost"
      port 24224
    </service>
    <buffer>
      @type "file"
    </buffer>
  </match>
</ROOT>
2020-11-17 19:48:40 +0900 [info]: starting fluentd-1.11.4 pid=94982 ruby="2.6.3"
2020-11-17 19:48:40 +0900 [info]: spawn command to main:  cmdline=["/path/to/ruby", "-Eascii-8bit:ascii-8bit", "/path/to/fluentd", "-c", "multi_file_buffer.conf", "--under-supervisor"]
2020-11-17 19:48:41 +0900 [info]: adding match pattern="pattern" type="forward"
2020-11-17 19:48:41 +0900 [info]: #0 [out_fwd] adding forwarding server 'localhost:24224' host="localhost" port=24224 weight=60 plugin_id="out_fwd"
2020-11-17 19:48:41 +0900 [info]: #1 [out_fwd] adding forwarding server 'localhost:24224' host="localhost" port=24224 weight=60 plugin_id="out_fwd"
2020-11-17 19:48:41 +0900 [info]: #0 starting fluentd worker pid=94997 ppid=94982 worker=0
2020-11-17 19:48:41 +0900 [info]: #1 starting fluentd worker pid=94998 ppid=94982 worker=1
2020-11-17 19:48:41 +0900 [info]: #0 fluentd worker is now running worker=0
2020-11-17 19:48:41 +0900 [info]: #1 fluentd worker is now running worker=1
@msamoylov
Copy link
Author

@msamoylov msamoylov commented Nov 17, 2020

This configuration leads to the above-mentioned error message with fluentd 1.11.5, but uncommenting the path parameter fixes it. Any clues?

<system>
      root_dir /var/log/fluentd
      workers 2
</system>

# Skipped

<match>
        @type copy
        @id internal_out

        <store>
          @type elasticsearch
          host "#{ENV['ES_HOST']}"
          port 9200
          scheme https
          user "#{ENV['ES_USER']}"
          password "#{ENV['ES_PASSWORD']}"
          logstash_format true
          logstash_prefix internal
          logstash_dateformat %Y%m%d
          include_tag_key true
          include_timestamp true
          time_key time
          type_name _doc
          reconnect_on_error true
          tag_key @log_name
          prefer_oj_serializer true
          reconnect_on_error true
          reload_on_failure true
          reload_connections false
          verify_es_version_at_startup false
          default_elasticsearch_version 7
          ssl_version TLSv1_2
          suppress_type_name true
          request_timeout 60s
          <buffer>
            @type file
            # path /var/log/fluentd/kubernetes.internal.system.buffer
            flush_mode interval
            retry_type exponential_backoff
            flush_thread_count 6
            flush_interval 5s
            retry_forever
            retry_max_interval 30
            retry_randomize true
            total_limit_size 5GB
            overflow_action "block"
          </buffer>
        </store>        
      </match>
    </label>
@repeatedly
Copy link
Member

@repeatedly repeatedly commented Nov 18, 2020

You forgot to add @id to elasticsearch setting.

root_dir/@id parameter
These parameters must be specified when you use the file buffer.

@msamoylov
Copy link
Author

@msamoylov msamoylov commented Nov 19, 2020

Are you saying I need @id in both <match> and <store> definitions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.