puppet-filebeat

Upgrading from <= v 0.10.4

Be sure to read the changelog as there are up to 2 breaking changes introduced in v0.11.0 related to processors and the default registry path

Table of Contents

1. Description
2. Setup - The basics of getting started with filebeat
   • What filebeat affects
   • Upgrading to Filebeat 5.x
   • Setup requirements
   • Beginning with filebeat
3. Usage - Configuration options and additional functionality
   • Adding a prospector
     • Multiline Logs
     • JSON logs
   • Prospectors in hiera
   • Usage on Windows
   • Processors
4. Reference
   • Public Classes
   • Private Classes
   • Public Defines
5. Limitations - OS compatibility, etc.
   • Pre-1.9.1 Ruby
   • Using config_file
6. Development - Guide for contributing to the module

Description

The filebeat module installs and configures the filebeat log shipper maintained by elastic.

Setup

What filebeat affects

By default filebeat adds a software repository to your system, and installs filebeat along with required configurations.

Upgrading to Filebeat 5.x

If you use this module on a system with filebeat 1.x installed, and you keep your current parameters nothing will change. Setting major_version to '5' will modify the configuration template and update package repositories, but won't update the package itself. To update the package set the package_ensure parameter to at least 5.0.0.

Windows users should set major_version to 5 and update the download_url parameter to the correct download.

If you're on a Debian based system, you need to make sure that the apt-transport-https package is installed if you want this module to manage the repository for you (it does by default).

Setup Requirements

The filebeat module depends on puppetlabs/stdlib, and on puppetlabs/apt on Debian based systems.

Beginning with filebeat

filebeat can be installed with puppet module install pcfens-filebeat (or with r10k, librarian-puppet, etc.)

The only required parameter, other than which files to ship, is the outputs parameter.

Usage

All of the default values in filebeat follow the upstream defaults (at the time of writing).

To ship files to elasticsearch:

class { 'filebeat':
  outputs => {
    'elasticsearch' => {
     'hosts' => [
       'http://localhost:9200',
       'http://anotherserver:9200'
     ],
     'index'       => 'packetbeat',
     'cas'         => [
        '/etc/pki/root/ca.pem',
     ],
    },
  },
}

To ship log files through logstash:

class { 'filebeat':
  outputs => {
    'logstash'     => {
     'hosts' => [
       'localhost:5044',
       'anotherserver:5044'
     ],
     'loadbalance' => true,
    },
  },
}

Shipper and logging options can be configured the same way, and are documented on the elastic website.

Adding a prospector

Prospectors are processes that ship log files to elasticsearch or logstash. They can be defined as a hash added to the class declaration (also used for automatically creating prospectors using hiera), or as their own defined resources.

At a minimum, the paths parameter must be set to an array of files or blobs that should be shipped. doc_type is what logstash views as the type parameter if you'd like to apply conditional filters.

filebeat::prospector { 'syslogs':
  paths    => [
    '/var/log/auth.log',
    '/var/log/syslog',
  ],
  doc_type => 'syslog-beat',
}

Multiline Logs

Filebeat prospectors (versions >= 1.1) can handle multiline log entries. The multiline parameter accepts a hash containing pattern, negate, match, max_lines, and timeout as documented in the filebeat configuration documentation.

JSON Logs

Filebeat prospectors (versions >= 5.0) can natively decode JSON objects if they are stored one per line. The json parameter accepts a hash containing message_key, keys_under_root, overwrite_keys, and add_error_key as documented in the filebeat configuration documentation.

Prospectors in Hiera

Prospectors can be declared in hiera using the prospectors parameter. By default, hiera will not merge prospector declarations down the hiera hierarchy. To change the behavior in puppet 3 use the prospectors_merge parameter. In puppet 4, you can use prospectors_merge, but can also use the lookup_options flag.

When prospectors_merge is set to true, prospectors will be replaced by the output of hiera_hash('filebeat::prospectors').

Usage on Windows

When installing on Windows, this module will download the windows version of Filebeat from elastic to C:\Temp by default. The directory can be overridden using the tmp_dir parameter. tmp_dir is not managed by this module, but is expected to exist as a directory that puppet can write to.

Processors

Filebeat 5.0 and greater includes a new libbeat feature for filtering and/or enhancing all exported data through processors before geing sent to the configured output(s). They can be defined as a hash added to the class declaration (also used for automatically creating processors using hiera), or as their own defined resources.

To drop the offset and input_type fields from all events:

class{"filebeat":
  processors => {
    "drop_fields" => {
      "params" => {"fields" => ["input_type", "offset"]}
    },
  },
}

To drop all events that have the http response code equal to 200:

class{"filebeat":
  processors => {
    "drop_event" => {
      "when" => {"equals" => {"http.code" => 200}}
    },
  },
}

Now to combine these examples into a single definition:

class{"filebeat":
  processors => {
    "drop_fields" => {
      "params"   => {"fields" => ["input_type", "offset"]},
      "priority" => 1,
    },
    "drop_event" => {
      "when"     => {"equals" => {"http.code" => 200}},
      "priority: => 2,
    },
  },
}

For more information please review the documentation here.

Processors in Hiera

Processors can be declared in hiera using the processors parameter. By default, hiera will not merge processor declarations down the hiera hierarchy. To change the behavior in puppet 3 use the processors_merge parameter. In puppet 4, you can use processors_merge, but can also use the lookup_options flag.

When processors_merge is set to true, processors will be replaced by the output of hiera_hash('filebeat::processors').

Reference

• Public Classes
  • Class: filebeat
• Private Classes
  • Class: filebeat::config
  • Class: filebeat::install
  • Class: filebeat::params
  • Class: filebeat::repo
  • Class: filebeat::service
  • Class: filebeat::install::linux
  • Class: filebeat::install::windows
• Public Defines
  • Define: filebeat::prospector
  • Define: filebeat::processors

Public Classes

Class: filebeat

Installs and configures filebeat.

Parameters within filebeat

• major_version: [String] The major version of filebeat to install. Should be either undef, 1, or 5. (default 5 if 1 not already installed)
• package_ensure: [String] The ensure parameter for the filebeat package If set to absent, prospectors and processors passed as parameters are ignored and everything managed by puppet will be removed. (default: present)
• manage_repo: [Boolean] Whether or not the upstream (elastic) repo should be configured or not (default: true)
• service_ensure: [String] The ensure parameter on the filebeat service (default: running)
• service_enable: [String] The enable parameter on the filebeat service (default: true)
• param repo_priority: [Integer] Repository priority. yum and apt supported (default: undef)
• service_provider: [String] The provider parameter on the filebeat service (default: on RedHat based systems use redhat, otherwise undefined)
• spool_size: [Integer] How large the spool should grow before being flushed to the network (default: 2048)
• idle_timeout: [String] How often the spooler should be flushed even if spool size isn't reached (default: 5s)
• publish_async: [Boolean] If set to true filebeat will publish while preparing the next batch of lines to transmit (defualt: false)
• registry_file: [String] The registry file used to store positions, must be an absolute path (default is OS dependent - see params.pp)
• config_file: [String] Where the configuration file managed by this module should be placed. If you think you might want to use this, read the limitations first. Defaults to the location that filebeat expects for your operating system.
• config_dir: [String] The directory where prospectors should be defined (default: /etc/filebeat/conf.d)
• config_dir_mode: [String] The permissions mode set on the configuration directory (default: 0755)
• config_file_mode: [String] The permissions mode set on configuration files (default: 0644)
• purge_conf_dir: [Boolean] Should files in the prospector configuration directory not managed by puppet be automatically purged
• outputs: [Hash] Will be converted to YAML for the required outputs section of the configuration (see documentation, and above)
• shipper: [Hash] Will be converted to YAML to create the optional shipper section of the filebeat config (see documentation)
• logging: [Hash] Will be converted to YAML to create the optional logging section of the filebeat config (see documentation)
• conf_template: [String] The configuration template to use to generate the main filebeat.yml config file
• download_url: [String] The URL of the zip file that should be downloaded to install filebeat (windows only)
• install_dir: [String] Where filebeat should be installed (windows only)
• tmp_dir: [String] Where filebeat should be temporarily downloaded to so it can be installed (windows only)
• use_generic_template: [Boolean] Use a more generic version of the configuration template. The generic template is more future proof (if types are correct), but looks very different than the example file (default: false)
• shutdown_timeout: [String] How long filebeat waits on shutdown for the publisher to finish sending events
• beat_name: [String] The name of the beat shipper (default: hostname)
• tags: [Array] A list of tags that will be included with each published transaction
• queue_size: [String] The internal queue size for events in the pipeline
• max_procs: [Number] The maximum number of CPUs that can be simultaneously used
• fields: [Hash] Optional fields that should be added to each event output
• fields_under_root: [Boolean] If set to true, custom fields are stored in the top level instead of under fields
• prospectors: [Hash] Prospectors that will be created. Commonly used to create prospectors using hiera
• prospectors_merge: [Boolean] If true, hiera_hash() will be used to build the the prospectors parameter (default: false)

Private Classes

Class: filebeat::config

Creates the configuration files required for filebeat (but not the prospectors)

Class: filebeat::install

Calls the correct installer class based on the kernel fact.

Class: filebeat::params

Sets default parameters for filebeat based on the OS and other facts.

Class: filebeat::repo

Installs the yum or apt repository for the system package manager to install filebeat.

Class: filebeat::service

Configures and manages the filebeat service.

Class: filebeat::install::linux

Install the filebeat package on Linux kernels.

Class: filebeat::install::windows

Downloads, extracts, and installs the filebeat zip file in Windows.

Public Defines

Define: filebeat::prospector

Installs a configuration file for a prospector.

Be sure to read the filebeat configuration details to fully understand what these parameters do.

Parameters for filebeat::prospector

• ensure: The ensure parameter on the prospector configuration file. (default: present)
• paths: [Array] The paths, or blobs that should be handled by the prospector. (required)
• exclude_files: [Array] Files that match any regex in the list are excluded from filebeat (default: [])
• encoding: [String] The file encoding. (default: plain)
• input_type: [String] log or stdin - where filebeat reads the log from (default:log)
• fields: [Hash] Optional fields to add information to the output (default: {})
• fields_under_root: [Boolean] Should the fields parameter fields be stored at the top level of indexed documents.
• ignore_older: [String] Files older than this field will be ignored by filebeat (default: 24h in filebeat < 1.2.0, infinite in filebeat >= 1.2.0)
• close_older: [String] Files that haven't been modified since close_older, they'll be closed. New modifications will be read when files are scanned again according to scan_frequency. Introduced in filebeat 1.2.0 (default: 1h)
• log_type: [String] (Deprecated - use doc_type) The document_type setting (optional - default: log)
• doc_type: [String] The event type to used for published lines, used as type field in logstash and elasticsearch (optional - default: log)
• scan_frequency: [String] How often should the prospector check for new files (default: 10s)
• harvester_buffer_size: [Integer] The buffer size the harvester uses when fetching the file (default: 16384)
• tail_files: [Boolean] If true, filebeat starts reading new files at the end instead of the beginning (default: false)
• backoff: [String] How long filebeat should wait between scanning a file after reaching EOF (default: 1s)
• max_backoff: [String] The maximum wait time to scan a file for new lines to ship (default: 10s)
• backoff_factor: [Integer] backoff is multiplied by this parameter until max_backoff is reached to determine the actual backoff (default: 2)
• force_close_files: [Boolean] Should filebeat forcibly close a file when renamed (default: false)
• include_lines: [Array] A list of regular expressions to match the lines that you want to include. Ignored if empty (default: [])
• exclude_lines: [Array] A list of regular expressions to match the files that you want to exclude. Ignored if empty (default: [])
• max_bytes: [Integer] The maximum number of bytes that a single log message can have (default: 10485760)
• json: [Hash] Options that control how filebeat handles decoding of log messages in JSON format See above. (default: {})
• multiline: [Hash] Options that control how Filebeat handles log messages that span multiple lines. See above. (default: {})

Define: filebeat::processor

Installs a configuration file for a processor.

Be sure to read the processor configuration details to fully understand what these parameters do.

Parameters for filebeat::processor

• ensure: The ensure parameter on the prospector configuration file. (default: present)
• priority: [Integer] Used to help alpha-numerically sort the processor files in the config dir location. (default: 10)
• processor_name: [String] The name of the processor. (default: $name)
• params: [Hash] The key-value pairs to pass to the processor as parameters. Not required on all processors, review documentation for more details. (default: undef)
• when: Optional[Hash] Run this processor on any event that matches these conditions. Populates the when option. (default: undef)

Limitations

This module doesn't load the elasticsearch index template into elasticsearch (required when shipping directly to elasticsearch).

When installing on Windows, there's an expectation that C:\Temp already exists, or an alternative location specified in the tmp_dir parameter exists and is writable by puppet. The temp directory is used to store the downloaded installer only.

Registry Path

The default registry file in this module doesn't match the filebeat default, but moving the file while the filbeat service is running can cause data duplication or data loss. If you're installing filebeat for the first time you should consider setting registry_file to match the default.

Be sure to include a path or the file will be put at the root of your filesystem.

Debian Systems

Filebeat 5.x and newer requires apt-transport-https, but this module won't install it for you.

Pre-1.9.1 Ruby

If you're on a system running a Ruby pre-1.9.1, hashes aren't sorted consistently, causing puppet runs to not be idempotent. To fix this, a limited template is used if the rubyversion is pre-1.9.1. The limited template only supports elasticsearch, logstash, file, and console outputs, and not all options may be supported (there is no warning when an option is omitted). Unlike with newer rubies, as new versions of filebeat are released, this template may not work until it's updated, even if you're using an updated configuration hash.

If you don't care about keeping puppet idempotent, this can be overridden by setting the conf_template parameter to 'filebeat/filebeat.yml.erb'.

See templates/filebeat.yml.ruby18.erb for the all of the details.

Using config_file

There are a few very specific use cases where you don't want this module to directly manage the filebeat configuration file, but you still want the configuration file on the system at a different location. Setting config_file will write the filebeat configuration file to an alternate location, but it will not update the init script. If you don't also manage the correct file (/etc/filebeat/filebeat.yml on Linux, C:/Program Files/Filebeat/filebeat.yml on Windows) then filebeat won't be able to start.

If you're copying the alternate config file location into the real location you'll need to include some metaparameters like

file { '/etc/filebeat/filebeat.yml':
  ensure  => file,
  source  => 'file:///etc/filebeat/filebeat.special',
  require => File['filebeat.yml'],
  notify  => Service['filebeat'],
}

to ensure that services are managed like you might expect.

Development

Pull requests and bug reports are welcome. If you're sending a pull request, please consider writing tests if applicable.