Installing logstash-filter-tld plugin for newer Logstash versions

The logstash tld plugin (logstash-filter-tld) is used to extract the individual labels of a fqdn. This is great for analysis of e.g. domains from a Domain Generation Algorithm (DGA). Unfortunately this plugin does not work on newer logstash versions. (See https://github.com/logstash-plugins/logstash-filter-tld/pull/10)
This is caused by java version requirements. To make it work, all that is needed is changing that requirement, then (re)building the gem. This is a few simple steps (done on Debian below - substitute with your distros package manager).
On the development machine:

• apt-get -y install jruby bundler
• gem install bundler
• git clone https://github.com/logstash-plugins/logstash-filter-tld.git
• cd logstash-filter-tld/
• nano (or vi) logstash-filter-tld.gemspec
• change logstash-filter-tld.gemspec, '=1.4.6' '' to logstash-filter-tld.gemspec, '~> 3'
• gem build logstash-filter-tld.gemspec

The finalized plugin is now saved as logstash-filter-tld-3.0.3.gem. Copy that to your logstash instances, then install it using:

• /usr/share/logstash/bin/logstash-plugin install --no-verify ./logstash-filter-tld-3.0.3.gem

For the logstash .conf-file(s) here's an example using the tld plugin:

filter {        

   tld {

        source => "computer_name"

    }


    mutate {

        rename => { "[tld][domain]" => "highest_registered_domain" }

        rename => { "[tld][trd]" => "sub_domain" }

        rename => { "[tld][tld]" => "top_level_domain" }

        rename => { "[tld][sld]" => "parent_domain" }

    }

}

Take a look at these logstash configurations too: https://github.com/HASecuritySolutions/Logstash

This post was first published on Peerlyst: https://www.peerlyst.com/posts/installing-logstash-filter-tld-plugin-for-newer-logstash-versions-martin-boller