treasure-data / omnibus-td-agent

td-agent (Fluentd) Packaging Scripts

Notice: This repository is deprecated

omnibus-td-agent is for td-agent 2 and td-agent 3, no longer be updated.
For td-agent 4 or later, check fluent-package-builder repository.

About

The event collector daemon, for Treasure Data. This daemon collects various types of logs/events via various way, and transfer them to the cloud. For more about Treasure Data, see the official website, and the documentation.

td-agent is open sourced as Fluentd project. In other words, td-agent is a distribution package of Fluentd.

td-agent package is based on Omnibus

If you want to know the td-agent changelog, check td-agent changelog article.

Installation

We'll assume you have Ruby 2.2 and Bundler installed. First ensure all required gems are installed and ready to use:

$ bundle install --binstubs

Usage

Build

At first, you should download dependent gems using downloder. This is for avoding broken gem download and reduce the build time by avoiding internet access.

$ bin/gem_downloader core_gems.rb
$ bin/gem_downloader plugin_gems.rb

Create required directory and add permission

$ sudo mkdir -p /opt/td-agent /var/cache/omnibus
$ sudo chown [USER] /opt/td-agent
$ sudo chown [USER] /var/cache/omnibus

After that, you create a platform-specific package using the build project command:

$ bin/omnibus build td-agent3

The platform/architecture type of the package created will match the platform where the build project command is invoked. So running this command on say a MacBook Pro will generate a Mac OS X specific package. After the build completes packages will be available in pkg/.

Build td-agent 2.5

Add BUILD_TD_AGENT_VERSION=2_5 to each command.

$ BUILD_TD_AGENT_VERSION=2_5 bin/gem_downloader core_gems.rb
$ BUILD_TD_AGENT_VERSION=2_5 bin/gem_downloader plugin_gems.rb
$ BUILD_TD_AGENT_VERSION=2_5 bin/gem_downloader ui_gems.rb

And use td-agent2_5 instead of td-agent3 for omnibus command.

$ bin/omnibus build td-agent2_5

Build on CentOS 5

td-agent build doesn't work on CentOS 5 by OpenSSL related issues. Follow this setup instruction before use omnibus: https://gist.github.com/repeatedly/97d4746e83a5ec135abf3eb77f46ff30

Clean

You can clean up all temporary files generated during the build process with the clean command:

$ bin/omnibus clean

Adding the --purge purge option removes ALL files generated during the build including the project install directory (/opt/td-agent) and the package cache directory (/var/cache/omnibus/pkg):

$ bin/omnibus clean --purge

Help

Full help for the Omnibus command line interface can be accessed with the help command:

$ bin/omnibus help

Build with customized plugins, files and package name

Follow steps below to build packages with customized gem lists, configuration files and original package name.

1. make plugin list for your own environment

• add your_plugin_gems.rb to define the plugin list for your package
• rm plugin_gems/*
• bin/gem_downloader your_plugin_gems.rb

2. make your default configuration file

• edit templates/etc/td-agent/td-agent.conf and td-agent.conf.tmpl

3. copy config/projects/td-agent2.rb to config/projects/YOUR_PACKAGE_NAME.rb
4. edit config/projects/YOUR_PACKAGE_NAME.rb

• fix name, maintainer, homepage and description
• change install_dir as /opt/YOUR_PACKAGE_NAME
• change build_version and build_iteration
• comment out td and td-agent-ui if you want not to install them

5. build package by bin/omnibus build YOUR_PACKAGE_NAME
6. test your package file

Build script generates file paths with YOUR_PACKAGE_NAME, from templates. Leave file names with td-agent in templates.

NOTE: edit project_name in Vagrantfile if required.

Vagrant-based Virtualized Build Lab

td-agent omnibus ships will a project-specific Berksfile and Vagrantfile that will allow you to build your projects on the following platforms:

• CentOS 6.9 64-bit
• CentOS 7.2 64-bit
• Ubuntu 14.04 64-bit
• Ubuntu 16.04 64-bit
• Ubuntu 18.04 64-bit
• Debian 9.3 64-bit
• Debian 10.0 64-bit

NOTE: CentOS 8.0 is not supported because chef client has a problem. For CentOS8, need to build the package on CentOS8 environment.

Please note this build-lab is only meant to get you up and running quickly; there's nothing inherent in Omnibus that restricts you to just building CentOS or Ubuntu packages. See the Vagrantfile to add new platforms to your build lab.

The only requirements for standing up this virtualized build lab are:

• VirtualBox - native packages exist for most platforms and can be downloaded from the VirtualBox downloads page.
• Vagrant 1.2.1+ - native packages exist for most platforms and can be downloaded from the Vagrant downloads page.

The vagrant-berkshelf and vagrant-omnibus Vagrant plugins are also required and can be installed easily with the following commands:

$ vagrant plugin install vagrant-berkshelf
$ vagrant plugin install vagrant-omnibus
$ vagrant plugin install vagrant-vbguest

Exceute berks command to setup cookbooks

$ berks vendor cookbooks

Once the pre-requisites are installed you can build your package across all platforms with the following command:

$ vagrant up

If you would like to build a package for a single platform the command looks like this:

$ vagrant up PLATFORM

The complete list of valid platform names can be viewed with the vagrant status command.

Amazon Linux build

You need to install vagrant-aws 0.5.0 and add --provider option to vagrant up.

AWS_SSH_KEY_PATH=/path/to/your_aws_key_file vagrant up amazon --provider=aws

After build package, you need to copy rpm file from ec2 instances. No automatic sync for now.

pkg_build command

You can build all environments via bin/pkg_build command.

./bin/pkg_build

After that, each package is stored in td_agent2_pkg directory.

% ls td_agent2_pkg/
centos-6.7/   centos-7.2/   debian-7.10/  debian-8.4/   ubuntu-12.04/ ubuntu-14.04/ ubuntu-16.04/

Tested environment

% vagrant --version
Vagrant 1.7.4

% VBoxHeadless --version
Oracle VM VirtualBox Headless Interface 5.0.4
(C) 2008-2015 Oracle Corporation
All rights reserved.

5.0.4r102546

% ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

NOTE

Vagrant syncs current directory in each platform. Downloaded gems are also installed automatically.

And you should not use rbenv local in project root because Ruby environment is built on top of rbenv in Vagrant. So if you set different Ruby verion in .ruby-version, running ruby code will fail during pacakging process.

Kitchen-based Build Environment

Every Omnibus project ships will a project-specific Berksfile and Vagrantfile that will allow you to build your omnibus projects on all of the projects listed in the .kitchen.yml. You can add/remove additional platforms as needed by changing the list found in the .kitchen.yml platforms YAML stanza.

This build environment is designed to get you up-and-running quickly. However, there is nothing that restricts you to building on other platforms. Simply use the omnibus cookbook to setup your desired platform and execute the build steps listed above.

The default build environment requires Test Kitchen and VirtualBox for local development. Test Kitchen also exposes the ability to provision instances using various cloud providers like AWS, DigitalOcean, or OpenStack. For more information, please see the Test Kitchen documentation.

Once you have tweaked your .kitchen.yml (or .kitchen.local.yml) to your liking, you can bring up an individual build environment using the kitchen command.

$ bundle exec kitchen converge default-ubuntu-1204

Then login to the instance and build the project as described in the Usage section:

$ bundle exec kitchen login default-ubuntu-1204
[vagrant@ubuntu...] $ source load-omnibus-toolchain.sh
[vagrant@ubuntu...] $ cd omnibus-td-agent
[vagrant@ubuntu...] $ bundle install
[vagrant@ubuntu...] $ ...
[vagrant@ubuntu...] $ sudo install -d /opt/td-agent -o vagrant
[vagrant@ubuntu...] $ ./bin/omnibus build td-agent3

For a complete list of all commands and platforms, run kitchen list or kitchen help.

License

Treasure Agent package is released under the Apache2 license.

omnibus-td-agent's Issues

td-agent installation fails

hi

I attempted to install td-agent using:
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh | sh
on ubuntu Xenial. However, I get the following error:

---

Setting up td-agent (2.3.3-0) ...
Conffile /etc/td-agent/td-agent.conf has been modified. Remain untouched.
Job for td-agent.service failed because the control process exited with error code. See "systemctl status td-agent.service" and "journalctl -xe" for details.
invoke-rc.d: initscript td-agent, action "start" failed.
dpkg: error processing package td-agent (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
td-agent
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Sub-process /usr/bin/dpkg returned an error code (1)

Output of 'systemctl status td-agent.service':

● td-agent.service - LSB: data collector for Treasure Data
Loaded: loaded (/etc/init.d/td-agent; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2016-12-04 17:11:43 AEDT; 2min 3
Docs: man:systemd-sysv-generator(8)
Process: 5574 ExecStart=/etc/init.d/td-agent start (code=exited, status=1/FAIL

Dec 04 17:11:43 db.ufsrv systemd[1]: Starting LSB: data collector for Treasure D
Dec 04 17:11:43 db.ufsrv td-agent[5574]: Starting td-agent: start-stop-daemon: u
Dec 04 17:11:43 db.ufsrv td-agent[5574]: * td-agent
Dec 04 17:11:43 db.ufsrv systemd[1]: td-agent.service: Control process exited, c
Dec 04 17:11:43 db.ufsrv systemd[1]: Failed to start LSB: data collector for Tre
Dec 04 17:11:43 db.ufsrv systemd[1]: td-agent.service: Unit entered failed state
Dec 04 17:11:43 db.ufsrv systemd[1]: td-agent.service: Failed with result 'exit-

Your help is appreciated.

reagrds

Restarting td-agent for fine work

Hello!
We have strange issue with td-agents. They working fine in lxc containers (around 100)
Version of td-agent is 2.2.0-1 (we cant update right now, sorry)
When some of lxc restarting - td-agent inside is up, but it does not send data to remote servers (socket and logs sources)
In strace i can find strings like that:
futex(0x7f375dc2d644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f375dc2d640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
epoll_wait(87, {}, 64, 998) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 998) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 998) = 0
epoll_wait(87, {}, 64, 999) = 0
write(6, "!", 1) = 1
futex(0x7f375dc2d610, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f375dc2d640, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f375dc2d644, FUTEX_WAIT_PRIVATE, 69673, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f375dc2d67c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f375dc2d678, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f375dc2d610, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f375dc2d644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f375dc2d640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f375dc2d610, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 999) = 0
write(6, "!", 1) = 1
futex(0x7f375dc2d610, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f375dc2d640, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f375dc2d644, FUTEX_WAIT_PRIVATE, 69977, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f375dc2d67c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f375dc2d678, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f375dc2d610, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f375dc2d644, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f375dc2d640, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f375dc2d610, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_wait(87, {}, 64, 998) = 0
epoll_wait(87, {}, 64, 999) = 0
epoll_wait(87, {}, 64, 998) = 0

And i should restarting td-agent few times (1-2-3..10 times) to make it works... Can you help me with this issue?

Gem file compatibility issues are causing builds to fail

With a fresh rvm install with ruby 2.2.1 when running bundle install --binstubs the following errors come up.

Happens with release omnibus-td-agent v2.2.1 through to current and ruby-1.9.2-p330 through to 2.2.1, although the incompatible gems may change slightly.

Changed versions of td-agent with git checkout i.e

git checkout v2.2.1
bundle install --binstubs

Bundler could not find compatible versions for gem "celluloid":
   In Gemfile:
     berkshelf (~> 3.0) was resolved to 3.0.0, which depends on
       celluloid (~> 0.16.0.pre)

     berkshelf (~> 3.0) was resolved to 3.0.0, which depends on
       celluloid-io (~> 0.16.0.pre) was resolved to 0.16.5.pre0, which depends on
         celluloid (>= 0.17.0.pre12)
 Bundler could not find compatible versions for gem "multipart-post":
   In Gemfile:
     berkshelf (~> 3.0) was resolved to 3.0.0, which depends on
       ridley (~> 3.0) was resolved to 3.0.0, which depends on
         faraday (~> 0.9.0) was resolved to 0.9.2, which depends on
           multipart-post (< 3, >= 1.2)

     omnibus was resolved to 5.0.0, which depends on
       pedump (~> 0.5) was resolved to 0.5.0, which depends on
         multipart-post (~> 1.1.4)

Add /usr/sbin/td-agent-gem command

It is usable for td-agent users on RVM environment.

#!/bin/sh
export GEM_HOME="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/"
export GEM_PATH="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/"
/opt/td-agent/embedded/bin/fluent-gem "$@"

daemon start/stop behavior on install/remove

I wanted to open this topic for discussion - should the post-install/remove scripts start/stop the daemon?

Currently, for rpm only, post-install will restart a running td-agent (but won't start it on initial install). Otherwise (also for deb), no action is taken.

What are the best practices here? Should the scripts:

• start the daemon at install time? seems debatable.
• restart the daemon for deb as is done for rpm? keeping them consistent would be good.
• stop the daemon at uninstall time? does it make any sense to keep it running? is the answer the same if we didn't start it during install?

LSB dependency not working for CentOS

#70 added dependencies on the LSB functions for init.d scripts, but the logic does not work for CentOS platforms. In particular:

case ohai["platform"]
when "linux"
  case ohai["platform_family"]
  when "debian"
    dependency "lsb-base"
  when "rhel"
    dependency "initscripts"
    dependency "redhat-lsb"
  end
end

For CentOS, the value of ohai["platform"] is centos, so no dependencies are added. This means the redhat-lsb-core package needs to be manually installed for the generated td-agent init.d scripts to function (they rely on /lib/lsb/init-functions, which is provided by redhat-lsb-core).

Separately, I don't know where the redhat-lsb and initscripts software are defined. They are not in this code base nor do I see them in the omnibus-software repo. If I fix the logic in the td-agent.rb software to include the dependencies for CentOS, I get the following error:

I could not find a software named `initscripts' in any of the software locations:"

    /home/vagrant/omnibus-td-agent/config/software
    /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-software-46b5677a209d/config/software

LD_PRELOAD libjemalloc.so error

Hi.

The following error occurs while using fluentd. (start, stop and restart)

$ /etc/init.d/td-agent start
Starting td-agent: ERROR: ld.so: object '/opt/td-agent/embedded/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/opt/td-agent/embedded/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded: ignored.
                                                           [  OK  ]
ERROR: ld.so: object '/opt/td-agent/embedded/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded: ignored.

• OS: Centos 5.8
• Fluentd: http://packages.treasuredata.com/2/redhat/5/i386/td-agent-2.2.0-0.i686.rpm (install using rpm command)

Thank you.

td-agent 2.3.3

Update:

• fluentd v0.12.29
• jemalloc v4.2.1: #91

New:

• fluent-plugin-kafka

JSON parser for Elasticsearch 2.x

We are running into an issue where Elasticsearch 2.x now does not allow field names to use dots i.e (params.id). We use the json parser within td-agent to parse our json logs and it adds dots to the output when outputting nested items inside an object.

{ data: { attr1: 1, attr2: 2 } }
is outputted as
{ data.attr1: 1, data.attr2: 2 }

Could you assist in pointing me in the right direction to solve this. Possibly taking the current json parser built into td-agent and reworking it to add underscores instead of dots?

Where is the code for that default json parser in this repo?

Missing versions in apt repositories

Dear All,

I wanted to install an older version of td-agent on a Debian like system (to keep consistent boxes), but I wasn't able because in the apt repositories of treasure data the only available version is always the latest. (This is not the case in the yum repo.)

vagrant@vagrant-ubuntu-trusty-64:~$ sudo apt-get install td-agent=2.2.1-0*
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '2.2.1-0*' for 'td-agent' was not found

Can you check if it's possible to list all available versions in the apt repo as well?

Thank you!

cc @edsiper

compression during out_forward

I am looking for a option to compress (snappy/gzip) a message during out_forward. Could someone help me?

Do we still have "compress true" for out_forward plugin (fluent/fluentd#504)?

I am using td-agent2 and tried to use compress true but getting below error in logs.

[warn]: parameter 'compress' in is not used.

td-agent-gem install fails on fluent-plugin-graphite

hey guys,

I'm running into an issue when trying to install the fluent-plugin-graphite gem(0.0.6). FYI, i'm running Ubuntu 14.04.01 and td-agent version 2.3.0 and td-agent-gem version 2.2.1. Any thoughts? Thanks!

root@gateway-ubuntu-1404:~# td-agent-gem install fluent-plugin-graphite
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-graphite:
    ERROR: Failed to build gem native extension.

    /opt/td-agent/embedded/bin/ruby extconf.rb
checking for main() in -lcrypto... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/opt/td-agent/embedded/bin/ruby
    --with-ssl-dir
    --without-ssl-dir
    --with-ssl-include
    --without-ssl-include=${ssl-dir}/include
    --with-ssl-lib
    --without-ssl-lib=${ssl-dir}/lib
    --with-openssl-config
    --without-openssl-config
    --with-pkg-config
    --without-pkg-config
    --with-cryptolib
    --without-cryptolib
/opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:467:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:552:in `try_link0'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:567:in `try_link'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:747:in `try_func'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:978:in `block in have_library'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:923:in `block in checking_for'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in `block (2 levels) in postpone'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in `open'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in `block in postpone'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in `open'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:347:in `postpone'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:922:in `checking_for'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:973:in `have_library'
    from extconf.rb:8:in `block in check_libs'
    from extconf.rb:8:in `each'
    from extconf.rb:8:in `all?'
    from extconf.rb:8:in `check_libs'
    from extconf.rb:95:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/eventmachine-1.0.9.1 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/eventmachine-1.0.9.1/gem_make.out

Unable to build Deb package

Am building under Teamcity using the openjdk:8 base image with Ruby 2.1.8 and following the steps outlined.

The build fails with:

.rvm/gems/ruby-2.1.8@tdagent/bundler/gems/omnibus-software-d55178fb6d2a/config/software/openssl.rb:26:in evaluate': undefined method fips_mode?' for #<Omnibus::Software (Cleanroom)> (NoMethodError)

Any thoughts please?

Bundle elasticsearch and kafka plugin

We heard many people use these systems with fluentd/td-agent.
So bundle these plugins reduces deployment task in many environment.

TODO: fluent-plugin-kafka's input plugins might not work with kafka v0.9 or v0.10. So replacing poseidon with ruby-kafka is needed bofore bundle.

#DEBHELPER# tags are not expanded during deb build process

The postinst/prerm/postrm scripts all have a '#DEBHELPER#' tag, i.e.:

https://github.com/treasure-data/omnibus-td-agent/blob/master/templates/package-scripts/td-agent/deb/postinst#L107

These are supposed to be replaced by actual code by debhelper/dh_installdeb but this is not happening, so the tag ends up in the deb package verbatim (see below for example).

This leads to some issues:

• In postinst, the generated code runs update-rc.d which is what makes the agent start at boot time (via links in /etc/rc*.d), but this is not happening, so td-agent will not automatically run if the machine reboots.
• It should also do 'invoke-rc.d ... start' in postinst and 'invoke-rc.d stop' in prerm, which would appear to be the root cause of the behavior I reported in #14

$ curl -O 'http://packages.treasuredata.com.s3.amazonaws.com/2/debian/wheezy/pool/contrib/t/td-agent/td-agent_2.1.4-0_amd64.deb
$ dpkg-deb -R td-agent_2.1.4-0_amd64.deb .
$ tail DEBIAN/postinst 
if [ ! -e "/etc/default/td-agent" ]; then
  cat > /etc/default/td-agent <<EOF
# This file is sourced by /bin/sh from /etc/init.d/td-agent
# Options to pass to td-agent
DAEMON_ARGS=""

EOF
fi

#DEBHELPER#

systemd support

Check list:

• How to override environment variable:

  omnibus-td-agent/templates/etc/init.d/rpm/td-agent

  Line 15 in 77f2ae9

  TD_AGENT_NAME=<%= Shellwords.shellescape(project_name) %>

• How to set ulimit parameter:

  omnibus-td-agent/templates/etc/init.d/rpm/td-agent

  Line 190 in 77f2ae9

  ulimit -n 65536 1>/dev/null 2>&1 || true

• How to export LD_PRELOAD:

  omnibus-td-agent/templates/etc/init.d/rpm/td-agent

  Line 166 in 77f2ae9

  export LD_PRELOAD="${TD_AGENT_HOME}/embedded/lib/libjemalloc.so"

One fluentd example: https://gist.github.com/saidie/591157d2c9e306ff43d4

Failed to install fluent-plugin-elasticsearch

On Ubuntu 12.0.4 Precise

/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
WARN: Unresolved specs during Gem::Specification.reset:
      yajl-ruby (~> 1.0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-elasticsearch:
        ERROR: Failed to build gem native extension.
    /opt/td-agent/embedded/bin/ruby extconf.rb
checking for curl-config... no
checking for main() in -lcurl... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
Provided configuration options:
        --with-opt-dir
        --with-opt-include
        --with-curl-include
        --without-curl-include=${curl-dir}/include
        --with-curl-lib
        --without-curl-lib=${curl-dir}/lib
        --with-curllib
        --without-curllib
/opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:456:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:541:in `try_link0'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:556:in `try_link'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:742:in `try_func'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:973:in `block in have_library'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:918:in `block in checking_for'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in `block (2 levels) in postpone'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in `open'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in `block in postpone'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in `open'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:347:in `postpone'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:917:in `checking_for'
        from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:968:in `have_library'
        from extconf.rb:33:in `<main>'
extconf failed, exit code 1
Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/patron-0.4.18 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/patron-0.4.18/gem_make.out

$ td-agent --version
td-agent 0.10.50

Agent install scripts points to Trusty repo over plain HTTP

Hello!
If you open https://td-toolbelt.herokuapp.com/sh/install-ubuntu-trusty-td-agent2.sh you'll see that it runs curl against an HTTPS URL, but then sets up the APT repo with a plain HTTP URL. Is this intended? I guess the key is protected, so any man-in-the-middle attack could, at worse, only cause denial of service, but these days TLS overhead should be negligible.
Thanks in advance.

Missing AWS related certificate

In next td-agent, we should use latest cacert config which uses github's cacert, not curl.haxx.se.

Provide two packages

td-agent now bundles fluentd-ui to easy to configure.
But for production, fluentd-ui is not needed so providing fluentd-ui less td-agent is good for large scale deployment by reducing package size.

Support for Amazon Linux 2016.03

Amazon Linux 2013.03 has been released.
https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/

Please apply new vesion to install-redhat-td-agent2.sh script.
The script currently failed because http://packages.treasuredata.com/2/redhat/2016.03/x86_64/repodata/repomd.xml is not found.

Using digital ocean to build packages for Ubuntu 14.04

Currenlty, VirtualBox / Chef bento combination has a bug and it blocks automatic build for Ubuntu 14.04.
To avoid this problem, we may use digital ocean provider instead of VirtualBox.

https://github.com/smdahlen/vagrant-digitalocean

dry run failed: Operation not permitted

I am running td-agent on ubuntu trusty. I install via your script for ubuntu/trusty, which just started automatically pulling in 2.3.3-0. Suddenly I am seeing this through install phase and on service restart:

2016-10-03 18:33:14 +0000 [info]: starting fluentd-0.14.6 as dry run mode
2016-10-03 18:33:14 +0000 [error]: dry run failed: Operation not permitted
2016-10-03 18:33:23 +0000 [info]: fluent/supervisor.rb:636:read_config: reading config file path="/etc/td-agent/td-agent.conf"
2016-10-03 18:33:23 +0000 [info]: fluent/supervisor.rb:450:dry_run: starting fluentd-0.14.6 as dry run mode
2016-10-03 18:33:23 +0000 [error]: fluent/supervisor.rb:456:rescue in dry_run: dry run failed: Operation not permitted
2016-10-03 18:33:28 +0000 [info]: fluent/supervisor.rb:636:read_config: reading config file path="/etc/td-agent/td-agent.conf"
2016-10-03 18:33:28 +0000 [info]: fluent/supervisor.rb:450:dry_run: starting fluentd-0.14.6 as dry run mode
2016-10-03 18:33:28 +0000 [error]: fluent/supervisor.rb:456:rescue in dry_run: dry run failed: Operation not permitted
2016-10-03 18:45:06 +0000 [error]: dry run failed: Operation not permitted
2016-10-03 18:46:03 +0000 [error]: dry run failed: Operation not permitted

These "dry run failed: Operation not permitted" happen with each attempted service restart. I don't get much output from that configtest code, but it seems related to the following command and error output:

root@xxxxx:/etc/td-agent# /usr/sbin/td-agent --verbose --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid --user td-agent --group td-agent
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/privilege.rb:51:in change_privilege': Operation not permitted (Errno::EPERM) from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/privilege.rb:51:inchange'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:154:in block (2 levels) in daemonize_with_double_fork' from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:150:infork'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:150:in block in daemonize_with_double_fork' from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:142:infork'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:142:in daemonize_with_double_fork' from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:107:inmain'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/serverengine-2.0.0/lib/serverengine/daemon.rb:68:in run' from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:524:insupervise'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:402:in run_supervisor' from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.6/lib/fluent/command/fluentd.rb:269:in<top (required)>'
from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in require' from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:inrequire'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.14.6/bin/fluentd:5:in <top (required)>' from /opt/td-agent/embedded/bin/fluentd:23:inload'
from /opt/td-agent/embedded/bin/fluentd:23:in <top (required)>' from /usr/sbin/td-agent:7:inload'
from /usr/sbin/td-agent:7:in `

'

This is on a freshly provisioned ec2 host running their ubuntu 14.04. The exact same automated provisioning process works fine with 2.3.2-0. Was there a change introduced with 2.3.3-0 that would cause this?

Also is 2.3.2-0 no longer available? I was hoping to revert.. I can see the old version out there at: (key = 2/ubuntu/trusty/pool/contrib/t/td-agent/td-agent_2.3.2-0_amd64.deb) but can't get apt to see / install it. I could manually pull it down and install, but was hoping to avoid that.

Unable to install td-agent 3 : RedHat repo is not available

Following:

http://docs.fluentd.org/v0.14/articles/install-by-rpm

When trying to source the repo, a call to http://packages.treasuredata.com/3/redhat/7Server/x86_64/repodata/repomd.xml

results in:

<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist.</Message>
  <Key>3/centos/7Server/x86_64/repodata/repomd.xml</Key>
  <RequestId>F9895CA0A3E0BED3</RequestId>
  <HostId>
    miVG5bI1ZsEpw+f5QzGsh5mgjbOG4M4pBhOIUqQOf8J1PlrxKkqiIYqI59rfpG9q9GWCtsAFcaY=
  </HostId>
</Error>

unable to decompress the message which is read from kafka

I am doing a snappy compression in Java and sending the data to Kafka through fluentd but I am unable to read the original message from kafka since fluentd is sending the data as a string not as a byte[]. Any suggestion on this? what should be the configuration I should use in this case?

here is the fluentd configuration.

<match *.**>
  @type               kafka_buffered
  brokers             <broker1_host>:<broker1_port>,<broker2_host>:<broker2_port>
  default_topic      topic
  flush_interval      10s
  buffer_type        file
  buffer_path /app/logs
  buffer_chunk_limit 64m
  buffer_queue_limit 700
  disable_retry_limit true
  output_data_type json
  output_include_tag true
  output_include_time true
  max_send_retries 3
  required_acks 1
  ack_timeout_ms 1500
  compression_codec none
</match>

Issue Installing fluentd

Hi,

I'm trying to install fluentd on a CentOS box and I'm getting this error:

curl: (7) Failed connect to packages.treasuredata.com:443; Connection timed out
error: https://packages.treasuredata.com/GPG-KEY-td-agent: import read failed(2).
Loaded plugins: fastestmirror, langpacks
base | 3.6 kB 00:00
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/repodata/repomd.xml: [Errno 12] Timeout on http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/repodata/repomd.xml: (28, 'Connection timed out after 30179 milliseconds')

This happens when I run this command:
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

Can someone help me with this?

Thanks,

Rodaan

Custom package build fail

Hi, I try to build custom debian package with some plugins and also custom name to recogize it.

But build end with ackward

root@9a7373d2c11f (master):~/dev# bin/omnibus build sklik-td-agent2
                              [CLI] I | Using config from 'omnibus.rb'
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
Building sklik-td-agent 2.3.2...
            [Software: preparation] I | Resolving manifest entry for preparation
         [NullFetcher: preparation] I | Fetching `preparation' (nothing to fetch)
               [Software: jemalloc] I | Resolving manifest entry for jemalloc
                   [Software: zlib] I | Resolving manifest entry for zlib
                [Software: cacerts] I | Resolving manifest entry for cacerts
                 [Software: xproto] I | Resolving manifest entry for xproto
            [Software: util-macros] I | Resolving manifest entry for util-macros
           [Software: config_guess] I | Resolving manifest entry for config_guess
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
        [Software: pkg-config-lite] I | Resolving manifest entry for pkg-config-lite
             [Software: makedepend] I | Resolving manifest entry for makedepend
                [Software: openssl] I | Resolving manifest entry for openssl
                [Software: ncurses] I | Resolving manifest entry for ncurses
                [Software: libedit] I | Resolving manifest entry for libedit
                [Software: libtool] I | Resolving manifest entry for libtool
                 [Software: libffi] I | Resolving manifest entry for libffi
                [Software: libyaml] I | Resolving manifest entry for libyaml
               [Software: libiconv] I | Resolving manifest entry for libiconv
                   [Software: ruby] I | Resolving manifest entry for ruby
                [Software: liblzma] I | Resolving manifest entry for liblzma
                [Software: libxml2] I | Resolving manifest entry for libxml2
                [Software: libxslt] I | Resolving manifest entry for libxslt
               [Software: rubygems] I | Resolving manifest entry for rubygems
               [Software: nokogiri] I | Resolving manifest entry for nokogiri
            [NullFetcher: nokogiri] I | Fetching `nokogiri' (nothing to fetch)
             [Software: postgresql] I | Resolving manifest entry for postgresql
                [Software: fluentd] I | Resolving manifest entry for fluentd
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                       [GitFetcher] W | git fetch on a sha1 is not guaranteed to work
                       [GitFetcher] W | Specify a ref name instead of dc4e6d7dbd28f2c00123a395519114a81f280d31 on {:git=>"https://github.com/fluent/fluentd.git"}
              [GitFetcher: fluentd] I | Fetching from `https://github.com/fluent/fluentd.git'
               [Software: td-agent] I | Resolving manifest entry for td-agent
            [NullFetcher: td-agent] I | Fetching `td-agent' (nothing to fetch)
         [Software: td-agent-files] I | Resolving manifest entry for td-agent-files
      [NullFetcher: td-agent-files] I | Fetching `td-agent-files' (nothing to fetch)
             [Software: fluentd-ui] I | Resolving manifest entry for fluentd-ui
             [Software: fluentd-ui] W | Version 8a322e7716370a1e2fdf69fda3266a1be477f221 for software fluentd-ui was not parseable. Comparison methods such as #satisfies? will not be available for this version.
             [Software: fluentd-ui] W | Version 8a322e7716370a1e2fdf69fda3266a1be477f221 for software fluentd-ui was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                       [GitFetcher] W | git fetch on a sha1 is not guaranteed to work
                       [GitFetcher] W | Specify a ref name instead of 8a322e7716370a1e2fdf69fda3266a1be477f221 on {:git=>"https://github.com/fluent/fluentd-ui.git"}
           [GitFetcher: fluentd-ui] I | Fetching from `https://github.com/fluent/fluentd-ui.git'
                     [Software: td] I | Resolving manifest entry for td
                  [NullFetcher: td] I | Fetching `td' (nothing to fetch)
            [Software: td-agent-ui] I | Resolving manifest entry for td-agent-ui
         [NullFetcher: td-agent-ui] I | Fetching `td-agent-ui' (nothing to fetch)
       [Software: td-agent-cleanup] I | Resolving manifest entry for td-agent-cleanup
    [NullFetcher: td-agent-cleanup] I | Fetching `td-agent-cleanup' (nothing to fetch)
       [Software: version-manifest] I | Resolving manifest entry for version-manifest
    [NullFetcher: version-manifest] I | Fetching `version-manifest' (nothing to fetch)
         [GitFetcher: config_guess] I | Fetching from `git://git.sv.gnu.org/config.git'
            [Software: preparation] I | Restored from cache
               [Software: jemalloc] I | Restored from cache
                   [Software: zlib] I | Restored from cache
                [Software: cacerts] I | Restored from cache
                 [Software: xproto] I | Restored from cache
            [Software: util-macros] I | Restored from cache
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
           [Software: config_guess] I | Restored from cache
        [Software: pkg-config-lite] I | Restored from cache
             [Software: makedepend] I | Restored from cache
                [Software: openssl] I | Restored from cache
                [Software: ncurses] I | Restored from cache
                [Software: libedit] I | Restored from cache
                [Software: libtool] I | Restored from cache
                 [Software: libffi] I | Restored from cache
                [Software: libyaml] I | Restored from cache
               [Software: libiconv] I | Restored from cache
                   [Software: ruby] I | Restored from cache
                [Software: liblzma] I | Restored from cache
                [Software: libxml2] I | Restored from cache
                [Software: libxslt] I | Restored from cache
               [Software: rubygems] I | Restored from cache
               [Software: nokogiri] W | No version given! This is probably a bad thing. I am going to assume the version `0.0.0', but that is most certainly not your desired behavior. If git caching seems off, this is probably why.
               [Software: nokogiri] I | Restored from cache
             [Software: postgresql] I | Restored from cache
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                [Software: fluentd] I | Restored from cache
               [Software: td-agent] W | No version given! This is probably a bad thing. I am going to assume the version `0.0.0', but that is most certainly not your desired behavior. If git caching seems off, this is probably why.
               [Software: td-agent] I | Restored from cache
         [Software: td-agent-files] W | No version given! This is probably a bad thing. I am going to assume the version `0.0.0', but that is most certainly not your desired behavior. If git caching seems off, this is probably why.
         [Software: td-agent-files] I | Could not restore from cache
          [Builder: td-agent-files] I | Starting build
          [Builder: td-agent-files] I | <Dynamic Ruby block>: 0.0289s
          [Builder: td-agent-files] I | Build td-agent-files: 0.0296s
/opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1396:in `copy': unknown file type: /root/dev/resources/sklik-td-agent/etc (RuntimeError)
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:469:in `block in copy_entry'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1506:in `call'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1506:in `wrap_traverse'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:466:in `copy_entry'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:441:in `block in cp_r'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1579:in `block in fu_each_src_dest'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1593:in `fu_each_src_dest0'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1577:in `fu_each_src_dest'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:440:in `cp_r'
    from /root/dev/config/software/td-agent-files.rb:65:in `block (2 levels) in evaluate'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:1012:in `instance_eval'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:1012:in `run'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:843:in `block (3 levels) in execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:865:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:865:in `with_retries'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:842:in `block (2 levels) in execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `measure'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:841:in `block in execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:915:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:915:in `with_clean_env'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:840:in `execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:704:in `block (2 levels) in build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:703:in `each'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:703:in `block in build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `measure'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:702:in `build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/software.rb:1142:in `execute_build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/software.rb:1010:in `build_me'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/project.rb:1077:in `block in build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/project.rb:1076:in `each'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/project.rb:1076:in `build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/cli.rb:84:in `build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/cli/base.rb:33:in `dispatch'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/cli.rb:42:in `execute!'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/bin/omnibus:16:in `<top (required)>'
    from bin/omnibus:16:in `load'
    from bin/omnibus:16:in `<main>'

And I have just added ./config/projects/sklik-td-agent2.rb and change name, maintainer, homepage and description, and also add custom sklik_plugin_gems.rb

What is suspicious for me is resources folder name:

root@9a7373d2c11f (master):~/dev/resources# ls
sklik-sklik-td-agent  td-agent

Is there problem on my side? Thanks for any help.

Debian package for stretch/sid

Hi!
Would be great to dispose of a Strech/Sid package installer.
Can be evaluated and if proceed, included in the roadmap?

Thanks!

CentOS 5 build broken

Getting this error when executing vagrant up centos-5.11 using the v2.3.1 tag:

==> centos-5.11:  [NetFetcher: makedepend] I | 2016-05-11T07:07:23+00:00 | Downloading from `http://xorg.freedesktop.org/releases/individual/util/makedepend-1.0.5.tar.gz'
==> centos-5.11:
==> centos-5.11:
==> centos-5.11: Progress: |
==> centos-5.11: ===
==> centos-5.11: ==
==> centos-5.11: /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/openssl/ssl.rb:240:in `post_connection_check': hostname "www.x.org" does not match the server certificate (OpenSSL::SSL::SSLError)
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:925:in `connect'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:852:in `start'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:313:in `open_http'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:724:in `buffer_open'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:210:in `block in open_loop'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:208:in `catch'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:208:in `open_loop'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/core_extensions/open_uri.rb:51:in `open_uri'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:704:in `open'
==> centos-5.11:    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:34:in `open'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/download_helpers.rb:79:in `download_file!'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/fetchers/net_fetcher.rb:173:in `download'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/fetchers/net_fetcher.rb:86:in `fetch'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/software.rb:843:in `fetch'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/project.rb:1067:in `block (3 levels) in download'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:64:in `call'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:64:in `block (4 levels) in initialize'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:62:in `loop'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:62:in `block (3 levels) in initialize'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:61:in `catch'
==> centos-5.11:    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:61:in `block (2 levels) in initialize'
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

I can ssh to the VM reproduce the error using curl:

[vagrant@localhost ~]$ curl -v -I -L http://xorg.freedesktop.org/releases/individual/util/makedepend-1.0.5.tar.gz
* About to connect() to xorg.freedesktop.org port 80
*   Trying 131.252.210.176... connected
* Connected to xorg.freedesktop.org (131.252.210.176) port 80
> HEAD /releases/individual/util/makedepend-1.0.5.tar.gz HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: xorg.freedesktop.org
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Wed, 11 May 2016 07:15:59 GMT
Date: Wed, 11 May 2016 07:15:59 GMT
< Server: Apache/2.4.10 (Debian)
Server: Apache/2.4.10 (Debian)
< Location: https://www.x.org/releases/individual/util/makedepend-1.0.5.tar.gz
Location: https://www.x.org/releases/individual/util/makedepend-1.0.5.tar.gz
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1

* Connection #0 to host xorg.freedesktop.org left intact
* Issue another request to this URL: 'https://www.x.org/releases/individual/util/makedepend-1.0.5.tar.gz'
* About to connect() to www.x.org port 443
*   Trying 131.252.210.176... connected
* Connected to www.x.org (131.252.210.176) port 443
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server key exchange (12):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSL connection using DHE-RSA-AES128-SHA
* Server certificate:
*    subject: /CN=www.freedesktop.org
*    start date: 2016-05-06 14:40:00 GMT
*    expire date: 2016-08-04 14:40:00 GMT
* SSL: certificate subject name 'www.freedesktop.org' does not match target host name 'www.x.org'
* Closing connection #1
* SSLv3, TLS alert, Client hello (1):
curl: (51) SSL: certificate subject name 'www.freedesktop.org' does not match target host name 'www.x.org'
Closing connection #0

I can workaround the makedepend software problems by downloading the packages outside the VM and pointing the code to local paths, but this just leads to other SSL errors with other packages later in the build process:

/opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:923:in `connect': SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A (OpenSSL::SSL::SSLError)
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:923:in `block in connect'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:923:in `connect'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/net/http.rb:852:in `start'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:313:in `open_http'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:724:in `buffer_open'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:210:in `block in open_loop'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:208:in `catch'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:208:in `open_loop'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/core_extensions/open_uri.rb:51:in `open_uri'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:704:in `open'
    from /opt/rubies/ruby-2.1.8/lib/ruby/2.1.0/open-uri.rb:34:in `open'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/download_helpers.rb:79:in `download_file!'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/fetchers/net_fetcher.rb:173:in `download'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/fetchers/net_fetcher.rb:86:in `fetch'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/software.rb:843:in `fetch'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/project.rb:1067:in `block (3 levels) in download'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:64:in `call'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:64:in `block (4 levels) in initialize'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:62:in `loop'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:62:in `block (3 levels) in initialize'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:61:in `catch'
    from /home/vagrant/.bundler/ruby/2.1.0/bundler/gems/omnibus-b62510242ece/lib/omnibus/thread_pool.rb:61:in `block (2 levels) in initialize'

The CentOS 5 build was working for me previously, and I suspect the breakages are related to sites dropping SSLv3 support and/or updating updating their certs.

jemalloc is out of date and can cause runtime problems in systemd/journal

When using td-agent with the fluent-plugin-systemd package, td-agent will experience crashes when the underlying systemd-journal package is releasing memory that was allocated via LibFFI. To the user of td-agent, td-agent just stops running and the supervisor says it returned with a code 6.

The issue can be reproduced by installing the td-agent RPM on CentOS 7 along with the fluent-plugin-systemd package. However, the following script demonstrates the issue more succintly.

The issue can be reproduced via a script on CentOS 7 via the following:

1. Install the td-agent RPM ( 2.3.1-0.el7)
2. From a shell, execute export LD_PRELOAD="/opt/td-agent/embedded/lib/libjemalloc.so"
3. From a shell, execute the test script using the embedded ruby: /opt/td-agent/embedded/bin/ruby test.rb

Expected: The test script should output the systemd cursor string in an endless loop
Actual: The Ruby interpreter crashes with

*** Error in `/opt/td-agent/embedded/bin/ruby': munmap_chunk(): invalid pointer: 0x00007f5a903bee00 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7bc67)[0x7f5a97c8ac67]
/opt/td-agent/embedded/lib/libffi.so.6(ffi_call_unix64+0x4c)[0x7f5a8f5fdc6c]
/opt/td-agent/embedded/lib/libffi.so.6(ffi_call+0x2d5)[0x7f5a8f5fd775]
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/ffi-1.9.10/lib/ffi_c.so(rbffi_CallFunction+0x104)[0x7f5a8fdeab14]
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/ffi-1.9.10/lib/ffi_c.so(+0x12656)[0x7f5a8fdee656]
/opt/td-agent/embedded/lib/libruby.so.2.1(+0x1d8ac1)[0x7f5a98b01ac1]
/opt/td-agent/embedded/lib/libruby.so.2.1(+0x1cfb74)[0x7f5a98af8b74]
/opt/td-agent/embedded/lib/libruby.so.2.1(+0x1d3a3f)[0x7f5a98afca3f]
/opt/td-agent/embedded/lib/libruby.so.2.1(rb_iseq_eval_main+0x1f6)[0x7f5a98b04c86]
/opt/td-agent/embedded/lib/libruby.so.2.1(+0x7d8af)[0x7f5a989a68af]
/opt/td-agent/embedded/lib/libruby.so.2.1(ruby_exec_node+0x1d)[0x7f5a989a89cd]
/opt/td-agent/embedded/lib/libruby.so.2.1(ruby_run_node+0x1c)[0x7f5a989ab19c]
/opt/td-agent/embedded/bin/ruby[0x4008bb]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5a97c30af5]
/opt/td-agent/embedded/bin/ruby[0x4008e9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:00 271459960                          /opt/td-agent/embedded/bin/ruby
00600000-00601000 r--p 00000000 fd:00 271459960                          /opt/td-agent/embedded/bin/ruby
00601000-00602000 rw-p 00001000 fd:00 271459960                          /opt/td-agent/embedded/bin/ruby
0076e000-0078f000 rw-p 00000000 00:00 0                                  [heap]
7f5a89007000-7f5a89807000 r--s 00000000 fd:00 138420975                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system.journal
7f5a89807000-7f5a8a007000 r--s 00000000 fd:00 135658738                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@a3b031a0dd484280b6bc6f6ef0a5e906-0000000000041b5b-000532db6a8048d8.journal
7f5a8a007000-7f5a8a807000 r--s 00000000 fd:00 135600382                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@a3b031a0dd484280b6bc6f6ef0a5e906-000000000003566d-000530771e079b0b.journal
7f5a8a807000-7f5a8b007000 r--s 00000000 fd:00 135248892                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@a3b031a0dd484280b6bc6f6ef0a5e906-000000000002ec3a-00052e12d17d6921.journal
7f5a8b007000-7f5a8b807000 r--s 00000000 fd:00 135744131                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@a3b031a0dd484280b6bc6f6ef0a5e906-000000000001c594-00052bae84ce7aa9.journal
7f5a8b807000-7f5a8c007000 r--s 00000000 fd:00 137760851                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@a3b031a0dd484280b6bc6f6ef0a5e906-0000000000011633-00052949cedf5144.journal
7f5a8c007000-7f5a8c807000 r--s 00000000 fd:00 134937310                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@a3b031a0dd484280b6bc6f6ef0a5e906-0000000000000001-000526e58292cb4f.journal
7f5a8c807000-7f5a8d007000 r--s 00000000 fd:00 137781093                  /var/log/journal/3b716909b0d6446babff5862f5870a66/[email protected]~
7f5a8d007000-7f5a8d807000 r--s 00000000 fd:00 137662707                  /var/log/journal/3b716909b0d6446babff5862f5870a66/system@3ef64a015534415a8b3e84eef2f254b4-0000000000000001-00051ea2d53ba6bc.journal
7f5a8d807000-7f5a8d81c000 r-xp 00000000 fd:00 268592623                  /usr/lib64/libz.so.1.2.7
7f5a8d81c000-7f5a8da1b000 ---p 00015000 fd:00 268592623                  /usr/lib64/libz.so.1.2.7
7f5a8da1b000-7f5a8da1c000 r--p 00014000 fd:00 268592623                  /usr/lib64/libz.so.1.2.7
7f5a8da1c000-7f5a8da1d000 rw-p 00015000 fd:00 268592623                  /usr/lib64/libz.so.1.2.7
7f5a8da1d000-7f5a8da2c000 r-xp 00000000 fd:00 268592643                  /usr/lib64/libbz2.so.1.0.6
7f5a8da2c000-7f5a8dc2b000 ---p 0000f000 fd:00 268592643                  /usr/lib64/libbz2.so.1.0.6
7f5a8dc2b000-7f5a8dc2c000 r--p 0000e000 fd:00 268592643                  /usr/lib64/libbz2.so.1.0.6
7f5a8dc2c000-7f5a8dc2d000 rw-p 0000f000 fd:00 268592643                  /usr/lib64/libbz2.so.1.0.6
7f5a8dc2d000-7f5a8dc42000 r-xp 00000000 fd:00 268592706                  /usr/lib64/libelf-0.160.so
7f5a8dc42000-7f5a8de41000 ---p 00015000 fd:00 268592706                  /usr/lib64/libelf-0.160.so
7f5a8de41000-7f5a8de42000 r--p 00014000 fd:00 268592706                  /usr/lib64/libelf-0.160.so
7f5a8de42000-7f5a8de43000 rw-p 00015000 fd:00 268592706                  /usr/lib64/libelf-0.160.so
7f5a8de43000-7f5a8dea2000 r-xp 00000000 fd:00 268592610                  /usr/lib64/libpcre.so.1.2.0
7f5a8dea2000-7f5a8e0a2000 ---p 0005f000 fd:00 268592610                  /usr/lib64/libpcre.so.1.2.0
7f5a8e0a2000-7f5a8e0a3000 r--p 0005f000 fd:00 268592610                  /usr/lib64/libpcre.so.1.2.0
7f5a8e0a3000-7f5a8e0a4000 rw-p 00060000 fd:00 268592610                  /usr/lib64/libpcre.so.1.2.0
7f5a8e0a4000-7f5a8e0a8000 r-xp 00000000 fd:00 268592711                  /usr/lib64/libattr.so.1.1.0
7f5a8e0a8000-7f5a8e2a7000 ---p 00004000 fd:00 268592711                  /usr/lib64/libattr.so.1.1.0
7f5a8e2a7000-7f5a8e2a8000 r--p 00003000 fd:00 268592711                  /usr/lib64/libattr.so.1.1.0
7f5a8e2a8000-7f5a8e2a9000 rw-p 00004000 fd:00 268592711                  /usr/lib64/libattr.so.1.1.0
7f5a8e2a9000-7f5a8e2be000 r-xp 00000000 fd:00 268915671                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5a8e2be000-7f5a8e4bd000 ---p 00015000 fd:00 268915671                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5a8e4bd000-7f5a8e4be000 r--p 00014000 fd:00 268915671                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5a8e4be000-7f5a8e4bf000 rw-p 00015000 fd:00 268915671                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f5a8e4bf000-7f5a8e4ff000 r-xp 00000000 fd:00 268594543                  /usr/lib64/libdw-0.160.so
7f5a8e4ff000-7f5a8e6fe000 ---p 00040000 fd:00 268594543                  /usr/lib64/libdw-0.160.so
7f5a8e6fe000-7f5a8e700000 r--p 0003f000 fd:00 268594543                  /usr/lib64/libdw-0.160.so
7f5a8e700000-7f5a8e701000 rw-p 00041000 fd:00 268594543                  /usr/lib64/libdw-0.160.so
7f5a8e701000-7f5a8e717000 r-xp 00000000 fd:00 268592525                  /usr/lib64/libresolv-2.17.so
7f5a8e717000-7f5a8e917000 ---p 00016000 fd:00 268592525                  /usr/lib64/libresolv-2.17.so
7f5a8e917000-7f5a8e918000 r--p 00016000 fd:00 268592525                  /usr/lib64/libresolv-2.17.soAborted

Test script:

require 'systemd/journal'

j = Systemd::Journal.new
j.seek(:head)
j.move(1)

def get_cursor(j)
    j.cursor
end

while true do
  puts(get_cursor(j))
end

Workaround:
I downloaded the latest jemalloc (4.2.1) and ran the test script with LD_PRELOAD pointed at it and the issue did not manifest itself. I recommend that the embedded jemalloc be updated to a newer version.

deb package show Signature by key ... uses weak digest algorithm (SHA1)

Hi,

SHA1 is intended to shut off completely on January 1, 2017.
https://wiki.debian.org/Teams/Apt/Sha1Removal

When I install td-agent2 on ubuntu16.04, install script show the following warning.

Do you have any plan to use SHA2?

W: http://packages.treasuredata.com/2/ubuntu/xenial/dists/xenial/InRelease: Signature by key C901622B5EC4AF820C38AB861093DB45A12E206F uses weak digest algorithm (SHA1)

ubuntu@ip-172-31-2-176:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04 LTS
Release:    16.04
Codename:   xenial

ubuntu@ip-172-31-2-176:~$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   563  100   563    0     0    595      0 --:--:-- --:--:-- --:--:--   595
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   934  100   934    0     0   6962      0 --:--:-- --:--:-- --:--:--  6970
OK
Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://packages.treasuredata.com/2/ubuntu/xenial xenial InRelease [1,929 B]
Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Get:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease [92.2 kB]
Get:5 http://packages.treasuredata.com/2/ubuntu/xenial xenial/contrib amd64 Packages [398 B]
Get:6 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main Sources [868 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Hit:8 http://ppa.launchpad.net/staticfloat/julia-deps/ubuntu xenial InRelease
Get:9 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/restricted Sources [4,808 B]
Get:10 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/universe Sources [7,728 kB]
Get:11 http://ppa.launchpad.net/staticfloat/juliareleases/ubuntu xenial InRelease [17.5 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/main Sources [34.3 kB]
Get:13 http://ppa.launchpad.net/staticfloat/juliareleases/ubuntu xenial/main amd64 Packages [3,580 B]
Get:14 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [8,944 B]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [728 B]
Get:16 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [129 kB]
Get:17 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB]
Get:18 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [50.8 kB]
Get:19 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,201 kB]
Get:20 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main Translation-en [568 kB]
Get:21 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [38.6 kB]
Get:22 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [8,344 B]
Get:23 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/restricted Translation-en [2,908 B]
Get:24 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [23.0 kB]
Get:25 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,532 kB]
Get:26 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [1,176 B]
Get:27 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [628 B]
Get:28 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/universe Translation-en [4,354 kB]
Get:29 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB]
Get:30 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]
Get:31 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Sources [176 kB]
Get:32 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [64 B]
Get:33 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Sources [86.9 kB]
Get:34 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [3,220 B]
Get:35 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [362 kB]
Get:36 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [136 kB]
Get:37 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [64 B]
Get:38 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [64 B]
Get:39 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [313 kB]
Get:40 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [106 kB]
Get:41 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [5,488 B]
Get:42 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [2,428 B]
Get:43 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main Sources [1,400 B]
Get:44 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/restricted Sources [64 B]
Get:45 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe Sources [800 B]
Get:46 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/multiverse Sources [64 B]
Get:47 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [1,540 B]
Get:48 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [1,484 B]
Get:49 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/restricted amd64 Packages [64 B]
Get:50 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/restricted Translation-en [64 B]
Get:51 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [1,000 B]
Get:52 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [584 B]
Get:53 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/multiverse amd64 Packages [64 B]
Get:54 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/multiverse Translation-en [64 B]
Fetched 24.7 MB in 6s (3,968 kB/s)
Reading package lists... Done
W: http://packages.treasuredata.com/2/ubuntu/xenial/dists/xenial/InRelease: Signature by key C901622B5EC4AF820C38AB861093DB45A12E206F uses weak digest algorithm (SHA1)
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  td-agent
0 upgraded, 1 newly installed, 0 to remove and 179 not upgraded.
Need to get 64.0 MB of archives.
After this operation, 226 MB of additional disk space will be used.
Get:1 http://packages.treasuredata.com/2/ubuntu/xenial xenial/contrib amd64 td-agent amd64 2.3.2-0 [64.0 MB]
Fetched 64.0 MB in 7s (8,156 kB/s)
Selecting previously unselected package td-agent.
(Reading database ... 144008 files and directories currently installed.)
Preparing to unpack .../td-agent_2.3.2-0_amd64.deb ...
Unpacking td-agent (2.3.2-0) ...
Setting up td-agent (2.3.2-0) ...
Adding system user `td-agent' (UID 112) ...
Adding new group `td-agent' (GID 116) ...
Adding new user `td-agent' (UID 112) with group `td-agent' ...
Not creating home directory `/home/td-agent'.
Installing default conffile /etc/td-agent/td-agent.conf ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
W: --force-yes is deprecated, use one of the options starting with --allow instead.

Cannot use @ in field name in regex

For elasticsearch, I need to use the timestamp name as @timestamp. However, such regex is invalid:

(?<@timestamp>[^ ]*)

invalid char in group name <@timestamp>: /(?<@timestamp>[^ ]*)

Failed td-agent-gem install fluent-plugin-kinesis with td-agent 2.2.1 and 2.3.0

Hi,

I've been reported that my plugin install failed with td-agent 2.2.1.
awslabs/aws-fluent-plugin-kinesis#59

I tried with 2.2.1 and it was reproduced and also with 2.3.0 seems to have the same issue. Although, it worked with 2.3.1. So I recommend the user to upgrade to 2.3.1.

Is there any workaround for 2.2.1 or 2.3.0? Is this my plugin issue?

Below is the output.

# curl -O http://packages.treasuredata.com.s3.amazonaws.com/2/redhat/5/i386/td-agent-2.2.1-0.el5.i386.rpm
# rpm -ivh td-agent-2.2.1-0.el5.i386.rpm
# td-agent-gem install fluent-plugin-kinesis -i repo --no-rdoc --no-ri
WARN: Unresolved specs during Gem::Specification.reset:
json (>= 1.4.3)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Fetching: cool.io-1.4.3.gem (100%)
Building native extensions. This could take a while...
Successfully installed cool.io-1.4.3
Fetching: http_parser.rb-0.6.0.gem (100%)
Building native extensions. This could take a while...
Successfully installed http_parser.rb-0.6.0
Fetching: json-1.8.3.gem (100%)
Building native extensions. This could take a while...
Successfully installed json-1.8.3
Fetching: msgpack-0.7.4.gem (100%)
Building native extensions. This could take a while...
Successfully installed msgpack-0.7.4
Fetching: sigdump-0.2.4.gem (100%)
Successfully installed sigdump-0.2.4
Fetching: strptime-0.1.6.gem (100%)
Building native extensions. This could take a while...
Successfully installed strptime-0.1.6
Fetching: thread_safe-0.3.5.gem (100%)
Successfully installed thread_safe-0.3.5
Fetching: tzinfo-1.2.2.gem (100%)
Successfully installed tzinfo-1.2.2
Fetching: tzinfo-data-1.2016.3.gem (100%)
Successfully installed tzinfo-data-1.2016.3
Fetching: yajl-ruby-1.2.1.gem (100%)
Building native extensions. This could take a while...
Successfully installed yajl-ruby-1.2.1
Fetching: fluentd-0.14.0.pre.1.gem (100%)
Successfully installed fluentd-0.14.0.pre.1
Fetching: concurrent-ruby-1.0.1.gem (100%)
Successfully installed concurrent-ruby-1.0.1
Fetching: i18n-0.7.0.gem (100%)
Successfully installed i18n-0.7.0
Fetching: minitest-5.8.4.gem (100%)
Successfully installed minitest-5.8.4
Fetching: activesupport-5.0.0.beta3.gem (100%)
ERROR: Error installing fluent-plugin-kinesis:
activesupport requires Ruby version >= 2.2.2.

How to uninstall td-agent?

This is not documented anywhere.

Add td-agent-ui command to launch fluentd-ui

Like following command as /usr/sbin/td-agent-ui.

#!/bin/sh
export GEM_HOME="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/"
export GEM_PATH="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/"
export FLUENTD_UI_TITLE="Treasure Agent UI"
export FLUENTD_UI_BRAND="Treasure Agent"
export PATH=/opt/td-agent/embedded/bin:$PATH
/opt/td-agent/embedded/bin/fluentd-ui "$@" --pidfile /var/run/td-agent/td-agent-ui.pid

td-agent shuts down itself after a few minutes

My td-agent properly starts on boot (Ubuntu 14.04.4 LTS), but after a couple of minutes, it justs shut itself down? Is there a specific configuration to add to force it to stay alive?

2016-02-17 18:06:22 +0100 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-02-17 18:06:22 +0100 [info]: starting fluentd-0.12.20
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-s3' version '0.6.5'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-02-17 18:06:22 +0100 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-02-17 18:06:22 +0100 [info]: gem 'fluentd' version '0.12.20'
2016-02-17 18:06:22 +0100 [info]: adding match pattern="td.*.*" type="tdlog"
2016-02-17 18:06:22 +0100 [info]: adding match pattern="debug.**" type="stdout"
2016-02-17 18:06:22 +0100 [info]: adding match pattern="player-*.**" type="forward"
2016-02-17 18:06:22 +0100 [warn]: 'host' option in forward output is obsoleted. Use '<server> host xxx </server>' instead.
2016-02-17 18:06:22 +0100 [info]: adding forwarding server 'storage.carlipa.io:24224' host="storage.carlipa.io" port=24224 weight=60 plugin_id="object:3f81f977d3f4"
2016-02-17 18:06:22 +0100 [info]: adding source type="forward"
2016-02-17 18:06:22 +0100 [info]: adding source type="http"
2016-02-17 18:06:22 +0100 [info]: adding source type="debug_agent"
2016-02-17 18:06:22 +0100 [info]: using configuration file: <ROOT>
  <match td.*.*>
    type tdlog
    apikey xxxxxx
    auto_create_table 
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    <secondary>
      type file
      path /var/log/td-agent/failed_records
      buffer_path /var/log/td-agent/failed_records.*
    </secondary>
  </match>
  <match debug.**>
    type stdout
  </match>
  <source>
    type forward
  </source>
  <source>
    type http
    port 8888
  </source>
  <source>
    type debug_agent
    bind 127.0.0.1
    port 24230
  </source>
  <match player-*.**>
    type forward
    host storage.foobar.io
    flush_interval 3s
    port 24224
    <server>
      host storage.foobar.io
      port 24224
    </server>
  </match>
</ROOT>
2016-02-17 18:06:22 +0100 [info]: listening fluent socket on 0.0.0.0:24224
2016-02-17 18:06:22 +0100 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2016-02-17 18:10:15 +0100 [info]: shutting down fluentd
2016-02-17 18:10:15 +0100 [info]: shutting down input type="forward" plugin_id="object:3f81f96c70e0"
2016-02-17 18:10:15 +0100 [info]: shutting down input type="debug_agent" plugin_id="object:3f81f96c423c"
2016-02-17 18:10:15 +0100 [info]: shutting down input type="http" plugin_id="object:3f81f96c5bb4"
2016-02-17 18:10:15 +0100 [info]: shutting down output type="tdlog" plugin_id="object:3f81f6ae7fb0"
2016-02-17 18:10:15 +0100 [info]: shutting down output type="stdout" plugin_id="object:3f81f8cf1ebc"
2016-02-17 18:10:15 +0100 [info]: shutting down output type="forward" plugin_id="object:3f81f977d3f4"
2016-02-17 18:10:15 +0100 [info]: process finished code=0

# uname -a
Linux player-8cd9 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
# td-agent --version
td-agent 0.12.20

Problem with upgrading to v2.3.4

Hi there,

I'm experiencing issues trying to upgrade our version of td-agent to the latest version 2.3.4, which was apparently released on December 31, 2016.

Some system information:
OS: Ubuntu 12.04
td-agent Version: v2.3.4

Below is the output from running apt-get install td-agent:

11:08:40 Reading package lists...
11:08:44 Reading package lists...
11:08:44 Building dependency tree...
11:08:45 Reading state information...
11:08:45 The following packages will be upgraded:
11:08:45   td-agent
11:08:45 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
11:08:45 Need to get 0 B/54.6 MB of archives.
11:08:45 After this operation, 40.5 MB disk space will be freed.
11:08:45 (Reading database ... 99941 files and directories currently installed.)
11:08:45 Preparing to replace td-agent 2.3.3-0 (using .../td-agent_2.3.4-0_amd64.deb) ...
11:08:45 Stopping td-agent:  * td-agent
11:08:45 invoke-rc.d: initscript td-agent, action "stop" failed.
11:08:45 dpkg: warning: subprocess old pre-removal script returned error exit status 1
11:08:45 dpkg - trying script from the new package instead ...
11:08:45 Stopping td-agent:  * td-agent
11:08:45 invoke-rc.d: initscript td-agent, action "stop" failed.
11:08:45 dpkg: error processing /var/cache/apt/archives/td-agent_2.3.4-0_amd64.deb (--unpack):
11:08:45  subprocess new pre-removal script returned error exit status 1
11:08:45 No apport report written because MaxReports is reached already
11:08:45 Starting td-agent:  * td-agent
11:08:46 Errors were encountered while processing:
11:08:46  /var/cache/apt/archives/td-agent_2.3.4-0_amd64.deb
11:08:46 E: Sub-process /usr/bin/dpkg returned an error code (1)

Did some investigation, seems that when this happens invoke-rc.d has trouble stopping the process for whatever reason. So running invoke-rc.d td-agent stop produces the same error message to the console. The only way I could get around this was stopping the process with service or kill, then modifying /var/lib/dpkg/info/td-agent.prerm from this:

#!/bin/sh
set -e

if [ -x "/etc/init.d/td-agent" ]; then
        invoke-rc.d td-agent stop || exit $?
fi

to this:

#!/bin/sh
set -e

#if [ -x "/etc/init.d/td-agent" ]; then
#        invoke-rc.d td-agent stop || exit $?
#fi

And then running apt-get remove --purge td-agent and reinstalling.

Any help is appreciated, not entirely sure if this is an isolated issue or a widespread one. We haven't had any issues with upgrading in the past.

Travis

"Unresolved specs during Gem::Specification.reset" for yajl-ruby

I saw the following while using fluent-gem on Ubuntu 12.0.4 64 bit.

# /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-dstat
WARN: Unresolved specs during Gem::Specification.reset:
      yajl-ruby (~> 1.0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Fetching: fluent-mixin-rewrite-tag-name-0.1.0.gem (100%)
Successfully installed fluent-mixin-rewrite-tag-name-0.1.0
Fetching: fluent-plugin-dstat-0.2.5.gem (100%)
Successfully installed fluent-plugin-dstat-0.2.5
Parsing documentation for fluent-mixin-rewrite-tag-name-0.1.0
Installing ri documentation for fluent-mixin-rewrite-tag-name-0.1.0
Parsing documentation for fluent-plugin-dstat-0.2.5
Installing ri documentation for fluent-plugin-dstat-0.2.5
Done installing documentation for fluent-mixin-rewrite-tag-name, fluent-plugin-dstat after 0 seconds
2 gems installed

Is the warning something to be concerned about?

Could not generate td-agent RPM for linux

I have tried different approaches to generate td-agent 2 RPM but nothing worked out. Could you please provide some steps to generate RPM, it would be really great.

Thanks in advance. I appreciate your help.

failed to flush the buffer - Mongo_replset does not reconnect to replica set

We are getting the following error messages from time to time.
After the error message the output plugin out_mongo_replset does not reconnect to the replica set and stays in this stuck state until it's restarted. No new messages are saved in MongoDB.

td-agent[18558]: 2016-09-24 22:02:21 +0000 [warn]: Failed to connect to Replica Set. Try to retry: retry number = 1
td-agent[18558]: 2016-09-24 22:02:21 +0000 [warn]: retry succeeded. plugin_id="object:153dc74"
td-agent[18558]: 2016-09-24 22:09:52 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-09-24 22:09:33 +0000 error_class="NameError" error="uninitialized constant IRB::Abort" plugin_id="object:153dc74"
td-agent[18558]: 2016-09-24 22:09:52 +0000 [warn]: suppressed same stacktrace
td-agent[18558]: 2016-09-24 22:10:20 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-09-24 22:09:35 +0000 error_class="NameError" error="uninitialized constant IRB::Abort" plugin_id="object:153dc74"
td-agent[18558]: 2016-09-24 22:10:20 +0000 [warn]: suppressed same stacktrace
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-09-24 22:09:39 +0000 error_class="Mongo::OperationTimeout" error="Timed out waiting on socket read." plugin_id="object:153dc74"
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/networking.rb:336:in `rescue in receive_message_on_socket'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/networking.rb:330:in `receive_message_on_socket'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/networking.rb:191:in `receive_header'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/networking.rb:182:in `receive'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/authentication.rb:446:in `auth_command'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/authentication.rb:409:in `issue_scram'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/authentication.rb:225:in `issue_authentication'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:321:in `block in check_auths'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/set.rb:263:in `each_key'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/set.rb:263:in `each'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:320:in `check_auths'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:285:in `block (2 levels) in checkout'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:267:in `synchronize'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:267:in `block in checkout'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:260:in `loop'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/connection/pool.rb:260:in `checkout'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/mongo_replica_set_client.rb:413:in `get_socket_from_pool'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/mongo_replica_set_client.rb:371:in `block in checkout_reader'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/mongo_replica_set_client.rb:354:in `checkout'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/mongo_replica_set_client.rb:369:in `checkout_reader'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/cursor.rb:623:in `checkout_socket_from_connection'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/cursor.rb:550:in `block in send_initial_query'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:55:in `block in instrument'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:20:in `instrument'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:54:in `instrument'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/cursor.rb:547:in `send_initial_query'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/cursor.rb:532:in `refresh'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/cursor.rb:139:in `next'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/db.rb:607:in `command'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:353:in `block in batch_message_send'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:55:in `block in instrument'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:20:in `instrument'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/functional/logging.rb:54:in `instrument'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:352:in `batch_message_send'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:82:in `block in batch_write_incremental'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:57:in `catch'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection_writer.rb:57:in `batch_write_incremental'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection.rb:1184:in `batch_write'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.5/lib/mongo/collection.rb:411:in `insert'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-mongo-0.7.12/lib/fluent/plugin/out_mongo.rb:154:in `operate'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-mongo-0.7.12/lib/fluent/plugin/out_mongo_replset.rb:45:in `block in operate'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-mongo-0.7.12/lib/fluent/plugin/out_mongo_replset.rb:61:in `rescue_connection_failure'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-mongo-0.7.12/lib/fluent/plugin/out_mongo_replset.rb:44:in `operate'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-mongo-0.7.12/lib/fluent/plugin/out_mongo.rb:133:in `write'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:345:in `write_chunk'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:324:in `pop'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:329:in `try_flush'
td-agent[18558]: 2016-09-24 22:10:40 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:140:in `run'
systemd[1]: Stopping td-agent...

td-agent-gem install fails for fluent-plugin-elasticsearch

/usr/sbin/td-agent-gem install fluent-plugin-elasticsearch

Fetching: patron-0.4.18.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-elasticsearch:
    ERROR: Failed to build gem native extension.

        /opt/td-agent/embedded/bin/ruby extconf.rb
checking for curl-config... yes
checking for rb_thread_blocking_region()... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/opt/td-agent/embedded/bin/ruby
    --with-curl-dir
    --without-curl-dir
    --with-curl-include
    --without-curl-include=${curl-dir}/include
    --with-curl-lib
    --without-curl-lib=${curl-dir}/lib
/opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:456:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:541:in `try_link0'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:556:in `try_link'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:742:in `try_func'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:1027:in `block in have_func'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:918:in `block in checking_for'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in `block (2 levels) in postpone'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in `open'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in `block in postpone'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in `open'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:347:in `postpone'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:917:in `checking_for'
    from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:1026:in `have_func'
    from extconf.rb:47:in `<main>'


Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/patron-0.4.18 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/patron-0.4.18/ext/patron/gem_make.out

I believe this happens with any gem that requires a native extension?

ver 2.3.0 type copy can't use with tdlog

I try this config.
but [warn] section is not used in <match **> of copy plugin

  <match **>
    type copy
    <store>
      @type tdlog
      endpoint ybi.jp-east.idcfcloud.com
      apikey xxxxxx
      database hiroba
      table act
      auto_create_table true
      buffer_type file
      buffer_path /var/log/td-agent/buffer/hoge
      num_threads 4
      include_time_key true
      use_ssl true
    </store>
    <store>
      @type file
      path /var/log/td-agent/hoge
      time_slice_format %Y%m%d
      time_slice_wait 10m
      time_format %Y%m%dT%H%M%S%z
      timezone Asia/Tokyo
      compress gzip
      buffer_path /var/log/td-agent/hoge.*
    </store>
  </match>

td-agent 3.0

Update components:

• ruby 2.4
• fluentd v0.14
• other core libraries

New plugins:

• fluent-plugin-elasticsearch (after merge optimized patch)

New support OS:

• Windows

Drop OS:

• CentOS 5
• Ubuntu 10.04

Drop plugins:

• fluent-plugin-scribe
• fluent-plugin-mongo

permission error on td-agent2

OS: CentOS 6.2

Installing td-agent2 as

$ curl -L http://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

The permission of gems directory becomes like

ls -ld /opt/td-agent/embedded/lib/ruby/gems/2.1.0/
drwxrwxr-x 8 root root 4096  3月 24 17:17 2015 /opt/td-agent/embedded/lib/ruby/gems/2.1.0/

So, td-agent-gem install fails as followings:

$ td-agent-gem install fluent-plugin-bigquery
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /opt/td-agent/embedded/lib/ruby/gems/2.1.0/ directory.

I can add sudo, but it will install gems as root. Writing /etc/td-agent/td-agent.conf as

<match **>
  type bigquery
</match>

results in an error as below because td-agent user can not read gems

Starting td-agent: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in `require': cannot load such file -- google/api_client (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:144:in `require'

because it's permission (and permissions of all files in the gem) is set as

$ ls -l /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.8.3/lib/google/api_client.rb
-rw-r----- 1 root root 27249  3月 24 18:19 2015 api_client.rb

Changing owner of directories makes succeed:

sudo chown td-agent:td-agent -R  /opt/td-agent/embedded/lib/ruby/gems/2.1.0/

Investigate optimizing the size of the td-agent package?

Hi, I have noticed the td-agent package is quite large at ~75MB, and when installed it consumes ~200MB in /opt. I did a brief investigation, here are some notes (all of these are based on the installed size):

• none of the binaries/libraries are stripped - doing this saves about 50MB.
• could we remove the directory ruby/gems/2.1.0/cache? That is 28MB.
• is it necessary to preserve the static .a libraries? This is less of a gain after stripping but still 8MB.
• The 'td' gem includes ruby-2.0.0-p0.pkg which is 13MB - is that needed? I see some other large files packaged by the gems that look like they may be unnecessary, although these might be harder to remove:
  • ohai-6.20.0/spec/unit/plugins/darwin/system_profiler_output.rb (~3MB)
  • httpclient-2.4.0/sample/api_rest.pdf (~2MB)
  • td-0.11.8/data/sample_apache.json (~2MB)
  • nokogiri-1.6.3.1/test/files/slow-xpath.xml (~2MB)

If we are able to make all of those changes, the size of the installed /opt/td-agent is reduced from 206MB to 97MB. I didn't build a package but as an approximation I tar/gzipped the directory which was reduced in size from 76MB to 18MB (!).

This isn't an urgent problem but it would be nice to reduce the size.

Service Unable to Start with google-api-ruby-client Gem Version 0.9.22 Installed

This is with v2.3.4-0 of td-agent installed on Ubuntu 14.04:

/opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require': cannot load such file -- /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/google-api-client-0.9.22/generated/google/apis/bigquery_v2.rb (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:69:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-bigquery-0.3.4/lib/fluent/plugin/out_bigquery.rb:165:in `initialize'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:132:in `new'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:132:in `new_impl'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:59:in `new_output'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/agent.rb:131:in `add_match'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/agent.rb:64:in `block in configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/agent.rb:57:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/agent.rb:57:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:86:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:129:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:103:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:489:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:160:in `block in start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:366:in `call'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:366:in `main_process'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:339:in `block in supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `fork'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:156:in `start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.31/bin/fluentd:5:in `<top (required)>'
        from /opt/td-agent/embedded/bin/fluentd:23:in `load'
        from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
        from /usr/sbin/td-agent:7:in `load'
        from /usr/sbin/td-agent:7:in `<main>'

Downgrading the google-api-ruby-client gem to 0.9.20 resolves the issue.

liblzma 5.0.5/5.1.2 are no longer available

see chef/omnibus-software#504

This is an additional issue for td-agent since we force 5.1.2alpha on centos7. So, I think we have two options:

• Change the default version to 5.0.8 and change the centos7 version to 5.2.2.
• Just change to 5.2.2 for all platforms

Let me know what you think and I'll send a PR... We can also wait to see what the chef/omnibus folks think, if they decide to go with 5.2.2 then we can just remove the customizations here.

support for changing the name of the daemon

We discussed this topic with Kiyoto earlier in the week; I would like to investigate whether it's possible to support changing the name of the daemon more directly.

Our current approach is to fork this repo (https://github.com/GoogleCloudPlatform/google-fluentd), and manually replace 'td-agent' with 'google-fluentd' both in file contents and file names. Merging upstream changes always leads to a lot of conflicts; resolving them is labor-intensive and error-prone.

Ideally the name of the daemon could be specified once (either in a file or on the command line when building) and the scripts would take care of all the string/filename substitutions during packaging.

Add LICENSE file

T/O

td-agent rpm package resets /opt permissions

When installing td-agent from official rpm packages (packages.treasuredata.com), /opt permissions are set to 775. Apart from this not being cool, this prevents clean install on at least on amazon linux, where yum fails because of permissions conflict. I suspect this is related to omnibus build scripts, but I haven't been able to find the cause.

Tested on CentOS 6.5 and Amazon linux 2014.09 with td-agent-2.1.0-0.x86_64.rpm