Agent Tailer cannot find any records about amazon-kinesis-agent

[asafcombo]

For anybody interested : one cause of this is that you need 755 permissions on the entire path.
so if you tail logs that are on /home/ec2-user/something you need 755 for /home for ec2-user etc'

[sarah-story]

I had to touch each of the files I wanted it to stream, even though I designated START_OF_FILE as the initial position.

[cf1497]

And here is the logs for the agent:

2016-07-15 11:20:49.296-0500 axil7 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 810029ms
2016-07-15 11:21:19.295-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/.json].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/.json]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-07-15 11:21:19.296-0500 axil7 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 840029ms

[chaochenq]

Hi @cf1497 looking at your log, the FileTailer was tailing this file:

/usr2/cheng/tomahawk-server/logs/.json

which seems invalid.

Could you verify your filePattern in the config file? Or double check files under that directory?

[cf1497]

Hi @chaochenq , thanks for your respond.

I just double checked the fileParttern, it is "/*.json", and then I replaced it with a specific file name under that directory, however the problem still happened.

Thanks!

[chaochenq]

If there's a star in the config file, the FileTailer should look like FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/*.json] rather than FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/.json]

Did you do a restart?

[cf1497]

Hi @chaochenq ,

I restarted it several times, and I double checked the logs, it has a start in the config file.

2016-07-15 12:18:43.556-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/.json].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/.json]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-07-15 12:18:43.557-0500 axil7 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 2640029ms

Thanks！

[chaochenq]

That's weird... just by looking at your log, it looks like the Agent is ignoring the star in the configuration. You may want to add some debug messages here:

https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/SourceFile.java#L46

[cf1497]

It is very weird, I can see the start in my config file and log file, but not here.
Thanks, I will add some debug messages to see what happened.

Here is the log that I used a specific file name :

2016-07-15 12:20:06.540-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-13.json].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-13.json]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.

[chaochenq]

Ok a few questions:

1. Are you actively writing data to the file (tomahawk_2016-07-13.json)
2. If not, you may want to add "initialPosition" : "START_OF_FILE" to the config. Otherwise agent will always tail from the end by default.

See http://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html#config-start

[cf1497]

Hi,

1. I double checked the file, it has data in it, and the data looks like:
   {"timestamp":"......", "message":"......"}
2. I tried to add "initialPosition":"START_OF_FILE", however it didn't help.

Thanks!

[chaochenq]

You can add "log.level" : "DEBUG" or "log.level" : "TRACE" to see if the tail moves forward, or any state change that breaks the tail.

[cf1497]

Hi,

I just changed the log.level to Trace, then I got these logs:

2016-07-15 13:16:10.439-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [DEBUG] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: No checkpoints were found. Looking for new files to tail...

2016-07-15 13:16:10.443-0500 axil7 (AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisherService [DEBUG] AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: Starting up...

2016-07-15 13:16:10.443-0500 axil7 (AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisherService [TRACE] AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: Main loop started

2016-07-15 13:16:10.443-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json] is refreshing current tailed file.

2016-07-15 13:16:10.443-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.

Thanks!

[chaochenq]

Only these are not sufficient for me to troubleshoot. I'd recommend going over the log more and tracking the tailing progress.

[cf1497]

Hi,

I just restart it, and here is the log:

2016-07-15 13:28:56.773-0500 axil7 (main) com.amazon.kinesis.streaming.agent.Agent [INFO] Reading configuration from file: /etc/aws-kinesis/agent.json
2016-07-15 13:28:56.807-0500 axil7 (main) com.amazon.kinesis.streaming.agent.Agent [INFO] null: Agent will use up to 288 threads for sending data.
2016-07-15 13:28:56.816-0500 axil7 (main) com.amazon.kinesis.streaming.agent.tailing.checkpoints.SQLiteFileCheckpointStore [DEBUG] Connecting to database /var/run/aws-kinesis-agent/checkpoints...
2016-07-15 13:28:56.942-0500 axil7 (main) com.amazon.kinesis.streaming.agent.tailing.checkpoints.SQLiteFileCheckpointStore [INFO] Deleted 0 old checkpoints.
2016-07-15 13:28:56.954-0500 axil7 (Agent STARTING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Starting up...
2016-07-15 13:28:56.961-0500 axil7 (Agent STARTING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Starting tailer for file fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json
2016-07-15 13:28:56.978-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [DEBUG] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: Starting up...
2016-07-15 13:28:56.979-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [DEBUG] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: No checkpoints were found. Looking for new files to tail...
2016-07-15 13:28:56.982-0500 axil7 (AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisherService [DEBUG] AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: Starting up...
2016-07-15 13:28:56.983-0500 axil7 (AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.AsyncPublisherService [TRACE] AsyncPublisher[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: Main loop started
2016-07-15 13:28:56.983-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json] is refreshing current tailed file.
2016-07-15 13:28:56.983-0500 axil7 (Agent STARTING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Startup completed in 28 ms.
2016-07-15 13:28:56.983-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.
2016-07-15 13:28:57.084-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.
2016-07-15 13:28:57.184-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.
2016-07-15 13:28:57.285-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.
2016-07-15 13:28:57.385-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.
2016-07-15 13:28:57.486-0500 axil7 (FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]) com.amazon.kinesis.streaming.agent.tailing.FileTailer [TRACE] FileTailer[fh:Tomahawk:/usr2/cheng/tomahawk-server/logs/tomahawk_2016-07-15_0.json]: There's no file being tailed.

Thanks!

[chaochenq]

Did you grant aws-kinesis-agent-user r+x permission for /usr2/cheng/tomahawk-server/logs? It needs x permission to list the files and determine which file to tail based on the latestModificationTimestamp.

[cf1497]

Hi,

I double checked the permission of that file and logs directory.

-rwx------ 1 aws-kinesis-agent-user aws-kinesis-agent-user 1025 Jul 15 13:14 tomahawk_2016-07-15_0.json

drwx------ 3 aws-kinesis-agent-user aws-kinesis-agent-user 4096 Jul 15 13:14 logs

Thanks!

[chaochenq]

You can dive through the code from here:

https://github.com/awslabs/amazon-kinesis-agent/blob/master/src/com/amazon/kinesis/streaming/agent/tailing/FileTailer.java#L237

[cf1497]

Hi, @chaochenq

Thanks for your help. I still have a question:

My linux version is 6.6, I am not sure if it will cause that problem (cannot find local files)?

Thanks!

[chaochenq]

Which distribution is that? RedHat or Ubuntu or something else?

[DiptiNemade]

I am having the same issue. 0 records sent successfully. No error in the log file. But s3 bucket is empty. Did you have a resolution on this topic. I have spent 3 days researching on this topic and no solution. Where do you want me to add log.level? I dont have any template to compare where exactly and in which file(path) this needs to go?

[nimboya]

Still having this issues, anybody with the solution on this ?

[Scapeuce]

I just did what asafcombo commented above and my problem was solved.

[vicks316]

Hi All,

For all those who stil haven't got it, please follow the steps below which worked for me

1. In the agent.json file, add "filePattern": "/tmp/_access_log*.log". Note that the agent will look for files in the /tmp/ library since it can't look for files in the ec2-user folder
2. Run the Python Fake log generator file using the command below
   python apache-fake-log-gen.py -n 0 -o LOG -p /tmp/

Please let me know if this does not work for you.