rittmanmead / obi-enhanced-usage-tracking Goto Github PK

Version 5 of the Elastic stack was released in late 2016, and it would be good to test this code with it and update the README section above to indicate if it works - or submit the required changes needed for it to do so.

At the moment only clicks in Presentation Services are captured and analysed. I bet the same can be done for Data Visualization/Visual Analyzer too ...

Hi! :)

First, I want to thank you for this excellent solution. It helps me a lot.

Second, I need to see whole URL's some user have run from OBIEE portal. I found in "rm_http*.log" that data I need is stored at "Referer" and "Page" tags. How can I "forward" this part of logs to CSV and after that, to DB table?

Thank you in advance for help!

Best regards from Croatia,
Drazen

Click events have defined 'Request' types, and these I have roughly grouped together into 'Request Groups' to help describe what the user was doing (e.g. Logon / Edit Report / Run Report). Not all requests have been assigned to request groups. It would be useful to identify all request types, and refine further the groupings.

I am getting this error running the setup sql file:
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
error opening file /apps/obia12c/obi-clickdata/http_requests.log

Here are my steps:

1. Install the following rpms on the server with the presentation server logs

• oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64.rpm
• oracle-instantclient11.2-sqlplus-11.2.0.4.0-1.x86_64.rpm

2. Go to /usr/lib/oracle/11.2/client64/bin and run
   ./sqlplus sys/@BIFSCDEV12C as sysdba

Am I missing something with my approach?

At the moment the scripts are manual to invoke and run. It would be neat to package this up into a service (or set of services) that could run automagically at server boot.

Until then, using GNU screen is a handy hack for setting scripts running and being able to disconnect from the server without terminating them. It's like using nohup ... &, except you can reconnect to the session itself as and when you want to.

There's lots of possibilities for this data. Auditing who did what, when, is useful (e.g. who deleted a report?). Taking it a step further, are there patterns in user behaviour? Certain patterns of clicks that could be identified to highlight users who are struggling to find the data that they want? For example, opening lots of presentation folders in the Answers editor before adding columns to the analysis? Can we extend that to identify users who are struggling to use the tool and are going to "churn" (stop using it) and thus contact them before they do so to help resolve any issues they have?

Hi Robin, This is a nice work. Thanks much. I am trying to set this up. Do we need to keep the logstash running? If I close this the click data is not writing anymore. How to keep this running in background?
Please help. thanks.

$ ./opt/logstash/bin/logstash -f home/oracle/obi-enhanced-usage-tracking/logstash-obi-clickstream.conf
Settings: Default pipeline workers: 2
Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads {:count_was=>2, :filters=>["multiline"], :level=>:warn}
Pipeline main started

Hi,

I need logstash for IBM AIX machine. When i tried to install the logstash it is giving error.

I am receiving this error:

dodrlobiaapp2:~# /opt/logstash/bin/logstash -f /apps/software/obi-enhanced-usage-tracking-master/logstash-obi-clickstream.conf
Settings: Default pipeline workers: 4
Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads {:count_was=>4, :filters=>["multiline"], :level=>:warn}
Pipeline main started
Errno::EACCES: Permission denied - /apps/obia12c/logs
initialize at org/jruby/RubyFile.java:370
new at org/jruby/RubyIO.java:871
open at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-2.2.5/lib/logstash/outputs/file.rb:264
receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-csv-2.0.5/lib/logstash/outputs/csv.rb:39
multi_receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/outputs/base.rb:109
each at org/jruby/RubyArray.java:1613
multi_receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/outputs/base.rb:109
worker_multi_receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130
multi_receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114
output_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301
each at org/jruby/RubyHash.java:1342
output_batch at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301
worker_loop at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232
start_workers at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201

The sawlog is a rich source of lots of data, but the Logstash script has to know how to parse it. It's all down to the grok statement which identifies fields to extract and defined their deliniators. Use grokdebug.herokuapp.com to help master your syntax. From there, the data can be emitted to CSV and loaded into Oracle.

Here's an example of something yet to build - when items are moved and deleted in the Catalog, it is all logged. What, who, and when. The Logstash grok currently scrapes this, but the data isn't included in th CSV output, nor loaded into Oracle.

Don't forget to submit a pull request for any changes to the code that would benefit others in the community!

You'll also find loading the data directly into Elasticsearch easier than redefining the Oracle table DDL and load script each time, since in Elasticsearch the 'schema' can evolve based simply on the data that Logstash sends to it.