nixnodes / glutil Goto Github PK

glFTPd binary logs utility

License: GNU General Public License v3.0

Shell 14.40% C 79.80% C++ 0.02% Makefile 5.27% M4 0.51%

glutil's Introduction

 General purpose glFTPd binary logs utility

  This tool serves as an I/O mechanism between user's scripts and 
  binary logs or online users data from the shared memory segment.
  Additionally, it can traverse a filesystem directory tree much the
  same way as it would other sources, honoring all of the the extensive 
  filtering options available, making it a powerfull filesystem search 
  application.
      
    Check /scripts folder in this repo for examples on what can be done 
    with this tool. This is a C source project, scripts are NOT required
    for any part of glutil to function.  However, scripts depend on 
    glutil to read/write binary log/shm/filesystem/config data. 
    Scripts are NOT maintained and tested on different evironments. 
    That said, they should work fine on most newer linux environments 
    and provide scripters with a base for porting to theirown platforms.

  See ./glutil --help for full list of options with short descriptions.

 MAIN FEATURES:

  - Import individual/missing records or rebuild entire dirlog based 
      on file-system and nukelog data (read below)
  - Rebuild existing binary glFTPd logs (see full list below) 
      using regex/custom filtering methods and sorting
  - Display various binary glFTPd logs/online users/config data 
      (filtered/sorted) using custom formatting and pre-processing
      hooks (-print, -execv)
  - Execute external tools/scripts for each data record matched, passing 
      along argument strings converted from binary data to shell (-execv)
  - Write to any binary log, using an ASCII formatted text or binary 
  	  source
  - Calculate CRC32 of files

 INSTALLATION:

  - Unpack and compile:

      git clone https://github.com/nixnodes/glutil.git
      cd glutil
      cp /glroot/bin/sources/glconf.h src/
      ./configure --prefix=/glroot && make      
      
      Some warnings might pop up (should be safe)
      glconf.h (found in /bin/sources) is needed to compile properly,
      make sure to put it into 'src/' folder.
      
  - Install:
      
      make install


 SUPPORTED glFTPd BINARY LOGS (with default paths):

  - Directory log    (/ftp-data/logs/dirlog)
  - Nuke log         (/ftp-data/logs/nukelog)
  - Dupe file        (/ftp-data/logs/dupefile)
  - Last-online log  (/ftp-data/logs/laston.log)
  - Oneliners file   (/ftp-data/logs/oneliners.log)
  - Online users     (shared memory segment)

  glutil specific binary logs:

  - Internet Movie Data Base log   (/ftp-data/logs/imdb.log)
  - TVRAGE info log                (/ftp-data/logs/tv.log)
  - Game info log                  (/ftp-data/logs/game.log)


 Credits : Bsugar for the original idea, design and beta testing

glutil's People

Contributors

Stargazers

Watchers

Forkers

kijdam h-0-s-h peps1 hrxcodes

glutil's Issues

Excluding Directories

Is there a way to exclude scanning directories like /private ?

Thanks!

tvmaze support

Would it be possible to add tvmaze support?

Fails to compile on newer gcc "multiple definition of"

Edit src/Makefile and edit:

From: CFLAGS = -Wall -O2
To: CFLAGS = -Wall -O2 -fcommon

It's due to this change: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85678

macro imdb-installch: could not compile exec string

Hi,

It seems there might be a syntax error in the imdb-installch macro:

box:# /glftpd/bin/glutil -m imdb-installch

result

ERROR: MACRO: [2603]: could not compile exec string: '{exe} -noop  --preexec `! updatedb -e "\{glroot\}" -o /tmp/glutil.mlocate.db && echo "updatedb failed" && exit 1 ; li="/bin/perl /bin/curl /bin/xmllint /bin/date /bin/egrep /bin/sed /bin/expr /bin/recode /bin/awk"; for lli in $li; do lf=$(locate -d /tmp/glutil.mlocate.db "$lli" | head -1) && l=$(ldd "$lf" | awk '{print $3}' | grep -v ')' | sed '/^$/d' ) && for f in $l ; do [ -f "$f" ] && dn="/glftpd$(dirname $f)" && ! [ -d $dn ] && mkdir -p "$dn"; [ -f "\{glroot\}$f" ] || if cp --preserve=all "$f" "\{glroot\}$f"; then echo "$lf: \{glroot\}$f"; fi; done; [ -f "\{glroot\}/bin/$(basename "$lf")" ] || if cp --preserve=all "$lf" "\{glroot\}/bin/$(basename "$lf")"; then echo "\{glroot\}/bin/$(basename "$lf")"; fi; done; rm -f /tmp/glutil.mlocate.db`'

dirlog rebuild/update follows symlinks even if not told to do so

Hi,

First I would like to say that it's an excellent tool for glftpd you coded here. Thanks for sharing it.

In an attempt to use it to update a dirlog, I've found out that it's following symlinks even I'm not using -y, --followlinks Follow symbolic links (default is no)

Here is an example:

root@box:# chroot /glftpd /bin/glutil --folders=/etc/glutil.folders.incoming -r -u -v
NOTICE: /ftp-data/logs/dirlog: explicit write pre-caching enabled
NOTICE: /ftp-data/logs/dirlog: created data backup: /ftp-data/logs/dirlog.bk
NOTICE: requested dirlog update only
SCANNING: '0DAY', depth: 2
DIRLOG: /site/incoming/0DAY/today/SOME.DIR1 - 217 Mbytes in 14 files - created 03 Apr 2017 00:43:55 by 200.1100 [0]
DIRLOG: /site/incoming/0DAY/today/SOME.DIR2 - 399 Mbytes in 23 files - created 03 Apr 2017 00:47:38 by 200.1100 [0]
DIRLOG: /site/incoming/0DAY/today/SOME.DIR3 - 2 Mbytes in 3 files - created 03 Apr 2017 14:16:56 by 245.200 [0]
... etc

today in this case is a symlink

glutil.folders file says :

incoming/0DAY 2

Any idea what I'm doing wrong ?

Thanks!

loopexec inexistant

Hi,

Just wanted to let you know this:

From the FAQ file

This will print out the users connected to the site and place in an organized table and loop it for 10 seconds,
clearing the screen each time.
        glutil -w --comp --loop 10 --loopexec clear

and from ChangeLog

 *   - + --loopexec (rev. 14)

but it seems this argument doesn't exists

/glftpd/bin/glutil -w --comp --loop 10 --loopexec clear
ERROR: [-2] invalid option '--loopexec'
See --help for more info

Also i've searched the code for loopexec, nothing is returned.
Maybe you should remove the examples from FAQ if this is deprecated. :)

dirlog update - nuked dirs duplicated

Hi,

Just noticed after running multiple times the dirlog update that it readds nuked directories even if they are already in the dirlog:

DIRLOG: /site/incoming/stuff/DIR1 - 167 Mbytes in 14 files - created 26 Mar 2017 07:44:01 by 206.500 [1]
DIRLOG: /site/incoming/stuff/DIR2 - 196 Mbytes in 16 files - created 26 Mar 2017 07:44:00 by 206.500 [1]
DIRLOG: /site/incoming/stuff/DIR3 - 218 Mbytes in 18 files - created 26 Mar 2017 07:44:01 by 241.900 [1]

[12:39:22] [L] site search DIR1
[12:39:22] [L] 200- (Values displayed after dir names are Files/Megs/Age)
[12:39:22] [L] 200- Doing case-insensitive search for 'DIR1':
[12:39:26] [L] 200- /incoming/stuff/DIR1 *NUKED*
[12:39:26] [L] 200- /incoming/stuff/DIR1 *NUKED*
[12:39:26] [L] 200- /incoming/stuff/DIR1 *NUKED*
[12:39:26] [L] 200- /incoming/stuff/DIR1 *NUKED*

delay

is it possible to artificially insert a delay?

In glutil/scripts/fxp_sync.sh double verify-certificate?

https://github.com/nixnodes/glutil/blob/master/scripts/fxp_sync.sh

LFTP_SYNC_OPTS="set ssl:verify-certificate off;
set ssl:verify-certificate off;

two times defined the same?

nixnodes / glutil Goto Github PK

glutil's Introduction

glutil's People

Contributors

Stargazers

Watchers

Forkers

glutil's Issues

Excluding Directories

tvmaze support

Fails to compile on newer gcc "multiple definition of"

macro imdb-installch: could not compile exec string

dirlog rebuild/update follows symlinks even if not told to do so

loopexec inexistant

dirlog update - nuked dirs duplicated

delay

In glutil/scripts/fxp_sync.sh double verify-certificate?

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent