octosearch's People
octosearch's Issues
HTTP indexer error
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Permission filter does not work correctly
Try harder to match filename
For example in pseudocode ('simplematch' => user_query, 'match_url' => user_query)
Files without parser should be indexed without content
So search can still match by the filename
Ignore files for which no parser is configured
And make it possible to configure files to be parsed with the fallback parser
Docs removed from fs are not removed from index
last_seen is passed as a timestamp in seconds since epoch to elastic, should this be a regular datetime?
Find a way to open local files
With a browser plugin or local application?
purge files from index that have been removed
purge files that have been skipped with the last update?
record last_seen with entries, so all files last_seen before indexing started can be purged.
Parser configuration should not use mimetypes for plugin names
It also makes configuring exceptions a bit awkward:
[parser]
application/vnd.ms-excel=octosearch:application/octet-stream
Rename project to Octosearch
Add web crawler plugin
Searching through authenticated and non-auth'ed sources while logged in does not work
It doesn't show results for the non-auth'ed sources
Add message flashing
For stuff like 'you've logged in' or 'saved your stuff'
Indexer hits ES index.mapping.total_fields.limit with Tika fields
Add Word parser
Make web the interface nicer
Use a lightweight css framework like PureCSS, Spectre or Kube?
Update permissions also when file contents haven't changed
Write help page for search box
Mostly what is on https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html, but a little more user-friendly.
Add option to make login mandatory
Add Excel parser
Indexer crashes on weird path / filename
Traceback (most recent call last):
File "octosearch.py", line 63, in
app.start(args)
File "octosearch.py", line 38, in start
index_job.index(indexer_conf)
File "/usr/src/octosearch/octosearch/indexer.py", line 32, in index
for file in indexer.index(conf):
File "/usr/src/octosearch/octosearch/indexers/mountedcifs/mountedcifs.py", line 42, in index
for file in super().index(conf):
File "/usr/src/octosearch/octosearch/indexers/localfs/localfs.py", line 26, in index
yield LocalFile(full_path)
File "/usr/src/octosearch/octosearch/indexer.py", line 180, in init
self._set_properties(path)
File "/usr/src/octosearch/octosearch/indexer.py", line 186, in _set_properties
statdata = os.stat(path)
OSError: [Errno 22] Invalid argument: ''
Add proper configurable logging (to syslog, etc)
Change 'auth' config option in indexer config block to not take the auth name
Because there can only be one auth provider configured.
Change it to check_read_permissions or something like that.
Add option to load config from a folder (like /etc/octosearch/conf.d)
Save thumbnails for images
Make sure this works properly for non-public files
Change cmdline args to make more sense
--index, --webserver and --truncate are all optional now. if nothing is specified the program connects to elasticsearch and then exits without an error or message.
Find out if it's possible to specify a main command that has a number of possible options attached, like the mercurial cmdline for example.
Searching must match the filename too
Use FUSE filesystems?
Convert code to Python 3
Make it possible for indexer classes to provide a title
Indexer supplied data like title should probably have more priority than what the parser returns, because the indexer has more domain knowledge.
Write a html parser class
Parser plugins must return utf-8
Add SFTP indexer
Move mountedcifs indexer to seperate package
Build external plugin
To see if that actually works, and if the indexer *File classes can be used.
Add Dropbox indexer
Sub-indexers; allow for indexing of zip files, or images linked in HTML pages
Lightweight local protocol handler for opening files from octosearch site?
Instead of a local search app.
For example octosearch://smb://server/path or octosearch://smb/server/path
It looks like the fallback parser inserts spaces in the content where they don't belong
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.