Comments (4)
Maybe it can be linked to this issue on Mwdb
CERT-Polska/mwdb-core#560
If we can have a different ursadb tag by mwdb bucket
from ursadb.
But we have some use cases where we want to add a lot of temporary files for a short duration.
Hi, do I understand correctly that you want to index some files temporarily and later remove them all? You can remove datasets (groups of files) (as you already know from documentation you linked), just not individual files.
The only problem is that it's hard to know which datasets to remove after indexing, but it's actually very simple to fix. You can group files during indexing by adding a specified tag (known as taint
in ursadb world), for example:
index "/opt" with taints ["temporary123"]
This will create datasets with tags, similarly to the screen below:
Datasets with one set of tags will never merge with datasets that have different tags. So after your temporary work is done, you can just query for all datasets with your tag temporary123
and issue dataset "xxxx" drop
commands.
I think this is even easier than keeping track of which files to remove after processing. And the added benefit is that (if you want to) you may use the tags to scope your mquery queries to for example only query your temporary files. Would this work for you?
from ursadb.
Yes I think it should be fine for my use case when the issue CERT-Polska/mwdb-core#560 will be closed. Indeed I need a way to split the sample files by tag to be able to index them correctly.
Having a different s3 bucket or a different root path for the samples should fixed that.
On the other hand, I still have errors in mquery with missing files. But as described in the documentation mquery should ignored them. Maybe the issue should be open on mquery rather than here.
from ursadb.
On the other hand, I still have errors in mquery with missing files. But as described in the documentation mquery should ignored them. Maybe the issue should be open on mquery rather than here.
If you remove the dataset from ursadb it shouldn't be an issue - the files will be gone from mquery as well. But if you just remove files from disk then you'll get errors.
I agree that CERT-Polska/mwdb-core#560 may help with implementing this in practice
from ursadb.
Related Issues (20)
- [META] Release v1.5: Watch Your Language
- Investigate github pages build fails HOT 1
- Update contact information in readme HOT 1
- Computer resources consuming HOT 1
- Outdated Dockerbuild HOT 1
- Improve startup error messages for corrupted state
- Investigate random test failure HOT 1
- Improve the command line interface - parameters
- Bump the Catch version HOT 1
- [META] Ursadb performance improvements HOT 1
- Add a performance counter for unique ngram reads
- Panic when a query is attempted on dataset removed from disk
- Consider (optionally?) compacting in the background
- Implement (optional?) query timeouts
- Implement a GC for iterators
- Automate releases
- Improve query performance during indexing HOT 2
- Add information about locks to the API
- Question/bug(?): multithreading in indexing operation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ursadb.