ahmia / ahmia-site Goto Github PK

Installing the requirements through pip to the virtualenv currently fails (as can be seen in the travis's output #16) with Error: could not determine PostgreSQL version from '10.1', similarly to psycopg #594 .

Moreover cryptography version seems to also need an updated version.

Updating to psycopg2==2.7.4 and cryptography==2.1.4 fixes this issue.

Should I make a new branch in order to test these versions after ahmia setup, or submit directly to master?

I believe this is what you are looking for https://ahmia.fi/blacklist/banned/

Originally posted by @chamalis in #64 (comment)

Where is the blacklist of hashes at?

I'm trying to make a tool that a person enters in a Onion URL that automatically hashes it and then searches that you have generates on your website.

Not sure if this is the right place to report it, but ahmia.fi replies with 502 error

On the site https://ahmia.fi the search result is always:
"
Internal Server Error
Sorry! :(
"

Warnings+ that happen in manage.py commands should be logged appropriately

did you forgot something?! seems like pagination is broken and not working.

Investigate if we can get better search results by using Elasticsearch fuzzy queries

Add some more django tests

the search is too slow these days on both tor or clearnet. In most cases id does not return anything for long while

For unknown reason, it seems that the postgres database is not populated with all of the tables when running python3 ahmia/manage.py migrate

django.db.utils.ProgrammingError: relation "ahmia_searchquery" does not exist
LINE 1: ...urrences", "ahmia_searchquery"."search_term" FROM "ahmia_sea...

From what i see the versions of dependencies you guys are using extremly outdated versions of dependincies.
Python 3.6 opposed 3.12
Django 5.0 opposed to 1.11
Elastic 6 opposed to 8
If there is a particular choice behind this choice(Assuming later versions dont work nicely with Tor) i would love to be informed because i am actively considering contributing to this project.

Current search uses a copy_to field that combines title, meta, anchors equally (as far as I can tell). [1]

Then multi_match search is performed on that composed field, discarding stemming, etc [2]

Try to order results based on weighted coefficients, e.g

...
"multi_match": {
    "query": query,
    "type": "most_fields",
    "fields": [
           #"fancy",
           #"fancy.stemmed",
           #"fancy.shingles",
          'title^4', 'meta^2', 'content^2',' anchors^2'
          # TODO find a way to use stemmed, shingles filters here 
    ],
    "minimum_should_match": "75%",
    "cutoff_frequency": 0.01,
}
...

and compare the results.

[1] https://github.com/ahmia/ahmia-index/blob/master/mappings_tor.json#L139
[2] https://github.com/ahmia/ahmia-site/blob/master/ahmia/search/views.py#L82

Hi !
I'm working on a 100% legal-content deepnet search engine and would like to use your blacklist to kick unwanted websites.

Fact is, when crawling the deepnet, no onion URL did never match the blacklist. Either my crawler never meets any banned stuff (sadly doubtful), either I really suck at coding (quite probable).

Did someone know a real banned onion url to test it against the blacklist ? (an old and unaccessible V2 onion URL would be perfect for that) ?

Or would it be possible to add a dummy onion url in the blacklist for testing purpose, like "trytoencodethistoseeifitsmatchahmiasblacklist.onion" ?

Thank you for the answer and for the project :)

The way logger.exception(e) writes to the logs is somehow bloated and suboptimal.

• Avoid writing the whole stack trace
• Include the timestamp

The onion add form returns same generic message for both "already exists" case and "invalid" case.

It would be better to distinguish between them.
Also replace TemplateView with FormView, that better fits for this use case.

Include a popularity metrics algorithm, that will have some influence on the results ordering.

This will use the backlinks from onion addresses. If spam link farmers are detected, we could use some sort of detection mechanism to reduce the influence of those spam websites.

We have to find an appropriate formula to combine that rating with the Elasticsearch's ordering score that's already applied.

Compare with the current results to find out if we managed to improve results ordering.

See duckduckgo/duckduckgo#258 (comment)

Can you do the same thing for https://ahmia.fi/ ?

You are listed at https://directory.fsf.org/wiki/Collection:Search_engines#Requested_sites, I'll move https://ahmia.fi/ to the main list of that page in return if you implement it.