sontek / notaliens.com

A website for showcasing pyramid powered sites and the hackers that use it!

notaliens.com

A website for show casing pyramid powered sites and the hackers that use it!

The site is a combination of these namespaced packages: - notaliens.core: Shared templates, assets, and base classes - notaliens.identity: Authentication and Sessions - notaliens.people: The section of the site that manages the list of Pyramid developers and their user profiles - notaliens.sites: The section of the site dedicated to managing the list of Pyramid powered websites - notaliens.jobs: The section of the website dedicated to posting and searching for Pyramid jobs

We are written with Python 3.3 and Python 2.7 in mind.

Features

We should have a mix of features that are similar to http://djangosites.org and http://people.djangoproject.com/

People

• Developers should be able to register accounts, list their basic profile information such as github, twitter, and linked in, along with their location and if they are looking for work. They should also be able to tag themselves for what type of technology they work with.

Sites

• Registered users should be able to add 'sites' that are powered by Pyramid. This will include adding screenshots, tags for type of site, URL, should have stats about backend db (pgsql, redis, etc), what server is used (nginx, gunicorn), OS, and released version. What spoken language it is in.

Jobs

• This will be for people to post up jobs that are pyramid related

Getting Started

$ easy_install -U setuptools
$ python bootstrap.py

Before running buildout, you need to make sure you have postgresql installed.

Fedora

$ sudo yum install postgresql-server postgresql-devel
$ sudo su - postgres
$ initdb
$ exit
$ sudo systemctl start postgresql.service

Ubuntu

$ sudo apt-get install postgresql libpq-dev python-dev
$ sudo apt-get install libtiff4-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms1-dev libwebp-dev

Finally

$ sudo su - postgres
$ createuser notaliens -P
$ createdb notaliens -O notaliens
$ exit
$ mkvirtualenv notaliens
$ cdvirtualenv
$ mkdir src
$ cd src
$ git clone [email protected]:sontek/notaliens.com.git
$ cd notaliens.com
$ pip install -e .
$ notaliens_update_geoip development.ini
$ notaliens_create_db development.ini
$ notaliens_update_geoip_csv development.ini
$ notaliens_rebuild_index development.ini

Running the application server:

$ bin/pserve notaliens/development.ini

Running the task queue worker:

$ bin/notaliens_task_queue notaliens/development.ini

We also have support for caching SQL queries via redis and using elasticsearch for full text search, so use your distro to install those if you would like to take advantage of those features.

You will notice on the view timings that we are running at about 30-40ms per view, this is not an accurate number because we enable the debugtoolbar by default.

With the toolbar off we average around 10-15ms per view request.

Elastic Search

To test the elastic search:

Profile Schema:

curl -XGET 'http://127.0.0.1:9200/users/_mapping?pretty=1'

All profiles:

curl -XGET 'http://127.0.0.1:9200/users/_search?pretty=1'

Filter Profile:

curl -XGET 'http://127.0.0.1:9200/users/user/_search?q=first_name:john&pretty=1'

Or for more advanced querying:

curl -XPOST 'http://127.0.0.1:9200/usres/user/_search?pretty=true' -d'
{
    "query":{
        "term" : {
            "first_name" :"john"
        }   
    }
}
'

For geo location queries:

curl -XPOST 'http://127.0.0.1:9200/users/user/_search?pretty=true' -d'
{
    "query": {
        "filtered" : {
            "filter" : {
                "geo_distance" : {
                    "distance" : "12km",
                    "location" : {
                        "lat" : 37.4644,
                        "lon" : -122.2267
                    }
                }
            }
        }
    }
}
'

Hacking

To run our tests you can do:

$ bin/py.test notaliens/tests

To get code coverage:

$ bin/py.test --cov-report term-missing --cov notaliens

notaliens.com's Issues

Create a index rebuild script

Sometimes we might want to disable elastic search or it might just go down. In this case we need a command that will re-index all entries currently in the DB.

Speed up the geoip csv parsing (using multiprocessing/threads)

Evaluate using pyramid_tm instead of managing our transactions manually

Paging doesn't maintain search criteria

Get on travis-ci

Once we add tests, we should get them running on travis-ci so we can make sure we are testing in python 2.7 and python 3.3

Add unit tests

All development up to this point has been without any actual unit tests. We need to fix this.

Add dynamic map for displaying clusters of developers

Its nice to see an overview of all the pyramid developers around the world on a map. Figure out a way to cluster a small amount of users across the world and then pull more in as you zoom.

Add details section of sites

When you drill into the details of a site, we currently don't showing anything.

Lets add the ability to view the site, additional screenshots, and their description of how they use the site.

Add security/verification for adding new sites.

Clean up the profile view

Right now we don't expose all the information we collect on the profile page. We should all the user data if possible.

Add tags and looking for contract work boolean to search

Right now we are only searching on first, last, and e-mail. We should also be able to search on tags like python and if someone is looking for contract work (this should probably be an advanced search dialog?)

Setup the sites section

1. Users should have the ability to post a website description, url, and a pip freeze of the libraries they use.
2. We should use a task queue to automatically go out and take a screenshot of the main site.
3. We should have full text search for looking for sites powered by certain technologies

get_region_by_postal fails for some postal_code

When going to the profile view for username "housewifehacker9" the postal_code "75017" gives the following error for get_region_by_postal in core.models.meta:

sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()

The postal_code of "75017" denotes both Irving, Texas and Paris, France and is making the query().one() fail due to returning two results.

Remove the multiple command entry points

We should build 1 command, notaliens and use something like cliff or argparse to have all the arguments we need to handle. That way you can control your whole web app from 1 single command.

Add searching for the country in the dropdown (select2.js?)

Add geo location support

We want to support both slow operations via the database and quick operations using elastic search. So geo location support should have the following:

1. In Elastic Search we should have geo points:

http://www.elasticsearch.org/guide/reference/mapping/geo-point-type/

1. For the database operations, we should use a lat/long range + haversine algorithm

Add fulltext search using Postgresql full text features

Add SQLAlchemy events to the user/userprofile for regenerating the elasticsearch index

404 Not found to Jobs

So, what's should to do here?

Using Postgres FTS to minimise the maintenance

First, Great initiative.
It will be great to have something like that online, we have got some very cool project which has been build with pyramid.

Recently I did loads of works with Elasticsearch for at the end falling back on the Postegres Full-Text search. The main arguments was that for the project needs, Elasticsearch was not bringing any advantage but only more maintenance issues with maintaining the cluster.

I also wrote a post to explain FTS with Postgres: http://blog.lostpropertyhq.com/postgres-full-text-search-is-good-enough/

Only a suggestion to make like easier but otherwise I have an ansible role that I can share for elastic search.

Create the jobs section

Users should have the ability to:

1. Post job descriptions (similar to http://www.python.org/community/jobs/ and https://djangogigs.com/ )
2. Search for jobs based on description, tags, location, etc
3. Should give links to other job sites

Add geo features using PostGIS