drupal-ukraine / ppdorg

ppdorg - Drupal.org users statistics

ppdorg project

ppdorg project is for getting drupal.org and github.com commits statistics for a list of users

(minimal version of installed Vagrant should be 1.8.1)

You could start your work with

vagrant up
vagrant ssh

If system will ask you a password - use "vagrant" pass phrase.

Drupal project is located at /var/www/docroot folder.

use

sh reinstall.sh

for drupal reinstallation from scratch

Virtual host for accessing ppdorg drupal installation

http://ppdorg.192.168.56.112.xip.io

If xip.io not working - create row with

192.168.56.112 ppdorg.192.168.56.112.xip.io

in /etc/hosts Drupal Vagrant Dev box for CIbox support

#Installation

• Vagrant
• VirtualBox
• Useful Vagrant's plugins
  • Hosts Updater
  • vbguest

#Usage

vagrant up && vagrant ssh

Drupal reinstallation from scratch

Unix users

sh reinstall.sh

Windows users

sh reinstall.sh --windows

By default your site will be accessible by using this url.

http://ppdorg.192.168.56.112.xip.io/

If xip.io not working - create row with

192.168.56.112 ppdorg.192.168.56.112.xip.io

in /etc/hosts or just use another ServerName in apache.yml

If you have Vagrant HostUpdater plugin, your hosts file will be automatically updated.

VirtualBox additions

For automatic update additions within guest, please install proper plugin

vagrant plugin install vagrant-vbguest

Tools

• XDebug
• Drush
• Selenium 2
• Composer
• Adminer
• XHProf
• PHP Daemon
• PHP, SASS, JS sniffers/lints/hints

##Adminer Adminer for mysql administration (credentials drupal:drupal and root:root)

http://192.168.56.112.xip.io/adminer.php

##PHP Profiler XHProf It is installed by default, but to use it as Devel module integration use:

drush en devel -y
drush vset devel_xhprof_enabled 1
drush vset devel_xhprof_directory '/usr/share/php' && drush vset devel_xhprof_url '/xhprof_html/index.php'
ln -s /usr/share/php/xhprof_html xhprof_html

After vset devel_xhprof_enabled it could return an error about "Class 'XHProfRuns_Default' not found" - ignore it.

Linux Containers

When your system enpowered with linux containers(lxc), you can speedup a lot of things by using them and getting rid of virtualization. For approaching lxc, please install vagrant plugin

vagrant plugin install vagrant-lxc
apt-get install redir lxc cgroup-bin

also you may need to apply this patch fgrehm/vagrant-lxc#354

When your system is enpowered by apparmor, you should enable nfs mounts for your host machine Do that by editing /etc/apparmor.d/lxc/lxc-default file with one line

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  ...
    mount options=(rw, bind, ro),
  ...

and reload apparmor service

sudo /etc/init.d/apparmor reload

and run the box by command

VAGRANT_CI=yes vagrant up

Do use

VAGRANT_CI=yes

environment variable, if you got issues with all vagrant commands.

Windows Containers

Install Cygwin according to provided steps.

Run Cygwin as Administrator user.

Use default flow to up Vagrant but run sh reinstall.yml --windows

##Windows troubleshooting

If you will see error liek ...[error 26] file is busy... during sh reinstall.sh modify that line:

before

name: Stage File Proxy settings
sudo: yes
lineinfile: dest='sites/default/settings.php' line='$conf[\"stage_file_proxy_origin\"] = \"{{ stage_file_proxy_url }}";'

after:

name: Copy settings.php
sudo: yes
shell: cp sites/default/settings.php /tmp/reinstall_settings.php

name: Stage File Proxy settings
sudo: yes
lineinfile: dest='sites/default/settings.php' line='$conf[\"stage_file_proxy_origin\"] = \"{{ stage_file_proxy_url }}\";'

name: Restore settings.php
sudo: yes
shell: cp /tmp/reinstall_settings.php sites/default/settings.php

MacOS users

Install vagrant

https://www.vagrantup.com/downloads.html

Install Virtualbox from here:

https://www.virtualbox.org/wiki/Downloads

cd to project folder

vagrant up && vagrant ssh

You will be logged into virtual machine.

Go to this path

/vagrant/docroot

and run the script

sh reinstall.sh

for drupal reinstall from scratch.

Configure Stage File Proxy to use the files from the correct source.

ppdorg's Issues

Define good features architecture.

Features architecture needs reorganisations according to the latest changes.
Variants:

• create one big feature for: CT, panels, Views.
• one for content.
  Or another one.

About text.

At home page, first widget that would be displayed to user is: Intro text.

We have to implement and discuss this text.
This text have to discover goal of the projects and value for end user.
It shouldn't be too long.
This text should be translated to: Ukrainian language first.
Ukrainian is first we need. Other languages could be added later.

Make project logo.

Now, project uses logo from drupal.ua.
We have to implement new logo, based on druplicon with honor places at the background.
Something like this but with the Druplicon at the top :)
Main idea is to combine druplicone with statistics bar.

Users ct node revisions

Save new node revision at user rescan if new user info is different to the user info, stored at DB.
Implement diff module https://www.drupal.org/project/diff that displays changes of the each profile, during all time he is stored at our project.

Create batches for all other types

// @todo create batches for all other types.

from ppgetstat.module

Implement autoreport feature, based on commits and closed issues.

We need implement feature which will grab all activity from users of specific company and generate report based on grabbed info.
Input arguments:

• date from
• tech company name on drupal.org

Expected result: Page with lists of issues like https://www.drupal.org/node/2685667

Update vagrantbox from cibox suite

according to #66 and @anpolimus request

Add users list field (taxonomy_term_reference) and save it during scanning.

Add to new taxonomy_reference field, called users_list.
Add info to this field during scan (for example, when we are scanning Ukrainian users a link to this users group should be at each user).

Do not forget to pack this field to the feature.

Fix bug with rounded people count at each country

Last users are not scanned by our code.
We have to resolve that.
http://screencast.com/t/20ScEsxMBR

Create a users list entity and configuration for it.

Create an object group that will define a group of users that we need to calculate and merge their statistic.
Initial target for us is to create:

• list of users with country=Ukraine;
  At current arch. the best way is to implement new taxonomy vocavulary with fields:
• title (title of the group);
• parameters arguments (key-value multivalue field)
  For example:
  field_sex : male
  field_country : Ukraine
  That would be transformed to https://www.drupal.org/api-d7/user.json?field_country=Ukraine&field_sex=1

This structure will give us ability to create different lists for reports like:
Ukraine community with world community
Ukraine vs USA

Also, this project could be reused by other communities around the world.
And this project could be published not just at github but also at distribution sections of d.org as a separate project.

We will use this Vocabulary for creating Drupalers from Ukraine report.
And then we will create and publish another report, based on group with users that marked themselves as a part of DrupalUA company at marketplace.

Commits batch grabber

Finish #8

If user was deleted his stats are still displayed on the frontpage

1. Remove already invalid ppgetstat table's entries from frontpage panes.
2. Remove ppgetstat entries on node deletion.

Create infographics with external service

Use current stored data for commits to display them with the service
http://infogr.am/

Create batches for all other type

After merging #8

// @todo create batches for all other types.

From ppgetstat.module

We should provide routines for getting issues, comments etcetera for a list of entered users

Ppgetstat table improvements

ppgetstat table:

• delete items field and asscianted code;
• add user_list_tid field and synchronise it with actual user data.
• rebuild main page widget based on new architecture.

user node add link to d.o. profile

would be great to be able to jump to user's drupal.org profile page. please add a link

Event CT.

We have to add event CT for displaying it at the community timeline.
Possible examples that could be defined as event:

• cafe code sprint;
• compay code sprint;
• camp code sprint;
• weekend code sprint;
• Dries birthday;
• celebration parties

This CT should be implemented with next fields:

• title;
• place;
• body;
• date (with finish date);

CT settings should be packed to the feature.

Need help block.

Let's use aggregator thing for getting list of issues we need help

frontpage

• show top 5 persons stats for last 2 months in all three categories
• allow anonymous users to add new users just by typing nickname
• show statistics of the queue (how many pages queued to be scanned).

Rescan button for development

After #16 merged, we have to provide easy way to do rescan from scratch

variable_del('ppgetstat_last_statsjobs_timestamp');

pull db from prod to dev

set up jenkins job to pull db from production to dev environment to test upgrade path

Move all widgets to the term panel with term object in context.

Now, all statistics block are displayed at home page and are displaying statstics for all scanned user lists.

Since, we want to scan different countries, we have to move that block to the new panel at users list term pages.

1. Add panel to the users list vocabulary term pages.
2. Use term object in widgets function for filtering country statistics.
3. Delete all blocks from home page and make Ukrainian team statistics term as home page.

Save stats to nodes

// @todo Save stats to nodes here. User will be passed count(nicks) time.

From ppgetstat.module

Add fields to the event CT

@andriyun

1. Add event type field with options:

cafe code sprint;
compay code sprint;
camp code sprint;
weekend code sprint;
Dries birthday;
celebration parties

2. Amount of attendees

3. Related links for announce, reports, photos

Change data saving using node revisions and properties

https://github.com/podarok/ppdorg/pull/16/files#diff-157
remove schema and add ability to save using node revisions and properties

Calculate empty profiles.

Calculate profiles that dont have:

• posts, commits
• contribute information
  and mark them as "empty"

Empty - should be logical field.
It will help us to calculate stat more quickly for active users.

Empty recalculation could be run once per week.

Rewrite core commits to parse locally hosted drupal core repo

git log --pretty="format:%an:::%ct:::%s" --date=raw --all --grep='podarok'

using https://github.com/kbjr/Git.php wrapper
It is fast, can be used realtime and can help ger rid of intencive pages scan at drupal.org

Fix warnings at cron

Warning: Invalid argument supplied for foreach() in _ppgetstat_cron_processItem_stats_jobs() (line 287 of/var/www/docroot/sites/all/modules/custom/ppgetstat/ppgetstat.module).
Warning: Invalid argument supplied for foreach() in _ppgetstat_cron_processItem_stats_jobs() (line 287 of/var/www/docroot/sites/all/modules/custom/ppgetstat/ppgetstat.module).

Users from Ukraine at d.org sync.

Write script that will sync all Ukrainian users at d.org and add new users to ppdorg with statistic data.

Rewrite stats jobs logic.

Now we have three different jobs: commits, core commis, posts and going to introduce github commits. We already have too much dublicated code and too dirty code structure. Lets rewrite everything with classes as ctools plugins?

Top mentors statistics.

Investigate, how to calculate mentors entitly links at each account.
Implement result table with the top accounts that are marked as mentors.

Get id for nickname rewrite

jQuery('#nav-content .links .posts a').attr("href")

Here is a selector for drupal.org/u/NICKNAME for getting user "/user/1845398/track" id contained string.
Let's get rid of dgo.to stuff

ratings pages

create separate pages with total ratings (most commits, comments, core commits) for last X period of time

Users listing

Page with list of all users sorted alphabetically with links to their profiles

Grab stats from github public profile

It would be nice to see how many actions were done on github as well.

Global widgets: top contributors

Made top contributors widget for home page without filtering by country.
This widget will show top of contributors list from all Countries under investigation.
For better architecture, it would be better to implement data fetching as one unique function for 2 top contributors widgets.

Scan patches, added by user.

We have to implement scanning of a new entity: patches that are added by d.org user.
This feature could be possibly added to the comments entity scanning code.

Setup travis

Setup travis + phpcs for project

Error

node/add

Fatal error: Unsupported operand types in /var/www/contributions/ppdorg/sites/all/modules/custom/ppgetstat/ppcc/ppcc.module on line 24

implement scan

https://www.drupal.org/u/alexpott/issue-credits/3060

Checkbox - out of the rating.

User should contain ability to skip itself from the list of TOPs

Cache downloaded pages to filesystem.

After #8 merged

// @todo Cache downloaded pages to filesystem: files/dorgpages/nodeid_week_year.html

from ppgetstst.module

Create demo site

Create demo site
propeople.druler.com with htpasswd credentials for internal usage with installed latest codebase and crontab activated

Obtain uid automatically

Change https://github.com/podarok/ppdorg/pull/16/files#diff-e3847fb2faeaa2a831c8429ae2aa4d14R15

To auto uid obtain using

function ppgetstat_get_id_by_nick($nick) {