Coder Social home page Coder Social logo

dp-rex's Introduction

DP|rEX VrAn

This repository contains an editor for collaborative maintenance of metadata for social media accounts. It is developed as the infrastructure of the data trustee for research on right wing extremism (VrAN).

VrAN is a sub project of DP|rEX and funded by the German Federal Ministry of Education and Reasearch.

Features

Currently the following features are supported:

  • View social media account metadata
  • Change the displayed metadata fields
  • Create new metadata fields
  • Edit Data
  • Upload batch data in .csv format and merge it with existing data, with automatic duplicate detection.

Running

You can run VrAN using docker compose. The following instructions use a single node Docker Swarm.

Setup Docker Swarm

  1. Install docker
  2. Setup swarm: docker swarm init

You can also edit the file located docker-compose.yml to use different ports than the standard 443 for HTTPS and 80 for HTTP (HTTP is only used SSL certificate renewal challenges). For more details how to edit the configuration please check the Docker Compose file reference Please substitute your values for patterns starting with a $ like $REPLACE_THIS.

  1. Start a container running a registry: docker service create --name registry --publish published=5000,target=5000 registry:2
  2. Create the secrets. you can either use the quick_setup.py script for setting up secrets or register the following secrets manually using echo '$SECRET_CONTENTS | docker secret create $SECRET_NAME -'
  • vran_db_password contains the database password.
  • vran_db_user contains the database user name.
  • vran_db_name contains the name of the database.
  • vran_pg_conf contains `vran_db:5432:$VRAN_DB_NAME:$VRAN_DB_USER:$VRAN_DB_PASSWORD
  • vran_pg_service_file containing
    [vran_service]
    host=vran_db
    port=5432
    dbname=$VRAN_DB_NAME
    user=$VRAN_DB_USER
    
  • vran_django_key contains the django secret key. You can generate it using < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-64};echo;
  • vran_redis_password contains the redis password
  • vran_redis_conf contains requirepass $REDIS_PASSWORD
  1. From the base directory of the repository run docker compose build to build the containers.
  2. Run docker compose push to push the images to the registry.
  3. Create empty directories:
    • /srv/vran/ssl
    • /srv/vran/contributions
    • /srv/vran/acme-challenge
  4. Set your user as owner of the directories: chown -R ${USER} /srv/vran
  5. Add a SSL certificate (vran.crt) and key (vran.key) while being in directory /srv/vran/ssl. you can create a temporary insecure certificate and key using
openssl req -x509 -out /srv/vran/ssl/vran.crt -keyout /srv/vran/ssl/vran.key \
   -newkey rsa:2048 -nodes -sha256 \
   -subj '/CN=localhost' -extensions EXT -config <( \
   printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

For proper credentials please use certbot or getssl 8. Run VrAN using docker stack deploy --compose-file docker-compose.yml vran 9. Check that VrAN is now accessible over HTTPS's default port 443 on your machine using a browser. 10. Navigate to http:127.0.0.1:8000 and login using the username admin and the password changeme. 11. Change the password and possibly the username in the django admin UI.

Development

There are two projects in this repository. A backend written in python and a frontend in TypeScript.

Requirements

Currently the only operating system supported is Linux. For the backend the project requires Python and poetry as a package manager. Please install Python with your distros package manager. Afterwards install poetry using pip install poetry.

Node is required for the frontend. You can find instructions on installing node at [https://github.com/nodesource/distributions]

Installing dependencies

  • For the backend run poetry install from the root of the directory.
  • For the frontend run npm install from the ui directory.

Preparing the backend

The following steps are required for the initial setup. You only need to run them once.

  • Create an instance of the environment file by running cp .template.env .env
  • Generate a secret key using < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-64};echo;.
  • Copy the secret key and paste it as the value of the variable DJANGO_KEY in the .env file.
  • Set the variable VRAN_DEBUG to true in the .env file.
  • Generate a local database by runing poetry run ./manage.py migrate.

Additional Services

You need to run a redis instance for managing task queues. The easiest way is to launch it in a container: docker run -p 6379:6379 redis.

Running

  • Start the backend by running poetry run ./manage.py runserver.
  • In a seperate shell switch to the ui directory and run npm start.

dp-rex's People

Contributors

mo-fu avatar

Stargazers

 avatar

Watchers

 avatar

dp-rex's Issues

Display Persons

About

As a social science researcher I want to view a list of relevant entities.

Steps

  • provide REST API for retrieving persons
    • persons have personal name, family name and display text
  • fill DB with script
  • display Entities in Table

Indicator for tags assigned to columns

About

Currently it is not obvious from the colum list in the column assignment view to which tag a column is assigned. This was a feature requested during the workshop on 22/01/2024

Outcome

Given a column
When it is assigned to a tag
Then the assignment is shown in the column list.

Reset Form on Tag Creation

About

Users noted that the feedback on successful tag creation is not obvious

Outcome

Given The tag creation form
When a user succefully submits a new tag definition
Then the form is cleared
And a notification is displayed.

Consistent spacing for curated check mark

About

Currently the checkmark, marking curated tags in the tag explorer is not spaced consistently.

Outcome

Given a curated tag
When it is displayed in the tag explorer
Then there is always space between the the last part of the name path and the curated check mark.

Containerisation

About

Containerisation of VrAN will aid in quick dpeloyment of the App

Outcome

As a VrAN admin
When I run docker-compose up
Then The app is accessible

Add Person

About

As the host of VrAN I want to add to the list of relevant persons in order to provide an up to date database

Steps

  • add REST API endpoint for adding/editing multiple persons
  • add form for entering a new person.

Outcome

  • when I open the table of relevant persons, I see a button for adding a new person above the table
  • when I click the button for adding new persons, an overlay for adding a new persons is opened
  • when the overlay for adding a new person is open, there are three form fields, family names, personal names, display text, a checkbox for determining the name order and a submit button
  • when I submit a new person the entered data is validated.
  • when I submit valid data the new entry is added to the database and displayed in the table of persons.
  • when I submit invalid data an error message is displayed.

Do not allow anonymous contribution.

About

Work in the database should always be attributed.

Outcome

Given
When uploading a contibution
Then There is no option to submit anonynmously.

Use notifications for all feedback

About

Currently there are multiple ways of providing feedback to the user. E.g., button text change, button pop over, notifications. The goal is to unify the mechanisms and only provide feedback using the notification mechanism.

Outcome

There are no button popover or button text change used in the app.

Code of Conduct

Link zu Document

  • CoC muss finalisiert werden (Ende Mai)
  • Datenschutzdokument muss aufgesetzt werden
  • Nutzungsvertrag muss aufgesetzt werden

UI Mockups bauen

  • evtl. Balsamiq oder Powerpoint oder was anderes für Wireframing / Design Mockup
  • Funktionsfähigkeit auf Mobile brauchen wir nicht (Darstellung auf kleinen Bildschirm + Touch sind kein Requirement)
  • dann größeres Treffen dazu mit anderen vom MRML Team, wo wir das fest klopfen

Show hand pointer for clickable items

About

Many clickable items in the trustee UI show a text cursor or normal pointer. It would be easier to see that the items are clickable if they would show a hand pointer.

Outcome

Given a clickable item
When it is hovered
Then a hand pointer is shown.

Do not cache id persistent as display text

About

Currently, the display text is computed from various tag values and stored in a cache. If there are no tag values available, the id persistent is stored in the cache as display text. By not storing the id and substituting it dynamically on cache misses, it is possible to spawn the display text computation on misses.

Define MVP Scope

Description

In the Kickoff meeting we defined a coplex usage scenario that captures the development goal before the handover to gesis.
Goal of this story is to outline possible increments that contribute to the envisioned solution.

Outcome

  • Definition of MVP Functionality
  • Outline additional Epics leading to handover goal.

User Management

About

As an admininstrator of a VrAN instance I want to restrict access to VrAN.
#Outcome
As a user
When I am not logged in
Then I will see a login mask and information on registering to VrAN

As a user
When I am logged in
Then I will see the VrAN table view

Display Default Columns

About

As a user I want to view tags associated with an entity.

Outcome

As a user
When I open VrAN
Then I see three columns (id, user, verified) each for social media platforms twitter, youtube, facebook, telegram.
The columns display the data from the DBOeS parlamentarier.csv

Implementation

  • Columns are modelled internally as tags
  • Create ORM Class for tag types. Fields: id, parentId, name, is_curated, type
  • Create ORM Class for tag instances. Fields: entity, tag, value
  • add endpoint for adding/modifying tags
  • add endpoint for getting tags for entities
  • Response is dict
  • fill db with tags and tag instances from parlamentarier.csv
  • add columns to UI state
  • get column content in UI data fetch.
    • columns are dicts
    • row_number to entity_id map

Initiales Definieren von Aufgaben für technische Umsetzung VrAN #3

  • Datenmodel (Individuen plus Institutionen, mehrere Plattformen, mehrere Accounts pro Plattform, Mehrfachaffilierung zu Institutionen, Mehrfachaffilierung zu Ideologien/Kategorisierungen, Veränderung über Zeit möglich)
    
  • Interface für Manuelle Bearbeitung
    
  • Schnittstelle für Batch Upload
    

Configure Columns

About

As a user I want to be able to select the columns displayed in VrAN table view.

Outcome

As a user
When I open the VrAN table view
Then I can remove a column from the view.

As a user
When I open the VrAN table view
Then I can add additional columns

As a user
When I add a new column
The I get suggestions for possible column views

Implementation

  • Add button to column head for removal
  • Button will remove column from currently displayed columns
  • Add endpoint for retrieving column/tag definitions
  • Add button for adding columns
  • When adding columns data is fetched

Externe Teilnehmer:innen einladen

Aktuell eingeladen:

Simon Greipl
Maik Fielitz

Vorläuftige Zusagen

HBI intern
Gregor Wiedemann
Jan Rau
Moritz Fürneisen
Daniel Wehrend

Extern
Christian Schwieter (HBI Budget)
Hannah Büttner (HBI Budget - abzüglich Übernachtung)
Cornelius Puschmann (HBI Budget)
Lena Frischlich (HBI Budget)
Heidi Schulze (HBI Budget)
Annett Heft (HBI Budget)
Axel Salheiser (HBI Budget)
Christoph Richter (?)
Pascal Kolkwitz-Anstötz (Selbstzahler)
Christina Dahn (Selbstzahler)

(HBI Budget aktuell: 7)

Fix scroll in column assignment view

About

Currently the column assignment view can scroll out of view and the original state can not be restored without refreshing the page.

Outcome

Given the column assignment view
When a user scrolls the column list
Then the finalize button is never out of view.

Given the column assignement view
when a user expands the tag definition hierarcht
then the create new tag button will always stay visible.

Feedback on Contribution Step Completion

About

When completing a contribution step, the success message in the button is not visible during mouse over.

Outcome

Given a contribution step
When the configuration by user is accepted
Then A clear visual indicator is shown

Display Relations

As a researcher I want to see the social media accounts belonging to a person.

Bug: Column Menu

Background

React 18 changed the way click events are handled.
This leads clicks opening a column menu to also register a click outside the menu.
See: glideapps/glide-data-grid#553
Therefore at the moment clicking outside the menu can not be used to close a menu.
There is already a fix in the codebase but it is not published yet.

This ticket is a reminder to recheck for new releases of datagrid

Resolve conflicts for merge requests

When a merge request is created and the destination tag is empty
Then the data is automatically mereged

When a merge request is created and there are no conflicts
Then the data is automatically merged

Given a merge request
When there are conflicts
Then The conflicts can be resolved from the UI

Upload CSV

About

As a User I want to add my data to the VrAN DB.

Outcome

As a user
When I upload a .csv file
Then I can see it in the list of uploaded data.

Implementation

  • Add ORM model for contributions.
    • Initial state is received
  • Add endpoint for retrieving list of contributions to the API
  • Add page for showing received contributions to the UI

Export Current Table View as CSV

About

As a user I want to download a copy of the data for local editing.

Outcome

As a user
When I open the VrAN table view
Then I can download the data contained in the table as .csv file

As a user
When I open a downloaded VrAN .csv
Then I see the table column headers in the first row.

Implementation

Edit Person

As a maintainer of VrAN I want to edit persons, in order to provide correct data

Steps

  • add edit functionality to form for adding persons
  • allow selection of persons to edit them
  • allow editing of persons through table.

Not in Scope

  • Batch Edit

Outcome

  • when I select a person I can edit their properties
  • when I update a table field the correspondign person is updated.

Order of columns in column assignment view.

About

Currently the order of columns in the column assignment view is not stable.

Outcome

Given the column assignment view
When a user enables/disables columns for import and reloads afterwards
Then the order of columns is the same as before the reload.

Disable column import per default

About

Users have noted that they often have csv files with many columns. Usaully only a small subset of columns should be imported into the trustee.

Oucome

Given a CSV file
When a user uploads it
Then all columns are deselected.

Enable Column width change in match view

About

Currently it is not possible to change the width in the entity match view.

Outcome

Given the entity match view
When a user drags the borders of a column
Then the column width changes.

Decide Initial DB Schema

Context

We need to decide how to structure an inital database for modelling VrAN

Outcome

  • Initial DB Schema

Assign display text should enable upload.

About

currently a column of a contribution is not marked for inclusion when selecting Display Text

Outcome

Given a column of an uploaded csv
When a user assigns Display Text
Then the column is marked as included.

Clarify password hint

About

Currently the password hint during registration does not accurately reflect the password requirements.

Outcome

Given the registration form
When the password is invalid
Then The error message states the correct requirements.

Konzeption Qualitätskontrolle

Workflow entwickeln für Datennutzung e.g. Nutzende verpflichten sich zur Überprüfung hinsichtlich Aktualität usw.? -> Würde Schwelle für Nutzung stark erhöhen

Abgestuftes Verfahren e.g. Daten hinsichtlich Richtigkeit, aber nicht hinsichtlich Vollständigkeit geprüft?

Automatic Contribution Step Progression

About

The contribution process would be easier to understand, if the progression of steps would be automatic.

Outcome

Given a contribution step
When it is completed
Then the UI is automatically forwared to the next step

Given a contribution step view
When a contribution is between the currently selected and the previous step requiring user input
Then a note and a loading indicator is shown.

Vorschlag für Autorenschaft der Datensätze formulieren

Wie kann angezeigt werden, von wem, welche Daten stammen, wer wie dazu beiträgt etc. -> Sichtbarkeit der beteiligten Forschenden sicherstellen, aber auch Sichtbarkeit von DP-REX sicherstellen

Weitere Gedanken:

  • Autorenschaft automatisch über Contributions tracken
  • Initialier Merge: Doppelte Einträge werden als Contribution zwischen Autor:innen aufgeteilt
  • Wenn bei späteren Merges Einträge schon vorliegen -> keine Autorenschaft mehr

Generelle Herausforderung: Veränderungen der Autorenschaft über Zeit

Remove hidden Tags from profile

About

Currently it is not possible to remove tags from a users default tag definitions.

Outcome

Given the table header menu
** When** a user clicks on hide column and refreshes the page
Then The tag definition is not shown as a column.

Update Merge Requests on Tag Owner Change

About

Currently when a tag changes ownership, merge requests associated with the tag remain with the previous owner. Instead the merge requests should be transferred to the new owner.

Outcome

Given A tag definition
When it changes ownership
Then all associated merge requests also change ownership.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.