proversity-org / edx-analytics-pipeline Goto Github PK
View Code? Open in Web Editor NEWThis project forked from openedx-unsupported/edx-analytics-pipeline
License: GNU Affero General Public License v3.0
This project forked from openedx-unsupported/edx-analytics-pipeline
License: GNU Affero General Public License v3.0
Setup and deploy pipeline on a server for harambee,
setup insights and data api on prod server.
Progress:
Pipeline server created and can execute the hadoop pi job without error, ImportsEnrollments job venv has been created but there is an error coming up from luigi staging missing parameter about defaul_log_level or something.
Insights installed and database migrated: requires SSO setup.
data api installation failed due to database rights of the insights user on the reports database. re-try tonight.
@renevatium shhht but fyi. sofar we looking mkay :)
Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
Hive metastore credentials/permissions are not right. Fix them.
Create LearnerCourseEngagement task including activity/engagement per unit of course.
Luigi (on EMR) not creating password stores, debug this.
Created date not being pulled through by ImportEnrollmentsIntoMysql task. Needed for dashboard to load.
Test CourseActivityDailyTask
Aggregate vertical module acceptance data
Test CourseActivityWeeklyTask
Test ModuleEngagementWorkflowTask
Test the AnswerDistributionWorkflow (performance) task(s)
Sync all imported hdfs data to an s3 bucket for backup.
Test InsertToMysqlCourseEnrollByCountryWorkflow task
Test working tasks with s3 input. Need to update edx-deployment/analytics/override.cfg
Connect to ElasticSearch for module engagement tasks to work as per Gabe's advice
CourseActivityDailyTask doesn't work properly for Insights.
Increase Hadoop/Yarn/Hive memory allocation using deployment configuration settings.
opaque_key_util.py:33 - Unable to parse course_id
Update master for production
Some of the tasks on harambee analytics pipeline server has been failing.
find and fix issues with.
"ImportEnrollmentsIntoMysql, InsertToMysqlAllVideoTask and ModuleEngagementWorkflowTask"
InsertToMysqlAllVideoTask and ModuleEngagementWorkflowTask are failing with the could not override table with empty result set error, looks like one of the database tables are not being transported due to this, either find and add a setting or just force it somehow. donno really.
ImportEnrollmentsIntoMysql fails due to the enrollment event task not being run for the certain period. i.e. the event task is not being run before the insert task
Upgrade luigi from 1.0.17 to 1.0.22. v1.3.0 works but has config issues.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.