Comments (5)
@rashadg1030 Let's have IW/Sync
directory and corresponding IW.Sync.*
modules for the forwker. Async
and Worker
seem to generic. Sync
is for synchronisation which is that it should do.
Regarding implementing the worker, I have some suggestions:
Implement a separate function that fetches data for a single repo or for a single issue. In this case we will be able to test such functions from GHCi and check the data in SQL tables. And it's easy to call such functions for all fetched repos.
from issue-wanted.
@rashadg1030 I'm going to describe a high-level overview of the sync algorithm.
Main function should look like this:
syncCache = forever $ syncWithGitHub `catch` \...
So we are trying to sync cache in the infinite loop, handling all errors.
syncWithGitHub
function looks like this:
syncWithGitHub = do
repos <- fetchAllRepos
upsertRepos
populateCategoriesAsync <- async $ populateCategories repos
issues <- fetchAllIssues
upsertIssues issues
wait populateCategoriesAsync
The idea is the following:
- First, we fetch all the repositories.
- Second, we populate categories for repositories in a separate thread.
- While we are populating categories, we can update all issues.
- Wait while we update all categories.
Does this make sense for now?
from issue-wanted.
@chshersh
I'm gonna use the algorithm you mentioned above now:
syncWithGitHub = do
repos <- fetchAllRepos
upsertRepos
populateCategoriesAsync <- async $ populateCategories repos
issues <- fetchAllIssues
upsertIssues issues
wait populateCategoriesAsync
from issue-wanted.
@chshersh This is perfect. Thank you!
from issue-wanted.
@chshersh Now that everything is in place, I'm gonna implement the sync algorithm for issues and then clear this issue. There will be a single function that syncs both repos and issues.
from issue-wanted.
Related Issues (20)
- Implement proper pagination HOT 2
- Wire up pagination on frontend
- Buy cool domain name HOT 3
- Figure out how to deploy this project
- Consider using named parameters in queries HOT 4
- Add `repoDefaultBranch` field to `Repo` type HOT 2
- Update README HOT 3
- Decide on how we want to model filtering issues and repos HOT 2
- Decide on sync scheduling strategy
- Implement function for fetching repos page by page
- Handle API rate limit more gracefully when syncing
- Think how to speed up syncing HOT 3
- Endpoint to get all labels
- Endpoint to get all categories
- Tables for labels and categories? HOT 6
- Add frontend check to CI
- Bump up to lts-14.* resolver
- Endpoint should return non-polymorphic data types HOT 2
- Update docker info docs
- Push docker image on docker hub HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from issue-wanted.