why? instagram is the biggest vangav backend template designed for big scale and covering all the core features/utilities: service oriented architecture, worker service, multi-entry-point api, multi-keysapce database, oauth 2, facebook authentication, facebook graph api, rest jobs, periodic jobs, vangav mighty, push notifications, logging, analytics, client generator, test service, bots service, ...
![instagram backend design](https://camo.githubusercontent.com/f895c6839af4686901719ed6332de364f52fa0ddbc525c673ac7e84ac820c172/68747470733a2f2f73636f6e74656e742d6d6164312d312e78782e666263646e2e6e65742f762f7433312e302d382f32303933343838385f313937363138353537393236333735325f323432313534343138313134373834343930345f6f2e706e673f6f683d3463303136313639656434336533636563643038623635386432633164633532266f653d3541324436424437)
- signup/login using email/facebook
- request authentication using oauth 2
- fetching/updating users' information
- follow/unfollow and get followers/following lists for users
- posting photos
- get posts by feed and by user
- interacting with posts (like, comment, ...)
- get top users/posts
- searching for users
- get server time
- is a helper service for instagram; responsible for distributing posted photos to followers (because on photo can be posted to tens of millions of followers; such an expensive operation should be distributable and shouldn't overload the main service)
- crawls for failed jobs in the database to retry executing them (e.g.: a dispense job, ...)
- uses vangav mighty to update top users (per geo grid, per country and world wide) every week
- uses vangav mighty to update top posts (per geo grid, per country and world wide) every day
- writes logs and analytics in the database
- sends push notifications (ios and android)
- handles fetching logs and analytics for system admin
- has a full-coverage test for all the services above
- simulates incremental usage behavior from many cities around the world with some celebrity users
- this service is based on vangav backend's instagram template
- this service has the 90+% of the vangav backend's generated code + the 10-% of the logic code needed to complete the service
- for first timers - follow the steps in the system requirements tutorial
- for first timers - follow the steps in the workspace initialization tutorial
- download instagram, instagram dispense, instagram jobs, instagram worker and instagram dash board projects (from the green
clone or download
button) inside the workspace directory created previously (my_services
) and unzip them
- rename unzipped directories, remove the
-master
from their names
- in the terminal
cd
to vos_instagram/cassandra/cql/
- execute
./_start_cassandra.sh
to start cassandra
cd
to vos_instagram/cassandra/cql/drop_and_create/
- execute the commands
./_execute_cql.sh ig_....cql
to initialize the services' database tables; repeat this step for all the .cql
files
cd
to vos_instagram_worker
and execute ./_run.sh
to start the instagram worker service on port 8000
cd
to vos_instagram_dispense
and execute ./_run.sh 3000
to start the instagram dispense service on port 3000
cd
to vos_instagram_jobs
and execute ./_run.sh 4000
to start the instagram jobs service on port 4000
cd
to vos_instagram_dash_board
and execute ./_run.sh 5000
to start the instagram dash board service on port 5000
cd
to vos_instagram
and execute ./_run.sh
to start the instagram service on port 9000
- from your prefered client (we recommned postman) start trying the service; refer to the features and service references sections for reference
- at the end to stop the services: press
control + d
in the terminal session where each service was started in (9, 10, 11, 12 and 13)
- to stop cassandra: execute
ps auwx | grep cassandra
to get cassandra's (pid)
then kill -9 (pid)
to stop cassandra
- follow the following steps to import the downloaded backend service in eclipse
cd
to the service's directory and execute the ./_eclipsify.sh
script
- file > import > general > existing projects into workspace > next > set "select root directory" to my_services > under projects make sure that vos_calculate_sum is selected > finish
- double check the java version used for compiling the project: right click the project > properties > java compiler > enable project specific settings > compiler compliance level > 1.7 or 1.8
- generate multiple services (main + dispense + jobs + worker + dash board) to work together in a service oriented architecture
- generate a multi-keyspace database
- oauth 2 and facebook authentication
- facebook graph api
- rest jobs
- periodic jobs
- vangav mighty
- push notifications
- logging
- analytics
- client generator
- test and bots services
controller(s) |
feature |
signup email, login email, login facebook, get access token, refresh access token and logout |
handle user signup, loging and requests' authentication |
update facebook info, update last active time, update location, update profile picture, get user info and get profile picture |
handle setting/updating users' info and getting it |
follow, unfollow, get followers and get following |
handle building the social graph and exploring it |
post photo |
handles posting a new photo to a user's followers |
get feed posts, get user posts, get post info, get post photo id, get photo, get photo thumbnail, get post likes and get post comments |
handle getting feed/users' posts and posts' info/content/likes/comments |
like post, unlike post, comment and delete comment |
handle interacting with posts |
get top users and get top posts |
handle exploring top users/posts |
search by email |
handles searching for users by their e-mail |
get server time |
getting server's time is important for various reasons like enabling the client to set how old a post is (2h, 1w, ...) |
controller |
feature |
start jobs |
stars the periodic job for retrying failed rest jobs and the periodic jobs for ranking users and posts |
controller |
feature |
test all controllers |
runs a full-coverage test for the instagram services above and returns a full report (success, failure, request-to-response time, ...) |
controller |
feature |
start bots |
starts simulating incremental usage behavior from many cities around the world with some celebrity users |
reference |
explanation |
routes |
api routes |
prop |
in addition to vangav backend's properties files, this directory contains instagram properties files: constants and dispense |
reverse_geo_coding |
reverse geo coding data |
controllers.json |
api request/response's elements |
ig_auth.keyspace |
ig_auth is the keyspace used for all authentication-related tables |
ig_app_data.keyspace |
ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...) |
ig_blobs.keyspace |
ig_blobs is the keyspace used for all blob-related tables (photos, thumbnails, ...) |
ig_jobs.keyspace |
ig_jobs is the keyspace used for all jobs-related tables -- a job is an operation that's either executed at a later time or issued by one service and executed by another where a jobs table keeps track of unsuccessful jobs which need to be reexecuted |
ig_logging.keyspace |
ig_logging is the keyspace used for all logging-related tables |
ig_analytics.keyspace |
ig_analytics is the keyspace used for all analytics-related tables |
Global.java |
initializes reverse geo coding data |
common |
handles common operations like authentication, cassandra's frequent-operations, omitting blobs from requests/responses for logging, ... |
controllers |
api implementation |
ig_auth |
ig_auth cassandra's keyspace client |
ig_app_data |
ig_app_data cassandra's keyspace client |
ig_blobs |
ig_blobs cassandra's keyspace client |
ig_jobs |
ig_jobs cassandra's keyspace client |
ig_logging |
ig_logging cassandra's keyspace client |
ig_analytics |
ig_analytics cassandra's keyspace client |
reference |
explanation |
routes |
api routes |
controllers.json |
api request/response's elements |
ig_auth.keyspace |
ig_auth is the keyspace used for all authentication-related tables |
ig_app_data.keyspace |
ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...) |
ig_jobs.keyspace |
ig_jobs is the keyspace used for all jobs-related tables -- a job is an operation that's either executed at a later time or issued by one service and executed by another where a jobs table keeps track of unsuccessful jobs which need to be reexecuted |
ig_logging.keyspace |
ig_logging is the keyspace used for all logging-related tables |
ig_analytics.keyspace |
ig_analytics is the keyspace used for all analytics-related tables |
controllers |
api implementation |
ig_auth |
ig_auth cassandra's keyspace client |
ig_app_data |
ig_app_data cassandra's keyspace client |
ig_jobs |
ig_jobs cassandra's keyspace client |
ig_logging |
ig_logging cassandra's keyspace client |
ig_analytics |
ig_analytics cassandra's keyspace client |
reference |
explanation |
routes |
api routes |
prop |
in addition to vangav backend's properties files, this directory contains instagram jobs properties files: constants , users rank and posts rank |
reverse_geo_coding |
reverse geo coding data |
controllers.json |
api request/response's elements |
ig_app_data.keyspace |
ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...) |
ig_jobs.keyspace |
ig_jobs is the keyspace used for all jobs-related tables -- a job is an operation that's either executed at a later time or issued by one service and executed by another where a jobs table keeps track of unsuccessful jobs which need to be reexecuted |
ig_logging.keyspace |
ig_logging is the keyspace used for all logging-related tables |
Global.java |
initializes reverse geo coding data and periodic jobs |
common |
handles common operations like instagram-jobs-service's properties |
periodic_jobs |
handles retrying failed jobs as well as ranking users and posts |
controllers |
api implementation |
ig_app_data |
ig_app_data cassandra's keyspace client |
ig_jobs |
ig_jobs cassandra's keyspace client |
ig_logging |
ig_logging cassandra's keyspace client |
reference |
explanation |
routes |
api routes |
controllers.json |
api request/response's elements |
ig_auth.keyspace |
ig_auth is the keyspace used for all authentication-related tables |
ig_app_data.keyspace |
ig_app_data is the keyspace used for all user-info-related tables which are directly relevant to the app's functionality (e.g.: posts, followers, ...) |
ig_logging.keyspace |
ig_logging is the keyspace used for all logging-related tables |
ig_analytics.keyspace |
ig_analytics is the keyspace used for all analytics-related tables |
common |
handles common operations like service-wide constants |
controllers |
api implementation |
ig_auth |
ig_auth cassandra's keyspace client |
ig_app_data |
ig_app_data cassandra's keyspace client |
ig_logging |
ig_logging cassandra's keyspace client |
ig_analytics |
ig_analytics cassandra's keyspace client |
reference |
explanation |
routes |
api routes |
setup_data |
contains a one-time setup data with all the cities (id, location, population, ...) where the bots are created |
prop |
in addition to vangav backend's properties files, this directory contains the clients properties file clients |
controllers.json |
api request/response's elements |
vos_instagram.client_java |
instagram client's api request/response's elements |
ig_bots.keyspace |
ig_bots is the keyspace used for all bots-related vos_instagream tables |
Global.java |
initializes cities and bots |
cities |
has the cities' json representation and loader |
bots |
has the bots' periodic jobs and the bots creator |
clients |
contains the generated clients for instagram service |
controllers |
api implementation |
ig_bots |
ig_bots cassandra's keyspace client |
- this section lists the 10-% code added after vangav backend generated 90+% of the code
file/dir |
change |
prop |
added instagram properties files: constants and dispense |
reverse_geo_coding |
added reverse geo coding data |
Global.java |
added the initialization of reverse geo coding data |
common |
added common operations like authentication, cassandra's frequent-operations, omitting blobs from requests/responses for logging, ... |
controllers |
added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages |
file/dir |
change |
controllers |
added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages |
file/dir |
change |
prop |
added instagram jobs properties files: constants , users rank and posts rank |
reverse_geo_coding |
added reverse geo coding data |
Global.java |
added the initialization of reverse geo coding data and periodic jobs |
common |
added common operations like instagram-jobs-service's properties |
periodic_jobs |
added the periodic jobs for retrying failed jobs as well as ranking users and posts |
controllers |
added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages |
file/dir |
change |
common |
added common operations like service-wide constants |
controllers |
added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages |
file/dir |
change |
common |
added common operations like representing instagram users |
controllers |
added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages |
file/dir |
change |
setup_data |
added a one-time setup data with all the cities (id, location, population, ...) where the bots are created |
Global.java |
added the initialization of cities and bots |
cities |
added the cities' json representation and loader |
bots |
added the bots' periodic jobs and the bots creator |
controllers |
added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages |
- following are the error codes of instagram services
class |
code : sub_code |
explanation |
Constants |
|
|
|
300 : 1 |
couldn't initialize kAuthCodeLifeTime |
|
300 : 2 |
couldn't initialize kAccessTokenLifeTime |
|
300 : 3 |
couldn't initialize kFacebookAppId |
|
300 : 4 |
couldn't initialize kDefaultRegion |
|
300 : 5 |
couldn't initialize kWorldRegion |
|
300 : 6 |
couldn't initialize kDefaultGridId |
|
300 : 7 |
couldn't initialize kGeoGridDimensionMetres |
|
300 : 8 |
couldn't initialize kGeoGridsConfig |
|
300 : 9 |
couldn't initialize kThumbnailDimension |
CommonPlayHandler |
|
|
|
400 : 1 |
no access token for user id and device token |
|
400 : 2 |
wrong access token |
|
400 : 3 |
already expired access token |
|
400 : 4 |
just expired access token |
HandlerComment |
|
|
|
401 : 1 |
can't comment on a post that doesn't exist |
|
401 : 2 |
non-fatal: invalid device type |
DeleteComment |
|
|
|
402 : 1 |
post/comment doesn't exist |
HandlerFollow |
|
|
|
403 : 1 |
can't follow self |
|
403 : 2 |
can't follow an non-existing user |
|
403 : 3 |
already following user |
|
403 : 4 |
non-fatal: invalid device type |
HandlerGetAccessToken |
|
|
|
404 : 1 |
auth code doesn't exist |
|
404 : 2 |
wrong auth code |
HandlerGetFollowers |
|
|
|
406 : 1 |
can't get followers for non-existing user |
HandlerGetFollowing |
|
|
|
407 : 1 |
can't get following for non-existing user |
HandlerGetPhoto |
|
|
|
408 : 1 |
photo doesn't exist |
HandlerGetPhotoThumbnail |
|
|
|
409 : 1 |
photo doesn't exist |
HandlerGetPostComments |
|
|
|
410 : 1 |
post doesn't exist |
HandlerGetPostInfo |
|
|
|
411 : 1 |
post doesn't exist |
HandlerGetPostLikes |
|
|
|
412 : 1 |
post doesn't exist |
HandlerGetPostPhotoId |
|
|
|
413 : 1 |
post doesn't exist |
HandlerGetProfilePicture |
|
|
|
414 : 1 |
user doesn't exist |
HandlerGetUserInfo |
|
|
|
418 : 1 |
user doesn't exist |
HandlerGetUserPosts |
|
|
|
419 : 1 |
user doesn't exist |
HandlerLikePost |
|
|
|
420 : 1 |
post doesn't exist |
|
420 : 2 |
already liked |
|
420 : 3 |
non-fatal: invalid device type |
HandlerLoginEmail |
|
|
|
421 : 1 |
e-mail isn't signed up |
|
421 : 2 |
wrong password |
|
421 : 3 |
e-mail isn't signed up - triggers in case authentication is disabled |
HandlerLoginFacebook |
|
|
|
422 : 1 |
couldn't get user's name from facebook graph api |
HandlerLogout |
|
|
|
423 : 1 |
no refresh token for this user/device_token |
|
423 : 2 |
wrong refresh token |
HandlerRefreshAccessToken |
|
|
|
425 : 1 |
no access token for user |
|
425 : 2 |
wrong access token |
|
425 : 3 |
no refresh token |
|
425 : 4 |
wrong refresh token |
HandlerSignupEmail |
|
|
|
427 : 1 |
e-mail already signed up |
HandlerUnfollow |
|
|
|
428 : 1 |
can't unfollow a user that doesn't exist |
|
428 : 2 |
a user can't unfollow another unless she/he is following that user already |
HandlerUnlikePost |
|
|
|
429 : 1 |
can't unlike a post that doesn't exist |
|
429 : 2 |
user didn't like this post before to unlike it |
class |
code : sub_code |
explanation |
Constants |
|
|
|
1000 : 1 |
couldn't initialize kVosInstagramUrl |
|
1000 : 2 |
couldn't initialize kVosInstagramDashBoardUrl |
class |
code : sub_code |
explanation |
Constants |
|
|
|
1000 : 1 |
couldn't initialize kVosInstagramUrl |
![linkedin share](https://camo.githubusercontent.com/6d242166ffe510c6ebc808d1fc16c4b334b8239b72bf192635e2fb15faf04e27/687474703a2f2f652d6169726c6c632e636f6d2f77702d636f6e74656e742f7468656d65732f6e6562756c612f696d616765732f736f6369616c5f626c61636b2f6c696e6b6564696e2e706e67)
![vangav's consultant](https://camo.githubusercontent.com/f12cbdf2df0564232cd756ba8411c7fba79eaebb57d2851ebbf0fcf66caabb36/687474703a2f2f7777772e666f6f7462616c6c686967686c69676874733234372e636f6d2f696d616765732f6d6f62696c652d73686172652f66622d6d657373656e6765722d36347836342e706e67)