prastut / centify Goto Github PK

@ALL We can keep all extra code inside a folder under trenity and then push this one big folder to Github and keep that repo private.

Bitbucket has the free organization feature.

Since we deal with live elements and the entire UX hinges around how fast we render elements when they happened, we need to deeply care about performance.

Some off the mind questions I want answers to:

• What is paint time and how best do we reduce the time the browser paints the interface -> to reactivity? Need to read more into how gatsby.js came up: https://www.gatsbyjs.org/blog/2017-09-13-why-is-gatsby-so-fast/

• Optimizing React components by optimizing componenentDidUpdate lifecycle. The more functional the components, the less they would update, the better we can optimize our codebase by returning false from the componentDidUpdate lifecycle.

This not a high priority issue but investigative work on how we would proceed will be really helpful.

The goal is there should be no jank on the website/app and the UX should be as buttery and smooth. Speed is one of the many things that lead to anxiety, confusion and irritation.

Data

• All entities for a team should have the team also as an entity -> with the corresponding flag.
• Trending data only contains one team's data at the moment.
• Investigate trending logic. Currently, it works on sorting difference between t+10 entities - t entities. Mario Mandazukich pops up but there are no tweets for him which is strange. Please see throttling rates from #28, probably I am missing something.

UX

• Country image.
• Both countries to always be displayed on the left in Carousel Menu.
• Player controls need fixing.
• Horizontal view of the video should integrate with second screen experience.
• flex properties need to be fixed for carousel items. Need to find the best way to fix it according to client properties.
• emojis on video need fixing.
• swipe fixing on second screen.
• replace events gif with events emoji

Pipeline

• entire snippet logic from the backend to the frontend

Question 1

Goal happened at time 't':

t - event
t+x - tweet
t+x+y - tweet to Kafka
t+x+y+z - tweet to Kafka to MongoDB
t+x+y+z+w - tweet to Kafka to MongoDB to backend
t+x+y+z+w+a - tweet to Kafka to MongoDB to backend to frontend

What is the time between t+x to t+x+y+z+w+a?

Question 2

What is the rate of Twitter API as in how many tweets are we able to get per sec? Our these tweets batched for Watson processing or not.

Add more questions here or in comments.

Every key inside MongoDB -> and then inside code.

• server has to shift to ssl.
• why do I have to use sudo for everything?

• Match Data - list of all matches, each match data, what teams are playing, start time. Simulated start time we can handle in memory because it is volatile and different for each client.

• Entities for teams. Currently, entity are players (as I see when I get from frontend) but entitityType has to be introduced where I can get the teams as well as players. What if we wanted to show the two teams by default on the frontend. We already process team tweets -> need a pipeline to show them on the frontend.

The entityList.js file should disappear from the code.

DB

Move to RethinkDB over MongoDB. It's perfect for our usage. https://rethinkdb.com/faq/

Auth

Use Passport.js (when we move to the model where users can sign in and reply)

API

Move to GraphQl (Apollo toolchain)

##Express
Setup it like it's here: https://github.com/withspectrum/spectrum/blob/alpha/api/index.js

Javascript Tooling

There is a lot of work that's been done in the JS ecosystem to finally put the fires down between developers. Why not use them?

Our team is primarily using VSCode as the editor, but feel free to use any which supports these tools (most popular editor have extensions for them):

• Prettier.
• Eslint
• Flow (will introduce when code has to scale)

Eslint

• Too many console.logs spoil the broth. Warning for console.logs
• Introduce Airbnb's eslint config.

Good extensions to have:
https://marketplace.visualstudio.com/items?itemName=burkeholland.simple-react-snippets

Data

• All entities for a team should have the team also as an entity -> with the corresponding flag.
• Trending data only contains one team's data at the moment.
• Investigate trending logic. Currently, it works on sorting difference between t+10 entities - t entities. Mario Mandazukich pops up but there are no tweets for him which is strange. Please see throttling rates from #28, probably I am missing something.

UX

• Country image.
• Both countries to always be displayed on the left in Carousel Menu.
• Player controls need fixing.
• Horizontal view of the video should integrate with second screen experience.
• flex properties need to be fixed for carousel items. Need to find the best way to fix it according to client properties.
• emojis on video need fixing.
• swipe fixing on second screen.
• replace events gif with events emoji

Pipeline

• entire snippet logic from the backend to the frontend

Instead of API service querying DB, create a db driver and its apis which would be called upon by NodeAPI service during socket emission.

We need to create a perfect sample dataset for product iterations.

state.db
      .collection(`${collection}_TRENDING`)
      .find(paramsForFind)
      .sort({ timeStamp: -1 })
      .limit(1)
      .toArray();

I don't want to convert to array since it's only one record that this command will return. If I don't use toArray then it throws a cursor object. I just want the data that is returned from the query which backend consumes and throws it to the frontend.

Some articles worth looking: https://stackoverflow.com/questions/20058614/stream-from-a-mongodb-cursor-to-express-response-in-node-js

• server has to shift to ssh in order for a faster pull
• why do I have to use sudo for everything?

Need to figure out the best trending logic as currently it's literally fucked up. We need to really sit down and think about the data schema, it's currently nuts how we organize data at the moment. The following code works but it shouldn't be this convoluted as implemented below

getTrendingEntities = async () => {
    console.log(this.state.sortedTrendingEntities);
    const { matchId, timeInsideMatch } = this.props;
    const dataForTrendingEntitiesCount = await axios.get(
      `/api/match/trending/${matchId}`,
      {
        params: {
          timeInsideMatch
        }
      }
    );

    const trendingEntitiesCount =
      dataForTrendingEntitiesCount.data[0].until_now;

    // console.log("Data Recieved->", trendingEntitiesCount);

    if (isEmpty(this.state.sortedTrendingEntities)) {
      const sortedTrendingEntities = sort(
        (a, b) => b[1] - a[1],
        toPairs(trendingEntitiesCount)
      ).reduce((accumulator, currentValue, index) => {
        return {
          ...accumulator,
          [currentValue[0]]: {
            count: currentValue[1],
            difference: 0
          }
        };
      }, {});

      //   console.log("Intial Dict Set->", sortedTrendingEntities);
      this.setState({ sortedTrendingEntities });
    } else {
      const unsortedTrendingEntities = {};

      Object.keys(trendingEntitiesCount).forEach(entity => {
        const prevDataForEntity = this.state.sortedTrendingEntities[entity];

        if (prevDataForEntity) {
          const oldCount = prevDataForEntity.count;
          const newCount = trendingEntitiesCount[entity];
          const difference = newCount - oldCount;

          unsortedTrendingEntities[entity] = {
            ...prevDataForEntity,
            count: newCount,
            difference
          };
        } else {
          unsortedTrendingEntities[entity] = {
            count: trendingEntitiesCount[entity],
            difference: 0
          };
        }
      });

      const sortedTrendingEntities = fromPairs(
        sort(
          (a, b) => b[1].difference - a[1].difference,
          toPairs(unsortedTrendingEntities)
        )
      );

      //   console.log("updated Dict Set->", sortedTrendingEntities);
      this.setState({ sortedTrendingEntities });
    }
  };

  getSortedTrendingEntities = entitiesDict => {
    const sortedEntitiesArray = sort(
      (a, b) => b[1].difference - a[1].difference,
      toPairs(entitiesDict)
    ).map(i => {
      const entity = i[0];
      const entityData = this.props.allEntities.find(
        data => entity === data.entityName
      );

      if (entityData) {
        return {
          entity,
          image: entityData.entityImageURL
        };
      } else {
        return {
          entity,
          image: null
        };
      }
    });

    return sortedEntitiesArray;
  };

What's the best and why?

facebook/react#9851

• EntitityList - should be an array of dictionaries. {entity: <entity_name>, image: <image_url>}

• TweetData - analysis should be combined into one key. userProfile should become user with the following format

{
  user: {
       name: <name>
       image: <image_url> 
     }
}

@ALL We can keep all extra code inside a folder under trenity and then push this one big folder to Github and keep that repo private.

Bitbucket has the free organization feature.

Product-event is every event that takes place inside the product. It is different from events/event which occur inside the real match.

For the second screen experience:

• Events - these need to be moved to a tailing socket. Currently, throttle = 1sec.
• TrendingEntities - throttle = 10sec
• TrendingEmojis - throttle = 2sec (to create quantity effect)

https://stackoverflow.com/questions/28712248/difference-between-mongodb-and-mongoose

Need to figure out the best trending logic as currently it's literally fucked up. We need to really sit down and think about the data schema, it's currently nuts how we organize data at the moment. The following code works but it shouldn't be this convoluted as implemented below

getTrendingEntities = async () => {
    console.log(this.state.sortedTrendingEntities);
    const { matchId, timeInsideMatch } = this.props;
    const dataForTrendingEntitiesCount = await axios.get(
      `/api/match/trending/${matchId}`,
      {
        params: {
          timeInsideMatch
        }
      }
    );

    const trendingEntitiesCount =
      dataForTrendingEntitiesCount.data[0].until_now;

    // console.log("Data Recieved->", trendingEntitiesCount);

    if (isEmpty(this.state.sortedTrendingEntities)) {
      const sortedTrendingEntities = sort(
        (a, b) => b[1] - a[1],
        toPairs(trendingEntitiesCount)
      ).reduce((accumulator, currentValue, index) => {
        return {
          ...accumulator,
          [currentValue[0]]: {
            count: currentValue[1],
            difference: 0
          }
        };
      }, {});

      //   console.log("Intial Dict Set->", sortedTrendingEntities);
      this.setState({ sortedTrendingEntities });
    } else {
      const unsortedTrendingEntities = {};

      Object.keys(trendingEntitiesCount).forEach(entity => {
        const prevDataForEntity = this.state.sortedTrendingEntities[entity];

        if (prevDataForEntity) {
          const oldCount = prevDataForEntity.count;
          const newCount = trendingEntitiesCount[entity];
          const difference = newCount - oldCount;

          unsortedTrendingEntities[entity] = {
            ...prevDataForEntity,
            count: newCount,
            difference
          };
        } else {
          unsortedTrendingEntities[entity] = {
            count: trendingEntitiesCount[entity],
            difference: 0
          };
        }
      });

      const sortedTrendingEntities = fromPairs(
        sort(
          (a, b) => b[1].difference - a[1].difference,
          toPairs(unsortedTrendingEntities)
        )
      );

      //   console.log("updated Dict Set->", sortedTrendingEntities);
      this.setState({ sortedTrendingEntities });
    }
  };

  getSortedTrendingEntities = entitiesDict => {
    const sortedEntitiesArray = sort(
      (a, b) => b[1].difference - a[1].difference,
      toPairs(entitiesDict)
    ).map(i => {
      const entity = i[0];
      const entityData = this.props.allEntities.find(
        data => entity === data.entityName
      );

      if (entityData) {
        return {
          entity,
          image: entityData.entityImageURL
        };
      } else {
        return {
          entity,
          image: null
        };
      }
    });

    return sortedEntitiesArray;
  };

• Server side code for events timeline
• Mocks for events timeline
• Frontend for events timeline

Prologue:

We use the Material UI library under the hood to provide us ready-made styles and config essential for rapid prototyping. I learnt a lot of what developers at the bleedging edge believe and how they work through the library. In Trenity, we specifically the Grid API and the withStyles HOC. For reasons, I can get on a call.

Since Material UI is heavily themable and customizable, that bodes well for us in terms of time-saving. We can quickly flesh out low fidelity prototypes and then when it comes time to change stuff a.k.a shift to our own design system, MUI using React's context API hydrates the entire UI with a theme object. It also gives us other time wins -> rather than making our silly components like Buttons and Modals etc, we use them and mod them for our needs and invest the time we saved by customizing the custom components if and when we make to suit our theme.

Therefore if you change a button once at the top level of your UI, all buttons change throughout your codebase. That is the huge advantage of using MUI + React.

I concur with this person's statement: https://medium.com/@NareshBhatia/i-use-material-ui-even-for-projects-that-dont-use-material-design-76ef88227cf9

Example

This is how a component is styled inside our codebase:

import * as React from 'react';
import { withStyles } from '@material-ui/core/styles';

const styles = {
   exampleStyle: {
      background: "red",
      fontSize: "2em", 
   }
}
  

class Example extends React.Component {
  render() {
    return <div className={classes.exampleStyle}>Hey</div>;
  }
}

export default withStyles(styles)(Example)

As you can see from the above example, we have a styles object that withStyles HOC takes in as one parameter and the other parameter is the raw component. It then renders out a new component with the classes prop which hydrates the component with the styles from the styles object.

This is pretty amazing as the component is rendered somewhat like this in the DOM tree:

<ComponentName-thenameoftheclasswhereyoudefinedthestyles-uniquehashsothatstylesdontmessupeachother>. 

The styles automatically become very descriptive + you don't need to find a unique name for every CSS selector i.e if there a root style in one component you can use the same name for root style for a different component and the naming problems in CSS won't be there because of the ComponentName + unique hash's combination.

Task

Since styles is a an object,

• how do we get to write styles like button:hover
• how do we extend base styles eg:
  • you have a button component where you define your base styles like font, size of the button.
  • now you use the buttom component and sprinkle some CSS magic to make an Error Button component which takes the base styles of the button component and add the Red color to the button.

To make changes to frontend and deploy to server is a very hectic process atm

Data

• All entities for a team should have the team also as an entity -> with the corresponding flag.
• Trending data only contains one team's data at the moment.
• Investigate trending logic. Currently, it works on sorting difference between t+10 entities - t entities. Mario Mandazukich pops up but there are no tweets for him which is strange. Please see throttling rates from #28, probably I am missing something.

UX

• Country image.
• Both countries to always be displayed on the left in Carousel Menu.
• Player controls need fixing.
• Horizontal view of the video should integrate with second screen experience.
• flex properties need to be fixed for carousel items. Need to find the best way to fix it according to client properties.
• emojis on video need fixing.
• swipe fixing on second screen.
• replace events gif with events emoji

Pipeline

• entire snippet logic from the backend to the frontend

@aviral254 please comment on all the resources that you found for events timeline. We can build a pipeline for every site and keep it for future.

(Specifically keep in mind the ones which we will probably use in EPL, use banter group to find resources)

What's the best and why?

facebook/react#9851

Automation requires:

• Setting Match config (preferably via a dashboard)

  • Match Time
  • Duration inside the match (time now)
  • Collection inside the DB (follow a name setting procedure in case optimization seems overkill)
  • Entities for the match. Ideally be automatic. Should be able to enter the team name -> go to FIFA website -> find the list of players and import it inside the database. A better solution is to make this script run for all teams beforehand since for a particular tournament all teams remain same.

• Ability to switch between matches. Currently, I can only simulate one match. When there is no match happening on the world, I should be able to switch between matches.

Product-event is every event that takes place inside the product. It is different from events/event which occur inside the real match.

For the second screen experience:

• Events - these need to be moved to a tailing socket. Currently, throttle = 1sec.
• TrendingEntities - throttle = 10sec
• TrendingEmojis - throttle = 2sec (to create quantity effect)

state.db
      .collection(`${collection}_TRENDING`)
      .find(paramsForFind)
      .sort({ timeStamp: -1 })
      .limit(1)
      .toArray();

I don't want to convert to array since it's only one record that this command will return. If I don't use toArray then it throws a cursor object. I just want the data that is returned from the query which backend consumes and throws it to the frontend.

Some articles worth looking: https://stackoverflow.com/questions/20058614/stream-from-a-mongodb-cursor-to-express-response-in-node-js

Latch onto (ranked in order of monetization preference):

• Hotstar video.
• Sony Liv video.
• Any YouTube videos for the world cup and show our UI elements. This could easily go on ProductHunt.

Every key inside MongoDB -> and then inside code.

https://stackoverflow.com/questions/28712248/difference-between-mongodb-and-mongoose

I bet https://www.nltk.org/ this can do everything that Watson does.

Task:

Port our codebase to yarn workspaces so that there is a common node_modules folder and we don't repeat package installation.

Reference:

https://medium.com/@kohlandpen/yarn-workspaces-cra-now-fullstack-react-%EF%B8%8F-35bcb9229f79

More info:

Currently, our client and server are entirely decoupled into different folders but housed under one repository for easier code management.

Most apps are moving towards a monolithic structure where codebase lives in one repo, but each folder can serve as a microservice. Essentially it's not mandatory to have different repos for decoupled structures when we can simply have one single repo. I was against the microservice folder structure from the start of the trend, I liked the concept in terms of abstraction but a concept shouldn't hamper developer productivity. Afterall tools are made for us, not the other way round.

Coming back to our architecture, we have client which is bootstrapped with create-react-app package and server which is bootstrapped with node/express combination. As you can see both folders have node_modules. What I would like is a central node_modules folder. Why? Because most of the libraries are shared between client and server for eg: I want to use moment both on the client as well as the server side. Currently, in our architecture, I have to install it twice.

We need to lift the node_modules to the parent folder and then symlink packages inside these folders. Yarn workspaces enables this.

Itna paisa kharch karke HTTPS nahi kiya toh kya kiya @aviral254

• server has to shift to ssh in order for a faster pull
• why do I have to use sudo for everything?

I bet https://www.nltk.org/ this can do everything that Watson does.

Data

• All entities for a team should have the team also as an entity -> with the corresponding flag.
• Trending data only contains one team's data at the moment.
• Investigate trending logic. Currently, it works on sorting difference between t+10 entities - t entities. Mario Mandazukich pops up but there are no tweets for him which is strange. Please see throttling rates from #28, probably I am missing something.

UX

• Country image.
• Both countries to always be displayed on the left in Carousel Menu.
• Player controls need fixing.
• Horizontal view of the video should integrate with second screen experience.
• flex properties need to be fixed for carousel items. Need to find the best way to fix it according to client properties.
• emojis on video need fixing.
• swipe fixing on second screen.
• replace events gif with events emoji

Pipeline

• entire snippet logic from the backend to the frontend

Thread for discussions/ideas on UX for our various consumer products.

• server has to shift to ssh in order for a faster pull
• why do I have to use sudo for everything?