This is a capstone project for Introduction to Data Science (DS-GA 1001) at NYU Center for Data Science.
The purpose of this project is to understand the state of Data Science and Machine Learning across industries and technologies.
- Inferential Statistics
- Data Visualization
- Predictive Modeling
- Clustering
- Supervised Classification
- Python
- Pandas, jupyter
- Sklearn
We are using Kaggle annual data science and machine learning survey responses data from 2022 and previous years. Data Sources:
- 2022 Kaggle Machine Learning & Data Science Survey
- 2021 Kaggle Machine Learning & Data Science Survey
- 2020 Kaggle Machine Learning & Data Science Survey
Inferential Statistics:
- Hypothesis: What Data Science jobs have the highest salaries?
- Hypothesis: Is the representation of Advanced degrees (Masters & above) among Data Professionals increasing over years?
Salary prediction: Predict data science job salaries of individuals in the United States based on covariates such as their job title, industry, skill sets and experience. Results:
Model | Train R² | Test R² | Adj Train R² | Adj Test R² | |
---|---|---|---|---|---|
1 | OLS | 0.159 | 0.18 | 0.389 | 0.393 |
2 | Ridge | 0.151 | 0.186 | 0.388 | 0.393 |
3 | Lasso | 0.157 | 0.184 | 0.386 | 0.392 |
4 | Linear SVR | 0.092 | 0.113 | 0.306 | 0.298 |
5 | Random Forest | 0.863 | 0.165 | 0.674 | 0.399 |
6 | XGBoost | 0.433 | 0.178 | 0.516 | 0.404 |
7 | Poisson GLM | 0.182 | 0.223 | 0.396 | 0.401 |
Clustering: Clustering the survey respondents into clusters based on their skills and expeiriences Approach:
- Traditional KMeans clustering using only numerical data.
- KPrototypes clustering, which is compatible with categorical information as described by Huang (1998)
Classification: Identify the most suitable jobs for a user based on the user’s responses about their skills, exposure and experience. Results:
Model | Accuracy | F1-Score | Top 2 accuracy |
---|---|---|---|
Dummy Classifier | 35% | 0.10 | 65% |
Decision Tree | 41% | 0.41 | 51.9% |
KNN Classifier | 42% | 0.33 | 62.6% |
Random Forest | 48% | 0.44 | 69.0% |
SVM | 47% | 0.45 | 78.6% |
Multinomial Logisitic Reg | 48% | 0.45 | 71.8% |
Multi-Layer Perceptron | 54% | 0.43 | 78.8% |
XGBoost | 54% | 0.45 | 79.1% |
LightGBM | 55% | 0.45 | 79.62% |
Findings and analysis: Report
Name | Github |
---|---|
Sargun Nagpal | sargun-nagpal |
Harsha Koneru | harshakoneru |
Sharad Dargan | sharaddargan |