zimiwang / data-mining

HW1: Python Programming for Data Analysis

Objective: This assignment focuses on utilizing Python to perform basic data manipulation and analysis tasks. It serves as an introduction to Python's capabilities for handling datasets and performing simple statistical analyses.

Tasks:

• Data Cleaning: Techniques for handling missing values, outliers, and errors in datasets.
• Data Transformation: Converting data into a suitable format for analysis, including normalization and scaling.
• Basic Statistical Analysis: Computing mean, median, mode, and standard deviation to understand the distribution of data.

HW2: Data Exploration and Visualization

Objective: The goal is to gain insights into a dataset through exploratory data analysis and visualization. This homework emphasizes the use of plotting libraries in Python to create histograms, scatter plots, and box plots that reveal underlying patterns, trends, and outliers in the data.

Tasks:

• Exploratory Data Analysis (EDA): Use statistical summaries and visualization techniques to explore datasets.
• Data Visualization: Implement various visualization techniques to communicate data insights effectively.

HW3: Distances and LSH

Objective: This assignment explores the concept of Locality-Sensitive Hashing (LSH) for efficient similarity search and retrieval in high-dimensional data. It introduces students to different distance metrics and their applications in identifying similar items within datasets.

Tasks:

• LSH Implementation: Implement LSH algorithms to hash data points into buckets to facilitate fast retrieval.
• Euclidean Distances: Calculate distances between data points to measure similarity.
• Jaccard Similarity: Estimate the best values of hash functions to provide good separation at a given similarity threshold.

HW4: Clustering

Objective: Investigate various clustering techniques to group similar data points. The assignment covers hierarchical clustering methods and point-assignment techniques, such as k-means and k-medians, with applications to different datasets.

Tasks:

• Hierarchical Clustering: Implement Single-Link and Complete-Link clustering algorithms.
• K-means++: Explore the k-means++ algorithm for initializing cluster centers in k-means clustering.
• Cluster Analysis: Analyze the clusters formed by different algorithms to evaluate their effectiveness.

HW5: Frequent Items in Data Sets

Objective: Focus on identifying frequent items in datasets, particularly in the context of streaming data. This homework introduces algorithms designed for finding frequent items in a single pass over the data.

Tasks:

• Misra-Gries Algorithm: Implement the algorithm to find frequent items in a data stream.
• Count-Min Sketch: Build and utilize a Count-Min Sketch data structure to estimate the frequency of items.
• Streaming Data Analysis: Apply streaming algorithms to simulate real-time data processing.

HW6: Regression Techniques

Objective: Explore regression models for predicting numerical outcomes based on input features. This assignment delves into linear regression and introduces regularization techniques to prevent overfitting.

Tasks:

• Linear Regression: Implement a linear regression model to fit the data and predict outcomes.
• Regularization: Apply L1 (Lasso) and L2 (Ridge) regularization techniques to enhance model performance.
• Cross-Validation: Use cross-validation to select the best model parameters and prevent overfitting.

HW7: Dimensionality Reduction

Objective: Learn dimensionality reduction techniques to simplify datasets while retaining their essential characteristics. The assignment covers Singular Value Decomposition (SVD) and Random Projection methods.

Tasks:

• SVD Implementation: Perform SVD on a dataset and analyze its components to understand data structure.
• Random Projection: Apply Gaussian and Sparse Random Projection techniques to reduce data dimensionality.
• High-Dimensional Data Analysis: Explore the effects of dimensionality reduction on data visualization and clustering.

HW8: PageRank

Objective: Implement the PageRank algorithm to rank web pages based on their importance. This assignment focuses on understanding the Power Iteration Method and the concept of teleportation in the context of web link analysis.

Tasks:

• Power Iteration Method: Implement this method to compute PageRank scores for a set of web pages.
• Teleportation: Incorporate teleportation into the PageRank calculation to deal with dead-end nodes.
• PageRank Analysis: Analyze the PageRank scores to identify the most important pages within a given dataset.