Coder Social home page Coder Social logo

intro2stats's Introduction

Introduction to Statistics

Bitdeli Badge

Inspired by Allen Downey's books Think Stats and Think Bayes, this is an attempt to learn Statistics using an application-centric programming approach.

Objective

Showcase real-life examples and what statistics to use in each of those examples. Almost every book teaches a concept and shows an example. Ultimately, every topic gets treated separately and no holistic view is presented. Here, we would take examples and see how to make sense out of it.

Topics covered

  • Mean, Median, Mode
  • Standard Deviation
  • Variance
  • Co-variance
  • Probability Distribution
  • Hypothesis Testing
  • t-test, p-value, chi-squared test
  • Confidence Intervals
  • Confidence levels and Sigificance levels
  • Correlation
  • Resampling (and uses in Big Data)
  • A/B Testing
  • A simple linear regression model

Workshop Plan

We would be using Marijuana prices in various states of the USA, along with demographic data of the USA based on the latest census data

There will be separate ipython notebooks - grouped by topic similarities. notebooks will be uploaded later Some examples include:

  • Find sum of people buying weed in a year, by various states.
  • Find mean of price in a week/month, by various states.
  • Find variance of price in selected states. Find variance of selected states by week of month
  • Define distribution. Plot histograms
  • Determining outliers (Plots, quantiles, box plots, percentiles) in weed price data
  • Continuous distributions(exponential distribution, normal distribution)
  • Introduction to Probability
  • Hypothesis testing. Check if weed price across states are similar or not. Check for different qualities of weed
  • Resampling
  • Simple regression model: Predict weed price for the next month. Understand the output and diagnostics
  • Introduction to A/B testing: Impact of regulation and deregulation on a couple of states

Prerequisites

  • Basics of Python. User should know how to write functions; read in a text file(csv, txt, fwf) and parse them; conditional and looping constructs; using standard libraries like os, sys; lists, list comprehension, dictionaries
  • It is good to know basics of the following:
    • Numpy
    • Scipy
    • Pandas
    • Matplotlib
    • Seaborn
    • IPython and IPython notebook - Everything here would be an IPython notebook
  • Software Requirements
    • Python 2.7
    • git - so that this repo can be cloned :)
    • virtualenv
    • Libraries from requirements.txt

Optional

Users could choose to install Anaconda, if they want. If using Anaconda or Enthought, please ensure that all libraries listed in the requirements.txt are installed.

Note to Windows Users: Neither of us use Windows. From past workshop experiences, Windows users have faced issues installing the way explained below. It is advisable to install Anaconda and ensure that all the libraries listed in the requirements.txt file are installed.

Setup Guide

Clone the repository

$ git clone https://github.com/rouseguy/intro2stats.git

Create a virtual environment & activate

$ cd intro2stats
$ virtualenv env
$ source env/bin/activate

Install reqirements from requirements file

$ pip install -r requirements.txt

Note: Make sure you have libraries for png & freetype.

Ubuntu users can install the below

apt-get install libfreetype6-dev
apt-get install libpng-dev

Script to check if installation is fine for the workshop

Please execute the following at the command prompt

$ python check_env.py

If any library has a FAIL message, please install/upgrade that library.


Creative Commons License
Introduction to Statistics using Python by Bargava and Raghotham is licensed under a Creative Commons Attribution 4.0 International License.

intro2stats's People

Contributors

bitdeli-chef avatar deborah-digges avatar raghothams avatar rouseguy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

intro2stats's Issues

Installing requirements is throwing error for functools

I am running Ubuntu 14.04 64 bit on a machine with Python 2.7 in the venv. Whenever I am trying to install requirements.txt or requirements_linux.txt, I am getting the following error

Could not find a version that satisfies the requirement functools32==3.2.3.post2 (from -r requirements.txt (line 6)) (from versions: 3.2.3-1, 3.2.3-1, 3.2.3-2, 3.2.3-2)

I bypassed this error by simply writing

functools32==3.2.3-2

I think both the requirements should be updated or is just a error specific to my machine only?

Potential Error in Resampling Notebook: shuffle_experiment function

In the shuffle_experiment function, I believe the accessing of the experiment_data array used to calculate the mean is off (see highlighted portion below). Specifically, the condition is grabbing the rows labeled 0 or 1 correctly, however, after that we need to only grab the second column of each row so as to exclude the label value from the mean calculation.

Current code:

def shuffle_experiment(number_of_times):
experiment_diff_mean = np.empty([number_of_times,1])
for times in np.arange(number_of_times):
experiment_label = np.random.randint(0,2,shoe_sales.shape[0])
experiment_data = np.array([experiment_label, shoe_sales[:,1]]).T
experiment_diff_mean[times] = experiment_data[experiment_data[:,0]==1].mean()
- experiment_data[experiment_data[:,0]==0].mean()

return experiment_diff_mean

Proposed code:

def shuffle_experiment(number_of_times):
experiment_diff_mean = np.empty([number_of_times,1])
for times in np.arange(number_of_times):
experiment_label = np.random.randint(0,2,shoe_sales.shape[0])
experiment_data = np.array([experiment_label, shoe_sales[:,1]]).T
experiment_diff_mean[times] = experiment_data[experiment_data[:,0]==1][:,1].mean()
- experiment_data[experiment_data[:,0]==0][:,1].mean()
return experiment_diff_mean

The same issue exists in this block:

experiment_diff_mean = experiment_data[experiment_data[:,0]==1][:,1].mean()
- experiment_data[experiment_data[:,0]==0][:,1].mean()

Statistical significance described the other way around

In the notebook 2 "Warm-up", the following statement is wrong:

Simulation. Run the experiment 100,000 times. Find the percentage of times the experiment returned 24 or more heads. If it is more than 5%, we conclude that the coin is biased.

The statement should be

Simulation. Run the experiment 100,000 times. Find the percentage of times the experiment returned 24 or more heads. If it is less than 5%, we conclude that the coin is biased.

This error was introduced by commit 1c3c897

Python linear regression package with statistical tests focus

In your linear regression Notebook, you mentioned about scikit-learn, but as we know that does not have good statistical tests for the linear regression (residual analysis or F-test etc. like R does).

I have developed a lightweight package called MLR which mixes the simple fit and predict of the scikit-learn package with statistical inference and residual analysis.

You can refer to it or use it in your Notebooks if you like. Here are the docs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.