Convert Figma logo to code with AI

donnemartin logodata-science-ipython-notebooks

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

27,106
7,827
27,106
38

Top Related Projects

⛔️ DEPRECATED – See https://github.com/ageron/handson-ml3 instead.

Python Data Science Handbook: full text in Jupyter Notebooks

Materials and IPython notebooks for "Python for Data Analysis" by Wes McKinney, published by O'Reilly Media

TensorFlow Tutorial and Examples for Beginners (support TF v1 & v2)

14,856

Jupyter metapackage for installation, docs and chat

aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

Quick Overview

The donnemartin/data-science-ipython-notebooks repository is a comprehensive collection of Jupyter notebooks covering various topics in data science, machine learning, and deep learning. It serves as a valuable resource for both beginners and experienced practitioners, offering practical examples and implementations of popular algorithms and techniques using libraries such as scikit-learn, TensorFlow, and PyTorch.

Pros

  • Extensive coverage of data science topics, from basic statistics to advanced deep learning
  • Well-organized structure with separate directories for different subjects
  • Includes practical examples and real-world datasets
  • Regularly updated with new content and improvements

Cons

  • Some notebooks may be outdated or use older versions of libraries
  • Not all notebooks have detailed explanations, which might be challenging for beginners
  • The sheer volume of content can be overwhelming for newcomers
  • Some advanced topics may require additional background knowledge

Code Examples

Here are a few short code examples from different notebooks in the repository:

  1. Basic data manipulation with pandas:
import pandas as pd

# Create a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Display basic statistics
print(df.describe())
  1. Simple machine learning model using scikit-learn:
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# Load dataset and split into train/test sets
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a Random Forest classifier
clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)

# Evaluate the model
print(f"Accuracy: {clf.score(X_test, y_test):.2f}")
  1. Basic neural network using TensorFlow:
import tensorflow as tf

# Create a simple sequential model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# Compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Display model summary
model.summary()

Getting Started

To get started with the donnemartin/data-science-ipython-notebooks repository:

  1. Clone the repository:

    git clone https://github.com/donnemartin/data-science-ipython-notebooks.git
    
  2. Install the required dependencies:

    pip install -r requirements.txt
    
  3. Launch Jupyter Notebook:

    jupyter notebook
    
  4. Navigate to the desired notebook and start exploring the content.

Competitor Comparisons

⛔️ DEPRECATED – See https://github.com/ageron/handson-ml3 instead.

Pros of handson-ml

  • More focused on practical machine learning implementations
  • Comprehensive coverage of modern ML techniques and frameworks
  • Structured as a cohesive learning path with clear progression

Cons of handson-ml

  • Less diverse in terms of data science topics covered
  • Fewer examples of data visualization and exploratory data analysis
  • May be more challenging for absolute beginners in programming

Code Comparison

handson-ml:

from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(n_estimators=100, random_state=42)
forest_clf.fit(X_train, y_train)
y_pred = forest_clf.predict(X_test)

data-science-ipython-notebooks:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.csv')
df['column'].plot(kind='hist')
plt.show()

The handson-ml repository focuses on implementing machine learning algorithms, while data-science-ipython-notebooks covers a broader range of data science topics, including data manipulation and visualization. handson-ml provides a more structured learning experience for machine learning, while data-science-ipython-notebooks offers a wider variety of examples across different data science domains.

Python Data Science Handbook: full text in Jupyter Notebooks

Pros of PythonDataScienceHandbook

  • More comprehensive and structured approach to data science topics
  • In-depth explanations accompanying code examples
  • Available as a published book for offline reference

Cons of PythonDataScienceHandbook

  • Less diverse range of topics compared to data-science-ipython-notebooks
  • May not cover the latest cutting-edge techniques in rapidly evolving fields

Code Comparison

PythonDataScienceHandbook:

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 100)
plt.plot(x, np.sin(x))
plt.show()

data-science-ipython-notebooks:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3)
svm = SVC()
svm.fit(X_train, y_train)

The PythonDataScienceHandbook example focuses on data visualization, while the data-science-ipython-notebooks example demonstrates machine learning with scikit-learn. This reflects the different emphases of the two repositories, with PythonDataScienceHandbook providing a more structured learning path and data-science-ipython-notebooks offering a wider variety of practical examples across different data science domains.

Materials and IPython notebooks for "Python for Data Analysis" by Wes McKinney, published by O'Reilly Media

Pros of pydata-book

  • Directly tied to the popular "Python for Data Analysis" book, providing a structured learning path
  • Focuses on core data analysis libraries like pandas, numpy, and matplotlib
  • Includes real-world datasets for practical examples

Cons of pydata-book

  • Less comprehensive coverage of machine learning topics
  • Fewer notebooks and examples compared to data-science-ipython-notebooks
  • May not cover the latest developments in data science tools and techniques

Code Comparison

pydata-book:

import pandas as pd
df = pd.read_csv('examples/ex1.csv')
df.head()

data-science-ipython-notebooks:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

The pydata-book example focuses on basic data manipulation with pandas, while data-science-ipython-notebooks includes more advanced machine learning concepts.

TensorFlow Tutorial and Examples for Beginners (support TF v1 & v2)

Pros of TensorFlow-Examples

  • Focused specifically on TensorFlow, providing in-depth examples for this popular deep learning framework
  • Includes a wide range of TensorFlow examples, from basic operations to advanced neural network architectures
  • Regularly updated to incorporate new TensorFlow features and best practices

Cons of TensorFlow-Examples

  • Limited to TensorFlow, lacking coverage of other data science tools and libraries
  • Less comprehensive in terms of general data science topics compared to data-science-ipython-notebooks
  • May be more challenging for beginners who are not yet familiar with TensorFlow or deep learning concepts

Code Comparison

TensorFlow-Examples:

import tensorflow as tf

# Create a constant tensor
hello = tf.constant('Hello, TensorFlow!')

# Start a TensorFlow session
with tf.Session() as sess:
    print(sess.run(hello))

data-science-ipython-notebooks:

import pandas as pd
import matplotlib.pyplot as plt

# Load and plot data
data = pd.read_csv('example_data.csv')
data.plot(x='Date', y='Value')
plt.show()
14,856

Jupyter metapackage for installation, docs and chat

Pros of Jupyter

  • Core project for the Jupyter ecosystem, providing the foundation for interactive computing
  • Actively maintained with frequent updates and improvements
  • Extensive documentation and community support

Cons of Jupyter

  • More complex setup and configuration compared to pre-packaged notebooks
  • Requires additional effort to curate specific data science examples and tutorials

Code Comparison

data-science-ipython-notebooks:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv('data.csv')
plt.plot(df['x'], df['y'])
plt.show()

jupyter:

from notebook.notebookapp import NotebookApp
import sys

if __name__ == '__main__':
    sys.exit(NotebookApp.launch_instance())

The data-science-ipython-notebooks repository focuses on providing ready-to-use examples for data science tasks, while Jupyter is the core project that enables the creation and execution of interactive notebooks. data-science-ipython-notebooks offers a curated collection of notebooks covering various data science topics, making it easier for users to learn and apply specific techniques. On the other hand, Jupyter provides the underlying infrastructure and tools for creating and running notebooks, offering more flexibility and customization options for advanced users.

aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

Pros of Probabilistic-Programming-and-Bayesian-Methods-for-Hackers

  • Focused, in-depth coverage of Bayesian methods and probabilistic programming
  • Cohesive, book-like structure with a clear learning path
  • Practical examples using PyMC3 for Bayesian inference

Cons of Probabilistic-Programming-and-Bayesian-Methods-for-Hackers

  • Narrower scope, covering only Bayesian methods and probabilistic programming
  • Less variety in topics compared to the broader data science coverage
  • May be more challenging for beginners due to its specialized focus

Code Comparison

data-science-ipython-notebooks:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Probabilistic-Programming-and-Bayesian-Methods-for-Hackers:

import pymc3 as pm
with pm.Model() as model:
    mu = pm.Normal('mu', mu=0, sd=1)
    obs = pm.Normal('obs', mu=mu, sd=1, observed=data)
    trace = pm.sample(1000, tune=1000)

The code snippets highlight the different focus areas of the repositories. data-science-ipython-notebooks demonstrates general data science techniques, while Probabilistic-Programming-and-Bayesian-Methods-for-Hackers showcases Bayesian modeling using PyMC3.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README



data-science-ipython-notebooks

Index


deep-learning

IPython Notebook(s) demonstrating deep learning functionality.


tensor-flow-tutorials

Additional TensorFlow tutorials:

NotebookDescription
tsf-basicsLearn basic operations in TensorFlow, a library for various kinds of perceptual and language understanding tasks from Google.
tsf-linearImplement linear regression in TensorFlow.
tsf-logisticImplement logistic regression in TensorFlow.
tsf-nnImplement nearest neighboars in TensorFlow.
tsf-alexImplement AlexNet in TensorFlow.
tsf-cnnImplement convolutional neural networks in TensorFlow.
tsf-mlpImplement multilayer perceptrons in TensorFlow.
tsf-rnnImplement recurrent neural networks in TensorFlow.
tsf-gpuLearn about basic multi-GPU computation in TensorFlow.
tsf-gvizLearn about graph visualization in TensorFlow.
tsf-lvizLearn about loss visualization in TensorFlow.

tensor-flow-exercises

NotebookDescription
tsf-not-mnistLearn simple data curation by creating a pickle with formatted datasets for training, development and testing in TensorFlow.
tsf-fully-connectedProgressively train deeper and more accurate models using logistic regression and neural networks in TensorFlow.
tsf-regularizationExplore regularization techniques by training fully connected networks to classify notMNIST characters in TensorFlow.
tsf-convolutionsCreate convolutional neural networks in TensorFlow.
tsf-word2vecTrain a skip-gram model over Text8 data in TensorFlow.
tsf-lstmTrain a LSTM character model over Text8 data in TensorFlow.

theano-tutorials

NotebookDescription
theano-introIntro to Theano, which allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.
theano-scanLearn scans, a mechanism to perform loops in a Theano graph.
theano-logisticImplement logistic regression in Theano.
theano-rnnImplement recurrent neural networks in Theano.
theano-mlpImplement multilayer perceptrons in Theano.

keras-tutorials

NotebookDescription
kerasKeras is an open source neural network library written in Python. It is capable of running on top of either Tensorflow or Theano.
setupLearn about the tutorial goals and how to set up your Keras environment.
intro-deep-learning-annGet an intro to deep learning with Keras and Artificial Neural Networks (ANN).
theanoLearn about Theano by working with weights matrices and gradients.
keras-ottoLearn about Keras by looking at the Kaggle Otto challenge.
ann-mnistReview a simple implementation of ANN for MNIST using Keras.
conv-netsLearn about Convolutional Neural Networks (CNNs) with Keras.
conv-net-1Recognize handwritten digits from MNIST using Keras - Part 1.
conv-net-2Recognize handwritten digits from MNIST using Keras - Part 2.
keras-modelsUse pre-trained models such as VGG16, VGG19, ResNet50, and Inception v3 with Keras.
auto-encodersLearn about Autoencoders with Keras.
rnn-lstmLearn about Recurrent Neural Networks (RNNs) with Keras.
lstm-sentence-genLearn about RNNs using Long Short Term Memory (LSTM) networks with Keras.

deep-learning-misc

NotebookDescription
deep-dreamCaffe-based computer vision program which uses a convolutional neural network to find and enhance patterns in images.

scikit-learn

IPython Notebook(s) demonstrating scikit-learn functionality.

NotebookDescription
introIntro notebook to scikit-learn. Scikit-learn adds Python support for large, multi-dimensional arrays and matrices, along with a large library of high-level mathematical functions to operate on these arrays.
knnImplement k-nearest neighbors in scikit-learn.
linear-regImplement linear regression in scikit-learn.
svmImplement support vector machine classifiers with and without kernels in scikit-learn.
random-forestImplement random forest classifiers and regressors in scikit-learn.
k-meansImplement k-means clustering in scikit-learn.
pcaImplement principal component analysis in scikit-learn.
gmmImplement Gaussian mixture models in scikit-learn.
validationImplement validation and model selection in scikit-learn.

statistical-inference-scipy

IPython Notebook(s) demonstrating statistical inference with SciPy functionality.

NotebookDescription
scipySciPy is a collection of mathematical algorithms and convenience functions built on the Numpy extension of Python. It adds significant power to the interactive Python session by providing the user with high-level commands and classes for manipulating and visualizing data.
effect-sizeExplore statistics that quantify effect size by analyzing the difference in height between men and women. Uses data from the Behavioral Risk Factor Surveillance System (BRFSS) to estimate the mean and standard deviation of height for adult women and men in the United States.
samplingExplore random sampling by analyzing the average weight of men and women in the United States using BRFSS data.
hypothesisExplore hypothesis testing by analyzing the difference of first-born babies compared with others.

pandas

IPython Notebook(s) demonstrating pandas functionality.

NotebookDescription
pandasSoftware library written for data manipulation and analysis in Python. Offers data structures and operations for manipulating numerical tables and time series.
github-data-wranglingLearn how to load, clean, merge, and feature engineer by analyzing GitHub data from the Viz repo.
Introduction-to-PandasIntroduction to Pandas.
Introducing-Pandas-ObjectsLearn about Pandas objects.
Data Indexing and SelectionLearn about data indexing and selection in Pandas.
Operations-in-PandasLearn about operating on data in Pandas.
Missing-ValuesLearn about handling missing data in Pandas.
Hierarchical-IndexingLearn about hierarchical indexing in Pandas.
Concat-And-AppendLearn about combining datasets: concat and append in Pandas.
Merge-and-JoinLearn about combining datasets: merge and join in Pandas.
Aggregation-and-GroupingLearn about aggregation and grouping in Pandas.
Pivot-TablesLearn about pivot tables in Pandas.
Working-With-StringsLearn about vectorized string operations in Pandas.
Working-with-Time-SeriesLearn about working with time series in pandas.
Performance-Eval-and-QueryLearn about high-performance Pandas: eval() and query() in Pandas.

matplotlib

IPython Notebook(s) demonstrating matplotlib functionality.

NotebookDescription
matplotlibPython 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
matplotlib-appliedApply matplotlib visualizations to Kaggle competitions for exploratory data analysis. Learn how to create bar plots, histograms, subplot2grid, normalized plots, scatter plots, subplots, and kernel density estimation plots.
Introduction-To-MatplotlibIntroduction to Matplotlib.
Simple-Line-PlotsLearn about simple line plots in Matplotlib.
Simple-Scatter-PlotsLearn about simple scatter plots in Matplotlib.
Errorbars.ipynbLearn about visualizing errors in Matplotlib.
Density-and-Contour-PlotsLearn about density and contour plots in Matplotlib.
Histograms-and-BinningsLearn about histograms, binnings, and density in Matplotlib.
Customizing-LegendsLearn about customizing plot legends in Matplotlib.
Customizing-ColorbarsLearn about customizing colorbars in Matplotlib.
Multiple-SubplotsLearn about multiple subplots in Matplotlib.
Text-and-AnnotationLearn about text and annotation in Matplotlib.
Customizing-TicksLearn about customizing ticks in Matplotlib.
Settings-and-StylesheetsLearn about customizing Matplotlib: configurations and stylesheets.
Three-Dimensional-PlottingLearn about three-dimensional plotting in Matplotlib.
Geographic-Data-With-BasemapLearn about geographic data with basemap in Matplotlib.
Visualization-With-SeabornLearn about visualization with Seaborn.

numpy

IPython Notebook(s) demonstrating NumPy functionality.

NotebookDescription
numpyAdds Python support for large, multi-dimensional arrays and matrices, along with a large library of high-level mathematical functions to operate on these arrays.
Introduction-to-NumPyIntroduction to NumPy.
Understanding-Data-TypesLearn about data types in Python.
The-Basics-Of-NumPy-ArraysLearn about the basics of NumPy arrays.
Computation-on-arrays-ufuncsLearn about computations on NumPy arrays: universal functions.
Computation-on-arrays-aggregatesLearn about aggregations: min, max, and everything in between in NumPy.
Computation-on-arrays-broadcastingLearn about computation on arrays: broadcasting in NumPy.
Boolean-Arrays-and-MasksLearn about comparisons, masks, and boolean logic in NumPy.
Fancy-IndexingLearn about fancy indexing in NumPy.
SortingLearn about sorting arrays in NumPy.
Structured-Data-NumPyLearn about structured data: NumPy's structured arrays.

python-data

IPython Notebook(s) demonstrating Python functionality geared towards data analysis.

NotebookDescription
data structuresLearn Python basics with tuples, lists, dicts, sets.
data structure utilitiesLearn Python operations such as slice, range, xrange, bisect, sort, sorted, reversed, enumerate, zip, list comprehensions.
functionsLearn about more advanced Python features: Functions as objects, lambda functions, closures, *args, **kwargs currying, generators, generator expressions, itertools.
datetimeLearn how to work with Python dates and times: datetime, strftime, strptime, timedelta.
loggingLearn about Python logging with RotatingFileHandler and TimedRotatingFileHandler.
pdbLearn how to debug in Python with the interactive source code debugger.
unit testsLearn how to test in Python with Nose unit tests.

kaggle-and-business-analyses

IPython Notebook(s) used in kaggle competitions and business analyses.

NotebookDescription
titanicPredict survival on the Titanic. Learn data cleaning, exploratory data analysis, and machine learning.
churn-analysisPredict customer churn. Exercise logistic regression, gradient boosting classifers, support vector machines, random forests, and k-nearest-neighbors. Includes discussions of confusion matrices, ROC plots, feature importances, prediction probabilities, and calibration/descrimination.

spark

IPython Notebook(s) demonstrating spark and HDFS functionality.

NotebookDescription
sparkIn-memory cluster computing framework, up to 100 times faster for certain applications and is well suited for machine learning algorithms.
hdfsReliably stores very large files across machines in a large cluster.

mapreduce-python

IPython Notebook(s) demonstrating Hadoop MapReduce with mrjob functionality.

NotebookDescription
mapreduce-pythonRuns MapReduce jobs in Python, executing jobs locally or on Hadoop clusters. Demonstrates Hadoop Streaming in Python code with unit test and mrjob config file to analyze Amazon S3 bucket logs on Elastic MapReduce. Disco is another python-based alternative.

aws

IPython Notebook(s) demonstrating Amazon Web Services (AWS) and AWS tools functionality.

Also check out:

  • SAWS: A Supercharged AWS command line interface (CLI).
  • Awesome AWS: A curated list of libraries, open source repos, guides, blogs, and other resources.
NotebookDescription
botoOfficial AWS SDK for Python.
s3cmdInteracts with S3 through the command line.
s3distcpCombines smaller files and aggregates them together by taking in a pattern and target file. S3DistCp can also be used to transfer large volumes of data from S3 to your Hadoop cluster.
s3-parallel-putUploads multiple files to S3 in parallel.
redshiftActs as a fast data warehouse built on top of technology from massive parallel processing (MPP).
kinesisStreams data in real time with the ability to process thousands of data streams per second.
lambdaRuns code in response to events, automatically managing compute resources.

commands

IPython Notebook(s) demonstrating various command lines for Linux, Git, etc.

NotebookDescription
linuxUnix-like and mostly POSIX-compliant computer operating system. Disk usage, splitting files, grep, sed, curl, viewing running processes, terminal syntax highlighting, and Vim.
anacondaDistribution of the Python programming language for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment.
ipython notebookWeb-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document.
gitDistributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows.
rubyUsed to interact with the AWS command line and for Jekyll, a blog framework that can be hosted on GitHub Pages.
jekyllSimple, blog-aware, static site generator for personal, project, or organization sites. Renders Markdown or Textile and Liquid templates, and produces a complete, static website ready to be served by Apache HTTP Server, Nginx or another web server.
pelicanPython-based alternative to Jekyll.
djangoHigh-level Python Web framework that encourages rapid development and clean, pragmatic design. It can be useful to share reports/analyses and for blogging. Lighter-weight alternatives include Pyramid, Flask, Tornado, and Bottle.

misc

IPython Notebook(s) demonstrating miscellaneous functionality.

NotebookDescription
regexRegular expression cheat sheet useful in data wrangling.
algorithmiaAlgorithmia is a marketplace for algorithms. This notebook showcases 4 different algorithms: Face Detection, Content Summarizer, Latent Dirichlet Allocation and Optical Character Recognition.

notebook-installation

anaconda

Anaconda is a free distribution of the Python programming language for large-scale data processing, predictive analytics, and scientific computing that aims to simplify package management and deployment.

Follow instructions to install Anaconda or the more lightweight miniconda.

dev-setup

For detailed instructions, scripts, and tools to set up your development environment for data analysis, check out the dev-setup repo.

running-notebooks

To view interactive content or to modify elements within the IPython notebooks, you must first clone or download the repository then run the notebook. More information on IPython Notebooks can be found here.

$ git clone https://github.com/donnemartin/data-science-ipython-notebooks.git
$ cd data-science-ipython-notebooks
$ jupyter notebook

Notebooks tested with Python 2.7.x.

credits

contributing

Contributions are welcome! For bug reports or requests please submit an issue.

contact-info

Feel free to contact me to discuss any issues, questions, or comments.

license

This repository contains a variety of content; some developed by Donne Martin, and some from third-parties. The third-party content is distributed under the license provided by those parties.

The content developed by Donne Martin is distributed under the following license:

I am providing code and resources in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code and resources is from me and not my employer (Facebook).

Copyright 2015 Donne Martin

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.