Published on

Dataflow Everything

Authors
  • avatar
    Name
    Dataflow
    Twitter

This documentation serves as a comprehensive guide to the resources we use for machine learning, data science, and hackathons.

Doing well in your courses a guide by Andrej Karpathy

Table of Contents

Essential ML Tools

  • numpy: A fundamental package for scientific computing with Python, providing support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on these arrays.
  • numba: JIT compiler that translates a subset of Python and NumPy code into fast machine code.
  • pandas: A powerful data manipulation and analysis library for Python, providing data structures like DataFrames for handling structured data.
  • polars: A fast DataFrame library implemented in Rust, designed for high performance and memory efficiency, particularly useful for large datasets.
  • XGBoost: An optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable, widely used for structured/tabular data.
  • YOLO: (You Only Look Once) revolutionary real-time object detection algorithm.
  • scikit-learn: A machine learning library for Python that provides simple and efficient tools for data mining and data analysis, built on NumPy, SciPy, and Matplotlib.
  • matplotlib: A plotting library for Python and its numerical mathematics extension NumPy, used for creating static, animated, and interactive visualizations in Python.
  • seaborn: A statistical data visualization library based on Matplotlib, providing a high-level interface for drawing attractive and informative statistical graphics.
  • plotly: A graphing library for Python that makes interactive, publication-quality graphs online, supporting a wide range of chart types.
  • TensorFlow: A comprehensive open-source platform for machine learning.
  • PyTorch: A flexible and powerful deep learning framework.
  • cudf: A GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
  • cuML: A suite of GPU-accelerated machine learning algorithms.

ML Platforms (Notebooks, Datasets, Models, and More)

  • Kaggle: A platform for data science competitions, datasets, and collaborative coding, offering a wide range of public datasets and a cloud-based Jupyter notebook environment.
  • Hugging Face: A platform for sharing and collaborating on machine learning models, particularly in natural language processing (NLP) and computer vision.
  • Google Colab: A free Jupyter notebook environment that runs entirely in the cloud, allowing users to write and execute Python code in the browser with access to powerful hardware like GPUs.
  • Jupyter Notebooks: An open-source web application that allows you to create and share documents containing live code, equations, visualizations, and narrative text.
  • Groq: Fastest inference with AI models (uses specifically designed chips - Language Processing Units (LPUs)).
  • OpenRouter.ai: Unified API gateway and marketplace for large language models. It lets developers access over 400 AI models from dozens of different providers using a single, OpenAI-compatible API key.
  • AI Datasets Platform in Kazakhstan by Astana Hub: Access government datasets, open source data, and more to kickstart your AI projects.
  • DSML.kz: Kazakhstan's largest AI community, fostering knowledge sharing and professional growth in AI/ML.

Hackathons List

ML Problem Solving Platforms

  • DeepML: A platform for practicing machine learning problems and improving your skills.
  • CodeRun: A platform by Yandex for solving machine learning problems.
  • LeetGPU: A platform for solving coding problems on GPUs (CUDA).
  • Hackerearth ML: A platform for practicing machine learning problems and improving your skills.

ML Hands-On Learning Platforms

ML Cheatsheets

ML/Algo/Math Books

Programming Languages

  • Python: The primary language for machine learning and data science.
  • C++: Used for performance-critical components and libraries.

Fast-Prototyping Frameworks

  • FastAPI: Utilized for creating high-performance APIs.
  • Streamlit: For creating quick and interactive data-driven web applications.
  • Gradio: A user-friendly framework for building machine learning demos and interfaces: UI for users / API for software / MCP for LLMs.

DevOps Tools

  • Docker: For containerization of applications, ensuring consistency across environments.
  • MLflow: For managing the machine learning lifecycle, including experimentation, reproducibility, and deployment.
  • Git: Version control system for tracking changes in code and collaborating with team members.
  • GitHub: Platform for hosting and managing Git repositories.
  • Ngrok (Connect anything, anywhere): ngrok is developer infrastructure that routes and secures traffic to your apps, APIs, and AI models.

Free Cloud GPU Providers

Cheap Cloud GPU Providers

  • Runpod.io: AI infrastructure with on-demand GPUs and serverless compute. Run training, inference, and batch workloads on the cloud with Runpod.

AI Tools

No Code AI Dev Tools

  • Teachable Machine by Google: A web-based tool that makes it easy to create machine learning models without any coding, allowing users to train models using their own data.
  • Roboflow: A platform for building and deploying computer vision models without extensive coding.
  • HuggingFace AutoTrain: A tool that simplifies the process of training machine learning models, particularly in natural language processing and computer vision, without requiring deep technical expertise.
  • Firebase Studio by Google: A no-code development environment for building and deploying applications on Firebase.
  • Lovable: No Code AI-powered applications development platform.
  • Bolt.new: No Code AI-powered applications development platform.
  • v0.dev: No Code AI-powered applications development platform.

ML/AI YouTube Channels

ML/AI YouTube Videos / Playlists

Tech Blogs

  • Vlad Feinberg: by Google DeepMind's pre-training area lead.
  • Ryan Peterman: by ex-SWE @instagram, @meta.
  • Andrej Karpathy: by ML/AI Legend.
  • Jeff Dean: Co-founder & CEO of Discovery Loop. Former Chief Scientist, Google. Helped build many Google products, TPUs, Gemini, TensorFlow, MapReduce, Bigtable.
Discuss on Twitter • View on GitHub