The life-changing magic of making with ML

One of the most fun ways to learn machine learning is by building projects for yourself. In this post, I’ll show you how.

An eon ago, I decided to learn to code by building my own personal website. I knew nothing about computers then, let alone what a server was, but it felt like I had infinite tech topics left to learn and an endless appetite to learn them. I laid awake at night thinking about all the ways I’d trick out my website–scrolling parallax stars, an obscene number of Google Fonts–and dreamt of all the personal projects I’d build next.

Years later, I’m a professional engineer who works on objectively cooler, more sophisticated technological problems than my hack-y personal website. Yet it’s hard to say I ever feel that, ahem, “childlike wonder” I did back when I was first learning to code.

If you work in tech, you surely know you’ve committed yourself to a lifetime of learning. Blink too long and every piece of software you thought was state-of-the-art will be entirely replaced (unless that software is Vim, which I still haven’t managed to exit).

In computer science, one great example is machine learning (ML). Most of us never learned this topic in university (if we studied CS in school at all), but it’ll soon be ubiquitous, transforming software development in every domain. It’s no surprise that the question I’m most frequently asked is, “Where should I start learning machine learning?”

Typically I’ll recommend resources like Google’s Machine Learning Crash Course, the book Hands-On Machine Learning, or Andrew Ng’s classic Coursera course, which cover the fundamentals from theory to practice.

But if you’re like me and your favorite way to learn is by building, consider learning ML by building software for yourself. Not only are personal projects a fun (and potentially useful) way to learn a new technology, they’re also a great way to learn about some of the challenges you’ll face deploying machine learning in production, rather than in the quixotic setting of a homework assignment.

At the beginning of the pandemic, when I suddenly found myself swimming in free time, this was the challenge I gave myself–can I learn more about machine learning by building personal projects that solve problems in my own life? Since then, I’ve used ML to intelligently search my family videos, improve my tennis serve, translate videos, invent new baking recipes, and a whole lot more.

Below you’ll find a roundup of these projects along with their source code, tutorial YouTube videos, and step-by-step blog posts. From introducing you to new techniques and tools to showing you how to weave this stuff together into functioning apps, I hope you’ll find these projects as fun to build as they are to learn from. Or, better yet, I hope they’ll inspire your own endeavors, made with ML. If you do build something neat, let me know. Until then, as always, happy making.

Smart family video archive

• What you’ll build: An archive that makes your home videos searchable by transcript (what people say) and by image (search for “birthday,” “bicycle,” “video game”).

• What you’ll learn:

• A super common Machine Learning use case: making complicated data types more easily sortable and searchable

• How to use the Video Intelligence API

• How to architect an ML-powered app, with a Flutter frontend, serverless Firebase backend, and search-as-a-service (powered by Algolia)

Discord moderator bot

• What you’ll build: A bot for the chat platform Discord that allows you to flag toxic, offensive, profane, or spam messages

• What you’ll learn:

• How to use the Perspective API to analyze text

• How to use ML in a chat application

• How to think about using machine learning for tricky problems that aren’t so black-and-white

Can AI make you a better athlete?

• What you’ll build: A Jupyter notebook that can track your tennis serve and the trajectory of a tennis ball (or golf swing or baseball pitch, etc) and that analyzes this data to give you tips to improve. Follow along in the Qwiklab.

• What you’ll learn:

• Hacks for doing sophisticated ML with small datasets

• How to combine pose tracking with simple math to understand human movement

• How to use the Video Intelligence API

• How to use AutoML Vision

Making smarter video game worlds with NLP

aka Build Apps Powered by Language with Semantic ML

• What you’ll build:

• A simple language-based system for having video game worlds respond to users’ free-text input

• What you’ll learn:

• How to use sentence embeddings, one of the most useful techniques in all of NLP

• How to implement semantic text search

• How to cluster text

• How to implement a basic conversational bot

• How to do all this stuff from a Google Sheet

PDF-to-audiobook converter

• What you’ll build: Code that converts PDFs to MP3 audiobooks

• What you’ll learn:

• How to extract text from PDFs using the Vision API

• How to speak words using the Text-to-Speech API

• How to use math hacks to parse document layouts

Translating and dubbing videos with ML

• What you’ll build: Code that can automatically transcribe, translate, and dub videos.

• What you’ll learn:

• How to combine Speech-to-Text, Translation, and Text-to-Speech

• How to improve transcription and translation quality

• How to work with videos and audio in Python

Can AI generate a good baking recipe?

• What you’ll build: A no-code machine learning model that classifies recipes and that can even generate new ones

• What you’ll learn:

• How to build a no-code machine learning model for tabular (“spreadsheet”) data using AutoML Tables

• How to ML explainability features to understand why a model is making the decisions it is

ML without code in the browser

• What you’ll build: A super quick machine learning model based on pose, vision, or sound

• What you’ll learn:

• What it takes to build a simple neural network (without coding)

• How to build a super fast model that runs in the browser with Teachable Machines

Making outfits with AI

• What you’ll build: An app that uses pictures of your own wardrobe and photos from fashionable social media influencers to recommend you outfits/

• What you’ll learn:

• How to use the Product Search and Vision API

• How to architect ML apps with React and Firebase