Following the tradition of 2023 and 2024, here are my favorite books from 2025:

Disco Elysium

“Disco Elysium isn’t a book, Alex!” Ok, sure, but it is one of the literary masterpieces of the 21st century. It does a miraculous job of immersing the player in its world, exploring the grip the past has on the present, and creating some of the deepest and most fully developed characters in any medium. It’s my favorite game of all time, and one of my favorite literary works, period.

• 
  Disco Elysium by Robert Kurvitz et al.

  ★★★★★

  Disco Elysium, written by Robert Kurvitz et al., is a role-playing game produced by ZA/UM. It’s the story of Harrier “Harry” Du Bois, a man who wakes up with no memories and has to solve a murder while learning who he is.

The Hyperion Cantos by Dan Simmons

I didn’t love the Hyperion Cantos the first time I read them. I found The Fall of Hyperion, the easier and more straightforward of the two, to be great, but I didn’t really get Hyperion. This year I re-read them for my book club and focused on finding and understanding the influences from John Keats’s works. That completely transformed the experience. I LOVED Hyperion, seeing it as far deeper and more complex than I did the first time through.

This year I’ll be tackling Endymion and The Rise of Endymion, which I’ve heard don’t live up to the greatness of the first two, but I’m still hopeful.

• 
  Hyperion by Dan Simmons

  ★★★★★

  Hyperion is Dan Simmons’s masterpiece. It is the first book in his Hyperion Cantos. It follows seven pilgrims as they travel to the Time Tombs on Hyperion to petition the Shrike. Along the way, each tells their own story, weaving together history, myth, and prophecy to tell of the impending downfall of man.

• 
  The Fall of Hyperion by Dan Simmons

  ★★★★★

  The Fall of Hyperion, by Dan Simmons, is the second book in the Hyperion Cantos, but really it’s the second half of Hyperion. It brings the seven pilgrims’ story to an end and depicts the war between the TechnoCore, the Ousters, and the Hegemony.

The Hydrogen Sonata by Iain M. Banks

Banks’s Culture series was one of my favorite reads in 2024. I deliberately delayed finishing it, spacing out the final few books because I knew that once I was done, it would be over for good. The Hydrogen Sonata was a fitting end to the series. It restates many of the themes Banks first explored in Consider Phlebas, but with a more hopeful bent.

It also feels like a fitting final book for Iain M. Banks himself, with its message that life only has the meaning you give it.

• 
  The Hydrogen Sonata by Iain M. Banks

  ★★★★★

  The Hydrogen Sonata, by Iain M. Banks, is the tenth and final book in the Culture series. It explores the last days of the Glitz people as they prepare to Sublime.

The Teixcalaan series by Arkady Martine

Another pair of books picked by my book club. I really enjoyed Martine’s subtle worldbuilding and the deep integration of themes and motifs across A Memory Called Empire and A Desolation Called Peace. The writing itself is beautiful, and often reminded me of Gene Wolfe’s The Book of the New Sun, with its mix of archaic prose and intentional vagueness.

• 
  A Memory Called Empire by Arkady Martine

  ★★★★☆

  A Memory Called Empire, by Arkady Martine, is the first book in the Teixcalaan series. It follows Mahit Dzmare, an ambassador from the space station Lsel, as she tries to save her home from being annexed by the Teixcalaanli empire.

• 
  A Desolation Called Peace by Arkady Martine

  ★★★★☆

  A Desolation Called Peace, by Arkady Martine, is the second book in the Teixcalaan series. It tells the story of Mahit and Three Seagrass trying to stop the war between the Teixcalaanli Empire and a mysterious alien race.

Roadside Picnic by Arkady and Boris Strugatsky

I picked up Arkady and Boris Strugatsky’s best-known novel, Roadside Picnic, because I wanted to broaden what I was reading. I ended up loving it. The characters feel incredibly real, the dialogue has a simple but earnest quality, and the atmosphere is completely different from Western sci-fi.

It wasn’t the only book of theirs that I read, but it was easily my favorite.

• 
  Roadside Picnic by Arkady Strugatsky and Boris Strugatsky

  ★★★★☆

  Roadside Picnic, by brothers Arkady and Boris Strugatsky, is a Soviet sci-fi novel. It’s essentially four short stories—each presented as a chapter—about the life of Redrick “Red” Schuhart, a “stalker” who illegally enters an alien-contaminated Zone to retrieve items for the black market.

The Murderbot Diaries by Martha Wells

Everyone loves The Murderbot Diaries, and I’m no exception. Wells hits the perfect mix of pulpy action, fast pacing, and memorable characters, while still digging into deep philosophical questions about what it means to be a person. I’ve got a few more to go this year, and I’m really looking forward to them. I fully expect to see them on my 2026 list.

• 
  Artificial Condition by Martha Wells

  ★★★★☆

  Artificial Condition, by Martha Wells, is the second book in The Murderbot Diaries. It follows Murderbot as it digs into its past and, once again, saves some scientists.

• 
  Rogue Protocol by Martha Wells

  ★★★☆☆

  Rogue Protocol, by Martha Wells, is the third book in The Murderbot Diaries. It follows Murderbot as it investigates a GrayCris terraforming station and, you guessed it, ends up saving a group of humans.

• 
  Exit Strategy by Martha Wells

  ★★★★☆

  Exit Strategy, by Martha Wells, is the fourth book in The Murderbot Diaries. It wraps up the GrayCris storyline as Murderbot returns to save its friends.

• 
  Network Effect by Martha Wells

  ★★★★☆

  Network Effect, by Martha Wells, is the fifth book in The Murderbot Diaries. It’s the first full-length novel in the series and features Murderbot getting kidnapped by ART to rescue its crew.

There Is No Antimemetics Division by qntm

I read qntm’s original edition edition in 2023 and thought it was packed full of amazing ideas, but ultimately let down by a poorly written second act. His rewrite this year fixed every single problem. The result is a tight, cohesive book that brings its wild concepts to life. Highly recommend.

• 
  There Is No Antimemetics Division by qntm

  ★★★★★

  There Is No Antimemetics Division, by qntm, is a book about researchers trying to control dangerous antimemes—ideas that can’t be thought—and how you might combat a foe you can’t even remember exists.

A Mote in Shadow by A. N. Alex

I found this book when my corner of BlueSky started talking about it, and I’m glad I paid attention. Alex’s debut novel is a fantastic mix of hard sci-fi and techno-thriller, set in a fractured human civilization in the not-too-distant future. He’s working on the sequel now, and I’m very much looking forward to it.

• 
  A Mote in Shadow by A. N. Alex

  ★★★★★

  A Mote in Shadow is A. N. Alex’s debut novel. It follows two down-on-their-luck outsiders dragged into a war between shadowy mercenary groups: exobiologist Chaeyoung No, whose disagreement with the scientific establishment leaves her in no position to question a too-good-to-be-true offer to fund her research expedition; and space hauler Frederik Obialo, who is more than willing to take a dangerous job if it brings him closer to his dream of giving his daughter a permanent home.

The Triumphant by Linda Evans, Robert R. Hollingsworth, and David Weber

I started reading the Bolo books when I was a teen and absolutely loved them. They feature giant tanks blowing stuff up, but, like The Murderbot Diaries, they also broach deeper questions about the meaning of being alive and whether honor and duty translate to machines. I recently started re-reading the series for nostalgia, and while most of the books are serviceable but not great, The Triumphant is the clear exception.

• 
  The Triumphant by Linda Evans, Robert R. Hollingsworth, and David Weber

  ★★★★★

  The Triumphant is the twelfth book in the Bolo series. It’s an anthology of Bolo stories written by three different authors. They explore the emotional bond between a Bolo and the people around them, and the dangers of caring too much about a machine built for war.

Your failure of courage raises some fundamental questions. What is the point of studying history if we learn nothing from it? What is the point of our own history if we simply repeat the same mistakes we made in the past?

We tell ourselves myths about Mario Savio and the Free Speech Movement, about the little circle of freedom in front of Sproul Hall, about how that history determined who we are now. But it is clear now that you have chosen to betray that legacy, siding not with Savio, but with the machine he warned us about. You aren’t willing to put your careers, never mind your bodies, upon the wheels and levers to make it stop.

You believe you’re saving the University. But you are destroying its soul.

Alexander Gude (‘08)

A computer’s job is to do work for you.

I hadn’t realized that before. Until I learned to program, computers could only do a small set of things for me—mainly the things other people had already decided they should do. But when I learned to code, I suddenly had the ability to make them do what I wanted.

Within a year I’d learned Python, picked up Vim, and switched my computers to Ubuntu. I started writing little scripts to make my life easier, like backing up my email, syncing files between my laptop and desktop, or converting Wikipedia pages into an archival format.

Back then, each script took me hours to write. I had to find the right libraries, learn their APIs, or sometimes write code from scratch. Because they took so much work I saved every one, sometimes even sharing them with the opensource community, just in case I ever needed them again. But all of that changed in the last two years when I learned to use an LLM for coding.

Now a Python script takes 30 seconds. Maybe a minute if I want to read it over and give some feedback. Gemini 2.5 Pro almost always writes a 100-line script correctly on the first try, just from a short description of what I want. It’s so fast and easy that I’ve stopped saving them—if I need another one, I can just prompt for it again in half a minute.

Python scripts are now free!¹ But I don’t think everyone has realized this yet. Once again, William Gibson was right:

The future is already here – it’s just not very evenly distributed.

At work, I use Codename Goose, which can directly interact with my local machine and files, switch between multiple LLM APIs, and connect to our infrastructure with Model Context Protocol (MCP) servers. At home, I’m more limited, but use GPT-4o and Gemini 2.5 Pro to generate code (that I have to copy back and forth between my browser and terminal).

After intentionally trying to shoehorn LLMs into my workflow to see where and how they work, this is what I’ve learned.

System Prompt

The system prompt, which the model “reads” before it gets started, is how you customize the way the model “thinks” and responds to you. I find a few things are useful to include:

1. A general overview of the task: This gives the LLM context about where you’re headed, even as it works line by line and file by file. “We’re refactoring a Jekyll website…”, “We’re building a set of machine learning signals in Java 17…”, etc.
2. A layout of the project: Locations of key files, what goes where, etc. Things like “a.py is where the main code lives, b.py provides helper functions…”
3. A style guide: Telling the LLM how you prefer to use the programming language. “Never use ternary operators”, “Always use {} with if statements”, “Prefer functional solutions”, etc.

A trick I use is, at the end of a session, I give the LLM the current system prompt and ask it to update it—adding anything new it learned and fixing anything that’s now out of date. Then I use that modified prompt when I spin up a new LLM to work on the project.

Unit Tests

English isn’t a very precise way to define what you want code to do, but that’s usually where I start, maybe with a few example code snippets. After that, I ask the model to write unit tests for the code we just wrote. This is where I make sure we’re on the same page.

I read through the unit tests and look for anything that doesn’t match what I was expecting. Sometimes it’s easy to just change the test to match the behavior I want. Other times, I try to understand why the LLM made the choice it did, then discuss the decision with it to come to an agreement. I often catch disagreements over when to throw errors, how to handle invalid inputs, or other rare edge cases in this process.

Once the behavior is codified in the tests, I ask the model to fix the original code so the tests pass. This iterative process of generation and verification is where LLMs truly shine. In this way, it’s sort of an inverse of test-driven development—I have the computer write the code and then the tests. I find that having some code to start with helps the LLM write broader, more useful tests, a practice whose importance I’ve previously discussed in the context of data science.

In some cases, I’ll even throw away the code, restart the LLM, and give it just the tests as the spec for what it should write. This is particularly useful when the model gets stuck and either won’t make a large enough change to the code, or loops back and forth between essentially the same few versions.

Iterate And Advise

When I’m working with an LLM, I see my job more as a manager or mentor than an engineer. My role is to define what we’re doing, then critique the result to get it into shape. It’s a lot like working with a junior engineer or intern—except one that can make a revision in 10 seconds instead of 4 hours. This lets me iterate a lot to get the code working exactly how I want.

The things I focus on are mostly high-level, like data structures and algorithms. The LLM often doesn’t need a lot of direction—just saying “Couldn’t we replace the double for loop with a single loop and a hash table?” is usually enough to get it on the right path. Only at the end, once the structure is solid, do I focus on nitpicky issues or edit the text directly.

The Future

I’m still learning how to use LLMs for coding. The technology and tooling is evolving so fast that I’m sure what I’m doing today will look antiquated in six months. Still, I hope it gives others some ideas about how they can use this technology to speed up their development.

ChatGPT 3.5 came out just over two years ago and sparked an explosion in Large Language Model (LLM) development. Dozens of companies released their own models, and the state of the art advanced by the hour.

At the time, I wrote about how I used ChatGPT to write. My method was primitive. With years of experience and improvements in the models, I have refined how I use LLMs to edit text. Here is my new method.

Drafting

I write the first draft entirely by hand. This helps me preserve my voice and prevents the writing from being overly influenced by the LLM. It also supports my main goal of writing: to clarify my thinking.

I used to write, edit, write, edit, and so on until I was nearly 100% happy with my work, but now I stop earlier and let the LLM handle an editing pass. Using the LLM early saves me multiple rounds of edits because they’ve become so good at fixing spelling and grammatical errors and slightly tweaking my writing without overpowering it.

The Prompt

The prompt you use with the LLM is important, because it strongly shapes how the model edits your writing. The prompt keeps the machine from filling my writing with phrases like “delve”, “showcasing”, and “underscores”. I currently use a slight variation of this prompt:

Help me edit this blog post I’m writing. Fix errors, make it clearer. Reword to make the arguments and sentences more coherent. Use the same sort of words I’m using, don’t substitute fancy synonyms. Maintain my voice. My work is below. Keep the formatting and wrap your output in ```.

Editing

Once I have the LLM’s version, I put it side-by-side with my draft to compare. Sometimes the edited version is perfect, and I’ll take an entire paragraph as is. Other times, I’ll borrow ideas about how to structure a paragraph or a transition, but I rewrite it in my own words. The model sometimes tries to fix sections that are fine, and I simply ignore it.

After that, I go through another human editing pass to ensure my voice comes through in every sentence. Sometimes, I will have the LLM focus on specific sentences or paragraphs that still need work and iterate. Once I’m happy, I commit my changes and publish.

Echopraxia by Peter Watts

Echopraxia is the sequel to Blindsight, one of my favorite books from last year. Although it wasn’t as well received by readers as Blindsight, it was my favorite read of the year. In fact, I liked it more than Blindsight because of its complex storyline, intriguing characters, and the broader perspective it offers on the world.

• 
  Echopraxia by Peter Watts

  ★★★★★

  Echopraxia, by Peter Watts, is the second book in the Firefall series, unfolding at roughly the same time as Blindsight. It follows parasitologist Daniel Brüks, who gets unwillingly dragged into a conflict between multiple transhuman factions, travels to the Icarus station orbiting the sun, and eventually back to Earth.

The Culture Series by Iain M. Banks

I hated Consider Phlebas when I read it in 2023, but I gave the Culture series another chance because I had spent the better part of two decades waiting to get my hands on the books. I’m glad I did, because I read The Player of Games with my book club and loved it! My favorite from the series was Surface Detail, closely followed by Use of Weapons. I even enjoyed Inversions, one of the least well-reviewed of Banks’s books. It will be bittersweet to finish the series with The Hydrogen Sonata in 2025.

• 
  The Player of Games by Iain M. Banks

  ★★★★★

  The Player of Games, by Iain M. Banks’s, is the second novel in the Culture series. It tells the story of Jernau Morat Gurgeh, a master game player who is recruited to play Azad, an incredibly complex game that serves as the basis for the Empire of Azad’s entire government.

• 
  Inversions by Iain M. Banks

  ★★★★★

  Inversions, by Iain M. Banks, is the sixth book in the Culture series, but it is very different from typical Culture novels: there are no spaceships and almost no advanced technology. Instead, it follows Culture citizens DeWar and Vosill as they manipulate a medieval society.

• 
  Look to Windward by Iain M. Banks

  ★★★★★

  Look to Windward, by Iain M. Banks, is the seventh book in the Culture series. It explores the aftermath of the Idiran–Culture War and Chelgrian civil war.

• 
  Surface Detail by Iain M. Banks

  ★★★★★

  Surface Detail, by Iain M. Banks, is the ninth book in the Culture series. It follows Lededje Y’breq as she seeks revenge for her own murder, set against the backdrop of a galactic conflict over virtual hells.

A Fire Upon The Deep by Vernor Vinge

A Fire Upon The Deep is a nostalgic favorite that I first read about 20 years ago and reread this year for my book club. It does a fantastic job of telling a story that feels small and personal while having galaxy-spanning implications. The Zones of Thought concept is also a unique way to structure the galaxy and explore how it shapes civilizations.

• 
  A Fire Upon The Deep by Vernor Vinge

  ★★★★★

  A Fire Upon The Deep is a sci-fi novel by Vernor Vinge. It tells the story of the Blight—a galactic-scale, transcendent evil—and the humans racing to stop it.

The Cheela Series by Robert L. Forward

The Cheela series consists of two hard sci-fi novels by Forward: Dragon’s Egg and Starquake. Even though the Cheela are extremely alien—living on the surface of a neutron star and experiencing time a million times faster than humans—their characters still pulled me in. It was exciting to watch them build their civilization from hunter-gatherers to a spacefaring society.

• 
  Dragon’s Egg by Robert L. Forward

  ★★★★☆

  Dragon’s Egg is a hard sci-fi novel by Robert L. Forward. It is the story of first contact between humans and the Cheela: beings who live on a neutron star.

• 
  Starquake by Robert L. Forward

  ★★★☆☆

  Starquake is the second book in the Cheela series by Robert L. Forward. It follows the Cheela as they rescue the humans and rebuild after a devastating starquake.

Childhood’s End by Arthur C. Clarke

Another nostalgic read, Clarke’s Childhood’s End was one of the first sci-fi books I ever read. Revisiting it years later was a treat, and I’m happy to say it holds up. The focus on humans’ psychic abilities feels a little dated, but Clarke’s crisp writing kept me engaged.

• 
  Childhood’s End by Arthur C. Clarke

  ★★★★☆

  Childhood’s End is a classic sci-fi novel by Arthur C. Clarke. It is about first contact between humans and the mysterious Overlords, and the end of the human race.

I tried multiple different chairs in person, and although my heart wanted an Aeron, my sit-test (for 10 minutes) told me I should get a Steelcase Gesture. Giving me confidence was the fact that The Wirecutter had rated it their best chair, saying “This is one of the most adjustable chairs available—anyone can make it comfortable, regardless of their height or size.” After eight months, I’ve decided they were wrong.

Overall, I am disappointed with the chair, which has an uncomfortable back, poor recline, and arms that just don’t get out of the way enough to get close to my desk. I have replaced it with an Aeron Remastered Size C.

My Chair

I bought a Steelcase Gesture with:

• Headrest

• Upholstered Wrap Back

• Cogent Lizard Upholstery (5S94)

• Platinum Metallic Frame

• Matching Base

• 360° Arms

• Lumbar Support

• Wheels for Carpet

The retail price for this chair is $1679, but I got it through my work’s office supplier for $1079.

Headrest

I debated whether or not I wanted a headrest and finally decided to get one. You can mostly fold it out of the way if you don’t like it, so I figured I’d rather give it a shot. It’s fine. I use it when lounging but not when sitting normally. I would not get it again because it is superfluous.

Arms

One of the selling points of the Gesture is how flexible the arms are. They are very easy to move, easier than the Aeron’s, but I didn’t find this to be a great selling point. Generally, I set the arms once and leave them forever. I don’t adjust them for each new position or task as Steelcase assumes.

Additionally, the arms aren’t as nice as the Aeron’s. The Gesture’s arms don’t go as low, about 6 1/2 inches off the seat compared to 5 1/2 for the Aeron. This is extremely important to me because I have very long arms compared to my torso; the Gesture’s arms force my shoulders up. The Gesture’s arms also stick out further when pushed all the way back (10 inches from the back of the chair, compared to 8 for the Aeron). This makes it harder to pull the chair under a desk to get close to the keyboard. The pads are a little thinner on the Gesture, making it slightly less comfortable.

The Back

The Gesture has a very upright back. It forces me into a very straight position, even though I normally prefer a little bit of recline. The back moves in two ways: it tilts top to bottom to try to adjust to the curve of your spine, and it reclines.

The back is also very tall, extending about 24 inches above the seat (not including the headrest). Because it is very straight, it makes contact with my shoulder blades in a way I found uncomfortable. For the first week of use, I got upper back pain which I assumed was just my body adjusting to “sitting correctly”. Eight months later and I’m not so sure; the upper-back pain persists (although far less severe) and my doctor thinks my posture is most of the problem. I had no problem before using the Gesture.

Recline

The Gesture’s back can be locked into 4 different positions, and the spring tension can be adjusted from “you can’t push this back if you tried” to “absolutely no resistance”. Unfortunately, the recline just doesn’t feel very good on my chair. It is either too stiff or too loose. There is no middle ground that feels supportive. I think my spring is broken because the Gestures we have at my company’s office have better tension control and a more comfortable recline.

The back moves separate from the seat, which feels OK. On the Aeron, the seat slides a bit when you recline which feels more natural. Overall, the recline on the Aeron feels much better.

Lumbar

The lumbar support is TERRIBLE. It pokes me in the back and feels like it’s trying to push me out of the chair. The curve of the backrest is already aggressive, and the lumbar support makes it more so. There is an entire thread on Reddit¹ about how the lumbar support ruins the chair. I followed Steelcase’s guide to removing the backrest pad to take out the lumbar support and it helped a lot (but I still didn’t find it comfortable). I would not buy the lumbar support again.

The Seat

The seat depth is adjustable. Because I’m tall (6 foot 1 inch), I have the seat all the way forward. This leaves a small gap between the back of the seat and the backrest. The height is also adjustable, and I can make it high enough to fit me.

Some people complain that the woven and padded seat retains heat and isn’t comfortable on hot days. I used the chair during the summer and didn’t notice heat problems, even when it was 110 °F outside and a little over 80 °F in my office.

Final Thoughts

I don’t like my Steelcase Gesture. For eight months, walking into my home office each morning came with a sinking feeling, knowing I’d have to sit in this chair. I replaced it with an Aeron, and the difference is night and day—I couldn’t be happier with the switch.

I review all the books I read, here is a selection of my favorites that I read in 2023:

Blindsight by Peter Watts

Blindsight does a great job of exploring the nature of consciousness and intelligence. Watts keeps the tension high and the plot moving quickly in this thought-provoking sci-fi novel. My favorite book of the year!

• 
  Blindsight by Peter Watts

  ★★★★★

  Blindsight is a hard sci-fi novel about first contact with aliens in the near future. A crew of four transhumans and a vampire are sent on a spaceship to investigate an anomaly in the solar system after a swarm of alien probes scan Earth.

The Chronicle of the Fallers by Peter F. Hamilton

Hamilton is known for his space opera, but The Abyss Beyond Dreams is more urban fantasy set during the Russian Revolution (in space) and Night Without Stars is a thriller set during the Cold War (again, in space). Both feature Commonwealth citizens with special knowledge as “Outside Context Problems”, pulling the stories into science fiction territory.

• 
  The Abyss Beyond Dreams by Peter F. Hamilton

  ★★★☆☆

  The Abyss Beyond Dreams starts off The Chronicle of the Fallers, another series in Peter F. Hamilton’s Commonwealth universe. Though billed as space opera, it often reads more as urban fantasy since most of the story occurs on the planet Bienvenido inside the Void where steam engines are their most advanced technology.

• 
  Night Without Stars by Peter F. Hamilton

  ★★★☆☆

  Night Without Stars is the second book in The Chronicle of the Fallers. It is action packed, with great pacing, and complex characters. It is my new favorite Peter F. Hamilton book.

The Fall of Hyperion by Dan Simmons

I enjoyed the sequel to Hyperion the most of the two books because it tied the personal story of the pilgrims to a much broader galactic conflict. Interestingly, you can see a lot of ideas in the Hyperion Cantos that Hamilton later adopted in his Commonwealth Saga including wormholes, a breakaway-but-helpful AI, and different factions of scheming AI who either want to eradicate the humans or uplift them.

• 
  The Fall of Hyperion by Dan Simmons

  ★★★★★

  The Fall of Hyperion is a sequel that outshines its predecessor. It is everything I was expecting from Hyperion and more! A true masterpiece.

The Commonwealth Saga by Peter F. Hamilton

Epic space opera with a massive cast of characters and incredible pacing.

• 
  Pandora’s Star by Peter F. Hamilton

  ★★★★☆

  I couldn’t put Pandora’s Star down! It is a sci-fi book that reads more like a thriller. There were always new mysteries that just a few more pages promised the answers to.

• 
  Judas Unchained by Peter F. Hamilton

  ★★★★☆

  The sequel to Pandora’s Star, Judas Unchained continues right where the last one left off, but with the action ramped up to 11. The various storylines and loose threads come together one by one until it’s the good guys racing against the bad guys for the fate of the universe.

Serpent Valley by Scott Warren

1980s mech sci-fi re-imagined for the 21st century. Warren’s self-published series takes a few books to really find its feet, but once it does, it’s a quick, fun, nostalgic read. The third book, Serpent Valley, exemplifies the series.

• 
  Serpent Valley by Scott Warren

  ★★★★☆

  Serpent Valley, the third book in the War Horses series, is another quick, action-packed read—but without the flaws holding back its predecessors. Easily my favorite of the series so far!

This is straightforward when the officer writes FORD or HONDA, which they mostly do. But since the officer can write anything, they occasionally make it a little harder on us by abbreviating or mistyping, for example VOLX and DODDGE. And sometimes they make it impossible by writing -- or ______.

The solution is to go through, one by one, and create a mapping like:

# Enumeration of common vehicle makes
@unique
class Make(Enum):
  CHEVROLET  = "chevrolet"
  GMC        = "gmc"
  HINO       = "hino"
  INFINITI   = "infiniti"
  MITSUBISHI = "mitsubishi"
  # Special Token for unknown make
  NONE       = None

# Dictionary mapping raw values to Make enum
make_map = {
  "CHEVRLT":  Make.CHEVROLET,
  "HINO/":    Make.HINO,
  "INFINITY": Make.INFINITI,
  "MITSUB":   Make.MITSUBISHI,
  "TAHOE":    Make.GMC,
  "UKNOWN":   Make.NONE,
}

As someone who did this mapping by hand for over 900 entries, it is quite tedious. Fortunately, making sense of mangled text is something Large Language Models (LLMs) are pretty good at!

Automating

The goal is to perform few-shot, multi-label classification of vehicle makes. Few-shot because we are going to give the model just a handful of examples of what output we expect, and multi-label because there are many possible vehicle makes it will have to map to.

Prompting

The first step is to write a prompt explaining the task to the model, the expected return value, and a few examples of input and correct outputs. Here is a shortened version, the full one is here, starting with the instructions:

Then the output format, with instructions to include an explanation of its logic first, which can help model accuracy:

And finally some examples of inputs and correct outputs:

Answers

Since I was manually copying the prompt into the model’s web interface, I used batches of 100–200 string sorted alphabetically. With API access, I could have used retrieval-augmented generation to create custom examples for each string while sending them one at a time.

Splitting the data into batches helped the model figure out very short entries. For example, the model failed when given WNBG (Winnebago) by itself, but succeeded when I gave it the list:

WINN
WINNE
WINNEBAG
WINNEBAGO
WINNI
WNBG
WNBGO

I believe seeing multiple short versions next to each other helped the model infer the right mapping.

Performance

I obtained the following performance on my 902 hand-mapped entries:

• The model correctly fixed 2 entries that I had gotten wrong.

• It matched 682 (75.6%) of my hand-labeled mappings.

• It missed 218 (24.1%) of the mappings, frequently using made-up enum values.

This is reasonably good performance, as finding wrong entries is pretty quick (and many could be fixed with find and replace).

We’ve now put 1850 miles on the bike, mostly taking the kids to local parks. After all the time spent with it, would we recommend it to another family?

In short: we love the utility an electric cargo bike offers, and I think we will always have one in the garage to supplement our minivan, but the RadWagon has some significant drawbacks that make recommending it difficult. Read on for the full review.

The Bike

We ordered the Rad Wagon from the Rad’s website for $1500. With a “caboose” enclosure and pads for the kids adding $250, plus tax, our total was $1901—not cheap, but almost a third of comparable cargo bikes which are in the $4000–6000 range.

Assembly and Adjustment

The bike comes in a box and you must assemble it yourself. The assembly was not too hard, but I also have a lot of experience with bikes and bike maintenance which helped.

The seat post and headset are highly adjustable, allowing people of varying heights to ride the RadWagon. Both my wife (5’4”) and I (6’1”) can ride it comfortably, but we’re both near the limits. I have the seat all the way up and she has it all the way down.

Motor

The 750W rear hub motor easily brings the bike to its 20 miles per hour computer-limited top speed. The bike has a throttle, which I love for getting started from a stop, and enough power to carry me, two kids, and some balance bikes up a steep hill. The downside though is the hub motor puts a lot of stress on the rear wheel.

Flaws

A flaw in the design is the rear brake. The RadWagon uses cheap, mechanical disc brakes, which are enough to stop the bike when they’re well aligned, but which need constant attention to keep them that way. The motor blocks the typical through-spoke access for adjusting the rear brake. Instead, Rad makes a special flat Allen wrench that fits between the brake and motor but adjusting remains hard.

A major flaw is the rear spokes. They are stressed by both the motor—which puts all its power through the rear wheel—and the cargo. The spokes were not tight enough from the factory and I broke several in the first 200 miles. I have broken fewer since replacing and retightening, but I still break one periodically, which is annoying for me and likely a dealbreaker for less experienced riders.

Support

I’ve contacted Rad’s support several times—to order spokes and the brake tool, and to replace a faulty accessory. They were generally quick and helpful but I’ve never needed support for other bikes. And the last interaction was terrible: Rad’s front basket was defective, and after I sent them photographic proof, they accused me of being unable to use a screw driver and stopped responding. It was the worst support experience I’ve ever had.

Final Thoughts

The RadWagon was a savior during the pandemic, letting us escape the house and ride when we’d otherwise be trapped inside. It also beats driving kids in a car—hauling them on the back makes getting to the park part of the fun.

But the bike needs constant maintenance that is difficult even for an experienced mechanic, and Rad’s support is not great. I wish I had purchased a higher-quality bike that wouldn’t fail so frequently.

You can probably already see the problem: APIs are meant to hide the inner workings behind them. But by returning the raw model scores, we revealed too much detail. Any changes to the model, like retraining it, could change the scores and break the front end.

In my guide to deploying machine learning models in shadow mode, I stated that deploying changes “in front of the API” has the advantage of giving the calling team control. This is precisely why we built the ATO API the way we did: to address the organizational issue that the engineering team did not trust the machine learning team.

But if your teams trust each other, there is a much better way to build.

What is a better way?

A better way is for the API to return a set of actions. For example, the ATO model API might return the following actions:

• Allow: The login looks fine, allow it.

• Step-up: The login looks odd, require the user to provide a second factor of authentication, such as a code sent to their email.

• Lock: The login looks clearly fraudulent, deny the login and lock the account until the user recovers it.

These actions do a really good job of hiding the implementation behind the API. You can freely change thresholds when the model performance changes, retrain the model, or even replace it entirely.

But you can do something else too, you can add more models!

Using multiple systems

A common fraud-prevention strategy is to train a model for each new fraud pattern identified. This allows each model to be highly precise, while also improving the recall of the overall system. These multi-model systems are often augmented with simple rules, such as “No logins from Russia allowed.” In the end, the system takes the outputs of the various models and rules and aggregates them in some way. In our ATO example, the system returns the most drastic action recommended by any model or rule.

In code:

def ato_api(event_token):
  # List of actions returned by all the models and rules,
  # consists of values from {'Allow', 'Step-up', 'Lock'}
  all_results = get_ato_system_results(event_token)

  if 'Lock' in all_results:
    return 'Lock'
  elif 'Step-up' in all_results:
    return 'Step-up'

  return 'Allow'

Of course, this is a great place to use enums and max:

from enum import IntEnum, unique

@unique
class Action(IntEnum):
  ALLOW = 0
  STEPUP = 1
  LOCK = 2

def ato_api(event_token):
  # List of actions returned by all the models and rules,
  # consists of values from Action() enum
  all_results = get_ato_system_results(event_token)

  return max(all_results)

But in the last 7 years language models have gotten much, MUCH better. I wondered how a state-of-the-art model, one too large to run on my computer, would perform on the same questions.

To find out, I ran the analogies through Anthropic’s biggest model: Claude.

Experimental Setup

I gave Claude the following instructions:

Then I gave it an analogy like:

One potential issue: I left the analogies and their possible answers in the sorted order from my word2vec post. This means the analogies word2vec got right were up first, with the correct answer in the first position. Then the ones word2vec had the correct answer second, and so on. I randomized a few of the analogies to check if this was a significant problem, and it did not seem to be. If I had API access, I would have randomized the order of all the potential answers and run multiple tests.

Results

Unsurprisingly, Claude did much better than Word2Vec. It got 25 out of 35 analogies correct¹ compared to 8 out of 36 for Word2Vec.

The most frequent failure appears to be Claude identifying the right relationship, but then incorrectly applying it to choose the answer. For example this analogy (correct answer bold, Claude’s answer in green underline):

With the following logic:

Claude correctly identifies that hospitable implies showing courtesy, but then picks the opposite relation, someone is despondent lacks glee.

All of Claude’s answers are here.

I gave an example in my last post of a good application for LLMs: editing prose. But what specifically makes this problem ideal for solving with a model? Succinctly, it is a problem where solving it is hard, but verifying the solution is easy. I will go into more detail in the rest of this post.

What Are They Good For?

In math, there are types of problems where finding a solution is difficult or impossible, but confirming a solution is easy. A common strategy to solve these problems is to guess the solution’s form and then verify it, such as for an integral where the solution can be checked by taking its derivative.

Large language models are particularly useful for exactly these types of tasks: where generating a solution is hard, but verifying it is easy. Editing a paragraph is a prime example of this kind of task since writing multiple versions is time-consuming, whereas verifying the quality of a single paragraph can be done quickly.

Other good use cases include automating data cleaning tasks or writing code, especially if you have tests in place to verify the code’s correctness.

What Are They Bad For?

LLMs are bad for problems where verification is hard compared to the generation of an answer.

Some people are using LLMs as a replacement for search engines.¹ This is a perfect example of a bad use of the technology because verifying the accuracy of the information provided by the model takes time and effort. In fact, it often involves additional searches to confirm the validity of the answer, which defeats the purpose of using an LLM in the first place.

• 
  How I Write with Large Language Models
  OpenAI's ChatGPT 3.5 transformed my writing process when it came out. After years of experience using it, I've further refined my method of using LLMs. This post explains how.

ChatGPT, the new chatbot from OpenAI, has received widespread attention for its versatility in conversation, information retrieval, and even song writing. However, its tendency to generate false but believable information has led many to see it as entertaining but not useful.^{1 (Click to see my edit prompts.)}

But ChatGPT is an incredibly useful tool for my writing due to its ability to rapidly iterate through alternative phrasings for ideas, sentences, and paragraphs. In these cases, the model’s habit of generating false information is not a concern because I wrote the original input so I know what it should say, and I can easily edit out any inaccuracies introduced by the model.²³

Let me illustrate how I use ChatGPT to write and copyedit my work.⁴

My Writing Process

I start by writing a rough draft which may consist of a few bullet points or a detailed set of paragraphs. Next, I give it to ChatGPT for refinement.

I don’t usually take the entire output from ChatGPT, rather I use it to inspire my own editing:⁵

I start by writing a rough draft which may consist of a few bullet points or a detailed set of paragraphs. Next, I give it to ChatGPT for refinement.

I iterate using this process, refining my text by sending the sections I’m unhappy with to ChatGPT for improvement until I’m satisfied.⁶

AI and Photography as Art

I come down on the side of AI-artists.

This is probably unsurprising because I am a machine learning engineer, it is my job to build the types of systems these artists are using. But what is less obvious is that my support is also because I am an artist, specifically a landscape photographer.

Photography—just like AI-generated art—has a complicated history as “art”. Although the first photograph was taken in 1826, it wasn’t until 1924 that an American museum recognized the medium as art by including photographs in its permanent collection. At first artists feared photography would replace traditional visual arts due to the ease of taking a picture. But eventually they realized it was a useful tool that could be combined with other art forms, even if they did not recognize photography as an art in its own right.^{3, 4}

The concerns and criticisms currently being directed towards AI-generated art are the same as those leveled against photography in the past. And just as photography eventually gained acceptance as a valid form of art so will AI-generated art. The resistance against it may be strong, but ultimately, it is a losing battle.

My Family’s Art

My family has a long history of painting. My great-great-great grandfather was the Norwegian landscape painter Hans Gude. My father, also named Hans Gude, was an accomplished oil painter.⁵ I too wanted to make art, but I did not have their skill with a brush so I picked up a camera instead.

I was drawn to photography specifically because it used technology. I like learning new technologies and how to master them. I also thought it would be easier to make art I was happy with using a camera. I have since learned that photography has its own set of skills to master, but after 15 years I think I was mostly right: it is much easier than oil painting.

I wonder what my great grandfather would think of my art. He spent months or years creating his seascapes, while my photographs are captured in a fraction of a second with the push of a button, and maybe a few hours adjusting tone curve and highlights back at my computer.

But I like to think that he would view my work as a continuation of our family’s artistic tradition. Maybe in the future, my descendants will find the camera too complicated and instead compose prompts for AI to translate into images. To me, that’s simply another evolution of the art form.

As per usual, the Jupyter notebook used to perform this analysis can be found here (rendered on Github).

Data Selection

I selected crashes involving pedestrians from the SQLite database with the following query:

SELECT collision_date,
       collision_time,
       pedestrian_killed_count
FROM collisions
WHERE Collision_Date IS NOT NULL
AND pedestrian_Collision = 1        -- Involves a pedestrian
AND collision_date <= '2020-12-31'  -- 2021 is incomplete
-- and it happens on Halloween
AND strftime('%m-%d', Collision_Date) = '10-31'

This gave me 1168 data points, of which 64 involve a pedestrian fatality, spanning the years 2001 through 2020. Incidents after 2020 are rejected because the database dump comes from mid-2021, and so that year is incomplete.

Incidents Per Hour

Alissa Walker wrote¹ that it is not just drivers that make Halloween deadly, it is commuters. The best way to explore this point is to look at when in the day crashes happen:

As we saw in the data for all dates, weekdays have two major peaks in collisions during the morning and evening commutes, as well as a peak during school pickup times. Examining the data for Halloween specifically, we see that when it falls on a weekday the three expected peaks (morning and evening commutes, and school pick-up) are present, but there is also a fourth peak at 18:00, likely due to a combination of darkness making it difficult for drivers to see pedestrians and trick-or-treating bringing more people out walking. This data supports Walker’s observation that commuter traffic contributes significantly to the number of pedestrian incidents.

Fatality Rates

But Walker makes a very specific claim: that fatalities involving children increase on weekday Halloweens. Does the data support this claim? To find out, we need to look at the fatality rate instead of the total number of fatalities because the number of people driving and walking changes year-by-year and using the rate helps to normalize some of this variation. Below is a plot of the fatality rates for each year’s Halloween, separated into weekday and weekend:

The data above includes all pedestrian fatalities, not just those involving children. At first glance, the distributions for weekday and weekend Halloween fatalities appear similar. A Mann–Whitney U test confirms this, with a p-value of 0.93, indicating that the difference between the two is not statistically significant.

But what about children alone (defined as pedestrians under 18)? Here is that data:

One interesting observation is that no children have been killed by cars on weekend Halloweens, whereas about half of the weekdays have seen at least one child death. This suggests that there is something about weekday Halloweens that makes them particularly dangerous for children, consistent with Walker’s claim.

Despite this, the data does not show a significant difference between the two distributions, with a p-value of 0.08. However, this lower p-value as compared to the all-ages data does indicate some evidence for the specific claim about child deaths.

So with all the new data, I wanted to look at the most vulnerable road users: pedestrians.

As per usual, the Jupyter notebook used to perform this analysis can be found here (rendered on Github).

Data Selection

I selected crashes involving pedestrians from the SQLite database with the following query:

SELECT collision_date
     , collision_time
     , pedestrian_killed_count
FROM collisions
WHERE collision_date IS NOT NULL
AND pedestrian_Collision = 1        -- Involves a pedestrian
AND collision_date <= '2020-12-31'  -- 2021 is incomplete

This gave me 282,039 data points to examine spanning the years 2001 through 2020. Incidents after 2020 are rejected because the database dump comes from mid-2021, and so that year is incomplete.

Crashes per Week

For bicycle involved incidents, I found there was an increase from about 2008 through 2013 followed by a decrease. For both bicycles as well as motorcycles, I found strong seasonality with many more crashes during the summer when people are out riding to take advantage of the weather. Pedestrian involved incidents defy both these trends:

Pedestrian involved incidents were flat or slightly down until about 2013, when instead of decreasing like bicycle collisions they increased strongly. Like both bicycles and motorcycle crashes, pedestrian incidents are strongly seasonal but they decrease in the summer (when there is a lot of light for drivers to see pedestrians) and increase in the winter when it gets dark early and drivers can’t see them. Of course there is also a massive decrease when COVID restrictions kept most people home starting in March 2020.

Day-by-day

Cars are involved in crashes on days when drivers have to commute to work and on holidays where people travel. The worst day is Halloween when people work and then go out and have fun after. I was curious if the large increase on Halloween was due to a large increase in pedestrian collisions; the answer is no:

On Halloween, drivers are more likely to hit pedestrians than on any other day of the year. But its only about 15 to 20 more incidents than on any other October day, and there are almost 200 additional car crashes on Halloween. The number of additional pedestrian incidents does not account for much of the increase in car crashes. For a more detailed analysis of pedestrian collisions on Halloween, check out my other post.

Otherwise there are some interesting patterns. Many holidays trend the same direction as cars: New Years, Memorial Day, Veterans Day, Thanksgiving, and Christmas all see a large reduction in both car crashes and pedestrian incidents. Halloween, as covered above, sees a large increase in both.

One outlier is the 4th of July. Car crashes decrease because people do not have to commute, but pedestrian incidents increase. I think this is because people are walking around in the dark going to and coming back from watching fireworks, and drivers have trouble seeing them.

Hour-by-hour

Finally we can look at when cars hit pedestrians by hour and whether it is a weekend or not:

The most striking feature is the large increase in the number of incidents during the morning commute (07–09) and again in the during the evening commute (17–19)! Commuters in cars are dangerous to pedestrians!

There is also an increase in incidents on both weekends and weekdays at about 17:00. This is probably because that is around sunset.¹

The weekend curve rises smoothly through the day, but the weekday curve has a large increase at 14. I suspect this is from school pickup which is generally earlier than the commute.

Finally, it is interesting that the number of late night and early morning incidents is much higher on the weekend. This is likely due to people going out to bars, as the number drops off at 02:00 which is when the bars close in California.

But one hard thing to do is to make sure you apply the exact same data manipulating steps to the training set as to the test set and the live data when the model is deployed. It is very easy to leak data or to forget a step, either of which can ruin your model.

To help solve this problem, Scikit-learn developed Pipelines. Pipelines allow you to define a sequence of transforms, including a model training step, that is easy to apply consistently. This post will go over how to use Pipelines with Pandas Dataframes.

Pandas and Pipelines: Formerly Not So Simple

It used to be tough to use Pandas Dataframes and scikit-learn pipelines together. There was a ColumnTransformer to work with dataframes, but it had some major limitations since the output of the transformer was a numpy array. This meant that if you used a second ColumnTransformer in your pipeline you would get the following error:

ValueError: Specifying the columns using strings is
only supported for pandas DataFrames

But scikit-learn version 1.2 updated the pipeline API to fix this! Now there is the option to output Pandas dataframes!

A working pipeline

Now that the set_output API exists, we can chain ColumnTransformer without error!

For example, we can impute for one column, and then scale it and a few others. First we set up the two ColumnTransformer, one to impute and one to scale:

# Apply each feature pipeline using a column transform
imputer = (
  "imputer",
  ColumnTransformer(
    [("col_impute", SimpleImputer(), ["x1"])],
    remainder="passthrough",
  ),
)

scaler = (
  "scaler",
  ColumnTransformer(
    [
      (
        "col_scale",
        StandardScaler(),
        ["col_impute__x1", "remainder__x2", "remainder__x3"],
      )
    ],
    remainder="passthrough",
  ),
)

Then we combined them in a pipeline:

pipe = Pipeline(
    steps=[
        imputer,
        scaler,
    ]
).set_output(transform="pandas")

And it works! There are two tricks; we have to:

1. Make the output of each step a dataframe with set_output(transform="pandas").

2. Adjust the columns names of the downstream steps because they get prepended with the name of the previous steps they’ve gone through.

Complete Example

Here is a Jupyter notebook (rendered on Github) with a toy dataset and a full Pandas pipeline example. Hope it helps!

• Should I allow this transaction?

• Should I allow this login?

• What items should I show this customer?

To make these decisions they need information about the event that they are scoring. These pieces of information are called “features”.

Up-to-date Features and Real-time Features

Some features are easy to update. Features like “The number of times the user has received a payment in the past week” are just counts that can be maintained in a feature store. When a new payment comes in, add one to the value. When another day has passed, subtract all the old payments out. Simple.

But some features are hard to keep up-to-date. What if we needed the value of the received payment feature above, but to make a decision on a received payment. Whatever system is keeping count now has to be very fast so it can update the feature and return the value to the model in order to make a decision without keeping the user waiting too long. In a payment system “too long” could be much less than a second!

Worse, some features can not be pre-computed at all. For example, the feature “Has this user ever logged in from this location?” is very useful for stopping account takeover fraud, but the model needs to make a decision before the login completes. However, the feature can only be computed once the login has started because only then do we know the location!

These features, that have to be computed during the event that a decision is being made on, are called real-time features. I will talk about one way to compute them below.

A Machine Learning System

Let’s consider a simplified machine learning system that looks like this:

The events (in purple and green on the diagram) are a never ending stream. Imagine the line of event boxes moving from right to left, each one taking a turn to dump its data into the feature store (in orange). The feature store uses the data in the events to update the values of features and reports those values to other parts of the system.

The model host (in blue) also operates on events but it handles them as they are generated, before they even have time to get to the feature store. The event that the model is currently making a decision on is the target event (in green).

The target event will eventually dump its data into the feature store (represented by a dotted line on the diagram) but has not done so yet. The target event does send some data to the model host (generally something simple like a user ID or event ID so the model knows what features to get) but this is fast compared to waiting for the feature store.

Machine Learning Model Host

Let’s take a closer look at the model host:

The data handling code (yellow in the diagram) gets IDs and other data from the target event so that it knows what features to get. For example, it might get a user ID so it can get all the features associated with that specific user. It then passes the features it receives from the feature store to the machine learning model (red in the diagram), which makes a decision and sends it back to the target event (or the system handling the target event).

Real-time Computation

To use this system to calculate real-time features, we make three changes:

• Add “proto-features” to the feature store.

• Send more data from the target event to the model host.

• Do additional processing in the data handling code to combine the proto-features and data from the target event into a real-time feature.

The diagram changes very slightly:

Here is how that would work for our example login feature:

• Add a proto-feature that is a list of previous login locations.

• Add the current location to the data passed in from the target event.

• Update the data handling code to check if the current location is in the list of previous login locations from the proto-feature.

Like last year, let’s explore how the race unfolded with data.

The code that generated the plots can be found here (rendered on Github). The data is here.

The Race for Yellow

The yellow jersey, awarded to the rider with the lowest combined time across the 21 stages of the race, was won by Tadej Pogačar the past two years, so he was the favorite going into this tour.

Other favorites this year were:

• Primož Roglič, who came in second in 2020.

• Jonas Vingegaard, who came in second last year.

• Geraint Thomas, who won in 2018 and finished second in 2019.

• Aleksandr Vlasov, who won the Tour de Romandie earlier in the year.

• Daniel Martínez, who won the Tour of the Basque Country earlier in the year.

After three weeks of racing, here is how the top five riders fared through the stages:

Pogačar started strong, taking the yellow Jersey on stage 6 after beating out Vingegaard in the final sprint. He held it through the first few Alp stages before getting isolated on stage 11. Roglič and Vingegaard, both riding for team Jumbo-Visma, force Pogačar to chase them, pulling him away from his supporting teammates. The duo then threw attack after attack at the yellow jersey, forcing him to constantly accelerate. By the time they got to the final climb on the Col du Galibier, Pogačar was so tired that Vingegaard was able to drop him and gain nearly three minutes, taking the yellow jersey.

Pogačar and Thomas stayed neck-and-neck until stage 16 when Vingegaard and Pogačar attacked in the Pyrenees and dropped the other contenders.

The Green Jersey

The race for the green jersey, awarded to the rider with the most sprint points which are earned by winning intermediate sprints and stages, was relatively boring this year. Wout van Aert ran away with it, never losing the jersey after winning it on stage 2.

Here is how the sprint race turned out, with sprint stages shaded in grey:

The Rest of the Race

176 riders started the race and just 136 finished, the lowest number to finish since 2000. Here is how each rider fared:

Wout van Aert, the green jersey winner, paced himself well. He finished just above the middle of the pack. This is in contrast to last year’s winner, Mark Cavendish, who finished near the bottom that year, because he is a much weaker climber than van Aert. Van Aert is an amazing talent, able to climb and sprint!

Another sprinter, Caleb Ewan, got the lanterne rougue, a prize awarded to the last place rider. Normally a strong contender, he crashed hard on stage 13 and struggled to stay with the other riders, but still managed to cross the finish line in Paris.

Pre-race hopeful Primož Roglič crashed and dislocated his shoulder on stage 5. He held on for 10 days—thankfully for Vingegaard who used Roglič to weaken Pogačar on stage 11—but dropped out during stage 15.

• I only collected dense data starting right before the property went pending.

• I collected data by hand so I often missed days.

When another nearby house put a sign out saying “coming soon”, I wrote a script to automate the scraping and collected a much denser time series from Zillow, Redfin, and Realtor.com. Let’s see what we can learn with more complete data!

You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here.

Data Collection

I wrote a script to download the entire page for the specific house from each of the three sites. I ran it on my Raspberry Pi everyday using cron. I parsed the HTML using Python and a wrote the cleaned data to JSON. That parsing notebook can be found here (rendered on Github). I won’t include the raw data, you will have to collect some yourself.

Plot

Here is a plot comparing the estimates for the sales price from Zillow and Redfin in time:

The daily price estimates from Redfin are shown using red circles, the estimates for Zillow are shown using blue triangles, the estimates for Realtor.com are shown using purple diamonds.

Comments

I wrote a script to ensure that I would get data for every day, but as you can see there are still many missing points.

Realtor.com

Realtor.com is the most frustrating! As soon as the house was actually listed on the market they stopped providing estimates! Realtor.com starts estimating again only after the sale price is posted. This defeats the entire point! The estimate is most important when the house is actually for sale and Realtor.com just punts completely. Embarrassing.

Still we can see that, when the house is not for sale, they update their estimate roughly every two weeks. Their initial estimates are not too bad, just about 6% low from the final sale price.

Zillow

Zillow similarly is missing estimates for most of the time when the home is actually for sale. Their page shows “Zestimate: None” with an error explanation blaming county transactional data.¹ I am sure that’s true but I am unimpressed. Dealing with missing data is a key part of building a robust machine learning model.

Zillow’s model updates more frequently than Realtor.com’s. It slowly climbs until it abruptly stops estimating a few days after the listing is posted. This suggests they make use of a different model for currently on-the-market homes and that that model requires more and different data than the off-the-market model.

The Zillow estimate does return at the end of the pending period but… I do not have anything nice to say about it. Just look at that variance!

Zillow underestimates the final price by about 10%.

Redfin

Redfin is the only company that keeps posting estimates once the house is actually for sale! Their pre-listing estimate is almost exactly right, but once the house is listed their on-the-market model overestimates by about 10%.

Before the listing is posted, Redfin updates its estimate roughly weekly, and like Zillow it takes 5 days to switch to the on-the-market model. This suggests that both sites get their data indicating the house is for sale from the same source. After the listing the model updates daily.

The on-the-market model trends upwards at first and then stabilizes after the house is pending, but because the time between listing and pending was so short it is impossible to tell if the stabilization was due to the listing going pending or not.

Conclusions

With the denser data and all three sites to compare, I conclude the following:

• Zillow and Redfin both use separate models for the time pre/post-listing and the time when the house is listed. It’s likely Realtor.com does as well which is why it stopped updating as soon as the house was listed.

• The on-the-market model, used during the listing period, are updated more frequently, this is probably due to the fact that they have more high-frequency data (views, time on market, etc.) and the possibility of making a commission on the property increases the amount of money they’re willing to spend running the models.

• Zillow and Redfin are both using the same data source to determine when to switch their models, and it appears to be different than the source they use to display if a home is actually listed or not. I don’t know why that would be. Possibly their models require data that is not immediately available?

• Zillow and Realtor.com require the same data for their model during the listing period and fail if that source is unavailable. Redfin continues estimating, but does so poorly.

Statement

I am currently a machine learning engineer at a major tech company. My career has taken a winding path, but I am very happy with where I am and where I am headed. I have worked hard, but that alone does not explain my success; I have had many lucky breaks, times when some event completely outside of my control worked out in my favor.

I think it is important to remember that no one is entirely self made; every career has some element of luck involved. I have cataloged my lucky breaks on this page in the form of a too-honest resume.

Education

• My GRE score was pretty bad. In 2008 I was rejected from every grad school I applied to. In 2009, Yuichi Kubota saw my application to Minnesota and thought “we can give him a shot”. He did that for a lot of people my year.

Experience

• My friend’s aunt worked at Lawrence Berkeley Labs as an executive assistant. My resume got passed to her (I don’t even recall how exactly) where it eventually found its way to the Supernova Cosmology Project under Nobel prize winner Saul Perlmutter. Saul’s post doc, Nao Suzuki, wanted a research assistant and invited me up to the lab.

• Weirdly, and lucky for me, Nao had decided not to use IDL, the standard language for astrophysics software, and instead wanted to work with Python. We used Numarray and Numeric (which would later become Numpy), giving me a head start in the skills I’d need later for machine learning.

• Nao also thought I should learn a text editor and introduced me to the one he used: Vim. It is my primary text editor to this day (I’m writing this post on it).

• I had decided not to go through the faculty lottery and was trying to figure out my next steps. I was looking through my saved bookmarks one day when I clicked on a blog written by my former student instructor at Berkeley, Jessica Kirkpatrick. The top post was Career Profiles: Astronomer to Data Scientist.¹ Inside was a link to Insight Data Science. There were just two days until applications to Insight were due so I threw one together and was accepted.

• My advisor told me there was no way I would graduate by January, 2015 (he was right). I emailed Insight and asked if I could delay starting until the next session. They said yes. I have since learned their policy is not to do that. I’m glad someone made an exception.

• My manager and director both left Intuit just after I joined, leaving our team rudderless. It gave me an opportunity to step into a management role first unofficially, and then officially several months later.

• While looking for a job after being laid off during COVID, I called my friend and former coworker from Lab41, Patrick Callier, asking if he had a lead on any positions at Square. He suggested I should work for Cash App and introduced me to my current boss and skip-level (also both Insight Alumni).

Most Vim users love tweaking their editor configurations and perfecting their editing habits. This is referred to as sharpening their saws by Drew Neil. I am no exception of course—I made a custom color scheme, Eldar, to make Vim that much more perfect for me—but there is one area where I am an iconoclast:

I still love hjkl.

Let me explain:

Movement

Moving around in Vim is a little different from other programs because there are so many ways to do it. Most users start with the arrow keys, but they quickly learn about using hjkl to move the cursor. Using hjkl might seem unnatural but it starts the user down the path towards thinking in modes, and besides it is nice to not have to move your hand off the home row.

Eventually, in the spirit of sharpening their saws, most users learn they can use search, marks, and GOTOs to more efficiently “jump” to where they want to go without mashing jjjjjjjjj. Some even argue that hjkl are irrelevant, as done here on Reddit² or here on Stack Overflow³.

And they are right on one point: hjkl can’t compete on pure speed of movement. But they are wrong when it comes to actually editing code.

Code Editing

When I am editing code, I am not writing most of the time. I am reading. I am thinking. I find hjkl amazingly effective for this.

Using hjkl lets me browse my code line by line, taking in the structure, the layout, and considering what changes I am going to make. I feel more connected with the logic of what the code describes than when I’m jumping quickly around the file.

If you think about it, this is exactly the type of editing Vim was designed for. Unlike most editors where you can just type and insert characters, Vim forces you to decide that you are going to make an insertion and then switch to insert mode to do it. Then you immediately leave insert mode. Vim expects you to spend only part of your time writing, it expects you to spend a lot of time reading, reorganizing, and navigating your code. And I find the lazy scrolling of hjkl to fit perfectly with this philosophy.

But what if the estimates change day-to-day? Christopher Moody posted this on Twitter:

I suspect that their algos factor in impressions. I bet there’s a fascinating time series in between pre- and post-listing price estimates

I decided to look.

You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here.

Data Collection

I recorded the estimated sales price everyday from Zillow and Redfin. I excluded Realtor.com and Xome because they do not update their estimate frequently (they seem to only update when the status of the house changes from pre-listing to listing to pending to sold). I occasionally missed a day of data because I was collecting it manually.

Unfortunately the idea to collect the time series came after the house was already pending, so I missed some of the most interesting changes. I have set up a Python script to scrape listings for a few other houses and should have a better time series for another article.

Plot

Here a plot comparing the estimates for the sales price from Zillow and Redfin in time:

The daily price estimates from Redfin are shown using red circles, the estimates for Zillow are shown using blue triangles.

Comments

There is a lot of movement in the estimates. Redfin and Zillow start off far apart. Redfin reverts strongly to the list price once the house is official on the market (I missed Zillow’s price on the day of listing). As the house goes pending, both estimates increase drastically.

This suggests that some impression metric is used in both models, because there is not much other information available that could cause such a large increase. It will be interesting to see what a better time series will reveal.

Both estimates vary day-to-day even after the house is pending. You would think all the information you need to determine the sale price would be present once the offer is made, but there are facts you can observe after the listing is pending—like how long the closing period is—that must have predictive power. Redfin’s estimate remains pretty tight while Zillow’s changes wildly, sometimes by 20% between days! The high variance really reduces my confidence in Zillow’s model.

Finally, both estimates completely miss the actual sales price. The estimates eventually jump to near the sales price, but it is not immediate; they take about a week to adjust. Possibly a model update is triggered immediately for newly listed properties—notice how the Redfin price adjusts the same day it was listed—but not for sales? This makes some sense. A listing is an event these business care about because they can profit off it, but a sale does not provide that opportunity.

Overwatch fans love debating which team each year is the “gatekeeper”. The gatekeeper is the team that beats lower ranked teams, often in a lopsided fashion, but can’t themselves move out of the midtable. In this way they gatekeep the standings: top teams can beat them, bottom teams can’t, and so they define the dividing line between the two groups.

Determining which team is the gatekeeper of each season seems like a good question to answer with data. I can look for a team that is good at beating low-level competitors, but fails against the top teams.

Gatekeeper Score

To find the gatekeepers of each season, I need to define some metric to measure them by. I will use what I call the gatekeeper score.

The gatekeeper score is the win percentage against teams that finished lower in the regular season standings minus the win percentage against teams that finished higher in the standings. A perfect gatekeeper team, one that beats all lower ranked teams but loses to all the teams above them, would have a gatekeeper score of 100 (100% win rate against lower teams minus 0% win rate against high teams).

This score captures most of what I want, but there are a few problems:

Undefined Scores

The best and worst team each season have an undefined score because there are no teams better or worse than them. This is not a big problem because to be a gatekeeper you must gatekeep someone, and you can’t do that at the top or bottom of the standings.

High Volatility Near the Top and Bottom

Teams near the top or the bottom of the standings have a small number of matches used to compute one component of their score. This tends to increase the volatility of their scores relative to midtable teams that have a lot of matches counted on either side.

This is a bigger problem because it means that teams near the top and bottom are more likely to have a high gatekeeper score just because they won or lost a single match, whereas teams in the middle need to win or lose many matches to change their score.

I could adjust the score by the number of games, or compute an estimate of the variance, but for now I will just call out this issue as I run into it.

The Data

I used two sources of data for this comparison. The first is a record of the outcome of every map played from the Overwatch League’s Stat Lab. I transform this data to get match-level¹ win-loss records for each team by opponent. You can find that data here and the notebook to parse it is here (rendered on Github).

The second data source is the regular season standings of all the teams. I scrape this data from Liquipedia. I used the regular season standings because the final standings are based on a handful of playoff games while the season standings incorporate many more matches and so provide a more accurate estimate of a team’s performance. The parsed standings data is here. The code to generate the data frame is here (rendered on Github).

To compute the gatekeeper score I used both regular season and tournament games, primarily because the dataset does not separate them and I don’t want to go label them by hand.

The final, combined data frame, can be found here. The notebook to read it is here (rendered on Github).

The Gatekeepers

Here are the gatekeeper scores for each season and region. The tables are ordered according to the regular season ranking. I have highlighted teams I think could rightfully be called gatekeepers.

2018 Season

The 2018 Florida Mayhem were a bad team with a possibly even worse uniform. They went 7-33 in the inaugural season and were saved from the bottom of the rankings by the team with the worst losing streak in professional sports history: the 0-40 Shanghai Dragons.

The Mayhem went 3-0 against the Dragons for a 100% win rate against worse teams, and 4-33 against better teams giving them an impressively bad 11% win rate. Combining those rates yields a gatekeeper score of 89!

But are they the gatekeeper? They run into the volatility problem I described above: 3 wins against the Dragons got them 100 points and 4 wins against better teams lost them only 11.

I think the 2018 Seoul Dynasty are a better candidate for the gatekeeper team. They went 22-18, solidly middle-of-the-pack, and they have a 93% win rate against lower ranked teams with only a 32% win rate against higher rated teams. They are also the first team with a positive win rate and map differential as you work your way up from the bottom.

2019 Season

The 2019 season added eight teams to the league and was dominated by the highly-technical GOATS meta in which teams played three tanks and three supports. During this meta, teams were forced to rigorously track their opponents ability usage and perfectly time their own in order to win fights. The Vancouver Titans, the San Francisco Shock, and to a lesser extent the New York Excelsior mastered this style of play and dominated the league while many other teams failed to achieve the high-level of coordination required and sunk down in the rankings.

Once again a bottom-ranked team has the highest score and for the same reason: volatility. The 2019 Washington Justice went 4-0 against Boston and Florida and 4-20 against better teams.

More interesting are the 2019 Hangzhou Spark, who placed fourth. They lost every game against the top three teams, going 0-7, including 2 lost playoff games against the eventual champions the San Francisco Shock. But the Spark made up for it with a 75% win rate against worse teams.

What feels right about the highly-ranked Spark being the gatekeepers is it follows the storyline of the 2019 season: that the Vancouver Titans and the San Francisco Shock were a tier above everyone else in the GOATS meta, that only a few other top teams were able to execute the GOATS composition with enough coordination (among them New York), and that everyone else floundered trying to play a meta they did not have the skill to.

2020 Season

The 2020 Overwatch League season was split into two regions with very few games played between them due to the 2020 COVID pandemic. For that reason I have split the comparison in two and look at the North American and Asian regions separately.

North America

The 2020 Atlanta Reign were the first time I remember a team specifically being referred to as a gatekeeper. They had flashy, lopsided games against teams below them in the standings, but constantly failed to advance by beating teams ahead of them. You can see it in their stats: the Reign have an 85% win rate against lower ranked teams—the third highest in the league behind the eventual champions the Shock and Washington who only had team below them—but just an 18% win rate against higher ranked teams.

However, I think the 2020 Dallas Fuel are the true gatekeepers of the league. They had almost as high a win rate against worse teams at 83%, but a much lower win rate against better teams at 7%—the lowest in the league that year!

Asia

There were two facts in the 2020 Asian region: the Shanghai Dragons were the best team in the region by a mile, and the London² Spitfire were the worst by a wide margin.

Amongst the remaining teams, the 2020 Hangzhou Spark look like a good candidate for the gatekeepers. They beat lower ranked teams 67% of the time, but only managing 17% against better teams, and they finished exactly in the middle of the pack in the standings.

2021 Season

The 2021 season was again split into two regions due to COVID, but there was a lot more cross-play between the regions as each of the four tournaments and the playoffs included teams from both regions. Still, I only look at the record against teams within the region as that is how regular season standings were determined.

North America

The 2021 London Spitfire were a disappointing team. The majority of the players had been called up from the Spitfire’s academy team the British Hurricane who had gone 12-0 and won the 2020 Overwatch Contenders season (Overwatch’s minor league). But they floundered in the Overwatch league, barely scrapping together a 1-15 season. They got their only win in the infamous Bread Bowl against the 1-15 Vancouver Titans. That gave the Spitfire a 1-0 record against lower ranked teams and a 0-15 record against higher ranked teams, resulting in a perfect 100 point gatekeeper score.

Of course a one-win team isn’t a gatekeeper, despite what the score says. Likewise it’s hard to call the 2021 Los Angeles Gladiators gatekeeper because their 0% win rate against higher ranked teams (just the Dallas Fuel) represents only two games.

For that reason, I think the 2021 Toronto Defiant are the gatekeepers of the North American region. They had a typical middle of the pack season: finishing 7 out of 12 in the West, a 9-7 match record, and a perfectly balanced map record with 32 wins and 32 losses. To round it out, they had an 82% win rate against worse teams and just a 20% win rate against better teams.

Asia

The 2021 Guangzhou Charge have the highest gatekeeper score, but it is, once again, due to only having a single team below them: the winless 2021 Los Angeles Valiant. So I do not think they’re a good choice for gatekeepers.

Instead, I would choose the 2021 Seoul Dynasty, who had a similarly bad win rate against better teams (22% for Dynasty, 21% for Charge), but who earn their 85% win rate against lower ranked teams honestly by beating teams that have actually won games, like the 10 and 10 2021 Philadelphia Fusion.

Recently a house in my neighborhood put up a “For Sale” sign which prompted me to look online for the listing. I couldn’t find one. None of the online real estate brokers had picked it up yet. I realized I had a chance to compare their current price estimates with the actual listing and sales price.

You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here.

Data Collection

I recorded the pre-listing estimates for four brokers: Redfin, Realtor.com, Zillow, and Xome. Xome provides both an estimate and a high and low range. All the others only present a single estimate. I also collected the same estimates after the listing was picked up and again after the sale was complete. The data is summarized in the table below:

I could not find Realtor.com’s estimate after the listing went up, so it is not included. Xome did not change their estimate after the listing was posted.

The house was listed at $948K and sold for $1070K.

Plot

Here a plot comparing the three estimates from each company to the list and sales price:

The three price estimates are:

• Pre-listing: Before the listing was picked up by the brokers, represented with a circle: ●

• Post-listing: After the listing was posted but before the sale, represented with a triangle: ▲

• Post-sale: After the sale was made public, represented with a square: ■

I have slightly offset the date points for each company—with the circle on left, the triangle in the middle, and the square on the right—to give a quick indication of how the price trended in time if you read from left to right. The Xome estimates include error bars for their high and low estimates. The listing and sale price are shown as lines. I have colored each company’s estimates to match their brand color. The companies are sorted from lowest to highest pre-listing estimate.

Comments

Zillow and Redfin strongly disagree about what the value of the house is initially, with a difference between their estimates of $200K. Both estimates poorly predict the final sales price, with Zillow underestimating by $132K and Redfin overestimating by $74K.

Both estimates revert towards the list price when it is posted, with Redfin’s slightly higher than the listing and Zillow’s slightly lower. This makes some sense, as the list price contains new information about the current market conditions and more importantly about the condition of the house and property relative to its neighbors. However, in this case the list price was obviously too low and likely intended to entice buyers.

Xome’s pre-list estimate is the closest to the sale value, missing by just about 3%, although they gave themselves a lot of room with their uncertainty. Here are the four pre-listing estimates ranked from lowest to highest absolute error:

The Data

The data was collected by my wife and I using a Google form on our phones. When we heard a new word we would log it. I then normalized the words (sometimes we would write down grandma and sometimes grandmother for the same word, for example) by hand and took the first occurrence of each word in each language as the date when they learned it.

I discuss data collection in more depth in Theo’s, Cory’s, and Ash’s data sections. You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here, here, and here.

Development

Below I have plotted the number of words each of my sons knew as a function of their age. Theo, our first son, is represented by dotted lines; Cory, our second son, is represented by dashed lines; Ash, our third son, is represented with solid lines.

Cory learned all of the languages much faster than his brothers. Theo was the slowest to learn English and Chinese, the primary languages of our household, but slightly faster than Ash on Spanish, Sign, and Animal Sounds. Interestingly, Ash’s Chinese started strong but has slowed; had we kept recording I suspect Theo would have caught up and passed him at 26 months of age.

I discuss the slow Chinese acquisition in the development section of Ash’s post, but briefly I think it is related to me working from home during the COVID pandemic. This meant that:

• We used a lot more English at home.

• I was able to better identify when he had learned new English words but not new Chinese words, which led to a selection effect in what was recorded.

With one additional month of living with Ash (not reflected in the chart and data) his Chinese has really taken off recently, using whole sentences instead of just a few words, which leads me to suspect that the slowdown in the data is mostly due to the selection effect mentioned above.

Also interestingly my wife tells me that Cory’s current Chinese is worse than Theo’s was at the same age. Perhaps Cory’s development slowed or there is more to proficiency than just number of words known.

The shape of the English curves are all very similar, just displaced in time. I would guess this has to do with the fact that they are always surrounded by English speakers to learn from whereas the other languages require special interactions with their family to learn.

Ash’s Spanish has taken off much slower than our other two sons, although in comparison he is not too far behind Theo. Theo’s long drought of Spanish words is due to an injury my mom sustained that prevented her and my father from watching Theo and so reduced his contact with Spanish. Ash doesn’t have the same excuse, but it does show it’s not too late for him to pick it up.

Ash never got much into sign language. He learned pointing quickly and got by just doing that until he could speak. We also didn’t emphasize it as much because we were very busy with our other two boys. I suspect this is why he didn’t know as many animal sounds as well: I used to sit with Theo and Cory and point at animals in books and teach them but I did not do this with Ash.

As a final note: our kids’ doctors were worried about both Theo and Ash’s language development being too slow. Some of that worry transferred to us as parents. Keeping track of their words like this was reassuring—we could see that Ash was on pace compared to Theo, and we knew Theo turned out fine!

Other Writings on Language Development

If you enjoyed this article, here are all the other articles I wrote about language development!

• 
  My Third Son’s Language Development
  We tracked my third son's language development word by word. Here, in plots, is how he learned to speak. Take a look!
• 
  Comparison of My Two Sons’ Language Development
  Being a nerd dad, I recorded all the words my first two sons spoke as they learned them. Now, I compare their language development rate!
• 
  My Second Son’s Language Development
  My second son is a little over two years old. We tracked every word he's spoken to watch his language development, and now you can observe it too!
• 
  My Son’s Language Development
  My son is a little over two and unfortunately he has two huge nerds for parents. We tracked every word he's spoken to watch his language development, and now you can join us!

The Data

The data was collected in the same manner as last two times. One difference was that Ash learned to speak during the COVID-19 pandemic, when we were much more isolated from family and the rest of the world, but also I was working from home so it was easier to enter my own data rather than have my wife text me when Ash said a new word.

The main difficulties in data collection were the same though:

• Deciding when Ash had associated a sound with a concept as opposed to just babbling.

• Deciding if Ash “knew” a word or was just repeating a sound he had just heard.

You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here.

Development

Ash’s first word was “milk” in Cantonese, spoken at 13 months old. Milk was something he loved back then and the word is still in frequent use. He often demands “milk milk milk” while shoving his empty cup towards us. His second word was “mom” in Cantonese and his third word was “dad” in English. Unlike my other children, Ash didn’t start using the Cantonese word for “dad” until very late. He preferred his own pidgin where he would simply use the English word “dad” in an otherwise Cantonese phrase.

Ash learned a few words of baby sign language, but unlike his other brothers was not too interested in learning much.

Ash’s language development is plotted below, showing the number of words he could speak in each “language” as a function of how old he was.

Ash picked up English and Cantonese at roughly the same rate, with Cantonese leading in the number of words spoken until he was 23 months old when he started speaking more English. I suspect the reason his English learning outpaced his Cantonese is because I was working from home during the pandemic. This had two effects:

• Ash got a lot more exposure to my wife and other sons speaking to me in English, and more time listening to me talk to him.

• I was able to record new English words he was speaking, but since my Cantonese is bad (really bad) I could not do the same for it and so relied on my wife. This made it more likely that I would write down a new English word while missing more Cantonese words.

Ash’s language development really took off at 17 or 18 months of age and went nearly vertical at 23 months. During his 23rd month he doubled the number of English words he knew to about 100 and overtook the number of Cantonese words, which climbed more slowly.

Ash’s Spanish development has been slow. He has had plenty of time with my parents (who both speak Spanish) but not much alone time with them. He normally visits with his older brothers who speak mostly English to my parents now, which has reduced Ash’s exposure to Spanish. Ash also did not love Spanish cartoons, which his older brothers did at his age.

The Words

I plotted a selection of some of Ash’s first words in each language below. Notice that I have switched to a log plot for the y-axis to better show the beginnings of each language.

Here is a selection of fun words Ash learned:

• Google (English): Like all my sons, Ash was fascinated by the Google Homes we have everywhere. It will be interesting to see how their concept of Google evolves over time. To me it is the search engine company, to them it is the little black speaker that sits on the shelf and has a personality.

• Gondola (English): All three boys love the gondolas at the Oakland Zoo. Ash learned to say “ganda” very quickly to indicate that he wanted to ride.

• Cookie (Spanish): All three boys learned to say cookie in Spanish very quickly because my parents give them cookies when they visit. Ash actually tells us that “cookie” in English is wrong and still only calls them “gagas”.

• Hulk (Animal Sounds): Cory loves Hulk and has a large action figure he plays with. Ash learned from Cory that Hulk makes a roaring sound while smashing things and would mimic it while playing with the toy.

• Little Brother and Big Brother (Chinese): Ash learned how to identify his brothers very quickly, mainly to complain to us when they took his toys!

Other Writings on Language Development

If you enjoyed this article, here are all the other articles I wrote about language development!

• 
  Comparison of My Three Sons’ Language Development
  I recorded the words my sons spoke as they learned our various languages and now I compare how each developed! Read on to find out how each son learned.
• 
  Comparison of My Two Sons’ Language Development
  Being a nerd dad, I recorded all the words my first two sons spoke as they learned them. Now, I compare their language development rate!
• 
  My Second Son’s Language Development
  My second son is a little over two years old. We tracked every word he's spoken to watch his language development, and now you can observe it too!
• 
  My Son’s Language Development
  My son is a little over two and unfortunately he has two huge nerds for parents. We tracked every word he's spoken to watch his language development, and now you can join us!

I have been on both tracks, and while it is true that you can continue to be promoted without going into management, there are some trade offs. Many of these trade offs are discussed in depth elsewhere, but there is one I want to highlight that is less obvious: There is more room at the top of the manager track than the IC track.

The Structure of the tracks

Every company is slightly different, but most of them have their tracks set up something like this:

    L1 → L2 → L3 → L4 → L5 → L6 ...
                  ↳ M4 → M5 → M6 ...

Where an L (short for “Level”) indicates an IC role and an M indicates a manager role. The higher the number, the more senior the role.¹

The only role open to the most junior engineers and scientists are IC roles, which makes sense: junior people lack the hard-earned experience companies want a manager to have. As they get promoted they move from L1 to L2 and so on until at some point they reach a level (L4 in my cartoon example above) where they can move onto the management track (prefixed with M). Of course not every IC gets this opportunity.

Both tracks narrow as they go up. There are generally a lot (perhaps tens of thousands for the tech giants) of L1, L2, and L3s in a company, but L4s and above are rarer. Likewise they have an army of front-line managers (M4) but only one CEO (or two in rare instances) at the top of the management track.

The Pyramid and the Spire

The IC track is very wide at the bottom. I was not exaggerating when I said that there are tens-of-thousands of lower level ICs at the tech giants. But what is different is how fast they narrow. The IC track gets narrow very quickly while the manager track stays broad longer. Let me share some examples from my previous company:

The Highest Levels

At the very highest level, there were a few (probably about five) executive vice presidents—M9 on my example tracks. There were no L9s at the company. There was a single L8, someone with more than 30 years of IC experience in Silicon Valley. The company had more than a dozen M8s.

In Data

In the data organization, there was one senior vice presidents (M8), two vice presidents (M7), and roughly five directors (M6). There were only two L6s, and one of them had almost 20 years of experience at the company. All of the managers had less tenure than that.

The Track Forward

I am very happy as an IC, I love solving problems and writing code. But I love helping people, setting strategy, and communicating with partners. I still haven’t decided which track I will stay on in the long term.

From a pure numbers game, if I want to climb to the highest levels, the manager track looks better. But it is easier to jump between companies on the IC track—I could leave my job today and have a stack of comparable IC offers within a month. Also the manager job is just different, I would no longer be an engineer.

I find Charity Majors’s post Engineering Management: The Pendulum Or The Ladder an excellent discussion of the drawbacks of being a manager and one strategy for jumping back and forth, one that has guided my career thinking. Give it a look!

This post was meant to be a chapter in a book where data scientists who had transitioned from academia to industry shared their experience and advice. Unfortunately, the 2020 pandemic seems to have killed the project, so I am sharing my chapter here. Enjoy!

I didn’t become a physicist for money—quite the opposite, in fact. I remember walking across UC Berkeley’s Sproul Plaza with my father on our way to our favorite lunch spot when I told him, “I love that Cosmology has so few practical applications! It’s amazing that someone will pay me just to expand human knowledge!” Perhaps that should have been my first hint that Physics did not have great long-term prospects, but I don’t think that would have changed my decision to pursue it.

The first time I visited CERN during my PhD, I got an ascetic feeling from looking at the dilapidated buildings and missing storm shutters. Its exterior reflected the same feeling I had expressed to my father a few years earlier—that science was worth more than dollars. I felt instantly at home.

When I ended up leaving academia several years later, it was for stability, not money. I had no intention of dragging my family around the globe just to end up six years older and once again hunting for a job. I’d seen my CERN colleagues try—and universally fail—to land tenure-track positions. So I decided to head back to where I grew up—the Bay Area—and see if I could make it as a data scientist with the help of the Insight Data Science program.

Compensation

As someone who’d never been motivated by money, I discovered during my job search that I was a little naive as to what all the numbers meant. Fortunately, Insight anticipated this and brought experienced data scientists, startup founders, and tech company executives to give us a crash course on life outside of academia. I learned everything there was to know about salary, bonuses, and equity.

I was already familiar with base salary: what you’re paid in semi-weekly paychecks in exchange for doing your job. It’s what I thought of when I heard “This job pays $120,000 a year.” But salary isn’t everything—data scientists often get paid in several different ways.

Every offer I received paid a yearly bonus of 15–20% based on performance. These were from standard tech companies, but I also heard from data scientists in finance and investment who’d be disappointed if their bonus wasn’t an “integer multiple of their salary.”¹

Insight also spent a lot of time teaching us about equity: owning a piece of the company you’re working in. For publicly traded companies, things were simple: I would get a certain number of RSUs that would vest over the next three or four years, at which point I could sell them (or hold them).²

Startups were more complicated. I didn’t get any offers from startups, but this is what I learned from Insight and from working with VCs for two years: particularly valuable early hires can receive up to a few percent ownership of the startup, but over time that share would “dilute” as new stock was issued. Complicated vesting structures could mean that even with a good exit, your shares might be worth less than you thought if you’re far down the liquidation preference stack.

After all the preparation, I had two offers to consider. But how to choose? Just pick the biggest number, right? Turns out it isn’t so simple. There are a lot of intangibles—like “what work would I be doing?” and “who would I work with?”—that matter a lot, possibly more than money. I decided to go with the offer that had a bunch of great people on the team and a very interesting problem, even though the total compensation was lower.

Negotiating

Once I had decided on an offer, I entered the most stressful part: negotiating. Insight had drilled into me that regardless of the offer I picked, I had to negotiate it. I spent a few days talking to my friends and family about the offer, trying to work up the courage to call the hiring manager. In the end, I took a deep breath and wrote a quick email:

Hi!

I’m very excited by your offer and the thought of working with the team, but I have another offer for $15,000 more. Can you increase yours? If so, I’ll sign today.

Alex

They wrote back a few hours later, having increased the offer by $7,500. I signed immediately. That five-minute email earned me $18,000 more over two years in increased salary and bonus. A pretty good return on investment for a few minutes of stress.

That time, I had leverage: a higher competing offer. A second offer shows your true market value—someone else is literally willing to pay you that much.

Two years later, I got an amazing offer from my current company, far higher than what I’d said I was looking for during the interview. I knew I had made a slight error giving the first number, and in the back of my mind I could hear my Insight advisor telling me to negotiate. But I was nervous. I did not have a counteroffer for leverage or to fall back on.

I talked to the VCs I worked with for a few days to work up the courage. They all gave the same advice: negotiate. One even said that when he was hired, he was sure his new boss would have seen it as a red flag if he hadn’t tried to get a better deal—after all, making deals was his job. Data science is a little different from investing, but the advice holds: as a hiring manager, I’ve never once minded when a candidate negotiated. I’m just excited to close on someone who’s going to help my team reach our goals.

I wrote an email asking for a lot and got a reply back asking for a phone call. I laid out my rationale—that my friends with similar experience were getting those kinds of offers. They kept me sweating for a few days, but finally returned an offer with a $5,000 higher base salary and $50,000 more in RSUs. It was a lot more stressful than my first negotiating experience because I did not have a fallback option, but again the return on investment was huge!

Helping Others Negotiate

Even if you are not motivated by money, as I was not, if there is some cause you care about—your family, the environment, helping others—you should negotiate. Why? Because money will give you the means to advance your aims. You can donate it, hire someone with it, or put it away for later. In the end, either you or your employer will have the extra dollars; who do you think will put it to better use?

One of my friends got a great offer for a software developer position in the Midwest and was reluctant to negotiate. He asked me, “I have no kids, we own our house. My wife and I don’t need anything. Should I still negotiate?” I encouraged him to do so anyway. With just a few emails, he got a larger signing bonus,³ which, true to his ideals, he donated to charity.

The only leverage he had was that they wanted to hire him and he had not signed their offer letter yet, but that is often enough. Hiring managers spend a lot of time screening candidates; when they find one they think is a good fit, they are anxious to close the deal before someone else does! Offering to sign immediately if your conditions are met is a great way to open a negotiation. You signal to the other party that if they do this one thing for you, they will get what they want: your signature. It’s a win-win.

Another one of my friends got an offer for her first data science job at an autonomous driving company. She had a few other offers pending, but this was her dream job. I had helped her through each part of her job search, and so she called me to talk over the offer. I told her it was a great offer, but since she still had offers pending, she should ask them for $10,000 more base salary to sign immediately.

She initially balked; it was her preferred company, after all, and she didn’t want to lose the offer. I told her they had already spent a lot of money interviewing her and determined that she was the right candidate. Asking for more was not going to jeopardize the offer; they would actually appreciate the opportunity to close on a candidate without having to compete. After a few minutes, she promised to send them an email. A few days later, she called: they’d said yes immediately.

Agreeing to sign immediately and having other offers are two good ways to start your negotiating that I have used and coached my friends in using, but there are many more. I highly recommend reading Patrick McKenzie’s guide to negotiating, where he covers everything from what the employer’s bargaining position is to a step-by-step guide through the whole negotiating process, and even what to say to avoid the dreaded “So what’s your current salary?” question. Patrick’s guide has helped engineers and data scientists earn combined millions of dollars in additional compensation; you should join their ranks!

Finally, remember that money isn’t everything. You can negotiate all the various parts of your compensation, including things you might not consider part of your compensation: vacation days, work location, and the ability to work from home. These can be easier to negotiate because, as an employee, we often value them much higher than the exact monetary value assigned by employers.

Negotiating an offer is stressful but lucrative. Data scientists get many forms of compensation and are in high demand. That gives us leverage. Use it. It’s easier than you think, and you truly have nothing to lose.

This plot isn’t perfect—I will fix it below—but even so it is striking. Before the stay-at-home order the number of collisions involving Toyotas was much higher than those involving Fords. After the order, the trend flips. Fords have more collisions. I had to figure out why.

The code for this analysis can be found here (rendered on Github). The data is available on Kaggle or Zenodo. There is a hosted Kaggle Notebook version of this post as well to help you dive right in.

Data

I select all collisions between 2019 and November 30, 2020 that involve a Toyota or a Ford, with this query:

SELECT c.collision_date
    , p.vehicle_make
    , count(1) as total
FROM collisions AS c
LEFT JOIN parties as p
ON p.case_id = c.case_id
WHERE c.collision_date IS NOT NULL
AND c.collision_date BETWEEN '2019-01-01' AND '2020-11-30'
AND p.vehicle_make IN ('ford', 'toyota')
GROUP BY 1, 2;

I start the data in 2019 because I need a sample from before the pandemic changed behavior, but I didn’t want to go too far back because collision rates vary drastically year-to-year. I cut off the data in November because the reporting is not yet complete for December.

Normalized Collision Rate

The number of collisions depends on many factors, primary among them is vehicle miles traveled.¹ To help control for VMT, I normalize the mean number of collisions for each make of vehicle from January through June of 2019. This gives me a baseline to compare against. Here is the normalized plot:

Interpretation

The normalized rates match up well through the Christmas and New Year holidays, which is the two-week dip caused by people taking time off work and hence not commuting. But right after, the series diverge:

• Toyota collisions trend down a few weeks before the stay-at-home order and drop off significantly the week before. Ford collisions stay constant until the order. This suggests that Toyota drivers made a decision to stay home by themselves while Ford drivers waited until the state mandated it.

• Toyota collisions drop much more the week of the order, likely indicating that more Toyota drivers stayed at home when told to do so.

• Ford collisions recovered towards their pre-pandemic level faster than Toyota collisions, indicating that Ford drivers were quicker to get back on the road when allowed to.

Taken together, I think these observations suggest the difference is due to a white-collar–blue-collar divide. White-collar workers generally have more flexible work arrangements and their jobs are easier to do from home, whereas blue-collar workers have to travel to a job site to perform their work. Blue-collar workers are more conservative than white-collar workers and more likely to buy American branded cars like Fords.²

Initially I thought this difference would be driven purely by the prevalence of Ford trucks, but as we shall see it is not just trucks versus cars.

Trucks

Is it just that there are more Ford trucks? No.

The same pattern holds, although both makes recover faster, with Fords returning to pre-pandemic levels and Toyota getting to 80%, which is much higher than the 50% Toyota reached when including non-trucks.

Location

Perhaps Ford owners just live in areas with looser restrictions, like the Central Valley? No. Here is data from Contra Costa County, part of the Bay Area:

It is the same pattern, but with a lot more noise due to the smaller population.

Age

Young drivers get in more accidents. Perhaps there is a strong age difference driving the trend? There is an age difference, see:

But that alone doesn’t account for the pattern:

Putting It All Together

A person’s identity is made up of many traits: their age, their politics, where they live, what job they do, and yes, what car they drive. I looked at three different traits—vehicle type, location, and age—and none of them explain the entirety of the collision rate difference between Toyotas and Fords after the COVID-19 stay-at-home order. My conclusion is that Ford drivers are just different from Toyota drivers, in multiple ways, each of which contributes to the trend.

The code that generated the plots can be found here (rendered on Github). The data is here.

The Race for Yellow

The top award in the Tour is the yellow jersey, which is awarded to the rider with the lowest combined time across the 21 stages of the race. Tadej Pogačar, the incredibly young¹ and surprisingly dominant winner of last year’s race, was the clear favorite this year.

Primož Roglič was a favorite again after taking second place in last year’s Tour. Since then he had defended his title in one of the two other grand tours, the Vuelta a España, winning for a second year in a row.

Ineos Grenadiers teammates Richie Porte, Geraint Thomas, and Richard Carapaz were also in the running. Porte had won the Critérium du Dauphiné—considered a warm-up race for the Tour used to test a rider’s form—and Thomas had placed third in the Critérium and was the 2018 Tour winner. Carapaz had just won the Tour de Suisse, the other Tour warm-up race, and had won the Giro d’Italia in 2019.

Unfortunately, the race for yellow turned out to be far less exciting than last year, indicated by the large time gap between Pogačar and the rest of the field that formed early in the race:

Pogačar only faltered on stage 7 as the race entered the Alps. He dropped to four minutes behind current race leader Mathieu van der Poel and three and half minutes behind second place Wout van Aert.² But Pogačar stormed back on stage 8 to take the lead, which he maintained for the rest of the race.

Ben O’Connor attempted to contest Pogačar’s lead on stage 9 with a solo break away win but fell two minutes short. O’Connor’s effort put him in second place, but he wasn’t able to hold his form and eventually placed fourth after losing time in the Pyrenees. By the end of stage 10, Pogačar had an unassailable lead of almost 6 minutes.

The Green Jersey

Although the race for yellow was uneventful, the race for the green jersey was incredibly exciting. The green jersey is awarded to the rider with the most points, which are earned by winning intermediate sprints and stages.

Mark Cavendish—considered by some to be the greatest sprinter ever³—entered the race having won 30 stages of the Tour, just four behind all-time great Eddy Merckx. But Cavendish had not won a Tour sprint since 2016 or any stage of any race since 2018. His performance had fallen so far that he had considered retiring before the 2021 season. But in early 2021, he showed a return to his winning form with four dominate sprint wins in the Tour of Turkey, which raised the possibility of him beating Merckx’s record.

Here is how the sprint race turned out, with sprint stages shaded in grey:

Cavendish took the lead in the points competition with a win on stage 4. He extended his lead on stage 6 with another win which brought him up to 32 all-time, just 2 behind Merckx. Cavendish had to survive the Alps in stages 7 and 8 if he wanted another shot at sprint wins. He managed to avoid the time cuts with the help of his team and went on to win twice more on stages 10 and 13, where second place Michael Matthews falls to his lowest point before clawing his way back over the next few stages.

Cavendish’s win on stage 13 tied Merckx’s record of 34 tour wins and set him up to beat the record during the final sprint of the tour on the Champs-Élysées. Unfortunately it was not to be, Cavendish came in third on the final sprint behind Wout van Aert and Jasper Philipsen. Cavendish may have another chance to beat the record in 2022, but as he is at the tail-end of his career it is not certain he will make the race.

The Rest of the Race

184 riders started the race and 141 finished. Here is how each rider fared:

Cavendish paced himself in the mountains, finishing with the slowest rider to save his energy. Cavendish’s teammate, Tim Declercq’s time is almost identical to Cavendish’s for the first 12 stages, as he stayed with the sprinter to ensure that Cavendish made it in under the time cut. Declercq was involved in a major crash in stage 13, where he lost almost 15 minutes. But he held on as other, slower riders dropped out, allowing him take the lanterne rougue awarded to the last place rider.

Finally, how did the other riders who held the yellow jersey during the race, Julian Alaphilippe and Mathieu van der Poel, do? Van der Poel dropped out when they hit the mountains to prepare for the Olympics. Despite his strength, he is not the type of rider who could have won this tour, being too heavy to climb quickly. Alaphilippe, although not a climber specialist, can compete in the mountains as we saw in the 2019 Tour, but he too started to lose time in the Alps. He held on and finished in Paris, but far down the leaderboard.

Although this year’s competition for the yellow jersey lacked the excitement of last year’s Tour, Mark Cavendish’s amazing return to form provided some rare sprinting tension. Hopefully Cavendish will return next year to attempt to break Merckx’s record!

The number of traffic collisions decreased as you would expect with the decreased volume but, surprisingly, the severity of the collisions increased. The rate of fatal accidents increased across the country. The National Highway Traffic Safety Administration attributes the increase to a change in behavior by drivers who stayed on the road: they drove more recklessly and wore their seatbelt less often.¹

I can’t test that hypothesis with my SWITRS data—it does not include much information about driving behavior, only about collisions—but I can look at the fatality rate on California roads.

The code for this analysis can be found here (rendered on Github). The data is available on Kaggle or Zenodo. There is a hosted Kaggle Notebook version of this post as well to help you dive right in.

Data

I selected all collisions in the dataset between the start of 2019 and November 30th, 2020, including whether there was a fatality as a result of the collision, with this query:

SELECT collision_date
    , 1 AS crashes
    , IIF(collision_severity='fatal', 1, 0) AS fatalities
FROM collisions
WHERE collision_date IS NOT NULL
AND collision_date BETWEEN '2019-01-01' AND '2020-11-30'

I start the data in 2019 because I need a sample from before the pandemic changed behavior, but I didn’t want to go too far back because collision rates vary drastically year-to-year. I cut off the data in November because the reporting is not yet complete for December.

Fatality Rate

I calculate the weekly fatality rate. It is the number of traffic collisions that resulted in a fatality divided by the total number of collisions during the week. Here is what that rate looks like before and after the stay-at-home order:

You can see the fatality rate immediately jumps up to over 1% for the first time in our dataset, and then goes even higher in the coming weeks. It stays elevated for the entirety of our data range.

Another way to look at this data is to plot a histogram of the rate before and after the stay-at-home order. Here it is:

The weeks with the highest fatality rate before the pandemic are between 0.8% and 1%. These overlap with the lowest fatality rate weeks after the stay-at-home order. These are clearly different distributions, but we can quantify that difference with a Mann–Whitney U test.

The Mann–Whitney test compares the probability that a value randomly drawn from the first distribution is larger than one randomly drawn from the second distribution, with a correction for ties. If this probability is not 50% (as it would be if they were the same) then the distributions must be different. The test is nonparametric and only assumes that the observations are independent, that they are orderable, that under the null hypothesis the distributions are equal, and under the alternative hypothesis the distributions are different.

The test confirms our eye test with a p-value of 3.6e-14. These distributions are significantly different, meaning that the California stay-at-home order increased the traffic fatality rate.

It is useful to know where you prefer to be on the data science spectrum, as it will determine what roles you should apply for. “What are the responsibilities of this position and what the key skills to be successful in it?” is one of the first questions I ask when applying to a new position. The answer lets me map that specific role to its place in the ecosystem and helps me determine if I would be interested in the job.

The Spectrum

You could define the spectrum of data science along multiple axes, but I find using just one works pretty well:¹

Engineeriness: Roughly, how close the role is to a traditional software engineering role.

On the “low engineeriness” side of the spectrum you have roles that work almost entirely with the contents of the data and domain-specific languages for data access, processing, and plotting. As you move towards the other end you start working with “lower-level” languages and often less on the data itself and more on supportive tooling around it.

A particular job at a company could fall anywhere on the spectrum, but the title gives a good idea of where exactly it fits. Below are five common job titles in a rough order from least to most “engineery”. Of course, in the real world, the responsibilities of these jobs overlaps heavily with their neighbors on the spectrum.

Business Analyst

A business analyst² uses data to help the company understand what has happened, what is happening, and what will likely happen so they can make better decisions. Their primary deliverables are internal-facing reports, dashboards, and presentations. They are generally really adept at SQL and making data visualizations, but are less likely to use general-purpose languages like Python.

Data Scientist

A data scientist³ is an expert at statistics and experimental design. They don’t just plot trends, they understand what causes them, and how you can influence them. They can clean a dataset, find biases, and then use it to power decisions and products. They work with more general programming languages like R or Python.

Modeler

Machine learning modeler⁴ is a rarer title, but I include it because I feel it fills the hole between data scientist and machine learning engineer. This role focuses on building models that directly impact customers. They find customer problems, build machine learning models to solve them, and own those models all the way from first iteration through hosting it in production. They use Python, machine learning frameworks like TensorFlow, sometimes Scala and Spark, Docker, and REST APIs.

This is the role I feel most comfortable in, with its mix of software development, machine learning, and direct impact on customers.

Machine Learning Engineer

A machine learning engineer⁵ focuses on the platforms underlying machine learning modeling and hosting. They often build ML tooling, hosting, and pieces of ML specific infrastructure like feature stores. They focus on making sure the machine learning models can scale to meet the demands of running in production and return answers fast enough to be used. They generally work with lower-level languages than the modelers like Scala or Java. Many MLEs come from a software engineering background.

Data Engineer

Data engineers build the infrastructure the data flows through. All the SQL databases, NoSQL, queues, streams, etc. that power the business and allow the other data roles to make use of it. They’re experts in a cloud services (where these systems are mostly hosted) and scaling systems to meet the demands of millions or billions of users while collecting and organizing their data.

The Split

At Insight I ran into lots of different types of data scientists. Some were really, really good at statistics; others lived for experiment design and power analysis; some—like me—loved writing clean, performant code and playing around with deep learning. Yet at the end of the program we mostly interviewed for jobs that were no more specific than “data scientist”. The companies were looking for someone to set up their data teams and extract value from their data.

This was the tail end of the unicorn era. Data engineering had already become its own specialty, but everything else was still under the umbrella term data scientist. Even as a neophyte I could see further splits coming.

Unicorn Era

The Unicorn Era was the time when data science was first being established. Companies had heard that “data was the new oil” and were desperate to hire someone to refine it for them. They looked to hire a person who could span the entire spectrum of data jobs, from setting up the infrastructure, to building analyses on top of it, to shipping and owning models in production. But people who could master all of these skills were not just rare, they were mythical. Hence, unicorns.

It worked OK for awhile. There were few data science jobs so companies needed to find only a scant handful of unicorns. Still it couldn’t last. Demand for data scientists exploded and the tasks they were asked to perform became more and more demanding and more and more specialized. It no longer made sense to try to find a single person to do it all, for many reasons.

First, finding unicorns had always been nearly impossible. Every data scientist is an expert in some part of the field, but almost no one is great at everything and interested in all of it. Splitting the role allowed people to focus on the areas that they were most passionate about.

Second, training a data scientist—waiting for them to finish a PhD in some other subject first while hoping the required data skills would rub off on them in the process—took too long. An undergraduate degree was the obvious solution and every college quickly spun up a data science undergraduate program. But how could you impart the ten years of knowledge gathered in the lead up to a PhD in just four? You can’t— unless you split it across three different people.

The Team Era

The data science role slowly split into multiple positions. Work that had before been owned by individuals with ill defined titles instead was distributed across an entire teams. Hence, the Team Era.

But it didn’t split into just one or two new roles, it split into a whole spectrum. The roles cover everything from setting up the underlying infrastructure, to improving specialized tooling, to building models, to reporting out the results. I discuss five of these roles in my next post:

• 
  The Data Science Spectrum:
  From Analyst to Machine Learning
  Data science has left the era of the Unicorn and entered the era of the team, but that means there is now a whole spectrum of data science jobs. Here is what they do.

It wasn’t enough. People still had to request the data, download it, and then run the scripts. Too much of a hurdle for most people. And worse, California no longer provided some of the oldest data. It was suddenly impossible for other people to reproduce my earlier work!

Luckily, I had saved all of the data. So I decided to host the dataset online to make it easy to start using right away.

But there was a problem: the dataset was so large that finding a site to host it was not easy. In the end I choose two places: Kaggle and Zenodo. In this post I’ll share the lessons I learned, the benefits of each site, and why I think using both is the right thing to do.

My Requirements

I had four requirements:

• Free: The service had to be free for me, and free for the end users as well. Requiring someone to pay for access to an open source dataset that I had volunteered to curate is unfair and would certainly drastically reduce the number of people making use of it.

• Easy: It had to be easy for me to set up. I wanted a service where I could get started without having to email someone for permission. I also wanted it to be easy for the end user to get the data so that the largest number of people could make use of it.

• Discoverable: It was important to me that people could easily find the dataset. There are dozens of sites that host large files, but they aren’t places where people would go look for data. To help people find it I would have to set up a web page pointing to the download, which wasn’t something I wanted to maintain.

• Permanent: Finally, there was no point in going through all this work if it was just going to disappear tomorrow. I wanted the data to be available for years and years.

AWS Open Data was one option I considered, but it looked like a lot of work to set up and it was unclear exactly how free it was. Further, getting the data wasn’t easy if you had never worked with S3 before. Ideally, I wanted a service that had a big button that said “Download this data!”

I also considered self-hosting on my Raspberry Pis, but quickly dismissed it. Availability would be terrible, download speeds would be even worse, and it would force me to perform a lot of maintenance to keep it running.

In the end I settled on two services:

Kaggle

Kaggle is a great place to host a dataset. It’s free, easy to use (with the exception that you need an account), and is a well known place to find datasets making it discoverable. The one downside is that Google is infamous for killing services, so the data might not last.

Kaggle allows users to download the file or work with the data directly in Kaggle’s hosted notebooks. The author can even set up a demo notebook to demonstrate how to work with the data. Kaggle will even help you set up a DOI for your data. Mine is: 10.34740/kaggle/dsv/1671261

Kaggle supports really deep data documentation. You can write an introduction for each table and each column. Additionally, Kaggle will automatically generate histograms of each column and some summary statistics.

Kaggle is more than just hosting; it is a community. Other people can share their work, set up challenges using the data, and ask questions in the forum. This community makes it easy for people to find the data and get started working with it.

Zenodo

Zenodo—hosted by CERN—is a much simpler and smaller service than Kaggle. It does not have a community built up around it. It does not have attached cloud compute. They have almost no users.¹

So why use Zenodo? Simple: Google kills products left and right while CERN knows a little something about keeping websites online. I trust CERN’s stewardship of the dataset. I am far more confident that you will be able to download it from Zenodo in 10 years than from Kaggle.

Zenodo has great support for academic dataset usage. It allows you to use any valid DOI, so I was able to reuse the one from Kaggle, although it will also generate one for you if you wish. It will track citations to your dataset and provides links to the papers. It will even let you export the citation to BibTex or generate a text citation on the website. Zenodo lets you link your identity to your Open Researcher and Contributor ID.

Unlike Kaggle, Zenodo makes downloading easy. You do not need an account, you just go to the page and click the download button. It also shows a MD5 hash of the file so you can verify your download is exactly the same as the file on the server.

Zenodo has a problem in addition to its low usage though: uploading a dataset often fails. I originally tried to upload the uncompressed database but that failed multiple times. After reaching out to their support (who were very responsive and helpful), I compressed the database tried again. The smaller file succeed where the larger one had failed.

Conclusion

I think using both Kaggle and Zenodo is the perfect way to host a public dataset. Kaggle has a great community and lets people quickly discover your dataset and make use of it. The downside is the uncertain longevity and the fact that you need an account to download the dataset. Zenodo perfectly complements Kaggle’s weakness as its backed by CERN, an organization that takes data hosting seriously, and makes it very easy to download the dataset.

Plotting the data as a time series is the best way to answer this question, but I never remember how to pivot the table, aggregate the events by type, and resample to the right frequency. So I made the Time Series Plotting Notebook to remember for me.

The Time Series Plotting Notebook

Suppose we are looking at the number of automobile collisions by make using my curated SWITRS dataset. We could extract one row for each collision and the associated vehicle, which would look like this:

The time series plotting notebook has two helpful functions to visualize this data: plot_time_series() and draw_left_legend().

Plot Time Series

The first function, plot_time_series() is simple. It takes a dataframe formatted like the above data and returns a plot showing the number of events for each value in the categorical column. For example, to plot the number of accidents per week by vehicle make, we would call:

plot_time_series(
  df,
  ax,
  date_col="datetime",
  category_col="vehicle_make",
  resample_frequency="W",  # Resample to 'W'eeks
)

Which would produce this plot:

The function accepts a few optional parameters:

• resample_frequency: controls the timescale over which the data is aggregated.

• aggfunc which controls how the data is aggregated.

• linewidth which can be used to make the lines larger if there are only a few of them, or thinner if there is lots of data.

Simple Legend

Simple legends are great. They convey their information effectively because the superfluous noise has been removed. My basic plotting notebook has a function to remove all the extra information from the legend box leaving only the color and the label. This time I have taken it a step further: I wrote a function to get rid of the box and label each line.

The function draw_left_legend() will draw labels on the end of each line, like so:

I’ve used this legend when Plotting the winners of the 2019 Tour de France as well as the 2020 Tour de France.

Putting It Together

The time series plotting notebook enables you to quickly plot your data in time with only a few lines of code. Here is the final version of the plot:

Which was produced by this short code snippet:

import seaborn as sns

fig, ax = setup_plot(title="Collisions by Make")

pivot = plot_time_series(df, ax, date_col=DATE_COL, category_col="vehicle_make", resample_frequency="W")

# Move labels slightly to avoid overlap
nudges = {"Toyota": 15, "Honda": -8}
draw_left_legend(ax, nudges=nudges, fontsize=25)

sns.despine(trim=True)

save_plot(fig, "/tmp/make_collision_in_time.svg")

I hope the notebook template library is useful to you! Let me know on Twitter or Github if it is. Your feedback helps make the project better for everyone!

Markdown provides a lot of syntax to simplify HTML, like **BOLD** to create <strong>BOLD</strong> text, or > Quote to create <blockquote>Quote</blockquote>. Recently, for my MiniFate project, I wanted to add a few custom <span> elements to highlight specific pieces of the text. I could have fallen back to writing it out in HTML each time, but doing so felt clunky when compared to how smooth writing Markdown normally is.

I created a way to define my own syntax based on Anatol Broder’s Compress and Sylvain Durand’s post on Improving typography on Jekyll. It uses Liquid to rewrite the web page after it has been compiled, giving you complete control of formatting and allowing you to define custom Markdown syntax. Since it uses only the default tools built in to Jekyll, it works natively on Github Pages! Here is how it works:

Layouts

The order of interpretation to build a page in Jekyll is:

1. Substitute site and page variables.

2. Run Liquid functions.

3. Compile Markdown to HTML.

4. Push the compiled HTML into its layout template, if there is one.

5. Write to file.

The problem is that Liquid runs before compiling, so we can’t use Liquid code embedded on a page to modify the final HTML. But there is a workaround: the fully compiled HTML is pushed to a layout (if one is specified) and that layout restarts the page build order! This means we can modify a page’s HTML using Liquid written in the page’s layout template!

To define our custom syntax then, we just need to write a simple layout and place it in _layouts/substitute.html:

{% comment %}
<!-- This is the code block to define custom syntax -->
{% endcomment %}
{% assign output = content
    | replace: '-!', '<u>'
    | replace: '!-', '</u>'
%}

{{output}}

Here content is the special variable that contains the compiled HTML from the page that is using the template.

We then change our primary layout (probably _layouts/default.html) to inherit from substitute:

---
layout: substitute
---

<!DOCTYPE html>
<html lang="en">
  <body>
    <main>{{ content }}</main>
  </body>
</html>

And that’s it! All the customization is controlled by changing the Liquid code in substitute.html. Below are some examples.

Defining Custom Markup

Markdown has no syntax for Underline, but we can define some like this:

{% assign output = content
    | replace: '-!', '<u>'
    | replace: '!-', '</u>'
%}

Now -!Underline!- compiles to <u>Underline</u>.

But we can go further: we can define anything we’d like in the substitution, for example a <span>:

{% assign output = content
    | replace: '-!', '<span class="book-title">'
    | replace: '!-', '</span>'
%}

Which can be fully customized with CSS.

This method has two limitations:

• We have to use characters that the Markdown parser won’t interpret.¹

• We must define unique opening and closing syntax to match the opening and closing HTML elements.

We can avoid these constraints by overriding standard Markdown syntax.

Overriding Markdown Syntax

I never use ~~Strike~~ in my writing, which inserts <del>Strike</del> to denote text that has been removed. We can override it to insert <u>Underline</u> instead as follows:

{% assign output = content
  | replace: '<del>', '<u>'
  | replace: '</del>', '</u>'
%}

Notice that I didn’t replace ~~, I replaced <del>. This is because the template Liquid substitutes after the Markdown is compiled to HTML, and so ~~ has already been removed from the page.

Replacement

Of course, we can replace anything using this method, not just custom Markdown syntax or HTML elements. We can define a macro that is replaced by an image or table. We can even reshape the page, for example, adding an <hr> above the footnotes automatically like this:

{% assign output = content
  | replace: '<div class="footnotes" role="doc-endnotes">', '<hr><div class="footnotes" role="doc-endnotes">'
%}

Conclusion

Using layouts gives us the full power of Liquid to update our web pages after the HTML is compiled, and it works natively on Github pages! I hope you use this to build awesome web pages and if you do let me know on BlueSky: @alexgude.com

I have a lot of practice with these types of interviews as a result of my time at Insight, my many experiences interviewing for jobs, and my role in designing and implementing Intuit’s data science interview. Similar to my last article where I put together an example data manipulation interview practice problem, this time I will walk through a practice case study and how I would work through it.

My Approach

Case study interviews are just conversations. This can make them tougher than they need to be for junior data scientists because they lack the obvious structure of a coding interview or data manipulation interview. I find it’s helpful to impose my own structure on the conversation by approaching it in this order:

1. Problem: Dive in with the interviewer and explore what the problem is. Look for edge cases or simple and high-impact parts of the problem that you might be able to close out quickly.

2. Metrics: Once you have determined the scope and parameters of the problem you’re trying to solve, figure out how you will measure success. Focus on what is important to the business and not just what is easy to measure.

3. Data: Figure out what data is available to solve the problem. The interviewer might give you a couple of examples, but ask about additional information sources. If you know of some public data that might be useful, bring it up here too.

4. Labels and Features: Using the data sources you discussed, what features would you build? If you are attacking a supervised classification problem, how would you generate labels? How would you see if they were useful?

5. Model: Now that you have a metric, data, features, and labels, what model is a good fit? Why? How would you train it? What do you need to watch out for?

6. Validation: How would you make sure your model works offline? What data would you hold out to test your model works as expected? What metrics would you measure?

7. Deployment and Monitoring: Having developed a model you are comfortable with, how would you deploy it? Does it need to be real-time or is it sufficient to batch inputs and periodically run the model? How would you check performance in production? How would you monitor for model drift where its performance changes over time?

Case Study

Here is the prompt:

At Twitter, bad actors occasionally use automated accounts, known as “bots”, to abuse our platform. How would you build a system to help detect bot accounts?

Problem

At the start of the interview I try to fully explore the bounds of the problem, which is often open ended. My goal with this part of the interview is to:

• Understand the problem and all the edge cases.

• Come to an agreement with the interviewer on the scope—narrower is better!—of the problem to solve.

• Demonstrate any knowledge I have on the subject, especially from researching the company previously.

Our Twitter bot prompt has a lot of angles from which we could attack. I know Twitter has dozens of types of bots, ranging from my harmless Raspberry Pi bots, to “Russian Bots” trying to influence elections, to bots spreading spam. I would pick one problem to focus on using my best guess as to business impact. In this case spam bots are likely a problem that causes measurable harm (drives users away, drives advertisers away). Russian bots are probably a bigger issue in terms of public perception, but that’s much harder to measure.

After deciding on the scope, I would ask more about the systems they currently have to deal with it. Likely Twitter has an ops team to help identify spam and block accounts and they may even have a rules based system. Those systems will be a good source of data about the bad actors and they likely also have metrics they track for this problem.

Metric

Having agreed on what part of the problem to focus on, we now turn to how we are going to measure our impact. There is no point shipping a model if you can’t measure how it’s affecting the business.

Metrics and model use go hand-in-hand, so first we have to agree on what the model will be used for. For spam we could use the model to just mark suspected accounts for human review and tracking, or we could outright block accounts based on the model result. If we pick the human review option, it’s probably more important to get all the bots even if some good customers are affected. If we go with immediate action, it is likely more important to only ban truly bad accounts. I covered thinking about metrics like this in detail in another post, What Machine Learning Metric to Use. Take a look!

I would argue the automatic blocking model will have higher impact because it frees our ops people to focus on other bad behavior. We want two sets of metrics: offline for when we are training and online for when the model is deployed.

Our offline metric will be precision because, based on the argument above, we want to be really sure we’re only banning bad accounts.

Our online metrics are more business focused:

• Ops time saved: Ops is currently spending some amount of time reviewing spam; how much can we cut that down?

• Spam fraction: What percent of Tweets are spam? Can we reduce this?

It is often useful to normalize metrics, like the spam fraction metric, so they don’t go up or down just because we have more customers!

Data

Now that we know what we’re doing and how to measure its success, it’s time to figure out what data we can use. Just based on how a company operates, you can make a really good guess as to the data they have. For Twitter we know they have to track Tweets, accounts, and logins, so they must have databases with that information. Here are what I think they contain:

• Tweets database: Sending account, mentioned accounts, parent Tweet, Tweet text.

• Interactions database: Account, Tweet, action (retweet, favorite, etc.).

• Accounts database: Account name, handle, creation date, creation device, creation IP address.

• Following database: Account, followed account.

• Login database: Account, date, login device, login IP address, success or fail reason.

• Ops database: Account, restriction, human reasoning.

And a lot more. From these we can find out a lot about an account and the Tweets they send, who they send to, who those people react to, and possibly how login events tie different accounts together.

Labels and Features

Having figured out what data is available, it’s time to process it. Because I’m treating this as a classification problem, I’ll need labels to tell me the ground truth for accounts, and I’ll need features which describe the behavior of the accounts.

Labels

Since there is an ops team handling spam, I have historical examples of bad behavior which I can use as positive labels.¹ If there aren’t enough I can use tricks to try to expand my labels, for example looking at IP address or devices that are associated with spammers and labeling other accounts with the same login characteristics.

Negative labels are harder to come by. I know Twitter has verified users who are unlikely to be spam bots, so I can use them. But verified users are certainly very different from “normal” good users because they have far more followers.

It is a safe bet that there are far more good users than spam bots, so randomly selecting accounts can be used to build a negative label set.

Features

To build features, it helps to think about what sort of behavior a spam bot might exhibit, and then try to codify that behavior into features. For example:

• Bots can’t write truly unique messages; they must use a template or language generator. This should lead to similar messages, so looking at how repetitive an account’s Tweets are is a good feature.

• Bots are used because they scale. They can run all the time and send messages to hundreds or thousands (or millions) or users. Number of unique Tweet recipients and number of minutes per day with a Tweet sent are likely good features.

• Bots have a controller. Someone is benefiting from the spam, and they have to control their bots. Features around logins might help here like number of accounts seen from this IP address or device, similarity of login time, etc.

Model Selection

I try to start with the simplest model that will work when starting a new project. Since this is a supervised classification problem and I have written some simple features, logistic regression or a forest are good candidates. I would likely go with a forest because they tend to “just work” and are a little less sensitive to feature processing.²

Deep learning is not something I would use here. It’s great for image, video, audio, or NLP, but for a problem where you have a set of labels and a set of features that you believe to be predictive it is generally overkill.

One thing to consider when training is that the dataset is probably going to be wildly imbalanced. I would start by down-sampling (since we likely have millions of events), but would be ready to discuss other methods and trade offs.

Validation

Validation is not too difficult at this point. We focus on the offline metric we decided on above: precision. We don’t have to worry much about leaking data between our holdout sets if we split at the account level, although if we include bots from the same botnet into our different sets there will be a little data leakage. I would start with a simple validation/training/test split with fixed fractions of the dataset.

Deployment

Since we want to classify an entire account and not a specific tweet, we don’t need to run the model in real-time when Tweets are posted. Instead we can run batches and can decide on the time between runs by looking at something like the characteristic time a spam bot takes to send out Tweets. We can add rate limiting to Tweet sending as well to slow the spam bots and give us more time to decide without impacting normal users.

For deployment, I would start in shadow mode, which I discussed in detail in another post. This would allow us to see how the model performs on real data without the risk of blocking good accounts. I would track its performance using our online metrics: spam fraction and ops time saved. I would compute these metrics twice, once using the assumption that the model blocks flagged accounts, and once assuming that it does not block flagged accounts, and then compare the two outcomes. If the comparison is favorable, the model should be promoted to action mode.

Let Me Know!

I hope this exercise has been helpful! Please reach out and let me know on BlueSky at @alexgude.com if you have any comments or improvements!

How can I prepare for dataset-based interviews? Do you have any examples of datasets to practice with?

I never had a good answer. I would tell them about how the interviews worked, but I wished I had something to share that they could get their hands on.

As of today, that’s changing. In this post I put together a series of practice questions like the kind you might see (or be expected to come up with) in a hands-on data interview using the curated and hosted dataset of California Traffic accidents. The dataset is available for download from both Kaggle and Zenodo, and I even have an example notebook demonstrating how to work with the data entirely online within Kaggle.

Interview Format

As I mentioned in my post about my most recent interview experience, data science and machine learning interviews have become more practical, covering tasks that show up in the day-to-day work of a data scientist instead of hard but irrelevant problems. One common interview type involves working with a dataset, answering some simple questions about it, and then building some simple features.

Generally these interviews use Python and Pandas or pure SQL. Sometimes the interviewer has a set of questions for you to answer and sometimes they want you to come up with your own.

To help people prepare, I have created a set of questions similar to what you would get in a real interview. For the exercise you will be using the SWITRS dataset. I have included a notebook to get you started in SQL (rendered on Github) or Pandas (rendered on Github). The solution notebooks can be found at the very end and I have included answers in this post, click the arrow to show them.

Good luck, and if you have any questions or suggestions please reach out to me on BlueSky: @alexgude.com

Questions

How many collisions are there in the dataset?

SELECT COUNT(1) AS collision_count
FROM collisions

What percent of collisions involve males aged 16–25?

SELECT
    COUNT(DISTINCT case_id)
    / (
        SELECT CAST(COUNT(DISTINCT case_id) AS FLOAT)
        FROM parties
        WHERE party_age IS NOT NULL
        AND party_sex IS NOT NULL
        )
    AS percentage
FROM parties
WHERE party_sex = 'male'
AND party_age BETWEEN 16 AND 25

How many solo motorcycle crashes are there per year?

SELECT
  STRFTIME('%Y', collision_date) AS collision_year,
  COUNT(1) AS collision_count
FROM collisions
WHERE motorcycle_collision = True
  AND party_count = 1
GROUP BY collision_year
ORDER BY collision_year

What make of vehicle has the largest fraction of accidents on the weekend? During the work week?

SELECT
  p.vehicle_make AS make,
  AVG(
    CASE WHEN STRFTIME('%w', c.collision_date) IN ('0', '6') THEN 1 ELSE 0 END
  ) AS weekend_ratio,
  AVG(
    CASE WHEN STRFTIME('%w', c.collision_date) IN ('0', '6') THEN 0 ELSE 1 END
  ) AS weekday_ratio,
  count(1) AS total
FROM collisions AS c
LEFT JOIN parties AS p
  ON c.case_id = p.case_id
GROUP BY make
HAVING total >= 10000
ORDER BY weekday_ratio DESC
LIMIT 1

WITH counter AS (
  SELECT
    p.vehicle_make AS make,
    AVG(
      CASE WHEN STRFTIME('%w', c.collision_date) IN ('0', '6') THEN 1 ELSE 0 END
    ) AS weekend_fraction,
    AVG(
      CASE WHEN STRFTIME('%w', c.collision_date) IN ('0', '6') THEN 0 ELSE 1 END
    ) AS weekday_fraction,
    count(1) AS total
  FROM collisions AS c
  LEFT JOIN parties AS p
    ON c.case_id = p.case_id
  GROUP BY make
  HAVING total >= 10000
)

SELECT * FROM (
  SELECT
    *
  FROM counter
  ORDER BY weekend_fraction DESC
  LIMIT 1
)

UNION

SELECT * FROM (
  SELECT
    *
  FROM counter
  ORDER BY weekday_fraction DESC
  LIMIT 1
)

How many different values represent “Toyota” in the Parties database? How would you go about correcting for this?

SELECT
  vehicle_make,
  COUNT(1) AS number_seen
FROM parties
WHERE LOWER(vehicle_make) = 'toyota'
  OR LOWER(vehicle_make) LIKE 'toy%'
  OR LOWER(vehicle_make) LIKE 'ty%'
GROUP BY vehicle_make
ORDER BY number_seen DESC

Solutions

So that’s it! I hope it was useful and you learned something!

Here are my notebooks with the solutions:

• The SQL solution notebook (Rendered on Github)

• The Python/Pandas solution notebook (Rendered on Github)

A special thanks to Quynh M. Nguyen who came up with some simplifications for my queries!

Let me know if you find any more elegant solutions!

• Finding out when automobiles, motorcycles, and bicycles crash.

• Quantifying the dangers of daylight saving time and the end of daylight saving time.

I even maintain a helpful script to convert the messy CSV files California will let you download into a clean SQLite database.

But requesting the data is painful; you have to set up an account with the state, submit your request via a rough web-form, and wait for them to compile it. Worse, in the last few years California has limited the data to 2008 and later!

Luckily I have saved all the data I’ve requested which goes back to January 1st, 2001. So I resolved to make the data easily available to everyone.

The SWITRS Hosted Dataset

I have combined all of my data requests into one SQLite database. You no longer have to worry about requesting the data or using my script to clean it up since I have done all that work for you.

You can download the database from either Kaggle (requires account) or Zenodo (no account required!) and get right to work! I have even included a demo notebook on Kaggle so you can jump right in!

The dataset also has its own DOI: 10.34740/kaggle/dsv/1671261

Read on for an example of how to use the dataset and an explanation of how I created it.

Data Merging

I have saved four copies of the data, requested in 2016, 2017, 2018, and 2020. The first three copies have data from 2001 until their request date, while the 2020 dataset only covers 2008–2020 due to the new limit California instituted. To create the hosted dataset I had to merge these four datasets. There were two main challenges:

1. Each dataset contains three tables: collision records, party records, and victim records; but only the collision records table contains a primary key. That key is the case_id.

2. The records are occasionally updated after the fact, but again only the collision records table has a column (process_date) indicating when the record was last modified.

I made the following assumptions when merging the datasets:

• The collision records table from the more recent dataset was correct when there was a conflict.

• The party records and victim records corresponding to that collision record were also the most correct.

These assumptions allowed me to write out the following join logic to create the hosted set. First I selected case_id from each copy of the data, preferring the newer ones:

-- Select all from 2020
CREATE TABLE outputdb.case_ids AS
SELECT case_id, '2020' AS db_year
FROM db20.collision;

-- Now add the rows that don't match from earlier databases, in
-- reverse chronological order so that the newer rows are not
-- overwritten.
INSERT INTO outputdb.case_ids
SELECT * FROM (
    SELECT older.case_id, '2018'
    FROM db18.collision AS older
    LEFT JOIN outputdb.case_ids AS prime
    ON prime.case_id = older.case_id
    WHERE prime.case_id IS NULL
);

-- and the same for 2017 and 2016

Then I selected the rows from the collision records, part records, and victim records that matched for each year:

CREATE TABLE outputdb.collision AS
SELECT *
FROM db20.collision;

INSERT INTO outputdb.collision
SELECT * FROM (
    SELECT col.*
    FROM db18.collision AS col
    INNER JOIN outputdb.case_ids AS ids
    ON ids.case_id = col.case_id
    WHERE ids.db_year = '2018'
);

-- and similarly for 2017 and 2016, and
-- for party records and victim records

The script to do this is here.

Using the dataset

Using the hosted dataset, it is simple to reproduce the work I did when I announced the data converter script: plotting the location of all crashes in California.

Just download the data, unzip it, and run the notebook (rendered on Github). This will produce the following plot:

I hope this hosted dataset makes working with the data fast and easy. If you make something, I’d love to see it! Send it to me on BlueSky: @alexgude.com

The code that generated the plots can be found here (rendered on Github). The data is here.

The Race for Yellow

The most prestigious award at the Tour is the yellow jersey, which is awarded to the rider with the lowest combined time across all 21 stages of the race. Egan Bernal was the favorite going into this year as he had won last year’s race. His team, Ineos, had also decided to dedicate all of their resources to him and left former winner and previous co-leads Chris Froome and Geraint Thomas off the roster.

Primož Roglič was another favorite. He had won last year’s Vuelta a España, taken 4th in a previous Tour, and his team, Jumbo–Visma, included a star-studded support roster.

After three weeks of racing, here is how the top five riders fared through the stages:

Stage 7 stands out in this plot. Although large time gaps normally occur during mountain finishes, this completely flat stage shook up the race for yellow. Instead of a steep climb, strong winds split the peloton in two and several top riders were stuck in the chasing group where they lost 1′21″.

After leading for most of the race, Roglič lost nearly a minute to Tadej Pogačar, a young Slovenian riding his first Tour ever, on the penultimate stage. Roglič had defended the jersey since stage 9, possibly as part of a strategy to take the jersey early in case the race had to be canceled midway through.

But the long defense left Roglič vulnerable. In a ride that caused 17-time Tour rider George Hincapie to declare it “the greatest Tour I have ever seen”, Pogačar stormed back on La Planche des Belles Filles, taking first on the stage by 1′21″. Every other top rider lost time on the stage as well, even Richie Porte who came in third for the stage and knocked Miguel Ángel López out of the top three overall.

Disappointing Results

Many riders set their sights on the yellow jersey but ultimately fall short. Crashes, illness, and simply not being in form can drag down even top riders. Here are the riders who went for the glory but could not keep it up for the full three weeks:

Notice that the y-axis now extends to over two hours behind the leader, not the minutes behind in the first chart.

It might be hard to call a top 6 finish a disappointment, but for Lopez is was. He was on the podium in 3rd place when he started stage 20, but he lost over 6 minutes in a disastrous time trial.

Guillaume Martin finished 11th, his highest ever place, but he had been in the top three for much of the early race, keeping up with favorites Bernal and Roglič. He lost time during stage 13 after holding strong during the first real test in the Pyrenees.

Both Bernal—last year’s winner—and Nario Quintana—two time runner up to Chris Froome—defended well in the early mountains but lost time in the high Massif Central. They were suffering from injuries incurred during crashes earlier in the race. In a controversial move, Bernal withdrew from the race after he lost time.¹ Quintana fought on and finished in Paris, but lost lots of time in the Alps.

Thibaut Pinot crashed on stage 1. Emanuel Buchmann crashed in a previous race and his ability to start was in question. Both lost time in the first mountains and never recovered, but nevertheless stayed in the race through the end.

The Rest of the Race

More than 100 riders finished the Tour, but most of them were not competing for the yellow jersey. Here are the paths taken by all 146 riders who finished in Paris:

Buchmann, the lowest placed rider in our previous plot, is actually ahead of most of the riders! We can also see the race started out tough, probably due to the chance that it might be canceled after the first rest day, with large time gaps opening up even before the first mountains.

The latter half of the second week was also tough with the hilly Massif Central and mountainous Alps. By the last few stages, the time gaps were pretty much set and most riders maintained their relative positions.

Two riders of note are Peter Sagan and Sam Bennett, who were competing for the green jersey. Both of them saved their energy, and hence lost time, on mountainous stages so they could give it their all in the sprints. Even though Sagan finished about an hour ahead of Bennett, he lost the Jersey. Bennett had done a better job of managing his energy and using it where it counted.

Finally, Roger Kluge won the lanterne rouge, finishing six hours behind Pogačar. His job in the race had been to escort his team’s sprinter, Caleb Ewan, through the race. This often meant falling back on climbs and waiting for Ewan so they could tackle the mountains together and avoid being cut for being too slow.

This year’s Tour was unique due to needing to adjust to the COVID pandemic, but it turned out to be one of the most exciting races in the history of the sport! And what’s more, it gave us some much-needed entertainment during these dark times.

The interview process was different from when I first interviewed as a young data scientist right out of Insight, and even different from my more recent interview experiences in 2017.

Observations

All of the interviews this year followed the structure I outlined in my post on not wasting a candidate’s time, which I appreciated. To summarize, that structure is:

• Prescreen: A resume or recruiter screen of the candidate, often offline.

• Technical Screen: Either a call with a team member or a take-home assignment to assess the candidate’s technical skills.

• On-site Interview Loop: An all-day interview with multiple team members and the hiring manager.

Although all of the companies structured the interviews well, several failed at the second area I emphasize in my post: Communication. One company took an entire month to convey feedback from an interview step while another did not get back to me about my on-site until I had emailed them several times.

Salary Negotiation

The first conversation with every recruiter in 2017 involved talking about present salary and future expectations. In 2020, not a single recruiter asked about compensation until after they had made a verbal offer. This surprised me because waiting to open negotiations until that late in the process offers the candidate an advantage as they now know that the company has strong interest in them. Previously I have tried to delay this conversation as long as possible as part of my negotiation strategy, but this time the recruiters did it for me!

This behavior might be explained by California’s new law which bans using salary history to determine an offer, but it specifically does not ban asking about salary expectations. It might be that the more senior roles I’m interviewing for now are hard enough to find candidates for that the companies don’t want to reject anyone without a chance to get the candidate committed to the role.

Technical Screens

In 2015 and 2017 I had real trouble with the technical screens. Often they were stereotypical engineering interviews where I was asked to do something difficult but irrelevant, like invert a binary search tree. I was able to solve these problems when I had seen them before in my studies or could come up with the “trick” on the fly. I passed about half of my technical screens.

In 2020, my experience was vastly different. Only one screen was even close to “invert this BST”, and that was a mistake where they later admitted that I was given the software engineering technical screen by mistake.

All of the other screens involved reasonable questions that would come up in a data scientist’s daily work, like manipulating a dataset, calculating some features, or implementing really simple algorithms or metrics. With these more applied questions, I passed all of my technical screens!

Virtual On-Sites

In 2017 on-sites were on site! In 2020 they are done via video conferencing. I thought virtual on-sites would be less draining, but I actually felt even more exhausted after them. However, I still felt I was able to connect on a personal level with the interviewers, despite not being in the same place.

Whiteboard Coding

There were fewer coding problems during the on-sites than previously and all of them were done in an online editor instead of on a whiteboard. This worked great!

I found myself looking forward to the coding challenges because, with the improvement of coding on an actual computer, they were a nice break from the other interviews. Just like the technical screens, these questions were all directly applicable to the work I would be doing.

Open-ended Problems and Behavioral Questions

This time there were more open-ended interviews that dug into some problem the business had (for example “How would you help us filter spam?”—similar to the machine learning case studies I’ve written about for interview prep) or went really deep exploring a project I had worked on previously. Although I was asked some of these questions during my previous years of interviewing they felt much more effective in the virtual format, perhaps because the lack of a whiteboard made it so the interviewer and I had to have a conversation instead of me giving a lecture.

This round of interviews was the first time I was asked behavioral questions. They were present in three of the five on-sites, including one company that had 90 minutes of them!

Results

I applied to seven companies using internal referrals from my network. Here is how I did during each round:

I am very happy with how well the technical screens went this time around, as mentioned above. I also feel good about the on-site to offer rate, although more offers would always be better of course.

I felt three interviews went really well: Square, Twitch, and Twitter. I thought I connected well with the teams and demonstrated that I had the skills they were looking for. So I was disappointed to not get an offer from Twitter, but excited for the offers from Square and Twitch.

The Stripe interview revealed the position to be less well aligned with my skill set² and career aspirations than I’d hoped, so I think their decision to not extend an offer was fair.

I generally do not apply to work for startups, for a variety of reasons both personal and financial.³ However, I agreed to interview at Grand Rounds after a friend reached out because I did not want to turn down any opportunities. In the end we all knew it was a bad fit.

So, after all that, I’m excited to get to work at Square and even more excited to be done interviewing during a pandemic!

But Python has some great functions to replace for loops, which I will cover below after a quick example.

Playing Cards

Given a list of playing cards as tuples, like so:

cards = [
  ("Spades", 14),
  ("Diamonds", 13),
  ("Hearts", 2),
  ("Spades", 8),
  ("Clubs", 11),
  ...  # etc.
]

We want to convert them to PlayingCard objects as defined in my previous post on enums. To do this, we need a function to convert a tuple into the class:

def tuple_to_card(card_tuple):
  suit, rank = card_tuple

  card = PlayingCard(
    CardSuit(suit),
    CardRank(rank),
  )

  return card

This makes it easy to parse the list with a quick loop:

new_cards = []
for card_tuple in cards:
  new_card = tuple_to_card(card_tuple)
  new_cards.append(new_card)

And we can even filter the cards so that we only keep hearts:

just_hearts = []
for card_tuple in cards:
  new_card = tuple_to_card(card_tuple)
  if new_card.suit is CardSuit.HEARTS:
    just_hearts.append(new_card)

These code snippets are fine: short and clean, with not a lot that can go wrong. But I love to replace custom code with Python built-ins whenever possible, because they are fast, well tested, and concise. Python provides two functions that can simplify even these already simple code fragments: map() and filter().

Map

The map() function replaces a for loop that calls a function on each item of a list, just as we did in the above when making PlayingCard objects. Here is how we could rewrite the above code using map:

new_cards = map(tuple_to_card, cards)

Simple!

If the function is not too complicated, it can be useful to define it inline with a lambda function:

get_card_rank = lambda card_tuple: card_tuple[1]
ranks = map(get_card_rank, cards)

But what if instead of tuples we had two lists: one of suits and one of ranks? We could use the zip function to combine the two lists like a zipper:

new_cards = []
for card_tuple in zip(card_suits, card_ranks):
  new_card = tuple_to_card(card_tuple)
  new_cards.append(new_card)

But map already allows us to do pairwise operations:

new_cards = map(tuple_to_card, card_suits, card_ranks)

Of course, we could write our own map using list comprehension:

new_cards = [tuple_to_card(card_tuple) for card_tuple in cards]

Which, perhaps, is a little more Pythonic.

Filter

But how would we filter the list so that we only keep hearts, as in our second example? We could wrap the map call in a for loop:

just_hearts = []
for card in map(tuple_to_card, cards):
  if card.suit is CardSuit.HEARTS:
    just_hearts.append(card)

But the filter() function does that for us! It takes a function and an iterable and returns only the elements of the iterable that evaluate to True when the function is called on them. This allows us to rewrite the above as:

is_a_heart = lambda card: card.suit is CardSuit.HEARTS

just_hearts = filter(
  is_a_heart,
  map(tuple_to_card, cards)
)

Of course, again, we could write this as a comprehension:

just_hearts = [
  tuple_to_card(card_tuple) for card_tuple in cards
  if tuple_to_card(card_tuple).suit is CardSuit.HEARTS
]

However, this is not as readable as the map and filter example, which is very short, very readable, and even a bit… functional.¹

The Plotting Notebook

Visualizing your data is a critical step in understanding it, and so it is appropriate that the first notebook in the library helps with making beautiful plots.

The notebook begins with boilerplate code that defines metadata for the resulting files and also changes some defaults, such as the figure size and resolution, font size, and legend frame. After that there are a few helpful functions which I will discuss below.

Draw Bands

One of my favorite functions is draw_bands(). It draws a set of alternating colored bands on the background of the plot based on the axis tick locations.

When called with just the axis, like draw_bands(ax), it produces this:

But you can also customize the color using draw_bands(ax, color="orange", alpha=0.05), which produces:

These bands are a subtle way of indicating where on the X-axis a point lies, which is especially useful when plotting a time series. I use them often. Here are some examples:

• Discussing my sons’ language development to highlight each month.

• Plotting the progression of the cycling hour record to show each decade.

• Exploring when cyclists are involved in traffic accidents to highlight the seasonality.

Draw Legends

I like minimal, but informative, legends. Color alone is often enough to differentiate lines or points, so I wrote a function to change the color of the legend text to match the line, called draw_colored_legend(). It produces a legend like on this plot:

This legend style can be seen in these posts:

• Plotting my son’s language development to label each language.

• Plotting Tour de France Prize Money to label the winner’s prize compared to the total.

• Comparing Data Science Salaries by Gender to differentiate the points for men and women.

Putting It Together

The plotting notebook enables you to make beautiful plots quickly and easily. For example, this plot:

Was produced by this short code snippet:

fig, ax = setup_plot(
    title="Title",
    xlabel="X-axis",
    ylabel="Y-axis",
)

ax.scatter(np.random.rand(500)-0.65, np.random.rand(500), label="First dataset")
ax.scatter(np.random.rand(500)-0.35, np.random.rand(500), label="Second dataset")

draw_colored_legend(ax)

draw_bands(ax)

save_plot(fig, "/tmp/output.svg")

If the notebook template library is useful to you, be sure to let me know on Twitter or Github. Your feedback helps make the project better for everyone!

In this post I will focus on a small piece of deployment: “How do I test my new model in production?” One answer, and a method I often employ when initially deploying models, is shadow mode.

If you’re interested in a broader overview of building and deploying machine learning products, I highly recommend Emmanuel Ameisen’s book: Building Machine Learning Powered Applications!¹

What Is Shadow Mode?

To launch a model in shadow mode, you deploy the new, shadow model alongside the old, live model.² The live model continues to handle all requests, but the shadow model also runs on some (or all) of the requests. This allows you to safely test the new model against real data while avoiding the risk of service disruptions.

When Would I Use Shadow Mode?

Shadow mode is a great way to test a few things:

• Engineering: With a shadow model you can test that the “pipeline” is working: the model is getting the inputs it expects, and it is returning results in the correct format. You can also verify that the latency is not too high.

• Outputs: You can verify that the distribution of results looks the way you expect (for example, your model is not reporting just a single value for all input).

• Performance: You can verify that the shadow model is producing results that are comparable to or better than those of the live model. This involves choosing the right machine learning metric based on the business use case to compare the model’s outputs against ground truth or against each other.

Shadow mode works well when the result of the model does not need a user action to validate it. Models where you try to influence the user—for example a recommendation model where success means more sales converted—are better tested using an A/B test. The big difference between an A/B test and shadow mode is that in an A/B test traffic is split between the two models whereas in shadow mode the two models operate on the same events.

How Do I Deploy In Shadow Mode?

There are two general methods that I use for deploying in shadow mode. Both are relative to the API for the live model: either in front of the live API or behind the live API.

In Front of the API

To put a model in shadow mode in front of the API, you host two API endpoints: one for the live model and one for the shadow model. The caller makes a call to both of them whenever they would normally call the live model. The caller can disregard the response, but they should log it so that the results can be compared. I have drawn this structure below:

This way of deploying is well-suited to situations where the calling team is change-adverse or has very strict requirements for how the shadow model must perform because it gives them control. I have found it useful for deploying models that have a large effect on some conversion funnel, like a model that runs at new user creation and blocks suspected bad actors.

The advantages of this method are:

• The caller has control. They decide when to switch the shadow model to live. They can roll back instantly if there are problems. They can even stop the experiment if it is hurting their system.

• The call can be different. If the shadow model requires different inputs (perhaps a new ID associated with the user), its API can be different than that of the live model.

The main disadvantages are:

• The change is closer to the customer. The calling code is generally closer to the core business, so any bug introduced during integration of the shadow model is likely to be more impactful.

• Tighter coordination is required. The team that owns the model and the team that calls it will both have to make changes to their code: the model team to spin up an endpoint, and the calling team to add the call to the second model as well as a logging action.

Behind the API

To put a model in shadow mode behind the API, you change the code that responds to API requests to call the live and shadow model. You log the results of both models³ but only return the result from the live model. I have drawn a schematic of this below:

This method is great when you want to move quickly (and break things), because you can change the shadow model without having to coordinate with the calling team. To the outside world the API looks unchanged and so hides the testing going on behind it.

The advantages of this method are:

• The model host has control. You can change the shadow model, turn it on, turn it off, and swap in a new one at a whim. You can log exactly what you are interested in recording.

• Little coordination with other teams is required. To the outside world the API looks the same as before; no one else has to change their code.

The main disadvantages are:

• The shadow model must be input compatible with the live model. Since the outside world is not changing what it passes to the API, your shadow model is restricted to the same inputs as the live model (although it can choose to use only a subset, or get additional inputs via some other method).

• You still have to change the calling code. Eventually, when the model is ready to replace the live model, you will need to change the API version and change the calling code to use this new version. This means there is a little extra work to be done once you are satisfied with your test results.

Conclusion

Deploying a model in shadow mode is an easy way to test your model on live data. It is flexible and allows you to empower the right team to control the experiment.

Luckily I have a good friend who had just bought one. She was happy with her Nookdesk and so I bought mine on the strength of her recommendation. As it turns out, that was a good decision. For a complete review, read on.

Desktop and Frame

The Nookdesk has a lot of customization options. The desktops are offered in multiple materials from various laminates to solid bamboo. The desktop is always 30″ deep with widths from 48″ to 72″. It is 3/4″ thick. The desk frame comes in three colors: white, black, and grey.

I chose a 30″ by 72″ (+$200) blackwood laminate (+$100) desktop with white frame. The wood laminate feels great: smooth with just a bit of depth from the grain. The white frame contrasts nicely with the black desktop.

The desk height ranges from 23.5″ to 49″ as measured from the ground to the top of the desktop. This is just enough range for me as use it at 26.5″ when seated and 47.5″ when standing.

Storage

The Nookdesk has some unique storage options. One of them is the “upper storage” add-on which I would describe more as a shelf. It is mounted on the top of the desk at the back edge. It is 12″ deep and rises 4″ above the desktop. I store my game controller and headphones under the shelf, and I put my monitors on top to bring them up to eye-level. The upper storage add-on cost $100, but was well worth the expense.

There is a similar “lower storage” shelf that mounts under the desk at the front edge. I did not order one because I like to have my desk surface very close to my knees in the sitting position, and the lower storage would add 4″ to the bottom of the desk. Like the “upper storage”, it also costs $100.

The ability to add some storage to my desk and raise my monitors was the feature that originally drew me to the Nookdesk and was what finally sold me on it.

Accessories

Nookdesk offers a wide variety of accessories, from keyboard trays to speakers. Below I review only the ones I purchased.

Controller

The Nookdesk comes with one of two controllers: a simple one which has only up and down buttons, and a programmable one with three memory settings for an additional $34. I got the programmable controller because getting my desk to exactly the right height each time was important to me. I only really needed two memory positions: one for standing and one for sitting. I still have not assigned the third position.

Powerstrip

Nookdesk sells a Freeport power strip that attaches to the underside of the desk for $80. It has eight outlets (including two with space around them for those extra-large plugs) and four USB-A outlets, three of which provide up 2.4A and one which supports Quick Charge 3.0. It has a 10 foot cord.

You can plug the desk’s motor into one of the eight outlets so that you only have to plug in the Freeport to the wall. I am satisfied with the power strip, but I suspect I could have found a cheaper one from a third party.

CPU Mount

Nookdesk also sells an under-desk CPU mount. It fits towers up to 9.25″ wide and 30″ tall, although if the desk is in its lowest position there is only 20″ of clearance for the tower.

I love the CPU mount; it is one of my favorite features of the desk. It was well worth the $100 it cost as it keeps my computer off the ground and frees up space on my desktop. I recommend it highly!

Cable Tray

The cable management tray is a metal tray that runs along the bottom of the desk at the rear edge and allows you to tuck your cables into it to keep them tidy. It costs $100. It does a good job of keeping cables from hanging down and getting in your way when you’re sitting, but seems expensive. However, I don’t know how I would organize my cables without it.

Ordering and Shipping

You can order the desk without creating an account on Nookdesk’s website, but if you do not create an account then retrieving your order status is difficult. You are supposed to use a special page on the site where you enter your email address, but it did not work for me. Instead I had to get customer support to send me the tracking number.

Delivery was free (due to a coupon code, NOOK150) but seems to run about $120 normally. The desk was sent by FedEx in two shipments totaling six boxes. The two shipments arrived on the same day a week after I ordered.

The box containing the desktop was very large and relatively heavy. I was able to lug it up my stairs by myself but it would have been much easier to have a second person. The box containing the desk frame was also really heavy (perhaps more so) because of the two motors that move the desk up and down, but it was thankfully pretty compact and so easy to maneuver.

Assembly

Assembling the desk was actually harder than I expected because there were a few problems:

• The hardware—screws, bolts, guides, and whatnot—was packed in unlabeled bags and the instructions reused the same icons to indicate different screws.

• The bill of materials listed both items that came in the bags and those that came preinstalled in the same count. When it said “10 screws of type X” it might mean there were 4 in the bag and 6 already attached to the frame.

• Hardware was included that is only used for optional accessories. I had screws leftover that I had not used at all. This made me worry that I had completely forgotten some critical step.

• Two hex keys are included, but one of them had such poor tolerance that it would not drive the bolts. I ended up using a set of hex keys that I already owned.

• The shelf is connected to the frame with four screws, but the contact area is so small it kept coming apart. I needed to add washers that I had lying around to make a solid connection. See my photo below:

Assembly took between 3 to 4 hours, but was not particularly difficult after I worked through the above issues. I made a minor mistake where I did not install the frame rails at the right time and had to backtrack and remove some pieces to get them in. Jon Walters says he made the same mistake in his review.

You need enough space to flip the desk over during assembly because the build is started with the desk upside down. The desk is unwieldy, so it helps to have a second person for this step, but I was able to do the entire build solo.

Final Thoughts

With taxes and shipping my desk was $1057, after a coupon code gave me $150 off and free shipping. Expensive, but I spend 10 hours a day at it so the price seems fair.

Although the assembly and shipping experience could be improved, the desk is solid and includes great accessories. I have really appreciated being able to switch positions instead of sitting for 8 hours, and so I am very happy with my Nookdesk.

But my workflow with notebooks is not very efficient; it is made up of the following steps:

1. Start a brand new, completely empty notebook.

2. Load the data and start cleaning it.

3. Begin making plots.

4. Realize I already have some code from a different project to make nice plots.

5. Dig through my repositories looking for the code.

6. Copy and paste the first code I find that sort of does what I need (and which probably is not the most recent or nicest version).

7. Hack the code up and make it even uglier.

After five years, I am ready for something better. And so I present the Jupyter Notebook Template Library, which has revolutionized my workflow and which I gladly share with you as well, gentle reader.

Jupyter Notebook Template Library

The Jupyter Notebook Template Library is a repository of notebook templates, each targeted at a different use case. The templates let me get right to working with the data as quickly as possible. And the library guarantees that my notebook will always have the latest and greatest helper functions without having to dig through my old work.

The Plotting Template

The first notebook in the library is the Plotting Template. Its goal is to change the above workflow to this:

1. Download the right template.

2. Load data and start cleaning it.

3. Make beautiful plots.

It lets me write this:

fig, ax = setup_plot(
    title="Title",
    xlabel="X-axis",
    ylabel="Y-axis",
)

ax.scatter(np.random.rand(500)-0.65, np.random.rand(500), label="First dataset")
ax.scatter(np.random.rand(500)-0.35, np.random.rand(500), label="Second dataset")

draw_colored_legend(ax)

draw_bands(ax)

save_plot(fig, "/tmp/output.svg")

And get this, complete with curated font sizes, a patented striped background, and a focused legend:

You can read more about the plotting notebook in detail here:

• 
  Jupyter Notebook Templates for Data Science: Plotting
  Jumpstart your visualizations with this Jupyter plotting notebook!

The Time Series Plotting Template

The second notebook in the library helps you take a dataframe of events and turn it into a time series plot with each item broken out into its own line.

You just write:

import seaborn as sns

fig, ax = setup_plot(title="Collisions by Make")

pivot = plot_time_series(df, ax, date_col=DATE_COL, category_col="vehicle_make", resample_frequency="W")

# Move labels slightly to avoid overlap
nudges = {"Toyota": 15, "Honda": -8}
draw_left_legend(ax, nudges=nudges, fontsize=25)

sns.despine(trim=True)

save_plot(fig, "/tmp/output.svg")

To make this plot:

You can read more about it here:

• 
  Jupyter Notebook Templates for Data Science: Plotting Time Series
  Jumpstart your time series visualizations with this Jupyter plotting notebook!

Conclusion

Enjoy the templates, I hope they make you more productive! And if you are feeling generous, I would love contributions!

The Problem

I had to handle setting some state based on two variables. Each variable could take one of a few discrete values. For simplicity, think of the variables and possible values:

All twelve combinations required doing something slightly different, so the first code I wrote looked something like this:

if (direction == "north" && travel_mode == "bike") {
  do_north_bike_stuff();
}
else if (direction == "north" && travel_mode == "car") {
  do_north_car_stuff();
}
else if ( ... ) {
  ... // etc.
}

This code wasn’t clever; it was boring and repetitive so I looked for a way to rewrite it! I had recently learned about a cool way to replace if/else in C++: the switch statement. I had to use it!

Making It Worse

But a switch statement needs integral values, so I had to map each state to numbers. Easy enough, but I quickly ran into a problem: I had to switch based on both values, so I had to combine the integers in some manner. Then I remembered a “useful” math fact: the product of unique primes is itself unique.² A horrible plan came together, it looked like this:

int NORTH = 2;
int EAST = 3;
int SOUTH = 5;
int WEST = 7:

int BIKE = 11;
int CAR = 13;
int PLANE = 17;

switch(direction * travel_mode) {
  case NORTH * BIKE:  do_north_bike_stuff(); break;
  case EAST  * BIKE:  do_east_bike_stuff(); break;
  ... // etc.
  case WEST  * PLANE: do_west_plane_stuff(); break;
}

My code was actually even worse; if you are morbidly curious I archived it here. I did not assign nice readable variables like NORTH but just used the numbers, so it looked like case 2 * 11: do_north_bike_stuff().

This code is way too clever; needing number theory to understand control flow is a huge warning sign. With ten years more experience, I actually prefer the verbose but understandable if/else method.

A Better Way

So how would I write it now? I think a hybrid method is actually the way to go, using enum³ and a few helper functions:

enum TravelMode { BIKE, CAR, PLANE };
enum Direction { NORTH, EAST, SOUTH, WEST };

switch(travel_mode) {
  case BIKE:  do_bike_stuff(direction); break;
  case CAR:   do_car_stuff(direction); break;
  case PLANE: do_plane_stuff(direction); break;
}

void do_bike_stuff(Direction dir) {
  switch(dir) {
    case NORTH: ...; break;
    case EAST: ...; break;
    ... // etc.
  }
}

This has a few nice advantages:

• It delegates the complexity of handling the direction to each travel mode. This is logical because it’s likely the way a car handles East and West are very similar, and very different from how a plane would.

• By using enum the compiler can check that we handle every case. If we forget case BIKE or case EAST, the compiler can warn us.

In the end, readable is better than clever, even if you have a bunch more lines to read!

The Data

The data was collected by my wife and I attempting to identify when the boys had learned a new word and writing it down. The most common error is writing down words one of the boys does not yet really know. This would lead to an increase in the number of words known at any time in the data; still the difference between the boys should be unaffected as the error pushes the data in the same direction for both of them.

I discuss data collection in more depth in Theo’s and Cory’s data sections. You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here and here.

Development

Below I have plotted the number of words each of my sons knew as a function of their age. Theo, our first son, is represented by dashed lines; Cory, our second son, is represented by solid lines.

Second children are known to have slower onset of language development.¹² They learn their first 50 known words more slowly, but catch up to their older siblings quickly, learning their first 100 words at about the same age. The advantage is small though; the average difference in time to first 50 words between the first and second child is only 1 month.

Theo and Cory do not follow this trend. Cory was 3 to 4 months faster than Theo to hit language development milestones in Cantonese, English, and Spanish; he also knew many more animal sounds. Theo was artificially limited in his Spanish acquisition though, as mentioned in his post, because my mother had injured herself and so Theo could not visit my parents for a few months.

Sign is the only area where Theo eventually learned faster than Cory. I suspect this is because he needed sign to communicate as he did not know as many words as quickly, whereas Cory gave up on sign once he could talk.

Finally, I owe Theo’s doctor an apology: she was always a little worried about Theo’s language development, but we did not take it seriously because we knew that bilingual children developed language more slowly. Looking at the data and comparing to his brother, I think our pediatrician was right to be worried. Thankfully, Theo has had no problems since and now talks incessantly.

Other Writings on Language Development

If you enjoyed this article, here are all the other articles I wrote about language development!

• 
  Comparison of My Three Sons’ Language Development
  I recorded the words my sons spoke as they learned our various languages and now I compare how each developed! Read on to find out how each son learned.
• 
  My Third Son’s Language Development
  We tracked my third son's language development word by word. Here, in plots, is how he learned to speak. Take a look!
• 
  My Second Son’s Language Development
  My second son is a little over two years old. We tracked every word he's spoken to watch his language development, and now you can observe it too!
• 
  My Son’s Language Development
  My son is a little over two and unfortunately he has two huge nerds for parents. We tracked every word he's spoken to watch his language development, and now you can join us!

The Data

The data was collected in the same manner as last time. The only difference was I added a new language category for “animal sounds” because when recording that data for Theo I realized it was hard to tell “moo” from “mu”. A catch-all category for animals made data logging much easier.

I had the same data collection difficulties as last time: when Cory was young, it was hard to decide if he was associating a sound with a concept. When Cory was older, he was so good at imitating sounds that it was hard to know if he knew the word or was just repeating what you had said. There was a new difficulty this time as well: we had our third child just as Cory’s language development exploded. Cory was with his grandparents for a few weeks as we adjusted to the new baby and so I was not able to record his new words during that time period.

You can find the Jupyter notebook used to perform this analysis here (rendered on Github). The data can be found here.

Development

Cory’s first baby sign was waving goodbye at 11 months. He learned four more—pointing at things he wanted, blowing kisses, shaking his head for “no”, and shaking his hands for “all done”—before speaking his first word. His first word, like his older brother, was in Cantonese, although it was “dad” and not “dog”. It was two more months before he said “mom” in Cantonese, at which point he already knew “older brother”, “ball”, “there”, and “pick me up”. My wife was disappointed it took so long, but I suspect it was because he spent so much time with her he did not need a word to get her attention, he always had it.

Cory’s sign language development outpaced his other languages until he was 20 months old. My suspicion is the same as it was with his brother: sign was his universal language. English could only be used to talk to dad, Cantonese could only be used to talk to mom, but sign could communicate with both of us and even his grandparents.

Cory’s language development is plotted below, showing the number of words he could speak in each “language” as a function of how old he was.

Cantonese and English exploded at about 19 months; Cory doubled his vocabulary in those two languages in just four weeks. He almost doubled it again in his 20th month. In his 22nd month, he picked up about 50 Cantonese words and almost 70 English ones!

Part of that growth in the 22nd month is a data entry artifact: that is when he went to stay with his grandparents. He had a week to learn words and I only recorded them when he came back to us and spoke them. You can see two flat areas in his English and Cantonese language development—one right before 22 months and one right after—that are due to this effect.

Spanish started pretty slowly because only my father speaks it to him. At 22 months we moved closer to my parents so Cory spent more time with them. You can see a bit of an increase in the amount of Spanish learned then; I’m sure we would see a much larger increase if we kept recording.

The Words

I plotted a selection of some of Cory’s first words in each language below. Notice that I have switched to a log plot for the y-axis to better show the beginnings of each language.

A few fun words:

• Grandpa (Spanish): As the only person who speaks Spanish full-time to him, it makes sense that Grandpa would be Cory’s first Spanish word. Cory has always had a special affinity for my father, so it is appropriate as well.

• Cow (English and Spanish) and Lola (Spanish): Cory started hearing La Vaca Lola (Lola the cow) and quickly learned most of the words. He still loves cows, and it is still one of his favorite songs.

• Older Brother (Cantonese): Theo is one of the defining features of Cory’s life as both Cory’s best friend and primary antagonizer. Cory learned to identify him quickly (and often would just point and say “big brother” when Theo had pushed him over).

• Older Sister (Cantonese): Cory doesn’t have a sister, but there were two girls who lived next door to our apartment with whom he would play and call “older sister”.

• Lion Dancer (Cantonese): Theo became obsessed with lion dancers during Lunar New Year last year. Cory has picked up on this obsession as well and often does a solo lion dance.

• Google and Deebot (English): We have a lot of technology in our house and Cory has learned all about it. We talk to our Google Home devices several times a day trying to get them to play music or animal sounds, and Deebot vacuums the kitchen every night as Cory watches in awe.

Other Writings on Language Development

If you enjoyed this article, here are all the other articles I wrote about language development!

• 
  Comparison of My Three Sons’ Language Development
  I recorded the words my sons spoke as they learned our various languages and now I compare how each developed! Read on to find out how each son learned.
• 
  My Third Son’s Language Development
  We tracked my third son's language development word by word. Here, in plots, is how he learned to speak. Take a look!
• 
  Comparison of My Two Sons’ Language Development
  Being a nerd dad, I recorded all the words my first two sons spoke as they learned them. Now, I compare their language development rate!
• 
  My Son’s Language Development
  My son is a little over two and unfortunately he has two huge nerds for parents. We tracked every word he's spoken to watch his language development, and now you can join us!

By the time I was running a team, I no longer had one thing I was focusing on a day; instead, I had dozens. Often these things would only take 15–30 minutes to do, but I would have to fit them in with all my meetings. My strategy of “remember what I need to do today” stopped working because I just had too many things to do, and items were constantly added and removed.

I needed a strategy; what I came up with was a version of David Allen’s Getting Things Done. Here is how I do it.

Trello

The key insight of the Getting Things Done framework is that there should be one place where tasks are recorded. I was storing tasks in my notes—both the electronic and paper versions—in my email using the “important” flag, and in my brain which meant I had to check multiple places when I had an opening in my schedule to complete a task. I decided the one place to track tasks would be Trello because adding and arranging tasks is easy, it supports my computer and phone, and has enough power-ups that I can use to customize my workflow.

I set up six columns named as follows (yes, with the emoji):

• 🗃️ Backlog: Tasks which are not urgent or not ready to start; a lot of “someday” tasks end up here.

• 📥 Inbox: The landing place for new tasks where I focus on quick entry, not on correctness.

• ☀️ Do This Week: Tasks to do this week.

• 📅 Do Today: Tasks to do today.

• ✅ Done: Finished tasks.

• 🎓 Learning: Books, articles, and papers that I want to read, classes I want to take, etc.

Triage

Every morning I triage my tasks by taking one of two actions:

1. If the task can be done in 5 minutes, I do it immediately.

2. If the task can’t be completed quickly, I move it to the correct column and make sure all the meta data like title are descriptive.

Triaging mostly means moving tasks from the Inbox and the Backlog to one of the Do columns. On Monday I also make sure to take a moment to think about my week and fill up the Do This Week column. Everyday I think about what I’m going to need to get done that day and move those tasks to the Do Today column.

When I have time to complete some tasks, I look at Do Today and pick something I can fit in, and do it. Then I move the task to Done.

The Learning column is special, it is where I store links to things I want to read, but did not have time to when I encountered them. If I’m honest, it’s mostly a place where I store links I find on Twitter when I’m on my phone.

Capturing Tasks

A good friend told me that the key to this system was that “you can’t block on IO”, meaning it must be easy to record a task and that you should not worry about getting it recorded perfectly. Triage is when you can clean up any typos and add context.

There are three places where I capture tasks:

• My Computer: Here I often write out a long task and put it directly into the correct column because the computer makes data entry easy.

• In My Email: I use an Outlook plugin that takes an email and turns it directly into a task in Trello. These tasks end up in the Inbox and I triage them later. This lets me go through the dozens (or hundreds) of emails and quickly get the ones that require action turned into a task.

• On The Go: I use the Trello app on my phone, and I have a widget that lets me add a card with the press of a button. Just like email, this puts it into the Inbox column for later triage. I find using the dictation feature of my phone makes adding tasks really fast.

Useful Power-Ups

Trello has some useful addons called “power-ups”. I found two power-ups to be useful (although the free plan only allows you to use one power-up at a time):

• Card Repeater: This power-up lets you schedule a card to reappear. I used it for things that have to happen on a fixed schedule like “plan for the weekly team meeting” and “fill out the project status update”. I also used it for things I wanted to get in the habit of doing, like “ask for feedback”. To this end I set three “ask for feedback” tasks appear on my board every Monday so I could get in the habit of asking.

• Card Aging: This power-up faded cards that hadn’t been touched for a while. This let me see what tasks were being ignored and let me re-evaluate if they were still worth doing.

This is how I get things done as a data scientist. It is pretty simple, but effective, and easy to put your own spin on it. Give it a try!

The plot is pretty good, at least at first glance! It is (appropriately) a log plot.¹ It labels all of its pieces. It even has gaps for when the race was not held. But the plot also has a few problems:

• The X-axis is wrong; the race did not start before 1900 and the two gaps are from the World Wars which did not happen in 1898 and 1921.

• The text is too small to read easily at Wikipedia’s default 200px image size.

• The axis labels are redundant and tick labels have a lot of zeroes.

I decided to fix it up using data from Bike Race Info, like I did last time when I fixed the Hour Record Plot.

Improvements

Here is my version:

The code that generated the improved plots can be found here (rendered on Github). The data is here, and the code that cleaned the data is here (rendered on Github).

I fixed the X-axis so that the dates are now correct! The first race is in 1903 as expected. I have also removed the axis label because I think the tick labels make it clear what is plotted. I have added my (patent pending 😛) grey stripes to the background to indicate each decade.

I changed the Y-axis to be more readable by abbreviating the numbers using K and M. I also removed the label and replaced it with the euro symbol (€) on each tick.

I made all the text larger and the lines thicker to improve legibility when the plot is downscaled. I have also changed from a line plot to a step plot because the amount of prize money changes at specific moments in time, not continually.

Finally, I have cleaned up the data a bit. The original plot used uncorrected Euro even though the original prizes were in old Franc, new Franc, and Euro depending on the year. I have normalized all values to 2013 Euro. I have included this information in the subtitle so that it survives even if the plot is separated from its caption on Wikipedia.

Overall I think it is an improvement, so I have contributed it back to the community here.

How would you measure the success of this model in production?

I have heard a lot of answers. They generally fall into two categories: machine learning theory focused, and business focused. The first is good, the second is better. I will go through each below.

To have something concrete to discuss, we will consider the following problem: “Train a model to classify customers as ‘high value customer’, and use it to decide if we are going to show them an up-sell page.”

Machine Learning Theory Focused

A common answer, especially from more junior interviewees, is “Accuracy”, which is the number of things your model classifies correctly divided by the total number of things.

Accuracy is rarely a good answer for real world problems, because the classes are often imbalanced. If only 1 in 1000 users is a ‘high value customer’, then a model that only returns ‘false’, despite having an accuracy of 99.9%, would be clearly worthless.

When pushed about accuracy’s obvious shortcomings, the candidate may fall back to something like F1 score, which is the harmonic mean of precision and recall. F1 is a lazy answer. It is better than accuracy because it is relatively less sensitive to imbalanced data, but rarely are precision and recall equally important (how the customer interacts with the model determines their relative weighting) and so the harmonic mean is not often applicable.

Customer Focused

The best candidates consider the problem more deeply. Instead of jumping to find a metric, they start by thinking about the experience of using a model from a business or user perspective. A good way to frame this is “What does a false positive cost my user?” and “How does that compare to the cost of a false negative?”

Sometimes a false positive is costly, as might be the case if the resulting action is drastic, like shutting down a user’s account. To avoid that, we need to be confident when we take action that we are only targeting the right users. In such a scenario, precision is more important since we want to make sure that most of the events the model flags are true positives. This is not the situation for our ‘high value customer’ model, because showing a user an up-sell page is unlikely to hurt them or cause them to churn.

Instead, in our case a false negative is costly, because we lose the chance at a large revenue increase from the up-sell. Recall is more important, as we would rather show a few extra users our up-sell page than miss the chance to convert a sale.

There are many other metrics that might be useful. The important part is not so much the metric itself, but what is motivating it, which should be the business use case and customer experience.

A Great Metric: Dollars

A great metric is the formalized version of our customer focused one: dollars. Assigning a dollar value to each model result (true positive, false positive, etc.) would allow us to optimize for revenue. This is often doable in simple models (like our ‘high value customer’ model), but in more complicated ones (like the fraud models I work on) it can be difficult.¹ For an example of using dollars in model optimization, see Airbnb’s post on Fighting Financial Fraud with Targeted Friction.

In short, a good metric is one which ties closely to the business or customer use. Consider their point of view when answering a metrics question.

• Install the software for each supported Python version.

• Run tests against each Python version.

• Publish exactly once to Pypi when all the tests in all the Python versions had passed.

Running a bunch of tests, and then deploying your software is exactly what Travis Stages were designed for. There was just one problem: I couldn’t find a good example of how to do it with multiple Python versions. This post will explain how to do it.

The Travis Configuration

Here is the full Travis Configuration file. I’ll go through a simplified version below that covers just the essentials:

Run Tests in Parallel

We start by setting up the Python versions to use for testing:

language: python
dist: xenial # Required for Python >= 3.7
python:
  - "2.7"
  - "3.7"
  # Also test pypy
  - "pypy3"

This will run tests in parallel on Python 2.7, 3.7, and Pypy3. Python 3.7 is only supported on Ubuntu Xenial, so we set that as the dist. I removed a bunch of versions for clarity; to add more, just write them in the list.

Next, we tell Travis how to set up the environment and test the code:

install:
  - pip install -r requirements.txt
script:
  # Unit tests
  - python -m pytest -v
  # Install and smoke test
  - pip install .
  - archiver --help

This installs the dependencies, runs the unit tests, makes sure we can pip install the package, and finally runs a quick “smoke test” on the installed package.

Build and Deploy

After the tests succeed (and only after) we build the Pypi package:

jobs:
  include:
    - stage: build
      python: "3.7"
      script: echo "Starting Pypi build"
      deploy:
        provider: pypi
        user: alexgude
        password:
          secure: Bq6I8x...sqslR # Hashed password
        distributions: "sdist bdist_wheel"
        on:
          tags: true
          branch: master
          repo: agude/wayback-machine-archiver
        skip_existing: true

This defines a new stage to build the package in 3.7 and then deploys it to Pypi, but only if it is the master branch on a tagged (from git tag) release.

Which gives us this:²

I hope that helps you set up your own Python packages for testing and deployment! In the future, I hope to migrate to Github Actions, but that is for another time.

The code that generated the plots can be found here (rendered on Github). The data is here.

The Race for Yellow

The yellow jersey is awarded to the rider with the lowest combined time across all 21 stages of the tour. Only a few riders are really in contention for yellow; the vast majority of the others are brought along to support their team leaders. Going into the 2019 Tour, defending champion Geraint Thomas was the favorite, but there were several strong challengers.

Below I show how the top riders did throughout the race by plotting how far behind the leader they were after each stage. Where a rider’s line is near the top they are close to taking the lead; when they drop down they are losing time.

Julian Alaphilippe held the jersey for the most days, even defending it on the individual time trial against expert time trialist Thomas. But Alaphilippe is not a climber, and after a brave defense of the jersey in the Pyrenees, he lost time in the Alps to Egan Bernal, Steven Kruijswijk, Emanuel Buchmann, and Thibaut Pinot. Pinot had looked out of contention earlier, but stormed back with a massive attack on stage 15. Unfortunately, he was forced to withdraw on stage 19 due to an injury.

Alaphilippe finally fell behind during that stage as well, losing the yellow jersey to Bernal. He lost his podium spot on stage 20 when he cracked during the final part of the climb. Alaphilippe finished 5th when the peloton rolled through the Champs-Élysées.

The Rest of the Race

From the above plot you might think that all the riders in the Tour finish within a few minutes of each other. But they do not. The last place rider, the lanterne rouge, was four and a half hours behind Egan Bernal.

Bernal and Alaphilippe, who looked so far apart in the first plot, are now seen to be neck-and-neck. Yoann Offredo looked like a lock to win the lanterne rouge when he fell ill on stage 8, but Sebastian Langeveld took it in the penultimate stage after suffering an injury in the second week of the race.

Peter Sagan, the green jersey winner, was only interested in sprints. He took it easy on the mountain stages to conserve energy, as you can see in the steep declines on stages 14 and 15 (in the Pyrenees) and Stages 18–20 (in the Alps). Most other riders performed similarly, although some recovered time on stage 17 when the favorites let a large breakaway group escape and gain a 20 minute advantage. Sagan did not make that group, which is clear from his lack of rise on the plot.

Romain Bardet, the polka dot jersey winner, was fighting for the yellow jersey until stage 14 where he cracked and lost 20 minutes. This forced him to pivot to trying to win the climbing jersey, which meant he needed to be one of the first riders to reach the top of the remaining climbs. For the last few climbing stages he stayed with the favorites or attacked early, keeping his time behind pretty consistent.

This plot gets the message across—twice, the distance went up quickly in a short amount of time—but could be much more effective. Here are some problems:

• It is missing a legend and title, which are both necessary to understand it.²

• It has too much precision in the date labels, which are down to the day but do not align with when the records were set.

• The label text is too small to read easily.

• It has a lot of unused space.

I love cycling, and I love plots, so I tried my hand at improving the plot.

Improvements

Here is my version: