CategoryGergely Orosz [The Pragmatic Engineer]

• Khan Academy is a US-based non-profit education provider, which teaches students about math, art, computing and many other topics for free.
• In 2017, the engineering team started experimenting with GraphQL for their API and decided to deprecate the REST interface and migrate existing endpoints to GraphQL.
• Python 2’s end-of-life announcement was the final spur to start the project and the team chose to move over to Go, largely for its first-class support in the Google App Engine, and the simplicity and consistency of its language and performance.
• The project was split into two parts: Phase 1: Minimum Viable Experience and Phase 2: endgame.
• The team chose a “field by field” approach for the migration, with the first migrated service being the simplest; a service hosting a single field.
• The migration strategy applied a similar side-by-side testing approach to all services, with optional shadowing of traffic on the new Go service and side-by-side testing of GraphQL services.
• The project took 3.5 years to complete, with around 100 software engineers working on it.
• Khan Academy migrated from Python to Go, a process which took 3.5 years and involved 100 engineers.
• The team used a side-by-side migration approach, which allowed them to track the percentage of traffic served from the new system.
• Engineers liked Go for its ease of reading and writing, documentation, tooling, and compiler speed.
• Performance was excellent compared to Python, with the service hour cost of operating the same code on Python and Go being up to 10 times cheaper for certain types of requests.
• The team kept shipping incrementally and only one service was allowed to write a given piece of data.
• The project was treated as a fixed scope, fixed timeline project, which was the right choice in hindsight.
• Switching to a brand-new language for the rewrite was worth it in the end, but the team played loose with the “port things exactly as they are” approach when it came to internal tools.
• Kevin Dangoor and Brian Anderson, two engineers from Khan Academy, shared their learnings from a 3 year-long project to migrate from a Python monolith to Go services and GraphQL.
• The team defined a “minimum viable experience” to help them focus and prioritize the right things.
• The team estimated the MVE phase would take about 2 years, and completed it almost exactly as per their original estimate two years previously.
• The migration added complexity in several ways, such as changing languages from Python to Go, moving to new versions of Google Cloud APIs, and splitting control of the data.
• Hard deadlines can be motivational, and having a hard deadline forced the team to align in creative ways to ensure a ‘critical path.’
• Just because you have services, you cannot ignore the broader ecosystem.
• Long-running migrations often feel thankless, never-ending and frustrating.
• To be a great product engineer, it’s worth familiarizing yourself on how to do migrations, so you can do them more efficiently and reliably.

Published February 7, 2023
Visit The Pragmatic Engineer to read Gergely Orosz’s original post Real-world Engineering Challenges #8: Breaking up a Monolith