Desktop and Mobile illustration

Why are you handholding your app releases?

The bigger context, and why we're building Tramline

Modern smartphones and their app stores are very new in the industry. iOS and Android saw exponential growth only after 2010. For a long time, this did not impact engineering because apps were either small in scope or secondary channels to a business.

This changed after the wild success of app-first businesses like Uber, Instagram, Swiggy, Gojek, etc. Today, it is common for businesses to be built around an app experience, which means apps are critical to the success of those businesses.
Engineering teams are acutely aware that the state of developer tooling for mobile teams is a decade behind what backend and frontend developers use regularly.

As a result, app-first businesses spend a lot of effort ensuring their apps are stable and perform reliably. This adds engineering overhead because now the organization must stitch together different tools to solve their needs. They must maintain this internal tooling in parallel with delivering business requirements.
Releasing apps is key to the success of an app-first business. It is a repeating transaction that delivers value to customers, ergo releasing an incorrect build can incur a high cost.
The App Store and Play Store are walled gardens with Apple and Google setting all the rules, which can and do change many times.

Continuous delivery, a proven valuable practice, is impossible in these stores and so is the rollback of bad builds.

Even the control of over-the-air updates rests with the user and the device. In that sense, apps are much more like on-premise software that cannot be frequently updated.
For the vast majority of the app ecosystem, releasing a new version involves many steps and multiple stakeholders.

The process is simple and coordination is easy when the team and product are small and does not move fast.

However, for large-scale fast-moving products, app releases become exponentially more complicated.
“It was the most terrifying thing to take 10,000 diffs, package it into effectively a bullet, fire that bullet at the horizon and that bullet, once it leaves the barrel, it's gone. I cannot get it back, and it flies flat and true with no friction and no gravity till the heat death of the universe. It's gone. I can't fix it. That fundamental model is terrifying in effect.”

What's in a release cycle?

  • Agreeing on a date to ensure all activities are completed on time
  • Deciding features, improvements, and bug fixes that will be shipped
  • Making builds for various (staging, pre-prod, production) API environments
  • Automated and manual testing of builds by developers and QA teams
  • Product sign off on the correctness of what is being shipped
  • Updating builds with fixes/changes so they align with the release decision
  • Notifying stakeholders of the progress of the release
  • Staged rollout of the final build and monitoring metrics to ensure stability
  • Creating and deploying "hotfix" builds if something critical is amiss
Such a release process is documented in a tool (e.g. Google Docs, Confluence) as a series of steps and checklists that have to be followed.

The process is run by a person assigned as the release pilot – someone who is accountable for a correct run.

The pilot’s role is one that often rotates within the engineering team because when a release is not being run the pilot can go back to their regular development duties. Some very large scale companies even have dedicated release engineering teams.
The release process checklist, like all checklists, is a maintenance burden because the process, tools, or steps change over time. Small out-of-band tasks may be done by a team (or specific engineers) that may never get mentioned in the checklist.

Human factors also play a role here. Churn happens in all teams, and every time that happens the team must make sure that all release-related knowledge has been transferred from memory and committed to documentation.

Our mission is to equip teams with tools that help them focus on delivering business value, and not on ancillary activities like driving a release checklist.

With Tramline, we want to remove the uncertainty around releases for everyone involved, and reduce the opportunity cost incurred by teams.