The High Bar: Learnings From Working on Amazon Go

On Friday, February 7th of 2020, I concluded the Amazon Go episode of my career! Working on such a unique problem space and with a remarkable team of talent has been encouraging of what tech-focused people are capable, and how quickly ideas can go from "that's a good idea, let's do that" to "wow, this is awesome!" I am beginning a new role in my machine learning career, working on a confidential project within Amazon, so I will continue to be very quiet about my specific work efforts. My time working on Go brought strong opportunities of education that I feel privileged to have encountered - a feeling that always strikes me as a signal to take those opportunities and present them to others. Doing that seems an optimal way to put a cap the experience before taking off for the next adventure.

This post is here for you. It is here to share learnings I've taken from the last couple years of my work, and covers work with a focus on both machine learning and software engineering - it is written for an audience of the same. If some come off as pretty obvious and you ask yourself, "Wait, who doesn't know that?", consider yourself among those who have already encountered that - not everyone has. It is my hope these will inspire or provide insight for you in some fashion with your own projects, whether these serve as reminders, or reach you as new points to consider.

As with everything I post, these are views of my own, and have no bearing on the views of my employer.

Learnings To Share

• Be honest about what does and doesn't work. Be data driven and honest when presenting results. Making your experiments defensible makes presentation of your work robust, and allows others to build off your work. If you simply "swear it worked" but can't back it up with data, another engineer working off your results can never be certain their own results are defensible.
• Machine learning projects are highly iterative. Just like any other deployed code, what you put into production will someday be supplanted. Embracing this will let you keep the bigger picture in mind for what purpose your model will serve, and guide you in finding what experiments to best put your effort toward.
• Iteration on a project, especially a project powered by machine learning, starts with you: move fast, move early. If you already know of a bug, or know that one experiment is likely to produce a significant improvement, already be starting on it. This is necessary because there will be an endless stream of good ideas to try - but only those that you dedicate your time to will see results. Shifting priorities will pull you farther and farther from those lingering tasks.
• Avoiding silo-ed knowledge early on helps in the long run. It's incredibly difficult to keep track of the debt taken on when an engineer sequesters knowledge, even unintentionally, until something breaks and everyone scrambles to figure it out. Documenting knowledge, even things that you believe are very intuitive or second-nature regarding a project, rarely hurts. It's easy to see when shared knowledge helps, hard to see when isolated knowledge hurts.
• Keep your operational bar high from the start of a project. Playing catch-up when it comes to operational excellence is painful and expensive (time-wise, and in other ways).
• Give appropriate attention to code reviews and change management reviews. If you're being rushed, be every bit as thorough as you would if you had no pressure. This is the one time that's officially dedicated to catching errors and asking questions and challenging assumptions, so make it time well-spent. It is tempting and unfortunately easy to rush these reviews, but maintain your integrity and it'll show in the results. Compromise that integrity, and it'll compromise the results.
• Out of date documentation happens, but having no documentation at all encourages the aforementioned point of holding isolated or tribal knowledge. Keeping documentation up to date is not always a specific tracked task, so covering it as part of your operational excellence helps keep your documentation relevant. This reminds me of the common adage applied to music production where if something would take you "2 minutes to fix" in the documentation now, it'll take you 20 minutes or longer to fix it later when you've forgotten it and trip over it again.
• Start with the customer, not the solution. Identify what you want the end result to be before getting started, and define the problem from there; once the problem is understood, define the solution. Starting at any of those latter two points, rather than starting with the customer, invites risk. For example, there's a risk of building a good solution to a good problem, but one that doesn't achieve the desired end result for the project, and ultimately misses the target. Start with the first step, and your actions become data driven. This makes for fewer course-corrective actions, and no surprises to pull the rug out from under you down the road.
• Change direction quickly when appropriate. If the problem space changes, the customer's needs change, the priorities of the business change, switch direction quickly to minimize lost effort.
• Keep well-defined work tasks; if the tasks are undefined or vague, clear the brush. You can't build without knowing the requirements, and you clearly don't know the requirements if you can't say what the work is to be done. This is especially important in an agile team setting where tasks are defined and reviewed by the team, but can be picked up by basically anyone. Having all the context written into the task to be done, as opposed to a vague five-word description of the problem, reduces the clarifying questions that need to be answered in an ad-hoc fashion later. The strength of this is that it assumes no knowledge from the reader, and facilitates a strong start for any engineer to pick up, even if they've never seen that problem before.
• Interview regularly. At least twice a year, sit for an interview, even if you have no desire to leave your current position. When you do no interviews for a long stretch, you find it takes more energy to take that first step to get back into the swing of being interviewed, whereas if you are interviewing multiple times per year, you have a regular expectation of what it takes to land a position elsewhere if you suddenly need to relocate. On top of this, you find areas where you are weak that you can put better time and attention into strengthening. This helps to keep you working on generalizable skills (rather than ones that are specific to your current work and not transferable anywhere else), which in turn strengthens you as a contributor in your current role since you pick up new knowledge. Even if the interviews you sit for don't bring you new technical areas to work on (for example, if you're stellar at interviews already), it can introduce perspective about what the industry values in a candidate if they ask you questions you didn't expect to hear, and can help you be a better interviewer yourself from sitting through a range of interviews, both bad and good.

Did these help you in some fashion? Feel free to reach out to me (on Twitter @jgreenemi for example) to discuss!