3.6 retrospective

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.” –Norm Kerth, Project Retrospectives: A Handbook for Team Review

Automated (Release) Testing: * 15 Nick: It is a good step that we have taken to document all the important workflows, but it is time to take the next step: we need to prioritize automated testing moving forward. Not everything all at once, but incrementally (e.g. automate search testing first since core services is already creating things that look like e2e tests for perf benchmarks). +3 Thorsten +2 Felix +1 Beyang +1 Stephen +3 Joe, +1 Farhan, +3 Keegan +1 Chris +3 Vanesa * 6 Issac: Let’s set this in motion by adding do-nothing scripts that tell the manual steps to perform one at a time, with stub functions that can be filled out with automation. +2 ijt +2 Joe, +1 Farhan +1 Chris

Manual Release Testing: * 11 Nick: Assignments in release testing grid feel too random. We aren’t taking advantage of testing affinity (e.g. multiple people might need to spin up a k8s cluster to test one thing instead of having a single person spin on a k8s cluster to test a few related things). It is nice that random assignments cause people to test things they haven’t tested before, but the cost is efficiency. I think we should optimize for efficiency in manual testing (as we move toward more automated testing). Release testing isn’t the appropriate time or process to introduce folks to parts of the product that they haven’t used/tested before. If we had each team own their own manual tests, this would align incentives for a team to invest in automating their tests. +2 Felix, +2 ijt, +3 Keegan, +2 Farhan +2 Chris +2 Vanesa * 1 Christina: Testing on pre-existing setups vs. testing customer setup experience. +1 Joe * 4 Joe: For the “Eng Owner” in test grid, probably we should directly assign someone instead of asking around, and if the person appears to be too busy, simply reroute to someone else and update docs (assume they have agreed upon rerouting). +2 Keegan +2 Beyang

Planning: * 7 Nick: As our team and product grow in size, we need to be more intentional about writing down our plans: the problems we are prioritizing, why we are prioritizing them, and how we are going to solve those problems. https://basecamp.com/shapeup +1 Thorsten, +2 Keegan, +1 ijt, +1 Chris, +2 Farhan * 2 Stephen: Strongly believe we should continue working on performance / stability. +2 Thorsten

Code Review: * Felix: Love that we now have code review guidelines * Nick: Still feels like there is ambiguity around whose approvals are “required” for a given PR, especially for PRs that touch multiple parts of the codebase, and especially for newer folks who don’t know whose review they should wait for.

Action Items * ( ) (Nick and Christina) Document agency, ownership, expectations of engineering owners * ( ) (Uwe) Look into improving alerting, stability, “ops love” * ( ) (Distribution team) Ops noise and monitoring * ( ) (@distribution) Deployment workflow - shorter and easier to deploy things * ( ) (@distribution) Make list of internal tooling needs that we can’t get to * ( ) (Thorsten -> Felix) chat about e2e testing pain points * ( ) (Farhan) each person writes one automated test for 3.7. Regression.tsx unless good enough for e2e * ( ) (Farhan) Release testing grid item is tested by owner (person). * ( ) (Christina) We have a roadmap that is a list of projects the team is working on. There is nothing missing on the list. Each item has a link to a doc/issue that has a good proposal.