“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.” –Norm Kerth, Project Retrospectives: A Handbook for Team Review
Release test grid * Farhan: How did Monday.com compare to Google sheets? (+1 Christina, +3 Joe) 4 * Polly survey results
Testing logistics * Farhan: Last retrospective, we decided that feature owners should also be testers, which we implemented this time. I would like feedback on how the tester assignments were. It seemed that splitting the rows by owner could lead to imbalances. For example, distribution has 48 total rows in the grid, whereas search has 23, and both teams had 3 team members available to test this iteration. (+2 Joe, +2 Uwe) 4 * Farhan: It is unclear who decides which rows should be tested for a given release cycle, and which environments (new Docker/new Kubernetes/k8s.sgdev.org,etc) should be tested. The grid has not changed in several iterations, so should we just remove the rows that aren’t marked as requiring testing? If not, we should make sure that the party responsible for deciding which rows and columns should be tested, takes the time to set up the grid at each testing cycle. (+1 loic, +2 BL, +3 Joe, +3 Farhan, +2 Vanesa) 11 * Farhan: k8s.sgdev takes a long time to sync repos, and I got many questions about this during testing. I suspect this will take a while, but is there any way we can improve this testing experience?
Automated testing * Farhan: It seemed that a lot of the testing grid was completed later than usual. One potential reason is that people were spending time doing automated tests. How can we still encourage people to write automated tests, but not sacrifice completing testing on time for next iteration? Should we increase the time for testing, or encourage people to do complete manual tests first, and then do automated tests with their free time? (+2 Geoffrey, +3 loic, +2 BL, +2 Farhan, +1 Joe) 10
Knowledge sharing/feature ownership * Farhan: Beyang was out of office this testing week, and it became clear that few other people had knowledge on auth, repo permissions, etc. Perhaps we need to improve the tests plans, or find a way to improve knowledge sharing for big features? (+4 BL, +1 Joe) 5
Release blockers * Nick: Multiple non-trivial features were merged in the hours leading up to the creation of the release branch. The fact that our release announcement is delayed until 3.7.1 is not a good outcome. We want releases to be calm and routine. If you are rushing, then you are going to probably make mistakes and skip things that should be done before shipping to customers (e.g. unit tests, perf measurements, PR comment followup). Better to call it a miss ahead of time and target the next release (patch if necessary). (+3 loic, +10 Keegan, +4 nick, +3 Chris, +3 Geoffrey, +2 Farhan, +10 Stephen, +2 vanesa, +5 Christina, +1 Joe, +2 BL) 45 * Christina: Communication around release blockers, what is the problem, and how does it impact the customer? (+3 nick, +3 loic, +4 vanesa, +2 Farhan, +3 Christina, +3 Chris) 18 * Stephen: I would like to have all projects outline potential problems users/admins could run into, as well as general questions admins might ask as part of speccing out a project. E.g., resources change a feature may need, rollout plan on large # of repos, etc. (+3 nick, +1 Farhan, +2 Chris, +2 Geoffrey) 8
RFCs * Nick: I have seen a lot more technical/product docs (now called RFCs) created since last retrospective. It is great!
Action Items * ( ) (Loic) Release captain should aggressively revert features that may cause delays. * ( ) (All) Default to tagging regressions as release blockers. * ( ) (All) Add high-level details like customer impact on issues so Christina can help determine whether or not it’s a release blocker. * ( ) (Farhan) Update release captain documentation to describe: * @Distribution to denote which rows should be tested in which environment (constant) * @Distribution to denote some rows that should be tested each iteration (constant) * Release captain will proactively reach out to teams owning “optional” rows to find out which of the optional rows need to be tested this iteration. * ( ) (All) Include a list of features you will add automated tests for in your tracking issue.