3.5 retrospective

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.” –Norm Kerth, Project Retrospectives: A Handbook for Team Review

Dev Process * 4 (+2 BL, +1 LG, +1 CF) @Beyang: Get rid of required review requirement as long as commit is tagged with “reviewed-by: $AUTHOR” or “i’m a clown” (search for “stamp” and “stamp please” in Slack)(a)(b) * @Issac: Our traces sometimes fail to be constructed on Lightstep when there are say 20k repositories because the app generates so many spans. I suggest we prevent this by no longer checking in code that generates spans within loops.

Release Process: * 23 (+2 BL, +3 LG, +3 ijt, +2 FA, +2 Geoffrey, +4 CF, +3 Thorsten, +4 Vanesa) @Stephen: We should not target release dates (“I will get this feature in by 3.5”) but instead target estimates (“I will get this done in N weeks”).© * We shouldn’t be afraid to miss the release train. * We regularly ship one-off high-value features to customers anyway (e.g. tiny features for Apple in a patch release), could we formalize that process? * Release testing could be feature-based instead of entire-app-based. I.e. like we did this iteration somewhat. * Release cadence, and how to handle features not making the release cut in time * What do Milestones mean? * Discussion: * BL: How often do we do regression testing? * BL: We already aren’t doing regression testing for patch releases * CF: How often can customers realistically upgrade? * IJT: Continuous deployment? Insiders build * CF: Planning vs. release - do we want them coupled? * CF: Why are we shipping features in patch releases? * High priority customers * Small features that aren’t big impact * Patch release will only contain low risk changes from when we did a full release testing pass * TB: Is this a technicality? Do we actually need to change things or is this just a mindset shift? * Two things: mindset, and to release more quickly key blocker is to make release testing faster * GG: Tied to “caveats” - if we had a defined set of requirements prior to merging to master. If there were clear criteria before merging to master, it will solve the push it when it’s done vs. land it then fix later. * 2 (+2 BL) @Geoffrey: @distribution needs to remove themselves from being testers (we don’t have the bandwidth and it slows other people down) * 10 (+2 BL, +1 ijt, +3 LG, +2 Geoffrey, +2 Thorsten) @Beyang: How did release testing go(d) * Release automation * Testing grid * Testing environments / customer proxies * Discussion * TB: Easier this time around, because we knew exactly what to test * GG: Made an effort to count number of items each person had and spread it out evenly * No automation this time around * ( ) Distribution to send out instructions to automate * LG: spreadsheet has limitations. Comments go unnoticed, not clear the correspondence between grid cells and corresponding issues * Tools: * Airtable * Notion * Monday.com * Difference between what testers want and what release captain wants * LG: tester prefer kanban view * ( ) Distribution look into that * 9 (+2 FA, +2 Geoffrey, +1 LG, +2 CF, +2 Vanesa) @Christina: Many of the items this release had caveats in them - what can we do to catch this sooner or make sure we identify this before the release testing week? * Discussion: * Testing at scale was not considered until release testing * Pre-merge testing/CI * Very specific use case that was brought up in the issue, but the feature made more sense in docs and blog post when considering general use case * Be clear when specing the feature in the beginning, what are the requirements * When you’re just getting it done for a customer you don’t always think about the broader sense * Docs earlier in the process: “This is what the docs will look like at the end of the feature” * 2 (+1 ijt, +1 Vanesa) @Christina: Related to Beyang’s above: Release testing week resourcing - was hard for distribution to do testing and support critical roll out at customer sites. Many vacations happening right now (summer!) but that was not accounted for in release testing grid

Planning: * 5 (+2 Geoffrey, +1 CF, +1 LG, +1 Vanesa) Figure out how to prioritize features vs. testing automation/stability * Discussion * Monitoring * Regression testing * Responsibilities * Better monitoring * Better admin experience * Search performance * Upgrade regression testing * N big customers took a lot of time * CUSTOMER is coming online, possibly CUSTOMER

Communication: * 4 (+2 ijt, +1 FA, +1 CF) @Stephen: For CUSTOMER-specific features(e), the search team & others didn’t understand or have a clear picture of the use case for regular users when they should have had this insight. Ryan wasn’t able to write the blog post sections for these easily, search team didn’t know how to test the features effectively, etc. * @Christina: Personal retrospective: I bit off more than I could chew with ramping up + tasks I took on. I take full responsibility for the last minute blog post rush, and incomplete retro items I was responsible for. I should have acknowledged this earlier and asked for help earlier where needed. * 5 (+ 2 Geoffrey, +2 BL, +1 ijt) @Stephen/Geoffrey: Our work / needs aren’t visible to other team members. Tools like Zulip can help with this. * How do other big companies use Slack effectively? * Small trial group? * 1 week trial to just see if it works * Slack has a lot of integrations - might be the case where things get worse before they get better. * Hackathon - good venue to try out new comms tool Proposed items

Action Items * ( ) (Beyang and Christina) Document agency, ownership, expectations of engineering owners * ( ) (ijt + @beyang) Graphql backend, @search owns search*.go files * ( ) (Beyang and Christina) Create checklist that product delivers to eng. Eng responsibility is to verify all requirements are met., * ( ) (Keegan, then Beyang and Geoffrey) Look into improving alerting, stability, “ops love” * ( ) (Christina) keep blogpost up to date over the course of the iteration, keep tracking issues up to date * ( ) (Distribution team) Ops noise and monitoring * ( ) (@distribution) Deployment workflow - shorter and easier to deploy things * ( ) (Christina) Schedule follow up discussion on release dates/milestones and planning mentality(f). Consider pre-planning and scoping of issues. When is a feature done. * () (@distribution) Make list of internal tooling needs that we can’t get to * ( ) (Dan) Set up Zulip trial for GopherCon+Hackathon weeks Prev retro action items * (x) (Beyang) Better define code owners * Identify gaps in the team structure, as well, and make sure everything is owned * (x) (Beyang) Discuss how to improve QA / release testing: Chris, Loic, Ryan, Geoffrey, Stephen, Issac, Felix, Christina * (x) (Christina) Figure out how to prioritize features vs. testing automation/stability (Stephen) * (x) (Christina) make a decision and communicate what we’re going to do about the PR pain point challenges * (x) (Beyang) Make people aware of the slack channels that exist and make it part of onboarding, bot in announce that announces new channels Discussion What feedback or thoughts would you like to share with the team? Here are some things you might want to consider: * Our previous retrospective * What went well? What did you like? * What didn’t go well? What didn’t you like? * Did you learn something? * What do you wish you had done differently? Add your feedback by editing this document directly.

(a)+1 related to action item from 3.4 and discussion on article in slack: https://sourcegraph.slack.com/archives/C07KZF47K/p1560467643125600

Suggest keeping the required PR, but drop the review requirement (to prevent direct push to master) which has been highlighted as a pain point. (b):+1: ©+1 (d)much smoother this time I found (e)Which ones? I thought multiline was tested effectively. It only took about 5 minutes chatting with Ryan for us to come up with a good example (decorated Python functions) for this. I guess you mean repoHasFile and repoHasCommitAfter. [email protected], did we get these wrong? If so, what needs to be changed? (f)( ) (Christina) Schedule follow up discussion on release dates/milestones and planning mentality [email protected] Assigned to Christina Forney