Proposed Themes: Sprint 20th Sep - 4th October (1.29.0 2/2)

[marcushyett-ph]

In order to give folks time to provide feedback and iterate ahead of the sprint planning meeting on Weds I wanted to share out an initial proposal for the focus of the next sprint, today

Current Sprint Themes: #5742

Assumptions
For the proposed next sprint themes I'm making the following assumptions and would love to validate / invalidate them

• "Nail Paths User Experience" - @EDsCODE we will have shipped all of the outcomes listed in the current sprint, but we will have a small number of other use-cases and tweaks based on customer feedback to complete during the next sprint?
• "Enterprise Unblock Project Based Permissioning" - @mariusandra @paolodamico @Twixes this will be completed by the end of this sprint?
• "Performance" - @macobo @fuziontech we've made tonnes of progress but we will have new outcomes related to performance of the next sprint?
• "Duplicate persons and distinct IDs & Cohorts" @yakkomajuri we will have worked out why this issue is happening: https://github.com/PostHog/product-internal/issues/126
• "Quantitative Analysis Design"- @clarkus @paolodamico we will have designs and some early feedback to unblock work on Quantitative analysis in the next sprint

Proposed Next Sprint Themes:

• "Paths User Experience" - Close Out specific (do we have a specific list @EDsCODE?) outstanding use-cases / improvements for paths
  • Why?
    • Paths are critical to enable people to diagnose causes, we have made a bunch of progress but we need to ensure our solution meets the needs of our customers based on their feedback
• "Quantitative Analysis MVP" - Ship an MVP for Quantitative Analysis (@EDsCODE @neilkakkar) - maybe this is too ambitious with the work above too?
  • Why?
    • To make diagnosing causes fast and easy enough for anyone to diagnose the possible primary causes - we need to automate the work by enabling One-click quantitative analysis showing the relationship between successful and unsuccessful users and properties / events
• "Session Recording" - Resolve (list specific session recording issues we can resolve within the sprint) @paolodamico @mariusandra
  • Why?
    • Session recording is a key differentiator in our product offering, but can be overwhelming and unreliable - to maximize the value our users get from it we should resolve reliability issues and make it trivial to find the most meaningful sessions to understand why something is or is not occurring
• "Performance" - Ship (list of specific further performance optimizations) @macobo @EDsCODE for your thoughts?
  • Why?
    • Performance has been a major challenge for our focus customers, we've made some enormous improvements but we still have a long way to go to ensure PostHog meets the performance expectations for larger businesses
• "Data Integrity" - @yakkomajuri it feels there are still some duplicate distinct IDs/ & cohort issues - would be great to get your perspective on a specific outcome we could achieve here in the next sprint
  • Why?
    • As called out at our offsite (Pre-mortem) https://github.com/PostHog/product-internal/issues/153 Data Integrity is critical, we should ensure we truly understand the causes of any potential discrepancies in our data and resolve them to avoid eroding customer trust

I would really appreciate feedback, changes, iterations, and adding more detail on specific outcomes for these proposed themes ahead of our session on Weds. Any major goals we should be focussing on instead of these? Especially from Platform @tiina303 Growth @samwinslow @kpthatsme perspective? Also any hackathon projects we need a major team focus on to get into production?

[paolodamico]

• We should take into account that we'll need to run quant analysis usability tests next week. It'd be ideal if we could have an MVP to run them, but unsure if it's feasible @neilkakkar ?
• In terms of session recording I think the goal of Core Experience Team can actually be ensuring a performant and seamless user experience when playing back recordings, as Core Analytics is working on finding the right recordings.
• +1 in coming up with a clear outcome on Data Integrity, at least for the problems we already know about.
• I know @mariusandra wanted to continue driving forward experimentation. My suggestion would be to only productionize multivariate testing which we know users are going to need, but hold off on anything else until we can do more research.

[tiina303]

• +1 to data integrity importance. "we will have worked out why this issue is happening" might too optimistic based on previous experience (a bunch of causes were identified and fixed, but it's still happening).
• offsite follow-up - Dive dashboard launch #5946
• Deprecate Postgres backed PostHog officially? (& Document migration from Postgres to Clickhouse posthog.com#1892)

[mariusandra]

Priorities that my hedgehog sees:

• "Paths User Experience" - if I'm not mistaken, this project is still in the design / exploration stage, and it'll take a good part of a sprint to get the product and implementation done. Don't underestimate how long getting all the frontend knobs turned just right takes.
• "Quantitative Analysis MVP" - too ambitious to think about it now.
• "Session Recording" - I vote yes to fixing the bugs that are 🥚-on-🤕 embarassing-enough to make competitors run attack ads.
• "Data Integrity" - P0. However, we're also not talking about two other parts of the puzzle:
  • Schemas. For at least some key customers, it's not clear what events are sent from posthog, what are their custom events (internal slack link to same customer). I once wrote an issue about how all API clients/libraries send different things. If we are to become the de-facto interconnection pipeline on the internet, we need to be well defined. There's PluginEvent that comes through Kafka, but that contains noise left over from ingestion (timestamp and now and sent_at, yes please!). Read this thread again to see the pain that will happen on a recurring schedule.
  • Durability. Right now if you export and import a posthog database to e.g. bigquery and back in again, you will not end up with the same data. It'll be close, but not the same. For example the $plugins_ran property will be dynamically updated. The order of $identify calls matters. However the biggest one is $set and $set_once for setting user properties. We should be able to always ingest all events backwards, but timestamping and/or denormalising person properties would take us a very long way (and also fix some concurrency issues).
• "Experimentation"
  • The feature flags in the toolbar task should be done this week. Once that's live (or flagged), it'll have scratched a big itch of mine. A final release of this toolbar view requires a few changes to the toolbar itself, as the green flag button will be floating aimlessly otherwise.
  • I wouldn't personally prioritise the follow up work with magic feature flags this sprint, but Team Growth who has been pushing for improvements in experimentation is absolutely free to roll with it and other improvements here.
• "Nail Fix Broken Stuff"
  • The "Compare Previous" feature has been causing problems for a while and should be fixed
  • Two paying customers have requested the feature to breakdown by data-attr in trends (might be done sooner?)
  • Our Onboarding Flow is out of this world 🥚-on-🤕.
  • We have some work to do to clean up our API. From the 🥚-on-🤕 issues with not being able to use multiple projects at the same time, to just publishing decent specs and client libraries, such as a fully typed posthog.api.getEvents() instead of api.get('/events').
  • I'd really like to start work on just making all the little things in the app look nice. From the "dashboard" headers, to the sessions filters, to the plugins page, our app looks like a proper hotchpotch of conflicting styles (don't mention the toolbar). That doesn't spark joy. Instead of nailing experimental features (paths), we should first nail core experience :).

[EDsCODE]

"Paths/Feature work"
The api for the updated paths is implemented and deployed (feature list: #5665 (comment))

+1 to Marius's note on "Path's User Experience". We need more user feedback to make a well-informed decision on how to proceed with the visualization

-1 on Marius's not on quantitative analysis. I think we should solve this process issue with this next sprint. We should try to wrap up paths feedback and design direction by the end of this week/early next week and use the next sprint to implement/deploy the paths feature. At the same time, we can design/gather feedback for quantitative analysis

"Performance"

• full context here from @macobo's doc
• continue to monitor and make sure materialized columns are stable
• Implement materialized columns for all insights (Make use of materialized person columns in other insight features #5888)
• funnel slowness on self-hosted deployments (Significant performance issues with funnels on Postgres #5519)
• Investigate why funnels run out of memory/query size limits (Funnel query Memory limit exceeded #5391)
• use materialized columns/new query patterns in the cohort calculation queries. They're severely outdated right now and have some bad practices that are making them slow

"Data Integrity"
+1

• We should definitely take a closer look at the materializedpostgresql bridge that clickhouse is working on. This would be a step in removing the inconsistencies in syncing persons between postgres and clickhouse

[mariusandra]

-1 on Marius's not on quantitative analysis. [...] At the same time, we can design/gather feedback for quantitative analysis.

That's the thing, I'm not sure design for quantitative analysis should be done "at the same time" or now/next. I just don't want to overload Chris, who's been hard at work on paths now for a while, at the cost of other work. I wouldn't want to tie him down with a similar large/undefined batch of work. My prediction is that it'll take longer to implement nailed paths than we think, and implementation work on nailing quantitative analysis won't start for at least another sprint or two. At the same time, there's so much other work and housekeeping Chris could help with.

[paolodamico]

Seems like we're aligned with the overall direction of the sprint! Let's discuss the outstanding details in the session today to avoid ending up with a super long issue.

[marcushyett-ph]

Discussion points for the meeting:

• Quick retro on current sprint outcomes
• How do we handle "Discovery" work ahead of implementation
• Prioritizing themes
  • Unanimous priorities
    • Paths User Experience (Core Analytics) - Design and iteration based on feedback internal / external complete and UI implemented for core use cases (listed here @EDsCODE )
    • Performance (Core Analytics + Platform) - Splitting ClickHouse out of helm chart (if they hit the wall) @EDsCODE @hazzadous to clarify these here - (specific performance improvements to queries)
    • Data Integrity (Core Analytics + Platform) - Ingestion - Get to Zero newly created duplicate distinct IDs (around sprint planning time), a plan for fixing duplicates for Self hosted and cloud users, Analytics - Take a look at where the problems might be @EDsCODE @hazzadous - (@kpthatsme can you share charts which are wrong)
    • Session Recording (Core Experience) - People don't feel like they're missing so many recording, It should be possible to get useful information out of any recording. (@paolodamico)
    • Quantitative Analysis MVP (Core Analytics + Design) - We really need to get Paths right first - this could continue to take up time due to unknowns @neilkakkar
  • Worth discussing further
    • Experimentation / Growth - FF are correct, Customer Success will take priority.
    • "Nail Fix broken stuff" - Ensure we have clear prioritization for deal-breaker issues (and if we find, fix and repri), larger UI changes requires larger focus for major change
• Aligning on specific outcomes for themes in sprint)
• Ensuring we don't overload design (@clarkus)

[paolodamico]

Alright @mariusandra has started #5974 for the actual sprint planning. I'd recommend each team add the specifics of their work in that issue. Please avoid having internal small teams discussions in that issue, and update the main description of the issue with your final conclusions. @EDsCODE @fuziontech @mariusandra @kpthatsme

[paolodamico]

Sprint is over 👋

[posthog-contributions-bot]

This issue has 1388 words. Issues this long are hard to read or contribute to, and tend to take very long to reach a conclusion. Instead, why not:

1. Write some code and submit a pull request! Code wins arguments
2. Have a sync meeting to reach a conclusion
3. Create a Request for Comments and submit a PR with it to the meta repo or product internal repo