The evolution of Azure Optimization Engine within the FinOps toolkit

[helderpinto]

With the upcoming migration of the Azure Optimization Engine (AOE) to the FinOps toolkit (see PR), it is now time to envision what should be its ideal evolution path. Either we keep its current architecture and keep developing its feature breadth or we promote its smooth integration with the FinOps toolkit ecosystem and leverage the opportunity to focus on the most valued FinOps-related features.

I support an evolution of AOE that transforms it into an add-on to the core of the FinOps toolkit - FinOps hubs - and shares synergies with other tools such as the Cost optimization or Governance workbooks. Instead of duplicating Azure billing and pricesheet data (as it currently does), AOE would leverage Azure cost management billing and pricesheet exports and share the same ETL and data repository. Some of the current AOE reports could possibly be built on top of this data, without additional processing. Other reports have overlap with some FinOps toolkit workbooks visualizations and could be migrated or merged into new workbooks. For the optimization recommendations, we'd need to collect additional metrics and other sources into the same data repository. For visualizations, we'd provide either (or both) Power BI or Azure workbooks reports, depending on the underlying data repository solution.

With this approach in mind, there is necessarily an important refactoring effort. If we aim to have soon a new version of AOE, better integrated in the FinOps toolkit ecosystem, some difficult choices will have to be made. AOE started as cost optimization tool and, based on customer feedback, has become over the years a kind of an optimization Swiss Army Knife, with features covering cost, governance, security, performance, reliability and operational excellence insights. This is not necessarily a bad thing for Azure customers, but it allows for development dispersion and will make its evolution effort much heavier. For this reason, we are inviting current AOE users to answer a short survey that will help prioritize the features that are more valued.

I am curious about the thoughts and ideas other FinOps community members have regarding this topic!

[helderpinto]

The AOE evolution survey has collected answers enough to conclude that the features customers value the most are custom cost recommendations (especially the Advisor VM right-size augmentation) and rate optimization-related workbooks. Customers are OK with AOE focusing and expanding only on these capabilities going forward.

Porting the most prioritized features to an architecture more aligned with the FinOps toolkit would require:

• ADF pipelines to extract/ingest the following data (ideally based on native exports to storage instead of API calls):
  • Azure Advisor cost recommendations
  • Pricesheet (for all types of consumption agreements - EA, MCA, etc.)
  • Reservations usage (a viable alternative could be calling the API directly from Power BI)
  • Savings Plans usage (idem)
  • Reservations prices (idem)
• Functions (Logic Apps might have some limitations for expressing complex logic) that query and combine data from multiple data sources to generate or augment cost recommendations:
  • Cost exports (from storage/ADX)
  • Pricesheet (from storage/ADX)
  • Azure Resource Graph (ARG)
  • Azure Advisor cost recommendations (either from ARG or from storage/ADX)
  • Azure Resource Manager (for Azure Monitor metrics, SKU properties, etc.)
  • Log Analytics (for guest OS performance counters collected by agents)

I also realize that there are multiple FinOps toolkit capabilities that share goals and even specific recommendations with AOE, such as the Cost Optimization Workbook, Waste Reduction Logic App and Azure Cost Optimizer proposed by @jamelachahbar. Maybe this is an opportunity for us to rethink how optimization recommendations in general should be generated, avoiding code and feature duplication across the toolkit. What are your thoughts on this, @flanakin, @arthurclares, @sebassem, @ro100e, @KevDLR, @jamelachahbar?

[jamelachahbar]

Hi @helderpinto, my first thoughts on this. I was thinking of a layered approach, where the foundational layer would be the finops hubs with ADF pipelines to pull in data transform it and export it to an ADLS Gen2 storage. AOE would be the recommendation engine, on top of the existing Advisor API/data, with historical data based on the exports, to create confidence in the recommendations where Advisor lacks context,...So AOE would leverage the data exports from finops hubs + enrich it further based on the needs for AOE. Then come the optimization workbook, logic apps, cost optimizer which cater to different personas and I believe, if pulling from the same source, we can avoid duplicating efforts from that aspect but allowing these tools to continue operating individually and providing their unique value, but still have a consistency in terms of data they pull in(cost data, advisor recommendations, custom recommendations from AOE,...). Speaking for Azure Cost Optimizer: I can pull in data from ADLS Gen2 containers, or doing API calls. While keeping the approach where an admin can craft their own cost policies and act upon them, I can pull in AOE's recommendations, Advisor,.... and allow admins to confidently take action, either by using devops practices like github action workflows or providing them with a User Interface, which would serve as an operational dashboard where they can pull in the info, make decisions and take corrective actions if needed.

[helderpinto]

I like the layered approach you suggested, @jamelachahbar. What do other contributors and the community think?

[007OM]

@helderpinto thanks a lot for creating such a great tool, I'm very happy AOE is becoming an essential part of FinOps toolkit, I second @jamelachahbar to create the foundational data layer, and I'd like to keep the lights on the other 4 pillars as well other than the cost pillar, from a business perspective the recommendations in the security pillar for example needs to be more accurate, makes more sense by actually understanding the customer deployments also should guide the customer to it's natural next step to evolve his security, there is another tool created by a friend of ours https://github.com/Azure/azqr it checks the resources against the best practices and it shows what is complaint and what is not and what should be done to be compliant. i believe if a foundational data layer is being created AOE can also add such a function (discussing with Carlos creator of AZQR) to provide customers with real 360 insights in all the 5 pillars of WAF. as the ROI is highlighted through the analysis AOE and finops hubs is doing on the cost pillar then we can guide the customers how to reinvest in the right areas (products & services) that they really need to invest in.

[helderpinto]

Thanks for the suggestion, @007OM. Definitely, AOE can reuse or be inspired by the work done in AZQR. On a different angle, improving AOE extensibility could, for example, allow to ingest AZQR outputs into the AOE recommendations.