Refine Dry Run

[wizenheimer]

Summary

Add support for usage and cost tracking, including caps, and implement a dry-run feature to mock the cost of operations.

Pain Points

What challenges are users encountering without this feature?

1. Users may lack visibility into usage and costs, leading to unexpected expenses and difficulty managing budgets.
2. Without cost caps, users might exceed their budget limits unintentionally.
3. The absence of a dry-run feature makes it challenging to estimate costs before running operations, potentially leading to unanticipated expenses.

Current Workarounds

Have you come across any alternatives or tried any workarounds?

1. Users might manually track usage and costs or use external tools for monitoring, which can be cumbersome and less integrated.
2. Cost caps might be managed through external budgeting tools or manual oversight, lacking automation.
3. Custom scripts or approximations might be used to estimate costs, but these methods may lack accuracy and integration.

Solution

What solution would you like to see?

• Add support for tracking usage and costs to provide users with detailed insights and control over their expenditures.
• Implement cost caps to automatically limit spending and prevent budget overruns.
• Introduce a dry-run feature to mock the cost of operations, allowing users to estimate expenses before committing to actual runs.
• Run the pipeline on a sample set
  • Add sampling utility for tasks with dry run flags
• Preview output prior to complete run
  • Fix tabulation
  • Add statistics

References

Have you seen similar solutions or examples elsewhere?

1. LiteLLM
2. Athina
3. Relari

Additional Notes

Is there anything else you’d like us to know?

1. N/A

Contribution

• I’d love help with this if needed!

---