Implement In-Process Queue Management for Email-Based ReFramework Workflow

[cprima]

Description

This feature implements a hybrid ReFramework solution tailored for email processing using an in-process queue managed at runtime. Unlike standard ReFramework implementations that rely on Orchestrator queues, this design will populate and manage transactions directly from a mailbox, ensuring transactions are processed in-memory. The in-process queue will support dynamic batching, real-time adaptability, and handle Orchestrator-specific functions like transaction status and error handling locally, reducing dependency on Orchestrator. This feature eliminates the need for a Dispatcher, handles a teaching context where multiple learners might run the process in parallel to each other, while providing the default TransactionItem for processing.

Acceptance Criteria

• In-process queue is populated directly from the mailbox at the initialization of each process run, adhering to the specified batch size.
• Transaction statuses are managed in-memory, with custom logic for marking items as “Successful” or “Failed.”
• Retry mechanisms for system exceptions are implemented in alignment with ReFramework retry standards.
• Logging and monitoring are integrated within the workflow, capturing transaction progress, errors, and retry details.
• Workflow maintains ReFramework’s standard structure, enabling seamless integration with GetTransactionData and Process states.

Subtasks

• Develop mailbox connection and email fetching logic to populate the in-process queue in batches.
• Implement dynamic batch size configuration (Target_Batchsize) to control memory usage.
• Create custom transaction status handling to track and update transaction outcomes locally.
• Design a retry mechanism for system exceptions, ensuring failed transactions adhere to retry limits.
• Integrate custom logging for tracking transaction statuses, retries, and errors.
• Conduct testing to verify the process handles real-time updates in the mailbox and processes new items adaptively.

Related Issues / Dependencies

• Related to: Refactoring monolithic legacy code #1 legacy code has no transactional integrity #10
• Depends on: -

Priority

• Priority: High

Attachments / Additional Resources

• Attachments:
  • none

[cprima]

Fixed with 03074ff