Improvement: Backend Improvements for Typing, Tests, Modularity and Pathlib

[omeryusufyagci]

Current app.py will be reorganized under backend/, with the following objectives:

• Typing: Add type hints throughout the backend
• Unit Tests: Develop unit tests for all utility and processing functions.
• Response Handling: Add a ResponseHandler class in backend/response_handler.py to handle all communication bwn BE-FE, BE-core using consistent JSON responses.
• Replace os with pathlib
• Reorganize the code under backend/:
  • Move Utils to backend/utils.py
  • Move MediaHandler to backend/media_handler.py
  • Keep app.py for route definitions and Flask app initialization.

This is a big step going towards v1.0 and offers a relatively straightforward first issue for contributors looking to make an impactful contribution to the project.

[prakash-2002-ramanathan]

Is this issue open? Shall I open a PR?

[omeryusufyagci]

Hi @prakash-2002-ramanathan, thanks for your interest. Indeed, it's open and much needed!

Would be great if you could open a PR!

[prakash-2002-ramanathan]

Hey, I have done all of the work, but not quite sure what you mean by "make the Backend to Core communication Consistent JSON "?
Like You want to start a server in C++ to handle JSON responses or pass the data in JSON format to the Media Handler subprocess ?

[omeryusufyagci]

Thanks for the question @prakash-2002-ramanathan,

We should ensure consistency across the application, using json responses between all layers. Currently the MediaProcessor writes to stdout and the backend is parsing directly these traces. Instead, the core should return a json object.

Likewise, anything between the frontend and backend should be managed through json objects for consistency.

Does this answer your question?

[prakash-2002-ramanathan]

I Appreciate your response. But how do I implement this. Do I pass the a JSON object as a string in the Std::cout from MediaHandler, and parse the JSON from result.stdout in python? Or is there a better way to communicate between the BE and the Core ? Any suggestions for improving this communication pattern?

I will give the input to the MediaHandler in subprocess. And shall I make it return the JSON output using HTTP to the sever?

[omeryusufyagci]

Well, we have an open issue to add a proper logger, e.g. spdlog. Until that comes, we're still writing plain text to cout/cerr.

In the future we'll probably use gRPC for inter-process communication (IPC), but at this stage of the project I'd rather get something significantly less error prone working first. In that sense, for now, we can dedicate cerr for logging, leaving cout isolated for our IPC.

Let's make sure the json structure stays consistent, so that we can write generic parsers.
I'll suggest something like:

{
    "type": "progress",
    "payload": {
        "status": "in_progress",
        "message": "Processing audio",
        "progress": {
            "percentage": 75
        }
    }
}

and for a response type we'd omit the progress. The parser should expect the absence of certain parameters and default to a sensible value.

{
    "type": "response",
    "payload": {
        "status": "success",
        "message": "Processing completed",
    }
}

Currently responses are more important than the progress packets, but still we should think ahead as we implement this.

Please let me know if you have any other questions or need help during the implementation. Thanks!