WIP: Feature/automated full interface generation #181

markdoerr · 2023-06-17T21:17:51Z

This is the related pull request to issue #180.

It is still work in progress, because the issue #178 needs to fixed.

The workflow for the interface generation is the following:

cd <to my app directory containing the model.py>
then run
./manage.py generategrpcinterface <my_app>

This will generate a grpc directory, containing three files:
handler.py serializers.py and services.py

and fill it with default content.

Then the protogenerator needs to be executed:
./mange.py generateproto <my app> # this should work after fixing #178

Finally one needs to remove the comments in serializer.py to assign the
proto_class and proto_class_list

That's it :)

(I also added the official python api call of protoc - so no external shell call is required - but there is also something with the paths I need to fix)

Looking forward to your comments....

… protoc import bug needs to be fixed

legau · 2023-07-19T11:19:31Z

django_socio_grpc/management/commands/generategrpcinterface.py

+INDENT_WIDTH = getattr(settings, 'INDENT_WIDTH', 4)
+
+
+class Command(LabelCommand):


If it's by app, couldn't it be AppCommand ? It would automatically handle some of your logic below

AppCommand

Looks like a good suggestion @legau . I will have a look into that...

legau · 2023-07-19T12:04:50Z

django_socio_grpc/management/proto_compiler.py

I don't see what this file has to do with the rest

It's a hack that is required because of a bug in python protoc.
See:
protocolbuffers/protobuf#1491
and
grpc/grpc#29459

legau · 2023-07-19T12:05:48Z

django_socio_grpc/management/commands/generateproto.py

+                    #     )
+                    # correcting protoc rel. import bug
+                    # 
+                    (pb2_files, _) = os.path.splitext(os.path.basename(file_path))


Prefer using pathlib for that

@legau - indeed, pathlib is the better choice. if you decide to use the buf-workflow for interface generation, this hack will be obsolete anyway. Let's discuss this tomorrow.

legau · 2023-07-19T12:15:55Z

django_socio_grpc/management/commands/generategrpcinterface.py

Could you provide the workflow you have in mind with this ? I have trouble understanding what are the requirements and what stage should you launch the command

Sure, @legau,
the goal should be to generate as much of the gRPC API as possible automatically:

For this Merge Request, the workflow would look like:

add the django_socio_grpc app to the django settings of a new project with one or several apps

cd to my app directory containing the model.py
then run
./manage.py generategrpcinterface <my_app>

This will generate a grpc directory within the app, containing three files:
handler.py serializers.py and services.py

and fill it with default content.

Then the protogenerator needs to be executed:
./mange.py generateproto # this should work after fixing #178

Finally one needs to remove the comments in serializer.py to assign the
proto_class and proto_class_list

By that one would have a good starting point for further extensions of the interface.
This is extremly handy, if you have a project with many apps and many fields in the models.

In the future, one should consider the buf workflow for the .proto compilation / API generation.

Also custom services could be added based on the .proto file in the future.

legau · 2023-07-19T12:16:19Z

django_socio_grpc/management/commands/generateproto.py

                    file_path.parent.mkdir(parents=True, exist_ok=True)
                self.check_or_write(file_path, proto, registry.app_name)

                if self.generate_pb2:
                    if not settings.BASE_DIR:
                        raise ProtobufGenerationException(detail="No BASE_DIR in settings")
                    os.system(
-                        f"python -m grpc_tools.protoc --proto_path={settings.BASE_DIR} --python_out=./ --grpc_python_out=./ {file_path}"
+                        f"python -m grpc_tools.protoc --proto_path={proto_path} --python_out={proto_path} --grpc_python_out={proto_path} {file_path}"


would this render your other PR obsolete ?

good point. Maybe we finish the first PR first :)

Dear Leni (@legau ),
I will rethink the workflow after what i learned from you today ...
(I think, the first PR still has it's validity, so let's complete this first.)

…-generation

AMontagu · 2023-11-28T16:44:20Z

@markdoerr after using a little bit more the library do you still think we need this ? Is yes can you describe your use case ?

markdoerr · 2023-11-28T20:59:46Z

Dear @AMontagu and @legau,
after working for a longer time now with your framework, many things got much clearer - and I will try to make suggestions for improvements of the documentations that beginners have it easier.
Your DSG workflow is very handy and convenient.
For large applications with a lot of models (in LARA we have in the range of 100 - 150 models, I guess), an initial generator of the service.py, serializer.py and handler.py file comes quite handy.
We could consider combining the generators in the generateproto routine: if not these three files do not exist, just generate them and populate them. This would make the whole interface building even more convenient. What do you think ?

New Workflow:

build a new django app from scratch (e.g. the startproject or cookiecutter)
define the models
run "manage.py generateproto" -> this builds the whole DSG / gRPC infrastructure (handler.py, serializer.py, services.py, grpc folder with .proto and stubs)
update / refine models and / or add services and actions as required
run "mange.py generateproto" again
new proto files / subs are generated , but service.py / serializer.py / handler.py are not affected.
reiterate until one is happy with the models and the gRPC / DSG interface, then create an api folder in the project and let buf generate all the stubs also for grpc-web / go / or whatever ... (optionally in a CI/CD pipeline and a separate python api package)

What do you think ?

legau · 2023-11-29T14:15:22Z

Hi,
I guess that would be a nice addition but it should be its own command. Something like generategrpcservices maybe.

In your use case, what you would expect is to have it generate one service by model with one serializer and this service added to an app handler ?

AMontagu · 2023-11-30T15:27:50Z

I am not a fan of generating code from models. I mean this could have been done with DRF but it is not. You have https://pypi.org/project/drf-generators/ but it's an external dependencies.

we could totaly copy drf generators and call it dsg-generators and making the really small changes we need but what is generated is the most easy part of the app and when you start a project from the begining developer often code API endpoint after endpoint instead of defining all the models and then endpoints.

Specially considering that running the command after developer made change can bring obfuscated automatic code change that can lead to bug due to misunderstanding of the tool.

markdoerr · 2023-12-29T14:38:00Z

Hi @AMontagu,
thanks for the hint - I was not aware of drf-generators.
I think it is a good idea, to copy the drf-generator code and adjust it to DSG needs.

Let's put it to the list of "nice-to-have" features of DSG :)

We could do this drf-generators "copy and adjustment", if there are more people interested in DSG to make the DSG start as painless as possible.

markdoerr added 5 commits May 27, 2023 12:01

fix: using --directory to set protoc output path

4237aa6

feat: first version of gRPC auto interface generation working nicely,…

76c5bae

… protoc import bug needs to be fixed

feat: improved proto compiler added - some parts need to be fixed

73e10b0

feat: protoc rel. import bug fix added

c7b0dc5

feat: allow to select the app name for building the interface

39d7fb2

legau reviewed Jul 19, 2023

View reviewed changes

markdoerr added 3 commits July 24, 2023 22:41

Merge branch 'socotecio:master' into feature/automated-full-interface…

280c14c

…-generation

Merge branch 'socotecio:master' into feature/automated-full-interface…

7d12b06

…-generation

Merge branch 'socotecio:master' into feature/automated-full-interface…

5f75635

…-generation

markdoerr closed this Dec 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Feature/automated full interface generation #181

WIP: Feature/automated full interface generation #181

markdoerr commented Jun 17, 2023

legau Jul 19, 2023

markdoerr Jul 24, 2023

legau Jul 19, 2023

markdoerr Jul 24, 2023

legau Jul 19, 2023

markdoerr Jul 24, 2023

legau Jul 19, 2023

markdoerr Jul 24, 2023 •

edited

Loading

markdoerr Jul 24, 2023

legau Jul 19, 2023

markdoerr Jul 24, 2023

markdoerr Jul 25, 2023

AMontagu commented Nov 28, 2023

markdoerr commented Nov 28, 2023

legau commented Nov 29, 2023

AMontagu commented Nov 30, 2023 •

edited

Loading

markdoerr commented Dec 29, 2023

		INDENT_WIDTH = getattr(settings, 'INDENT_WIDTH', 4)


		class Command(LabelCommand):

WIP: Feature/automated full interface generation #181

WIP: Feature/automated full interface generation #181

Conversation

markdoerr commented Jun 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markdoerr Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AMontagu commented Nov 28, 2023

markdoerr commented Nov 28, 2023

legau commented Nov 29, 2023

AMontagu commented Nov 30, 2023 • edited Loading

markdoerr commented Dec 29, 2023

markdoerr Jul 24, 2023 •

edited

Loading

AMontagu commented Nov 30, 2023 •

edited

Loading