Create an example to demonstrate how to use Application Apecific Management Groups

[nandojve]

I've been testing the library is I liked a lot. However I struggled to understand how I can use smp + smpclient + smpmgr with my private application specific management groups without upstream the content. I would like to use all the 3 apps or just have a very small app that glue all 3 + application specific management groups.

[JPHutchins]

Hi @nandojve! This is a good question and there is no official answer. Ideally we can work together to develop an approach that works.

Upstreaming smp, smpclient + smpmgr would be my preference.

If that is not possible due to licensing restrictions or other, then we need another approach. I think that it would be good to start with you stating how it would work in a perfect world. For example, you might like to define them ad-hoc from the command line or by referencing some file? smpmgr is intended as a developer tool, and it is a shortcoming that there is no command for sending/receiving arbitrary data!

I also have a few questions to kickoff:

• Would this be delivered to end users? Like, you create a private fork of smpmgr that will have your private groups? In that case, it seems that the private fork of smpmgr alone should be able to register the de/serialization and generics for use with smpmgr, without having to also fork and import smp and smpclient.
• You mentioned all 3; meaning smp, spmclient, and smpmgr. You need the smp library if you are defining your own transport layer, and you need the smpclient library if you are building your own application. But it sounds like you perhaps don't need them at all if you're just planning on using a modified smpmgr?

JP

[nandojve]

Hi @JPHutchins ,

Thank you for prompt answer!

Upstreaming smp, smpclient + smpmgr would be my preference.

Yes, I understand that and I know the value of keep things up to date in the upstream. I've been mostly contributing to Zephyr since 2019. Unfortunately there are few cases that you can expose core business if you do that and then it became prohibitive.

• Would this be delivered to end users? Like, you create a private fork of smpmgr that will have your private groups? In that case, it seems that the private fork of smpmgr alone should be able to register the de/serialization and generics for use with smpmgr, without having to also fork and import smp and smpclient.

No private fork. The idea is to use directly your libraries as is.

• You mentioned all 3; meaning smp, spmclient, and smpmgr. You need the smp library if you are defining your own transport layer, and you need the smpclient library if you are building your own application. But it sounds like you perhaps don't need them at all if you're just planning on using a modified smpmgr?

The issue is that there are specific commands that can not public. Those commands can already be build in Zephyr using an OOT module. Now I want to expose those using MCUmgr, at command line in development phase, and later integrate with other applications. In this case, I consume many of the public and I have my own commands.

I was brain storming the idea and realize that on the ideal world it will be nice to have an yaml file that could be read and the private application specific management groups be dynamically create based on that, maybe even the standard ones. The yaml(s) could be passed by command line argument or just read a directory. Those could be joined with the standard ones.

[]'s

CC: @otavio

[JPHutchins]

I was brain storming the idea and realize that on the ideal world it will be nice to have an yaml file that could be read and the private application specific management groups be dynamically create based on that, maybe even the standard ones. The yaml(s) could be passed by command line argument or just read a directory. Those could be joined with the standard ones.

Yeah, something like this would be perfect! I won't want to do anything with the standard commands since this sounds like meta-programming and it will break all of the static typing. But, testing with EchoWrite/Response seems a good first step.

I know TOML is not so great for nested objects, but at least it's builtin to Python - worth considering vs YAML?

Parsing the definition file will, at runtime:

• create pydantic models for de/serialization and validation
• register a usable Typer command

PS: smpmgr interface needs some breaking changes to support BLE + UDP transports. It looks like it will be something like smpmgr [global options] <transport> [transport specific options] <command group> <command> [args].

[JPHutchins]

@nandojve Can you try pulling https://github.com/intercreate/smpmgr/tree/example/custom-groups to your private fork and modifying it to see how it goes? So far I don't see any issues and it's probably better since writing actual python code gets the type safety vs the YAML def approach.

Still, for development purposes, I think that smpmgr should support arbitrary bytes request/response and maybe some convenient syntax for sending an arbitrary request. Maybe

smpmgr custom --group=80 --command=0 cbor={"my_field": 1}

[nandojve]

Hi @JPHutchins ,

Nice! I'll try to check this weekend.

[otavio]

@JPHutchins, I believe that sending a custom request directly from the command line is very useful for the development process. That said, defining the group and the comments using a TOML or a YAML file to describe this would be helpful because it allows for multiple comments definition and more complex object definition and validation.

[JPHutchins]

I'll plan on adding sending of custom commands ASAP including arbitrary bytes.

smpmgr bytes <hex string>

e.g.

smpmgr bytes 234af3782

would return some bytes

smpmgr smp <operation> <group> <command> [--version default V2] [--flags default 0] [--sequence default 0] [--format default cbor] <data ... JSON string interpreted as CBOR assuming that --format is not set to something else>

eg
smpmgr smp write 0 1 '{"my_cbor_field": 342}'

I'll need to be careful with header validation. It would return the SMP header and deserialized CBOR.

Fortunately the Python syntax is about as declarative as YAML or TOML and less prone to problems due to static type checking:

smpmgr/smpmgr/user/custom.py

Lines 17 to 96 in e056af0

	@unique
	class CUSTOM_RET_RC(IntEnum):
	OK = 0
	NOT_OK = 1
	class CustomErrorV1(smperr.ErrorV1):
	_GROUP_ID = 88
	class CustomErrorV2(smperr.ErrorV2[CUSTOM_RET_RC]):
	_GROUP_ID = 88
	class CustomWriteRequest(smpmsg.WriteRequest):
	_GROUP_ID = 88
	_COMMAND_ID = 0
	d: str
	class CustomWriteResponse(smpmsg.WriteResponse):
	_GROUP_ID = 88
	_COMMAND_ID = 0
	r: str
	class CustomWrite(CustomWriteRequest):
	_Response = CustomWriteResponse
	_ErrorV1 = CustomErrorV1
	_ErrorV2 = CustomErrorV2
	class CustomReadRequest(smpmsg.ReadRequest):
	_GROUP_ID = 88
	_COMMAND_ID = 1
	class CustomReadResponse(smpmsg.ReadResponse):
	_GROUP_ID = 88
	_COMMAND_ID = 1
	something: int
	class CustomRead(CustomReadRequest):
	_Response = CustomReadResponse
	_ErrorV1 = CustomErrorV1
	_ErrorV2 = CustomErrorV2
	@app.command()
	def write(ctx: typer.Context, message: str) -> None:
	"""Custom echo write."""
	smpclient = get_smpclient(cast(Options, ctx.obj))
	async def f() -> None:
	await connect_with_spinner(smpclient)
	r = await smpclient.request(CustomWrite(d=message))
	print(r)
	asyncio.run(f())
	@app.command()
	def read(ctx: typer.Context) -> None:
	"""Custom echo read."""
	smpclient = get_smpclient(cast(Options, ctx.obj))
	async def f() -> None:
	await connect_with_spinner(smpclient)
	r = await smpclient.request(CustomRead())
	print(r)
	asyncio.run(f())

So I'm not too excited about YAML or TOML. The only advantage I can think of is sending someone the main smpmgr build along with a file that adds the custom behavior. But it seems less error prone to fork smpmgr.

[nandojve]

Hi @JPHutchins ,

I validate the #27 and that works fine as expected.
Question: Is it possible to add a parameter in the cmdline that user can pass a folder and with that enumerate the py files and automatically register then? I believe with this functionality we don't need clone to any repo.

What do you think about it?

[JPHutchins]

Hi @JPHutchins ,

I validate the #27 and that works fine as expected.

Question: Is it possible to add a parameter in the cmdline that user can pass a folder and with that enumerate the py files and automatically register then? I believe with this functionality we don't need clone to any repo.

What do you think about it?

I think that it is possible, if somewhat complicated by pyinstaller. I'll be looking into it!