Skip to content
This repository has been archived by the owner on May 7, 2020. It is now read-only.

Add Serial-over-IP (rfc2217) support in ESH Serial API #5560

Merged
merged 11 commits into from
Jul 22, 2018

Conversation

msteigenberger
Copy link
Contributor

@msteigenberger msteigenberger commented May 9, 2018

The actual instance of the SerialPort is chosen at runtime depending on the port's name.
This allows to have different implementations for different protocols. This can be e.g. RXTX (default local) or RFC2217.
Implementations are discovered at runtime via OSGi DS.

A client application now only needs to set the correct port name.
e.g.

  • /dev/ttyUSB -> RxTxPort
  • rfc2217://x.x.x.x:3001 -> TelnetSerialPort

Signed-off-by: Matthias Steigenberger [email protected]

@msteigenberger
Copy link
Contributor Author

@maggu2810: Please have a look it is an addition to your PR: #5313

I want to have an easy way in the framework itself to allow bindings to switch between serial port implementations.
Actually I first thought to have a own bundle which holds the registry and all other implementations (like javacomm) can also use it and provide a "SerialPortCreator". I left it because I first wanted to know your opinion in general for this.
Thanks!

@msteigenberger msteigenberger reopened this May 9, 2018
@kaikreuzer
Copy link
Contributor

I want to have an easy way in the framework itself to allow bindings to switch between serial port implementations.

Could you elaborate on why you want to let the bindings to do the selection?

In essence, the current serial API expects a single implementation to be present, while you are now asking to have the option to have multiple implementations being present at runtime.

Imho the simplest (and non-intrusive) enhancement would be to leave it to the implementations to claim certain ports for them - e.g. any port that starts with rfc2217:// could be claimed by the TelnetSerialPort implementation. Would such a naming convention approach work?

@maggu2810 Any comments or ideas from your side?

@msteigenberger
Copy link
Contributor Author

Imho the simplest (and non-intrusive) enhancement would be to leave it to the implementations to claim certain ports for them - e.g. any port that starts with rfc2217:// could be claimed by the TelnetSerialPort implementation. Would such a naming convention approach work?

But then every implementation would need its own logic to decide which serial port to use AND - which is even worse - the implementation would need a dependency to the specific port (like TelnetSerialport which comes from nrjavaserial).
The whole intention was to exactly not depend on any implementation but rather let the framework decide depending on the users input which serial port implementation to use. If the user wants to communicate via rfc2217 he/she only has to state the rfc2217:// URL and the binding implementation doesn't care about it.

@maggu2810
Copy link
Contributor

Hm, for me the distributor should decide which implementations should be supported and not the user itself.
If I think about ESH as a framework also for commercial products the company choose the serial port implementation (stability, licenses, ...) and the consumer will use that implementation without needing to knowing the implementation itself.

@kaikreuzer
Copy link
Contributor

I agree with @maggu2810 that the user should be the last one to be bothered with choosing a certain implementation.

All that is asked from him is to specify the name of a serial port to connect to - our implementations are currently also supposed to provide a list of available names here.
Clearly, remote tcp ports that accept telnet connections and act like a serial port won't be discoverable and thus not be listed - this will already cause some issue in the Paper UI as you are currently not allowed to enter any other string. But let's assume the user could: What is he supposed to put in there? Is there any typical naming convention?

Imho it makes only sense to consider those "rfc2217" ports as serial ports, if they actually behave exactly the same, so from a binding perspective, it actually should not matter whether the port is local or remote - any existing binding that requires a serial connection should directly also work with remote ports; is this realistic? If so, the only way to distinguish between local and remote ports (and thus do proper handling in the serial implementation) is a naming convention wrt the port names.

@msteigenberger
Copy link
Contributor Author

Imho it makes only sense to consider those "rfc2217" ports as serial ports, if they actually behave exactly the same, so from a binding perspective, it actually should not matter whether the port is local or remote - any existing binding that requires a serial connection should directly also work with remote ports; is this realistic? If so, the only way to distinguish between local and remote ports (and thus do proper handling in the serial implementation) is a naming convention wrt the port names.

Exactly. Intention is that it makes no difference from the binding perspective whether it's a remote (rfc2217) or local serial port. Therefore the user can decide that (and is the only one that can do that because it's depending on his setup). Of course if the binding (or protocol) makes some timing critical things, it could happen that it is not working with rfc2217.

maggu2810 added a commit that referenced this pull request May 17, 2018
@maggu2810
Copy link
Contributor

Please have a look at: https://github.com/eclipse/smarthome/compare/master...maggu2810:serial-port-provider-id?expand=1

If we assume that :// will not be part of a currently existing port identifier's name, this one could be a working solution.

@kaikreuzer
Copy link
Contributor

If we assume that :// will not be part of a currently existing port identifier's name, this one could be a working solution.

Imho this misses the point. What we need is a naming convention for "remote ports". Just like for local ports, it should not at all matter (to the user), which implementation is eventually used. @msteigenberger wants to add an implementation based on rxtx, but likewise, we could have other alternative implementations available that know how to remotely access telnet serial ports.
We therefore do not need/want any providerId in the port names. And as I said above, any implementation will actually fail to return anything in getIdentifiers(), because you simply do not know, which devices on your network might be available with such a feature (unless we could identify them through mDNS or similar, which I doubt).

@maggu2810
Copy link
Contributor

If an user do not want to take care of the implementation, he just does not need to specify :// and the part in front of.

If the user want to force a special implementation, he can specify the provider ID.

So, e.g. if we don't want to differ between the specific implementation, but differ between types, we just need to change provider ID to provider type and the first one that matches will be used.

@maggu2810
Copy link
Contributor

With the PR I linked above...

... there could be multiple serial port provider implementations be present at runtime and the user can choose which one to use or he can not care about.

... every serial port provider has an ID applied. This ID currently refers to the specific implementation of the provider for the serial port identifiers.
So, there is javacomm and rxtx.

We could also think about providing the type of the port (local, telnet com port), so the user can choose the type but do not need to care about the implementation.

Should we support that the user can:

  • define port name, unset type, not define implementation => the first serial port identifier that matches the port name is chosen (perhaps the normal use case for local serial port access where the user do not care)
  • define port name, define type, not define implementation => the first serial port identifier that matches the port name and that serial port provider type matches is chosen (e.g. the user wants to force telnet com port)
  • define port name, not define type, define implementation => the first serial port identifier that matches the port name and that serial port provider implementation matches is chosen (e.g. someone would access one bridge by implementation A and the other bridge by implementation b)
  • define port name, define type, define implementation

@msteigenberger
Copy link
Contributor Author

@maggu2810: I admit that your solution is very flexible, but it also adds some complexity to binding developers.

IMHO we just have to consider following cases:
The solution provides the serial port implementation (rxtx, javacomm)
The user provides the serial port type (local, telnet)

Considering that we don't need to make any specific API for the binding developer.
But we need naming convention for port names.
This can be e.g. <PORT_TYPE>://<PORT_SPECIFIC_NAME>
For local port we need to be backward compatible and can't follow this convention unfortunately.

@msteigenberger wants to add an implementation based on rxtx, but likewise, we could have other alternative implementations available that know how to remotely access telnet serial ports

Exactly. In my first thought implementation I did that but then moved to a less intrusive change for now. If we decide to go for the solution, I would change it.

Clearly, remote tcp ports that accept telnet connections and act like a serial port won't be discoverable and thus not be listed

Yes, we don't have a discovery mechanism for remote ports. That's a little drawback but Imho acceptable.

@gersilex
Copy link

gersilex commented Jun 1, 2018

you simply do not know, which devices on your network might be available with such a feature (unless we could identify them through mDNS or similar, which I doubt).

@bodiroba told me about https://github.com/Tknika/serial2tcp-gateway where he is working on a daemon that watches the USB ports for known devices, creates TCP listen ports that relay to the physical serial port of that devices and advertises them on mDNS. Worked well in my short test. That can make the devices discoverable.

We could also think about providing the type of the port (local, telnet com port), so the user can choose the type but do not need to care about the implementation.

I agree on that, too. We could start by expecting only one implementation per provider (remote-port only uses rfc2217, local port only uses nrserial, ...).

Thinking about the socket naming. I think that keeping the unix-style everything is a file system approach (/dev/ttyUSB0) and for remote ports the URL-style absolute location approach (see rfc1738 Section 3.1 (//serialTelnetServer:5555).

This would also allow the user to directly specify an implementation (a scheme) if he (or the binding) really insists on it by prepending the scheme like in said rfc1738 (rfc2217://serialTelnetServer:5556 or even rxtx:/dev/ttyUSB1).

All of the URIs can be matched easily:

  • ^\/\w+ matches /dev/ttyUSB0: Use classic serial behaviour, local serial port.
  • ^\/{2}(\w+):(\d+) matches //serialTelnetServer:5555: Use remote serial port with best matching provider/scheme
  • ^(\w+):\/{2}(\w+):(\d+) matches rfc2217://serialTelnetServer:5556: Use remote provider/scheme from URI.
  • ^(\w+):([^:]+)$ matches rxtx:/dev/ttyUSB1: Use local provider/scheme from URI.

Show these regular expressions on regex101.com

What do you think?

@kaikreuzer kaikreuzer changed the title SerialPort creation dependent on port name Add Serial-over-IP (rfc2217) support in ESH Serial API Jun 18, 2018
@kaikreuzer
Copy link
Contributor

IMHO we just have to consider following cases:
The solution provides the serial port implementation (rxtx, javacomm)
The user provides the serial port type (local, telnet)

Makes sense to me like that.

Imho @gersilex suggestion looks pretty good - it would be fully backward compatible and "//" would tell us that we are talking about a remote port. If we agree that (for now) we leave it to the solution to choose the implementation, I would suggest to not support the optional scheme for now at all (as it only complicates the implementation and it can be easily added later without breaking anything).

creates TCP listen ports that relay to the physical serial port of that devices and advertises them on mDNS. Worked well in my short test. That can make the devices discoverable.

Yes, but I am not sure how much that is gonna be used; we are talking here also about embedded devices that offer such a port, but where you cannot add any additional services yourself. So in any case, we will have to set limitToOptions=false for the serial port parameter, so that the user can freely enter a string.

@maggu2810 & @msteigenberger Would that solution be ok for you? If so, @msteigenberger, could you rework the PR accordingly?

@maggu2810
Copy link
Contributor

I would suggest to not support the optional scheme for now at all (as it only complicates the implementation and it can be easily added later without breaking anything).

We already have the regular expression and we just need to expose the name in the specific implementation. I don't think it will complicate the handling. Or do I miss some point?

If it is just a few lines I would prefer to add it as it will help to test serial provider implementations a lot.
E.g. one receiver on serial interface 1, another receiver on serial interface 2 and different implementations chosen for the bridges... Sure, this is nothing the normal user will do and the framework does not need to be used to test serial provider implementations -- but if it can be added without blowing the code base 😉

@kaikreuzer
Copy link
Contributor

I don't think it will complicate the handling. Or do I miss some point?

Well, you obviously need to be able to identify different implementations by ID (which influences the API), implement the selection logic, document what it is about, wonder whether you need end user documentation and what options you want to allow for solutions, add localisation and Paper UI support in case that solutions should be allowed to leave the selection of an implementation to the user, etc.

That's why I simply think that as long as we do not have any requirement for it, better just completely leave it out for now...

@msteigenberger
Copy link
Contributor Author

I've added a new commit.
However the strategy is now following:

  • Every implementation provider has now a name/id which is used as the scheme ("rxtx", "rfc2217", "javacomm")
  • The user can use the scheme to use the concrete implementation
  • If no implementation for provided scheme is found, an appropriate one with same type (local or remote) is used
  • The user can use just "//" to indicate remote connection
  • In latter case the first found remote implementation is used otherwise a local implementation is used

@kaikreuzer
Copy link
Contributor

Travis fails with:

[ERROR] Cannot resolve dependencies of MavenProject: org.eclipse.smarthome.io:org.eclipse.smarthome.io.transport.serial.rxtx.rfc2217:0.10.0-SNAPSHOT @ /home/travis/build/eclipse/smarthome/bundles/io/org.eclipse.smarthome.io.transport.serial.rxtx.rfc2217/pom.xml: See log for details -> [Help 1]

@msteigenberger
Copy link
Contributor Author

Travis fails with:

@kaikreuzer: yes, thats because the current used version of nrjavaserial (3.12.0-OH) has no rfc2217 packages included.
We need to integrate newer version of nrjavaserial to get that build working.

@kaikreuzer
Copy link
Contributor

We need to integrate newer version of nrjavaserial to get that build working.

You formulated this nicely fuzzy. Digging into it, I realized that NeuronRobotics/nrjavaserial#109 still isn't merged, which is a prerequisite for us - and apart from this, it would be really time for a new official release of nrjavaserial...

If no release is in sight, we would have to do a 3.13.0.OH build ourselves (but we would need somebody, who is capable of doing so...).

Copy link
Contributor

@maggu2810 maggu2810 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added some comments, thanks for your effort


/**
*
* @author MatthiasS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be something like: forname surename - Initial contribution (and API)

* @author MatthiasS
*
*/
@Component(immediate = true, service = SerialPortRegistry.class)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need an immediate activiation?

}

public Collection<SerialPortProvider> getPortCreators() {
return Collections.unmodifiableCollection(portCreators);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unmodifiable collection is a view and so every change to the portCreators will affect the user of the view.
This will potential lead to a CME if register or unregister is called while someone uses the view.

You should perhaps use something like Collection.unmodifiableCollection(new HashSet<>(portCreators)).

*/
@Reference(cardinality = ReferenceCardinality.MULTIPLE, policy = ReferencePolicy.DYNAMIC, policyOption = ReferencePolicyOption.GREEDY)
protected void registerSerialPortCreator(SerialPortProvider creator) {
this.portCreators.add(creator);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you are using a dynamic reference the member portCreators will be accessed by different threads.
You could e.g. use a concurrent collection or other synchronization statements.

*/
@NonNullByDefault
@Component
public class SerialPortManagerImpl implements SerialPortManager {
@Component(service = SerialPortProvider.class, immediate = true)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

immediate?

*/
@NonNullByDefault
@Component
public class SerialPortManagerImpl implements SerialPortManager {
@Component(service = SerialPortProvider.class, immediate = true)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

immediate?

*
* @author Markus Rathgeb - Initial contribution
* @author MatthiasS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above


/**
*
* @author MatthiasS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above


/**
*
* @author MatthiasS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

*
* @author MatthiasS
*
* @param <T>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the interface does not contain a generic parameter

@kaikreuzer
Copy link
Contributor

Good news, we should soon have a 3.14.0 release of nrjavaserial, see NeuronRobotics/nrjavaserial#109!

@kaikreuzer
Copy link
Contributor

I was just checking how we can upgrade the tp to nrjavaserial 3.14.0. We imho cannot add it to our ESH 3rd party p2 site as it is an LGPL "works-with" dependency. We cannot consume it directly from Maven Central as we need it on a p2 site. So I guess, I will have to best deploy a p2 site to the openhab bintray, just where we have the forked version right now as well.

@maggu2810, @SJKA & @htreu Any better/other idea?

@maggu2810
Copy link
Contributor

$ find extensions/binding/ -type f -name "*.java" | xargs grep -l 'import org.eclipse.smarthome.io.transport.serial'
extensions/binding/org.eclipse.smarthome.binding.serialbutton/src/main/java/org/eclipse/smarthome/binding/serialbutton/handler/SerialButtonHandler.java
extensions/binding/org.eclipse.smarthome.binding.serialbutton/src/main/java/org/eclipse/smarthome/binding/serialbutton/internal/SerialButtonHandlerFactory.java
extensions/binding/org.eclipse.smarthome.binding.bluetooth.bluegiga/src/main/java/org/eclipse/smarthome/binding/bluetooth/bluegiga/handler/BlueGigaBridgeHandler.java
extensions/binding/org.eclipse.smarthome.binding.bluetooth.bluegiga/src/main/java/org/eclipse/smarthome/binding/bluetooth/bluegiga/internal/factory/BlueGigaHandlerFactory.java

@codeaholics
Copy link

Ha! Thanks for that. I’m away from home right now with just mobile phone access, so wanted to make sure I had as much information as possible to hand before I got back so I could hit the ground running!

@kaikreuzer
Copy link
Contributor

None of these hits is tested yet with rfc2217 support - in theory they should work, though.
@msteigenberger should probably the best person to talk about how to enable a binding to correctly work with rfc2217.

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/share-z-wave-dongle-over-ip-usb-over-ip-using-ser2net-socat-guide/34895/85

@dastrix80
Copy link

Hi All, would this fix the issues I see when the console displays I/O errors because the virtual USB (mapped using ser2net & socat) has been removed or restarted? ie would the zwave binding be able to reconnect if that mapping was removed?

Right now, as soon as the USB mapping is removed a restart of the Zwave binding is required.

Sorry thats about as best I can explain the issue.

More info here: https://community.openhab.org/t/share-z-wave-dongle-over-ip-usb-over-ip-using-ser2net-socat-guide/34895/85

Thank you!

@maggu2810
Copy link
Contributor

would this fix the issues I see when the console displays I/O errors because the virtual USB (mapped using ser2net & socat) has been removed or restarted?

I don't remember there has been any change in this PR that is related to your point(s).

@codeaholics
Copy link

Yes, because OH is no longer opening a serial port. There is no ser2net or socat on the OH end - only on the remote device (ZWave stick) end. OH then natively talks to that using TCP/IP. So, assuming that this plugin correctly deals with connection drops and reconnects, then yes, this would solve the issue.

@dastrix80
Copy link

Thats great news! I hope this gets implemented in the stable release soon!

@thefathefa
Copy link

Hi,

I was about to go this way : https://community.openhab.org/t/share-z-wave-dongle-over-ip-usb-over-ip-using-ser2net-socat-guide/34895 because I need such a setup but if there is a more "mainstream openhab way" that is coming, I'd rather wait... Is it possible to have an rough view of when this will be available in openhab 2.4 branches i.e. within weeks or months or quarters?
Thanks,

@msteigenberger
Copy link
Contributor Author

msteigenberger commented Oct 7, 2018

It is already available in latest snapshots of openhab. I tested it in my binding:
https://github.com/openhab/openhab2-addons/pull/2167

@dastrix80
Copy link

Hi msteigenberger, can you tell me how/ what this changes? I'm a little confused over how this helps the issue or what is now different with the latest snapshot. I'm constantly experiencing issues with my zwave stick connected to a remote device using socat/s2net and when that device has a restart, OH2 looses connection to the usb stick and requires a restart of the zwave binding.

@htreu htreu added this to the 0.10.0 milestone Oct 30, 2018
@msteigenberger
Copy link
Contributor Author

@dastrix80: You don't need socat anymore that is located at the same machine as your OH2 instance. Instead OH2 now directly supports RFC2217 (which socat/ser2net is using).
When your remote device now disconnects/restarts, OH2 will fail to read the inputstream. This needs to be handled by the binding and a reconnection needs to be done.

@dastrix80
Copy link

dastrix80 commented Oct 31, 2018

Which version of the snapshot was it added to? So I just change the Zwave binding to rfc://ip:port to make this work?

@msteigenberger
Copy link
Contributor Author

msteigenberger commented Nov 1, 2018

I'm not sure about the z wave binding. From what I can see, it's not yet integrated. Sorry, didn't recognize that you talked about a specific binding.
After it would be, you have to change the port like so: 'rfc2217://ip:port'

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/3rd-party-bluetooth-binding-beta-testers-needed/38492/738

@lolodomo
Copy link
Contributor

I will try this new feature with the Sony projector I am developing.
What is finally the required syntax ? Is the prefix rfc2217: required ?

Regarding the binding, is it necessary to suppress the context serial-port from the thing configuration setting to let the user enter a value such as rfc2217://192.168.0.10:3333 ?

@msteigenberger msteigenberger deleted the serialport_factories branch January 25, 2019 17:22
@msteigenberger
Copy link
Contributor Author

// is possible and just means that it uses any remote implementation. As only rfc2217 is available, it would pick this. To be more precise you should rather use rfc2217://

You don't need to suppress the context in the configuration, as it would allow to enter manual values as well.

@lolodomo
Copy link
Contributor

Thank you for your answer.
As I got difficulty to make it work with a new binding, can you tell me what bindings were confirmed to work with Rfc2217 ?
Regarding ser2net, shall I declare telnet or raw protocol ?

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/zwave-binding-updates/51080/662

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/share-z-wave-dongle-over-ip-usb-over-ip-using-ser2net-socat-guide/34895/180

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/preparation-for-2-5m2/75738/26

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/share-z-wave-dongle-over-ip-usb-over-ip-using-ser2net-socat-guide/34895/238

@openhab-bot
Copy link
Contributor

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/share-z-wave-dongle-over-ip-usb-over-ip-using-ser2net-socat-guide/34895/243

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants