Supporting 'informed user choice' in port selection

[JayBeavers]

I wanted to call out an idea @marcoscaceres posted in a separate thread:

I love this prototype and how it empowers the user to make an informed device choice. In a separate discussion, we're working out the details about exposing port properties that can be used to infer device capabilities on the other end of the port. Assuming we work this through and reach the ability to say, 'this port connects to a Sphero robot' or 'this port connects to an Arduino microcontroller with the Firmata protocol', how can we enable the building of a UI like above via our API?

I think it's going to be more complicated than a requestPorts() call popping a generic listbox. Do we want the webpage to be able to skin the choice box (e.g. if the Manufacturer is Orbotix and the Product contains Sphero, show an icon for a Sphero robot next to the port name)? I understand there's security risk here of masquerading the choice, but the benefit of enabling informed choice outweighs the risk imho.

[fbender]

All this is out-of-scope of the API (as any UI and user interaction), and the UA vendor is free to implement this however he wants to. Also, we don't need to expose all the properties in the API the above sketch would need to display in such a way (transferring this to Serial ports): The UA can check and use more data than the API provides. Also, adding hints in the API solely for supporting UI (re: manufacturer icons) is not something we should do (the UAs can do that on their side, e.g. match up manufacturer in a db).

While we can add some fuzzy implementation recommendations, I think the scope of this issue is too broad to begin with, so I'd like to close it. Any objections?

[marcoscaceres]

Just a quick note: @fbender, technically you are not wrong (in that it's outside the scope of what is exposed to script), but you are not fully right either 😸 ... we still need to evaluate if it's even possible to generate the UI above in a cross-platform manner. This is important to Mozilla and possibly other implementers because it underpins the security model for this API, so I would be in favor of us leaving this open to do the investigation - thus, this is requirement that should be captured in the spec.

However, you have made a good point that the problem needs to be viewed differently. There is "implementation side" and "developer side". I'll describe this in more detail in a follow up.

[fbender]

we still need to evaluate if it's even possible to generate the UI above in a cross-platform manner

Maybe we actually need a different UI for different platforms, e.g. on UNIXoid show TTY0, manufacturer, serial#, on Windows show COM0, manufacturer, pnpId (or whatever Windows has what the others don't). The goal is not cross-platform UI, but cross-platform API, while each platform does what it can do (while not relying on this API to get these infos) to aid the user in selecting the right port(s).

[marcoscaceres]

@fbender in an advanced view, yes. But I (as an end-user) am not interested in seeing all that stuff (TTY0, manufacturer, serial#, bla bla... yuck) - I just want to see a picture of my Arduino or 3D printer and click it... Seeing the other metadata should be an optional thing for advanced users.

Where the "cross-platform UI" thing comes into play is being able to show an icon for an Arduino or for a 3D printer, etc. For that, we need to make sure that the relevant data is available across multiple platforms - it will most likely be, because the USB standard provides it (probably BlueTooth too, but don't know anything about that standard yet).

[fbender]

But this is not what the spec is about!?

Sure, a good-looking UI is something for the users, though I don't believe that we can provide any of that. The devices themselves need to provide this identifying info. I imagine the UAs query a database that returns some pretty pictures and descriptions for some combination of manufacturer and device ID, but again, this is not a problem we can solve (unless we want to provide such a database as part of the group work).

[marcoscaceres]

@fbender I think we are talking past each other - what you are saying is correct (it has no bearing on the JavaScript API) - but it's only one side of the coin: If it's not possible for an implementer to generate such an UI, or if it's too hard/expensive or whatever, then that's a barrier to implementation. We need to think of the whole solution even if this just ends up being a sentence in the spec itself.

When I go back to Mozilla with the spec and they ask me, "but can this be done on Windows, MacOS, Linux, Android, etc.?" I can point them to this bug and say, "yes, the node-serial guys showed that they could get all the info from USB, etc. etc. in a cross platform way."

[marcoscaceres]

We've seen that we can get the list of manufacturers and products, for instance:
http://www.linux-usb.org/usb.ids

[reconbot]

I'd prefer enough of an api that developers can do their own fuzzy matching
and prefix lookups. I doubt this is something UA vendors will be able to
keep up with as well as user space could.
On Nov 13, 2013 2:27 PM, "Marcos Caceres" [email protected] wrote:

We've seen that we can get the list of manufacturers and products, for
instance:
http://www.linux-usb.org/usb.ids

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-28425507
.

[marcoscaceres]

@reconbot likely; that's another angle I'd like to see us explore as part of this.

[JayBeavers]

The way I was envisioning it is that we may design this api such that the web app would be allowed to generate the selection ui and could do the heavy lifting to translate the port properties into pretty pictures.

If however we think the browser has to generate the UI that appears between ..requestPorts() and the array of user selected ports being provided to the web app, I think we're going to have a tough time seeing an end result as usable as the midi sketch above. While Midi has a relatively constrained/consistent list of devices/pictures to support, USB has tens of thousands and changes over time. I think we're down this path of 'database of vid/pid -> friendly strings & images' being either a part of or associated with this spec.

Could we go with a simpler 'prompt for permission to use serial ports' approach that doesn't need the user to select the 'right' port for the app and then let the web app handle a generating a nice port selection UI?

[fbender]

The way I was envisioning it is that we may design this api such that the web app would be allowed to generate the selection ui and could do the heavy lifting to translate the port properties into pretty pictures. […]
Could we go with a simpler 'prompt for permission to use serial ports' approach that doesn't need the user to select the 'right' port for the app and then let the web app handle a generating a nice port selection UI?

IMO, that is not possible without breaking #4 and the security model, because as soon as we let a WebApp/website access all ports, we can't stop it from tinkering with all ports/devices. Also, users may want to be sure that a particular App/site/instance can only access the one port they specified.

If the interface (i.e. RS232, USB, BT) provides enough information, what would stop the UA from using this information in a sensible manner? … On a related note, what were/are the WebUSB guys doing about that? @marcoscaceres, can you ask them?

[tzikis]

I'm also in favor of pushing the UI selection to the website. Having this in the browser is hugely limiting, and confusing as well. The only reasoning behind that would be for security-wise, so as to limit access to specific websites for specific devices but in practice, this is not only a pain in the ass, but confusing, and pointless as well (i.e. when I give access to COM1 but not COM2, and next time I reboot, the devices swap places so the permissions are reversed). not to mention the fact that user doesn't even necessarily know, or care, or is even able to figure out, which device is which.

IMO, given the above, and other reasons, the only practical solution is a universal request for serial access, which the user provides, and then the website decides how to provide that info to the user.

Here's an example of what we do with codebender (via an NPAPI plugin currently, but a very good use case for this API).

In our case, we also decide to hide some of the serial ports from the user, since they can't use them. You can see we also provide the USB Flash button (which opens, reads/writes to the port and closes it, sometimes more than once, and at different baudrates), and the serial monitor which performs the explicit functions the user requests.

there are also other good use cases for that, for which i could go more detail, but let's just say a 3d printing app might want to show you the image of each the 3d printer, based on pndId, and hide some of the devices as well, or scan all ports and only show those who recognize a specific communication protocol (i.e. CISCO routers only).

[marcoscaceres]

IMO, given the above, and other reasons, the only practical solution is a universal request for serial access, which the user provides, and then the website decides how to provide that info to the user.

@tzikis,@JayBeavers that's really valuable feedback... ok, so, let's make it our working assumption that there will be a single blanket access request to ports and we leave it to the implementation to sort it out (assuming a door hanger thing, like @fbender said). We can tighten this later if security folks complain.

This also allows us to do what @JayBeavers suggested, but build a really nice selection UI within the webpage itself.

[fbender]

when I give access to COM1 but not COM2, and next time I reboot, the devices swap places so the permissions are reversed

The permissions will be reset by then.

not to mention the fact that user doesn't even necessarily know, or care, or is even able to figure out, which device is which.

Please don't forget that we are probably targetting h/w hackers at first. We can always expand our scope once we have a working solution, though for the beginning, let's stick with a simple design and high security, please!

Here's an example of what we do with codebender

Can you provide some (pseudo) code how you do this and display these information? Can you also map this to what we currently have, please?

USB Flash button (which opens, reads/writes to the port and closes it, sometimes more than once, and at different baudrates)

Can you elaborate? When I hear "USB Flash" I think of flashing a firmware …

serial monitor which performs the explicit functions the user requests

I don't understand that, too.

hide some of the devices as well, or scan all ports and only show those who recognize a specific communication protocol

We could add something to the request API here, e.g. strings to match some properties to (matchManufacturer = /arduino/i etc.) – that was already discussed in #4: So the UA can prioritize matching ports in the UI.

[tzikis]

The permissions will be reset by then.

that would be very inefficient in itself, but it's not just about resetting. if i disconnect the device, there is no guarantee it will be recognized as the same one (in fact, on windows, chances are against it). this is also true when pressing i.e. an arduino's reset button. a leonardo, for example, needs to be reset twice just to program it, and instead of trying to access the same port the second time, you look for a newly-connected port and connect to it.

Please don't forget that we are probably targeting h/w hackers at first. We can always expand our scope once we have a working solution, though for the beginning, let's stick with a simple design and high security, please!

i know more about serial ports than i would ever want, or wish upon anyone to have to learn about. i still have no ability to chose between COM1, COM3, COM5 and COM23 when I'm using windows. /dev/ttyUSB0 vs /dev/ttyUSB1 doesn't particularly help either. did i mention /dev/ttyS1, /dev/ttyACM0 and /dev/usbmodemRANDNUM?

Can you provide some (pseudo) code how you do this and display these information? Can you also map this to what we currently have, please?

using the NPAPI plugin gives us access to the OS functions. so it's basically something like this

ports = os.listPorts()
foreach ports:
 if port is could be an arduino
 then add it to the combo box

Can you elaborate? When I hear "USB Flash" I think of flashing a firmware …

well, it's because it's a fuzzy term, and the arduino guys don't help by using the word upload to refer to flashing. and flashing for flashing a bootloader in particular. yes, essentially, it send the new binary to the device over serial/USB, and the device loads the new program into memory, and executes it.

serial monitor which performs the explicit functions the user requests

I don't understand that, too.

yeah i wasn't very precise i guess. i meant that the open opens the port, disconnect disconnects, read reads etc...

[fbender]

(I fixed up your comment WRT quote formatting, please leave blank lines between quotes and non-quotes (and subquotes), otherwise GitHub does weird things.)

if i disconnect the device, there is no guarantee it will be recognized as the same one (in fact, on windows, chances are against it). this is also true when pressing i.e. an arduino's reset button.

A lost connection would reset the permission.

a leonardo, for example, needs to be reset twice just to program it, and instead of trying to access the same port the second time, you look for a newly-connected port and connect to it.

Uhh :( that can't work efficiently with a tight security model. Unless you install your App as an App and grant it (on install) the permission to always be able to access all ports – that would be possible for the UA to implement (though for "normal" websites, this is definitely not an option).

i still have no ability to chose between COM1, COM3, COM5 and COM23 when I'm using windows. /dev/ttyUSB0 vs /dev/ttyUSB1 doesn't particularly help either. did i mention /dev/ttyS1, /dev/ttyACM0 and /dev/usbmodemRANDNUM?

But UA could and should show any information it can aquire, e.g. manufacturer, in the permission prompt. What does the first select in your screenshot above contain? Manufacturer? Translated device ID? (see next quote-question)

using the NPAPI plugin gives us access to the OS functions. so it's basically something like this

What are those OS functions? Is if port is could be an arduino part of these OS APIs, or where/how do you do that, and what information do you need so you can display the UI in the screenshot above?

yes, essentially, it send the new binary to the device over serial/USB, and the device loads the new program into memory, and executes it.

Off topic: Can this functionality be done through the limited API we currently envision? If not, please file an issue with extensive explanation.

[tzikis]

A lost connection would reset the permission.

ouch. you have to understand how often that happens. let's just say, for the arduino, about 1-5 minutes when developing something. consider bluetooth as well.

the leonardo case is quite specific, but it is a user case, and definitely won't work well with tight security

in regards to selecting the correct port, in our case we only show ports that follow a naming pattern. on Windows, you only get COM1, COM2, etc, so we show everything. on Linux and OS X, since we care about arduino only, so we use ports that start with /dev/ttyUSB or /dev/ttyS on Linux, and /dev/tty.usbmodem or /dev/tty.FTDI (excluding the duplicate records i.e. /dev/cu.usbmodem for /dev/tty.usbmodem) for OS X. mind you, we do not identify devices (for example, two arduinos might have the exact same info, while being completely different devices, even talking with different protocols and baudrate settings), we are simply able to minimize the list of devices we might be interested in from 5-20 to 1-5.

Off topic: Can this functionality be done through the limited API we currently envision? If not, please file an issue with extensive explanation.

yes. it's simply a well-defined and well-timed sequence of serial.open() .read() .write() and .close() calls. in fact i intend to suggest it as a demanding use case once i am not in semi-traveling mode

[reconbot]

@fbender a lot of serial devices will disconnect and reconnect, even the usb ones. Devices with USB serial interfaces that drop the USB connection during normal operation that I've used include arduinos, 3d printers, displays, and rfid sensors. So it's not an arduino special case.

I don't like the idea of authing a website to use all serial ports at all times but its a possible outcome.

I'd like to know more about identifying reconnected devices on windows, is the path really not predictable? On OSX this isn't a problem unless you hit multiple resetting devices.

[fbender]

I don't like the idea of authing a website to use all serial ports at all times but its a possible outcome.

It's either by-port or all :(. Especially with the use cases you are giving.

[dhylands]

On Wed, Nov 13, 2013 at 3:37 PM, Francis Gulotta [email protected]
wrote:

@fbender a lot of serial devices will disconnect and reconnect, even the
usb ones. Devices with USB serial interfaces that drop the USB connection
during normal operation that I've used include arduinos, 3d printers,
displays, and rfid sensors. So it's not an arduino special case.

I don't like the idea of authing a website to use all serial ports at all
times but its a possible outcome.

I'd like to know more about identifying reconnected devices on windows,
is the path really not predictable? On OSX this isn't a problem unless you
hit multiple resetting devices.

My epxerience with windows, is that most USB devices, when plugged into the
same physical USB port, retain the same COMx
However, moving the USB device to a different physical USB port will cause
it to get a new COMx

Under linux, the ttyUSBx seems to be assigned on the order that the devices
are enumerated (which translates to the order that they're plugged in,
until you reboot your machine at which point its the order that the ports
get enumerated).

All the fancy recognition stuff for Arduinos only works for certain types
of arduinos.

This device: http://arduino.cc/en/Main/ArduinoBoardUno uses an ATMega16U2
to do its USB interface, so it can present custom strings during
enumeration.
This device: https://www.sparkfun.com/products/11575 (which looks more like
the classic Arduino) will look exactly the same to the OS as this device:
https://www.sparkfun.com/products/718
This device: https://www.sparkfun.com/products/11303 has only logic-level
serial interface, and will look like the USB-to-serial adapter used to hook
it up

So even with something "simple" like an Arduino, we can only do the fancy
UI with the first case, and not the 2nd or 3rd cases.

Any device which has an RS232 interface, as opposed to a USB interaface,
will look like the serial adapter used to connect to it.

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

[tzikis]

I'll agree with Dave, except that in my experience, windows devices don't retain the same COM port when connected. Generally with more than one you can never be sure. Even with one, in some cases, the OS will assign a different port, often because it hasn't released the old one first.

This happens quite often with the Leonardo for example. When you program it, it reconnects, gets a new port name because the old one hasn't been released yet, then the old port is released, and all of that happens so fast that you don't realize why the port name changed for no "obvious" reason.

On 14 Nov 2013, at 03:18, Dave Hylands [email protected] wrote:

On Wed, Nov 13, 2013 at 3:37 PM, Francis Gulotta [email protected]
wrote:

@fbender a lot of serial devices will disconnect and reconnect, even the
usb ones. Devices with USB serial interfaces that drop the USB connection
during normal operation that I've used include arduinos, 3d printers,
displays, and rfid sensors. So it's not an arduino special case.

I don't like the idea of authing a website to use all serial ports at all
times but its a possible outcome.

I'd like to know more about identifying reconnected devices on windows,
is the path really not predictable? On OSX this isn't a problem unless you
hit multiple resetting devices.

My epxerience with windows, is that most USB devices, when plugged into the
same physical USB port, retain the same COMx
However, moving the USB device to a different physical USB port will cause
it to get a new COMx

Under linux, the ttyUSBx seems to be assigned on the order that the devices
are enumerated (which translates to the order that they're plugged in,
until you reboot your machine at which point its the order that the ports
get enumerated).

All the fancy recognition stuff for Arduinos only works for certain types
of arduinos.

This device: http://arduino.cc/en/Main/ArduinoBoardUno uses an ATMega16U2
to do its USB interface, so it can present custom strings during
enumeration.
This device: https://www.sparkfun.com/products/11575 (which looks more like
the classic Arduino) will look exactly the same to the OS as this device:
https://www.sparkfun.com/products/718
This device: https://www.sparkfun.com/products/11303 has only logic-level
serial interface, and will look like the USB-to-serial adapter used to hook
it up

So even with something "simple" like an Arduino, we can only do the fancy
UI with the first case, and not the 2nd or 3rd cases.

Any device which has an RS232 interface, as opposed to a USB interaface,
will look like the serial adapter used to connect to it.

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com
—
Reply to this email directly or view it on GitHub.

[tzikis]

Guys, it looks to me that we've grown stagnant, due in some part to a bit of bikeshedding.

I'd like to get things rolling again if I can. The way I see it, based on the comments so far, the only alternative is what @marcoscaceres suggested. Unless anyone has an objection and a better way to tackle this, I'll go ahead and create a pull request with the necessary changes and then close this issue

[JayBeavers]

@tzikis I concur, thanks for moving us forward.

[marcoscaceres]

@tzikis, that's ok with me.

Please do take a look here tho:
#27

That's been sitting there for a while and it might be worth fixing up.

I'll try to re-engage with this spec soon also. I've added it as one of my Q1 goals for my work at Mozilla.

[fbender]

Yeah, please go ahead!

#27 is on my TODO for a while, I may get to it later next week.

[tzikis]

ah, cool.

from what i see, it's good to go, apart from an update on the "Security considerations" section (lines 273-290). i should be able to add a new commit to requestports and the pull request will be updated, right?

there's also some concerns by @JayBeavers in there, which i'm not in a position to decide

[reillyeon]

This was fixed by #105 with the specification of requestPort(), which behaves similarly to the proposal in the issue description) and getPorts(), which will return all ports that a site has previously been granted permission to access. User agents are free to define out-of-band mechanisms for granting a site permission to access ports such as administrator settings. Such ports would be automatically present in the list returned by getPorts().