Native Messaging signaling server

[guest271314]

I am currently using Native Messaging to stream system audio output ("What-U-Hear") to the browser.

Reading examples/signaling-server-python and examples/copy-paste I should be able to incorporate the pattern into https://github.com/guest271314/captureSystemAudio/blob/master/native_messaging/capture_system_audio/capture_system_audio.py without using WebSocket.

The issue with my current approach is I have to insert an <iframe> into the arbitrary web page to keep the Native Messaging connection active because ServiceWorker becomes inactive in 5 minutes and 15-30 seconds. I would prefer to not use an <iframe> at all.

WebRTC can be executed on any web page.

I have no experience with C++. I do not know how to substitute Python's subprocess to execute a shell command, in this case PulseAudio's parec and stream STDOUT from that command to the data channel, terminating the command when the data channel is closed on the client (browser) side. I read https://stackoverflow.com/questions/478898/how-do-i-execute-a-command-and-get-the-output-of-the-command-within-c-using-po today, though I am curious how you would achieve that requirement?

Thanks in advance.

[paullouisageneau]

This is a bit out of the libdatachannel scope, but there is no portable way to do so in C/C++, you have to use popen on POSIX systems and CreateProcess on Windows.

However, a portable C++ program could be made to read stdin and send in a Data Channel. You would then pipe parec into the program with the shell or Python.

[guest271314]

However, a portable C++ program could be made to read stdin and send in a Data Channel. You would then pipe parec into the program with the shell or Python.

How to do that?

This is a bit out of the libdatachannel scope

Should I close this issue?

[paullouisageneau]

Should I close this issue?

No, it's OK.

How to do that?

After connecting the Data Channel (like in the examples), you need to do something like this:

const std::size_t bufferSize = 4096;
std::byte buffer[bufferSize];

while(std::cin.good()) {
    // Read from stdin
    std::cin.read(buffer, bufferSize);
    std::size_t count = std::cin.gcount();
    if(count == 0)
        break; // end of file
    
    // Send in the Data Channel
    dc->send(buffer, count);
}

dc->close();

[guest271314]

Would I place that in

libdatachannel/examples/copy-paste/offerer.cpp

Lines 112 to 122 in 0975afa

	case 3: {
	// Send Message
	if (!dc->isOpen()) {
	cout << "** Channel is not Open ** ";
	break;
	}
	cout << "[Message]: ";
	string message;
	getline(cin, message);
	dc->send(message);
	break;

Again, I have no experience with C++. Which part is the STDOUT from one of the answers at the SO question linked?

  while (std::getline(proc.out(), line))
    std::cout << "stdout: " << line << '\n';

There is no EOF. No file is being read. The stream is live output of what you hear output to speakers and headphones.

[paullouisageneau]

Thinking about it, what you want to do is quite tricky, you can't just drag-and-drop some code into an example. For instance, how do you plan to pass the WebRTC signaling messages back-and-forth to the C++ process in order to open the Data Channel?

If you have no experience in C or C++, I'd recommend aiortc, a WebRTC Python library.

There is no EOF. No file is being read. The stream is live output of what you hear output to speakers and headphones.

End-of-file happens when stdin is closed, e.g. when parec is killed.

[guest271314]

Here I use clipboard as signaling server https://gist.github.com/guest271314/04a539c00926e15905b86d05138c113c, here an <iframe> https://plnkr.co/edit/2Y9B97LmGbl5XR0d?preview.

The expected flow is user clicks extension icon the browser WebRTC recv-only data channel is constructed, the C++ data channel is constructed as send-only and shell script is executed piping output to the data channel, necessary signaling occurs via appropriate means, when browser side closes the data channel, shell script terminates, C++data channel terminates. I should be able to repeat that procedure multiple times, ideally without ever writing to a file.

[guest271314]

Thinking about it, what you want to do is quite tricky, you can't just drag-and-drop some code into an example.

Technically, I can.

I can also write to and read from files if that would be simpler.

On the browser side I can read (static) local files listed in "web_accessible_resources" in the extension directory with fetch(). I can also write files with File System Access API (though that requires a prompt which can trim capture of audio from the front of the stream/recording if user is not quick enough with click, and I prefer only 1 click to initialize, start and stop the stream.

Thus I can read the send-only SDP of C++data channel. I could also use the Native Messaging host to write the browser recv-only SDP which C++side reads.

Where I have no knowledge in C++ is where to pipe the data to the data channel.

I looked at aioquic. Your library appears to be self-contained and minimal in size.

In lieu of closing the issue, if you have/find the spare time perhaps you could add an example of streaming STDOUT from a popen() call to the C++ data channel?

[guest271314]

If you have no experience in C or C++, I'd recommend aiortc, a WebRTC Python library.

I had no experience with Python either until I began experimenting with WebTransport, which I have not been able to live stream with.

(Ideally I would just use Native Messaging (12MB when running) directly at any origin, however that (currently) is restricted by Chromium source code ungoogled-software/ungoogled-chromium#1800, or something like Kagami didwith mpv and HTML <embed> element https://github.com/Kagami/mpv.js)

Have a great day.

[paullouisageneau]

Sure, if you can adapt the signaling then that's fine.

I looked at aioquic. Your library appears to be self-contained and minimal in size.

Yes thanks, I tried to make it that way.

In lieu of closing the issue, if you have/find the spare time perhaps you could add an example of streaming STDOUT from a popen() call to the C++ data channel?

It would look like this:

#include <stdexcept>
#include <stdio.h>
[...]
const char *command = "parec";
File *file = popen(command, "r");
if (!file)
    throw std::runtime_error("popen failed");

try {
    const std::size_t bufferSize = 4096;
    std::byte buffer[bufferSize];
    size_t count;
    while((count = fread(buffer, 1, bufferSize, file)) > 0)
        dc->send(buffer, count);

} catch(const std::exception &e) {
    std::cerr << e.what() << std::endl;
}

pclose(file);

[guest271314]

Thanks. I'll need a few days to try C++. I got to

$ make USE_GNUTLS=1 USE_NICE=0 NO_MEDIA=1 NO_WEBSOCKET=1

which throws

Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
...
src/hash.c:22:10: fatal error: nettle/md5.h: No such file or directory
   22 | #include <nettle/md5.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:41: src/hash.o] Error 1
make[1]: Leaving directory '/home/user/libdatachannel/deps/libjuice'
make: *** [Makefile:126: libjuice.a] Error 2

I could not locate a solution for this at #37.

[paullouisageneau]

To compile with GnuTLS, you need headers for both GnuTLS and its dependency Nettle. On Ubuntu or Debian you can install them with:

$ apt install libgnutls28-dev nettle-dev

[guest271314]

This is quickly becoming expensive. We are already at 1GB for libdatachannel folder. I am basically trying to determine if the <iframe> approach is the least expensive. 3.7MB for the <iframe>, 12.5MB for the Python Native Messaging host, 5.7MB for parec.

Nonetheless I will test libdatachannel later today or tomorrow.

[paullouisageneau]

This is quickly becoming expensive. We are already at 1GB for libdatachannel folder.

The whole source repository with submodules, git history, media examples, and temporary compilation objects may grow around 500MB, but the final compiled library (typically libdatachannel.so) should be only around 3-4MB.

Note the biggest thing in the repository is actually the git history of the json submodule, which you don't need. Therefore, you can reduce the size under 200MB by recreating a fresh clone and fetching the submodules with --depth=1:

$ git clone https://github.com/paullouisageneau/libdatachannel.git
$ cd libdatachannel
$ git submodule update --init --recursive --depth=1

[guest271314]

I installed nettle.

$ git submodule update --init --recursive --depth=1
...
$ make USE_GNUTLS=1 USE_NICE=0 NO_MEDIA=1 NO_WEBSOCKET=1

Still throwing error

/usr/bin/ld: hmac.c:(.text+0xd6): undefined reference to `nettle_hmac_sha256_update'
/usr/bin/ld: hmac.c:(.text+0xe6): undefined reference to `nettle_hmac_sha256_digest'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:55: tests] Error 1
make[1]: Leaving directory '/home/user/libdatachannel/deps/libjuice'
make: *** [Makefile:126: libjuice.a] Error 2

[paullouisageneau]

What is your OS? How did you install nettle?

You could use the main build method with CMake if the Makefile does not work for you.

[guest271314]

Linux 5.4.0-42-lowlatency.

This built your library

cmake -B build -DUSE_GNUTLS=1 -DUSE_NICE=0 -DNO_WEBSOCKET=1 -DNO_MEDIA=1

websocket.cpp, websocketserver.cpp were built anyway.

I'll begin experimenting.

[guest271314]

How did you install nettle?

$ sudo apt install libgnutls28-dev nettle-dev

[guest271314]

Is the examples/web example complete? I built again with WebSocket support and examples/web is not built.

I can use the WebSocket/web example as a pattern to use with Native Messaging. In the meantime I will try to adapt copy-paste.

[paullouisageneau]

How did you install nettle?

$ sudo apt install libgnutls28-dev nettle-dev

Is it on Ubuntu? I can't reproduce neither on Debian nor on Arch Linux.

websocket.cpp, websocketserver.cpp were built anyway.

This is expected, the build goes through everything irrelevant of the flags for the sake of simplicity but the corresponding code is not compiled if disabled.

Is the examples/web example complete? I built again with WebSocket support and examples/web is not built.

The web example is just a web page with an equivalent JS client that you can use to connect to the libdatachannel one. Please refer to the corresponding README.md file to run it in a browser. Also, note that like the libdatachannel client example, it requires one of the example signaling servers to be running.

[guest271314]

Yes, Ubuntu (studio).

Note, I'm pretty sure the WebSocket / web example is broken. When I copy/paste move /web to /build and copy signaling-server.py to /web in /build then run

$ python3 signaling-server.py
we get the error

Traceback (most recent call last):
  File "signaling-server.py", line 24, in <module>
    import websockets
ModuleNotFoundError: No module named 'websockets'

No (Python) websocket module exists in the repository that I could locate. Were you intending on converting websocket.cpp to Python module?

when I run python3 -m http.server --bind 127.0.0.1 8080

we get 404

WebSocket connection to 'ws://localhost:8080/xKBx' failed: Error during WebSocket handshake: Unexpected response code: 404

for with localId appended.

The ports are mismatched as well

const url = ws://localhost:8000/${localId};

though we are running on 8080.

What happens when you try to run the /web example with signaling-server.py on your machine?

I'll try to adapt the copy-paste code.

[paullouisageneau]

Note, I'm pretty sure the WebSocket / web example is broken. When I copy/paste move /web to /build and copy signaling-server.py to /web in /build

You don't need to copy anything to build, just run the programs from examples.

then run

$ python3 signaling-server.py we get the error

Traceback (most recent call last):
  File "signaling-server.py", line 24, in <module>
    import websockets
ModuleNotFoundError: No module named 'websockets'

No (Python) websocket module exists in the repository that I could locate. Were you intending on converting websocket.cpp to Python module?

You must install the websockets Python module:

$ sudo apt install python3-websockets

when I run python3 -m http.server --bind 127.0.0.1 8080

we get 404

WebSocket connection to 'ws://localhost:8080/xKBx' failed: Error during WebSocket handshake: Unexpected response code: 404

for with localId appended.

The ports are mismatched as well

const url = ws://localhost:8000/${localId};

though we are running on 8080.

The ports are not mismatched: the HTTP server serves pages to the browser on port 8080, but the signaling server listens on port 8000. Of course it breaks since you changed the port to the wrong one.

In the first place, the signaling server needs to be properly started, and then, don't change the signaling server port.

[guest271314]

You must install the websockets Python module:

$ sudo apt install python3-websockets

That should be in the documentation.

Right now I'm trying to adapt C++ copy-paste and just use Native Messaging host https://stackoverflow.com/a/26583974 instead of installing more packages.

[guest271314]

Using Native Messaging I should be able to do something like

// get SDP
chrome.runtime.sendNativeMessage(id, {"message":""}, (message) => {
   // message: SDP
   // I can get this to the web page where data channel is defined
});
// send SDP from data channel in web page
chrome.runtime.sendNativeMessage(id, {"message":sdp}, (message) => {
  
});
// repeat for candidate

where everything needed, including the while loop that writes to data channel is in one (1) compiled offerer program.

[guest271314]

I can't get the /web to work. I ran both python3 signaling-server.py and python3 -m http.server --bind 127.0.0.1 8080 from two (2) different terminals. The WebSocket is created but getting an error at setRemoteDescription().

Are you able to run the /web example without errors on your machine?

[guest271314]

This is what I have so far based on https://stackoverflow.com/q/68170912

nm.cpp

#include <iostream>
#include <cstdio>
#include <string>
#include <vector>

void sendMessage(std::string message) {
  auto* data = message.data();
  auto size = uint32_t(message.size());

  std::cout.write(reinterpret_cast<char*>(&size), 4);
  std::cout.write(data, size);
  std::cout.flush();
}

int main() {
  while (true) {
    std::uint32_t messageLength;

    // First Four contains message legnth
    std::cin.read(reinterpret_cast<char*>(&messageLength), 4);

    if (std::cin.eof()) {
      break;
    }

    std::vector<char> buffer;

    // Allocate ahead
    buffer.reserve(std::size_t(messageLength) + 1);

    std::cin.read(&buffer[0], messageLength);

    std::string message(buffer.data(), buffer.size());

    sendMessage("{\"text\": \"Hello World\"}");
  }
}

nm_cpp.json

{
  "name": "nm_cpp",
  "description": "datachannel",
  "path": "/home/user/nm-cpp/nm",
  "type": "stdio",
  "allowed_origins": [
     "chrome-extension://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/*"
  ]
}

$ g++ nm.cpp -o nm
$ make nm

manifest.json

{
  "name": "nm-cpp",
  "version": "1.0",
  "manifest_version": 3,
  "permissions": ["nativeMessaging"],
  "background": {
    "service_worker": "background.js",
    "type": "module"
  },
  "action": {}
}

background.js which we use here just for ability to call Native Messaging host from "service worker" DevTools inspected window at chrome://extensions

chrome.runtime.onInstalled.addListener(async(e) => {
  console.log(e);
});

Navigate to chrome://extensions, toggle "Developer mode", click "Load unpacked", then substitute the generated ID in nm_cpp.json for xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.

then

$ cp nm_cpp.json ~/.config/chromium/NativeMessagingHosts

nm-cpp.zip

Notice that we do not need to install anything.

Or execute the program at terminal. The program is executed by Chromium when sendNativeMessage() or connectNative() is called. Far simpler than installing and running signaling servers just to establish a data channel connection. I am currently using connectNative to stream directly from Python Native Messaging host.

https://github.com/guest271314/captureSystemAudio/blob/71f38294da096f256c00514485078a3d0a22d3c3/native_messaging/capture_system_audio/capture_system_audio.py#L36-L43

while True: 
     receivedMessage = getMessage() 
     process = subprocess.Popen(split(receivedMessage), stdout=subprocess.PIPE) 
     os.set_blocking(process.stdout.fileno(), False) 
     for chunk in iter(lambda: process.stdout.read(1024 * 1024), b''): 
         if chunk is not None: 
             encoded = str([int('%02X' % i, 16) for i in chunk]) 
             sendMessage(encodeMessage(encoded))                        

Where I am not progressing is when the initial empty object message is sent, given the C++ copy-paste example, the local description should be sent to the extension, read the remote description, read/write candidates.

[paullouisageneau]

I can't get the /web to work. I ran both python3 signaling-server.py and python3 -m http.server --bind 127.0.0.1 8080 from two (2) different terminals. The WebSocket is created but getting an error at setRemoteDescription().

You entered the local identifier as remote identifier, which is expected to fail as you can't connect a WebRTC Peer Connection to itself (peers take asymmetrical roles similar to client and server). You need to connect to another peer, for instance a second tab on 127.0.0.1:8080 or a libdatachannel example client.

[guest271314]

I successfully opened a connection using /examples/offerer and a RTCDataChannel at Chromium console. No messages were sent to the other peer and eventually error was thrown.

[paullouisageneau]

Did you set onmessage on browser side? Otherwise, you won't receive messages.

I can't see the error on the screenshot. If the error is readyState is not open it means the Data Channel was closed for some reason.

[guest271314]

Yes, onmessage is attached. I basically used the working code in answer.html from https://plnkr.co/edit/2Y9B97LmGbl5XR0d?preview at console. Did I miss a step? If I did how did the data channel fire open event?

local = new RTCPeerConnection();
[
  'onsignalingstatechange',
  'oniceconnectionstatechange',
  'onicegatheringstatechange',
].forEach((event) => local.addEventListener(event, console.log));
local.onicecandidate = async (event) => {
  console.log(event); // copy-paste candidate to libdatachannel /examples/offerer
  if (!event.candidate) {
    if (local.localDescription.sdp.indexOf('a=end-of-candidates') === -1) {
      local.localDescription.sdp += 'a=end-of-candidates\r\n';
      console.log(local.localDescription.sdp);
    }
  }
};

channel = local.createDataChannel('test', {
  negotiated: true,
  id: 0,
});
channel.binaryType = 'arraybuffer';
channel.onopen = async (e) => {
  console.log(e, local, channel);
};
channel.onmessage = async (e) => {
  console.log(e);
};

await local.setRemoteDescription({ type: 'offer', sdp }); // paste SDP from /offerer
const answer = await local.createAnswer();
local.setLocalDescription(answer); // Add your code here

[paullouisageneau]

Did I miss a step? If I did how did the data channel fire open event?

From what I can see on your screenshot, the DataChannel opened correctly then might have been closed. Again, I can't read the error on the screenshot.

[guest271314]

If I recollect accurate, the channel ready state is not open. Though it should be. You should be able to reproduce the experiment yourself.

[guest271314]

For my use cases I cannot really reconcile setting up and using a dedicated signaling server and WebSocket just to establish a data channel. I might as well just use the server itself to stream the data, in far less lines than it takes to exchange SDP and candidates

<?php 
//if (isset($_POST["capture_system_audio"])) {
    header('Vary: Origin');
    header("Access-Control-Allow-Origin: *");
    header("Access-Control-Allow-Methods: GET");
    header("Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers");    
    header("Content-Type: application/octet-stream");
    header("X-Powered-By:");
    echo passthru("parec -d @DEFAULT_MONITOR@");
    exit();
//  }

fetch(url, {signal})
  .then((r) => r.body.pipeTo(...))

I am just trying to avoid using an <iframe>, however that approach appears to be less steps and more efficient than other approaches I have experimented with.

[paullouisageneau]

If I recollect accurate, the channel ready state is not open. Though it should be.

It looks like your answerer calls createDataChannel() while the example offerer does it too. It means both sides will create their own DataChannel and ignore the remote one, meaning both will end up closed.

On answerer side, you want to set up the ondatachannel callback to get the remote DataChannel instead of calling createDataChannel().

You should be able to reproduce the experiment yourself.

I'm OK to help but I don't have time to debug your own code, sorry.

For my use cases I cannot really reconcile setting up and using a dedicated signaling server and WebSocket just to establish a data channel.

You don't actually need a WebSocket, it's just a protocol commonly used for large-scale signaling. Here, you just need a way to pass the SDP offer and get the SDP answer back (You don't have to transmit candidates separately if you wait for GatheringState to be complete and get the SDP offer/answer from localDescription)

[guest271314]

On answerer side, you want to set up the ondatachannel callback to get the remote DataChannel instead of calling createDataChannel().

That is the key and what I will try later today or tomorrow.

I'm OK to help but I don't have time to debug your own code, sorry.

Thanks for your time and effort so far.

Should I close this now to take this issue off of your plate?

[guest271314]

Got it working. Thanks.