@yume-chan/scrcpy documentation for standalone package

[thelicato]

Hi, I'd like to use only the yume-chan/scrcpy package to integrate Scrcpy in a React frontend that integrates with a Node.js backend. I was reading the specific README at this link but I found that something is missing, like:

• ScrcpyVideoStreamPacket
• ScrcpyControlMessageSerializer

Probably that page is not updated but I would very much like a guide to integrate your frontend package with a different backend implementation.

[yume-chan]

The README is up to date, in fact it's too up to date that it applies to the unreleased code.

ScrcpyVideoStreamPacket was called VideoStreamPacket in version 0.0.16, and ScrcpyControlMessageSerializer didn't exist in 0.0.16, I separated it out for easier integration to other frontends, just like what you need.

My plan for new release is to wait after next Scrcpy release, which includes my PR Genymobile/scrcpy#3369 to improve scrolling. Without that scrolling is barely working on Web. But if you want, I can do another release now.

[thelicato]

Got it! No, thank you. I'll wait.
Maybe I'll clone the repo and install it locally.

[thelicato]

I tried to install it locally, currently the scrcpy package folder has adb in the dependencies array. Maybe you should remove it since it could be used even without the adbpackage.

[yume-chan]

In exported types from @yume-chan/scrcpy, if the type name starts with Scrcpy then it doesn't require @yume-chan/adb, if the type name starts with Adb then it does.

If you don't use those types (with names start with Adb), code from @yume-chan/adb package should be tree-shaken out and won't in your final code bundle.

[yume-chan]

For a general usage guide, as you have your own backend, you need to supply the two socket streams from Scrcpy server to this package.

This package uses Web Streams API (https://developer.mozilla.org/en-US/docs/Web/API/Streams_API), it's similar to Node.js streams, but not same.

The video socket can only be read, so you need to create a ReadableStream from it. The control socket can be both read and written, so you need to create one ReadableStream and one WritableStream. Exactly how those streams should be created depends on how you transfer those data to your frontend, for example with one WebSocket for video socket and another WebSocket for control socket, you can use something similar to my WebSocket backend does (ignore the final StructDeserializeStream and AdbPacketSerializeStream, they are specific to AdbPacket streams):

ya-webadb/libraries/adb-backend-ws/src/index.ts

Lines 15 to 58 in c1128a3

	const socket = new WebSocket(this.serial);
	socket.binaryType = 'arraybuffer';
	await new Promise((resolve, reject) => {
	socket.onopen = resolve;
	socket.onerror = () => {
	reject(new Error('WebSocket connect failed'));
	};
	});
	const factory = new DuplexStreamFactory<Uint8Array, Uint8Array>({
	close: () => {
	socket.close();
	},
	});
	socket.onclose = () => {
	factory.dispose();
	};
	const readable = factory.wrapReadable(new ReadableStream({
	start: (controller) => {
	socket.onmessage = ({ data }: { data: ArrayBuffer; }) => {
	controller.enqueue(new Uint8Array(data));
	};
	}
	}, {
	highWaterMark: 16 * 1024,
	size(chunk) { return chunk.byteLength; },
	}));
	const writable = factory.createWritable(new WritableStream({
	write: (chunk) => {
	socket.send(chunk);
	},
	}, {
	highWaterMark: 16 * 1024,
	size(chunk) { return chunk.byteLength; },
	}));
	return {
	readable: readable.pipeThrough(new StructDeserializeStream(AdbPacket)),
	writable: pipeFrom(writable, new AdbPacketSerializeStream()),
	};

All chunks in those streams should be Uint8Arrays.

---

Then you need to create an option value (new ScrcpyOptions1_XX({ ...options })). Use the correct version as your server binary, and same options as given to the server. Some control messages differ between versions, and some options (like sendDeviceMeta) affect how the video stream should be deserialized.

To parse the video stream, pipe the video stream to the video stream deserializer
videoReadableStream.pipeThrough(options.createVideoStreamTransformer()), this returns another ReadableStream, but with ScrcpyVideoStreamPackets as chunks, you can consume it by calling stream.getReader() (https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader) and reader.read() (https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader/read), or you can pipe it to another WritableStream

videoSocket
  .pipeThrough(options.createVideoStreamTransformer())
  .pipeTo(new WritableStream({
    write(packet) {
      switch (packet.type) {
        case 'configuration':
          console.log(packet.data.croppedWidth, packet.data.croppedHeight);
          break;
        case 'frame':
          console.log(packet.keyframe, packet.pts, packet.data);
          break;
      }
    }
  }));

To send control messages to device, create a ScrcpyControlMessageSerializer using the control socket ReadableStream and the options

const controller = new ScrcpyControlMessageSerializer(controlSocket, options);

Then hook up your UI to call methods on the serializer:

await controller.backOrScreenOn(AndroidKeyEventAction.Down);
await controller.backOrScreenOn(AndroidKeyEventAction.Up);
await controller.injectTouch({
  screenWidth: 1920,
  screenHeight: 1080,
  action: AndroidMotionEventAction.Down,
  buttons: 0,
  pointerId: 0n,
  pointerX: 500,
  pointerY: 500,
  pressure: 0,
})

Exact parameters of each method are not documented too much, check my demo for examples.

[yume-chan]

I have released version 0.0.17 so you can use them directly.

[thelicato]

You are truly generous. Thanks for the super-detailed answer