Error "Inappropriate ioctl for device" on macOS

[ryanwebber]

On my macOS machine there's an error when marking the pts fd as nonblocking here:

pty-process/src/sys.rs

Line 53 in 74c223d

pub fn set_nonblocking(&self) -> rustix::io::Result<()> {

Steps to reproduce:

git clone https://github.com/doy/pty-process.git
cargo run --features tokio,async --example tokio

Logs:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Rustix(Os { code: 25, kind: Uncategorized, message: "Inappropriate ioctl for device" })', examples/tokio.rs:53:43
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Note: I'm not sure if this is actually unique to macOS. I haven't been able to find much info on this error with respect to pty or nonblocking fds unfortunately, but as I keep looking I'll post anything I find.

[ryanwebber]

Came across this thread with some info: https://developer.apple.com/forums/thread/734230

Taking inspiration from the main reply regarding opening the slave fds before setting O_NONBLOCK seemed to work, but I'm not sure this is the right solution for all platforms.

diff --git a/src/sys.rs b/src/sys.rs
index 927fe93..43b7693 100644
--- a/src/sys.rs
+++ b/src/sys.rs
@@ -51,10 +51,6 @@ impl Pty {
 
     #[cfg(feature = "async")]
     pub fn set_nonblocking(&self) -> rustix::io::Result<()> {
-        let mut opts = rustix::fs::fcntl_getfl(&self.0)?;
-        opts |= rustix::fs::OFlags::NONBLOCK;
-        rustix::fs::fcntl_setfl(&self.0, opts)?;
-
         Ok(())
     }
 }
@@ -140,10 +136,17 @@ impl Pts {
     pub fn session_leader(&self) -> impl FnMut() -> std::io::Result<()> {
         let pts_fd = self.0.as_raw_fd();
         move || {
+            let fd = unsafe { std::os::fd::BorrowedFd::borrow_raw(pts_fd) };
             rustix::process::setsid()?;
-            rustix::process::ioctl_tiocsctty(unsafe {
-                std::os::fd::BorrowedFd::borrow_raw(pts_fd)
-            })?;
+            rustix::process::ioctl_tiocsctty(fd)?;
+
+            #[cfg(feature = "async")]
+            {
+                let mut opts = rustix::fs::fcntl_getfl(fd)?;
+                opts |= rustix::fs::OFlags::NONBLOCK;
+                rustix::fs::fcntl_setfl(fd, opts)?;
+            }
+
             Ok(())
         }
     }

[mightyiam]

@ryanwebber what did you end up using? I couldn't find an alternative with async.

[ryanwebber]

I didn't find an alternative, I just used a version of this repo with the above patch applied to temporarily unblock myself.

[mightyiam]

#8

[samuela]

FWIW I am facing the same issue but when calling pty.resize(...):

pub fn main() {
  let pty = pty_process::blocking::Pty::new().unwrap();
  pty.resize(pty_process::Size::new(24, 80)).unwrap();
}

produces

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Rustix(Os { code: 25, kind: Uncategorized, message: "Inappropriate ioctl for device" })', src/bin/bug.rs:3:46
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace