Support for post-authorization - non gui app

[0x7464]

Hi,

Is there any way once the GUI icon and menu system have been used and the token and the rest of the credentials have been placed into the configuration file after successful authorization that the app could be run without needing a gui ? - like maybe a emailproxy_after_authorization.py or something as you don't really need to touch the GUI again after the settings are in the configuration file correct ?

Since it's bound to a GUI currently I haven't found a way to automate bringing it up in a users graphical session after boot without logging in manually (it's actually quite tricky) - I think a lot of users would love this functionality as running things without a GUI is normal for server usage.

Thanks

[simonrob]

Thanks for the suggestion - I agree that this mode could be useful in many situations.

One problem I can foresee is that there's no guarantee the script will always be able to refresh your OAuth credentials automatically, and you may be asked to log in again manually to authenticate.

This is relatively rare, but does sometimes happen. For example, some organisations may have policies to do this, e.g., every 30 days. Google also occasionally requires re-authentication based on various factors.

If the script was running in non-GUI mode and this happened then there would need to be a way to alert the user about this and prompt to re-authenticate. Currently this is done via GUI notifications and the menu bar icon, but when running as a background process there is often no way to capture input or provide output.

I don't have this requirement myself, so don't know what the best way to deal with this would be. What do you think?

[0x7464]

I was thinking about the point about it not being able to refresh automatically - would it be possible to (trying to think of the simplest way) have the script do a periodic poll (test auth) every so many minutes or something and have a toggleable setting in the config file to allow that to periodic poll to write its results to /var/log/syslog as a message ?

[simonrob]

This wouldn't be possible because the script doesn't actually initiate any communication with the server on its own; it simply edits the commands your email client sends to add the correct authentication parameters.

Perhaps it would be okay to simply ignore this aspect and rely on the timeouts and error messages of the email client and/or server?

You can simulate what would happen with this approach by just ignoring an authentication request with the current version of the script – after about 60 seconds the server will close its side of the connection, and the client will normally time out soon after. The proxy waits a little longer (10 minutes at the moment), but if there is no response to the request it eventually discards it and sends an error message to the client. This message is not currently written to the log, but it easily could be.

From your side, you could just watch for failure messages in your client or the log, and you'd know when authentication is required because sending/receiving email would fail. You could then open the graphical version of the proxy and respond to the request. Would this work?

[0x7464]

That sounds great - I think that should work perfectly.

[0x7464]

Actually, now that I think further about it, the main problem with that approach is that we still have the dependency on the GUI session running which requires a user to have initiated it. If the application had a daemon that only listens for requests and reads the config file which runs as a daemon, that would be beautiful - and then the GUI session could be started as required for reauthentication.

[simonrob]

I had a look at this just now, and it seems like it should be possible with a standard systemctl service with a few minor changes to the proxy script. Please could you test the following?

Download the no-gui branch and copy your current emailproxy.config file to this folder. (Note: no-longer necessary – non-GUI mode is now part of the main script).

Open a terminal, navigate to the no-gui folder, then run the following commands to enable Python virtual environments and create one with the proxy's dependencies:

python3 -m venv .env
source .env/bin/activate
python3 -m pip install -U pip
python3 -m pip install -r requirements-no-gui.txt

Create a systemctl file for the proxy: sudo systemctl edit --force --full emailproxy.service, and then add the following in the editor that opens:

[Unit]
Description=Email OAuth 2.0 Proxy

[Service]
ExecStart=/path/to/folder/.env/bin/python3 /path/to/folder/emailproxy.py --no-gui
Restart=always

[Install]
WantedBy=multi-user.target

Replace the two occurrences of /path/to/folder with the full path to wherever you saved the no-gui branch, then save the file, accepting the default path/filename (ctrl+x, y, enter with the default editor).

Finally, enable and start the service: sudo systemctl enable emailproxy.service --now

You should then be able to use your accounts via the proxy without any of its normal GUI features. (It's possible you may need to restart your computer first - please try that if things don't work initially.)

If you encounter problems please check the service's status and output with sudo systemctl status emailproxy.service and let me know what you see there. Also, it probably won't be needed, but you can enable the proxy's debug mode if required by adding the additional parameter --debug after --no-gui in the systemctl editor (just rerun the edit command to reopen the service file and then restart the service).

You can always start, stop or restart the service, or enable or disable it being run on system startup with sudo systemctl [command] emailproxy.service (replace [command] with what you want to do).

Please let me know how you get on.

[0x7464]

Really great work so far Simon.

OK so far with my testing results are as follows:

GMAIL (GSuite) - end to end testing - Works 100% (gui and non-gui)
Exchange Online - end to end testing - GUI app works, but segfaults on starting the no_gui app emailproxy.py with the office365 data in it.

Segfault always returns with the following:

May 13 21:24:05 TESTMACHINE kernel: [692290.204499] python3[76790]: segfault at 7ff6e1e46ab7 ip 00007ff6deca08ea sp 00007ffdd5352d30 error 7 in _imagingft.cpython-38-x86_64-linux-gnu.so[7ff6dec98000+b000]
May 13 21:24:05 TESTMACHINE kernel: [692290.204506] Code: e8 7b ae ff ff 48 85 c0 48 89 03 0f 84 bf 01 00 00 48 8d 35 28 11 00 00 48 89 c7 e8 10 b3 ff ff 48 8b 2d 29 1f 20 00 48 85 c0 <48> 89 45 00 0f 84 6c 01 00 00 48 8b 3b 48 8d 35 9a 10 00 00 e8 ed

Any ideas ?

[simonrob]

Thanks for trying this out. It's great to hear things work well for Gmail in this new mode.

To be honest, I am totally baffled by the segfault that you're seeing. Firstly, that it is a segfault at all rather than just a script error message, but also that exactly the same script works in the same mode for Gmail, and that the only differentiating factor (the emailproxy.config file) itself works in GUI mode.

Would you be willing to share the emailproxy.config file that you're seeing this crash with? Even better, could you zip up the whole folder (including the .env directory and the config file) and send me that for testing?

[0x7464]

Sure Simon, I have sent you an email with a link to download the zip of the whole non_gui folder.
Please ignore the first automated link sent - as that was when I made a mistake in sharing the file - just use the link sent in my email to you.

Thanks

[simonrob]

Thanks for sharing the files. Unfortunately, despite trying lots of different configurations with this setup I cannot reproduce the issue. I really don't understand how it could only happen with the Exchange Online version when Gmail works with no problems. Very strange!

Just to be extra clear: is this crash occurring when you launch the proxy, or when you try to log in to your account? Could you perhaps try systematically replacing the configuration entries with dummy data into the config file that causes this to help isolate the error?

Are you totally sure that this is happening in non-GUI mode? The reason I ask is that the imagingft module is I believe related to pillow, which is only used when there's a menu bar icon. It may be worth trying the latest commit in the non-gui branch as I've fixed a crash that happened on exit when the script couldn't be initialised. I don't really see how it could be related here, but I suppose there's a small chance that failing to start the proxy on the local IP you're using could have caused this.

The only other thing I can suggest is that it may be something to do with the local virtual environment configuration vs. your global setup (though this doesn't explain the Exchange vs. Gmail difference). Perhaps you could try some of the suggestions listed here, which look to be related to the same sort of issue?

[0x7464]

Hi Simon,

Greatly appreciate you looking in to this - I will try all your suggestions today and get back to you.
And yes, this is a crash when you running the proxy - i.e when running - python3 emailproxy.py, and yes - it is definitely only happening in the non-gui mode as i authorized the app in gui mode no problems, then stopped the gui mode proxy and copied the configuration file over to the non_gui app folder and started the non gui app leading to the segfault.

[0x7464]

Hi Simon,

OK so here is what I've done:

1. Test running non_gui with gmail configuration - still segfaults (I'm sure I tried this before so not sure)
2. Setup configuration file to have original dummy data - still segfaults
3. Setup completely from scratch with latest non_gui code code commit with same office 365 configuration file - WORKS!

I'm really quite sure I tested the non_gui with the gmail configuration and it did work before moving onto office365/EXO - so I'm really unsure as to what happened here - maybe when I did whatever I did to make it not work with office365 it broke it for everything ?

I'll run it through a few different scenarios today and bring it up and down and also run it as a daemon - I have not tested that part yet but I wouldn't say that would break it and let you know how it goes. Once again, really great work Simon!

*N.B as an aside - when you do the pip3 install -r requirements.txt on linux - you get the following output:

.env) me@TESTMACHINE:/opt/email-oauth2-proxy-no-guiv2/email-oauth2-proxy-no-gui$ pip3 install -r requirements.txt
Ignoring launchctl: markers 'sys_platform == "darwin"' don't match your environment
Ignoring pyobjc: markers 'sys_platform == "darwin"' don't match your environment
Collecting configobj
Using cached configobj-5.0.6.tar.gz (33 kB)
Collecting cryptography
Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting pillow
Using cached Pillow-8.2.0-cp38-cp38-manylinux1_x86_64.whl (3.0 MB)
Collecting pystray
Using cached pystray-0.17.3-py2.py3-none-any.whl (46 kB)
Collecting pywebview
Using cached pywebview-3.4-py3-none-any.whl (325 kB)
Collecting timeago
Using cached timeago-1.0.15-py3-none-any.whl
Collecting six
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting cffi>=1.12
Using cached cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting python-xlib>=0.17
Downloading python_xlib-0.30-py2.py3-none-any.whl (178 kB)
|████████████████████████████████| 178 kB 13.2 MB/s
Using legacy 'setup.py install' for configobj, since package 'wheel' is not installed.
Installing collected packages: six, pycparser, python-xlib, pillow, cffi, timeago, pywebview, pystray, cryptography, configobj
Running setup.py install for configobj ... done
Successfully installed cffi-1.14.5 configobj-5.0.6 cryptography-3.4.7 pillow-8.2.0 pycparser-2.20 pystray-0.17.3 python-xlib-0.30 pywebview-3.4 six-1.16.0 timeago-1.0.15

Now the launchctl one can most likely be ignored totally as that is just a MacOS init service, however with pyobj - do you think we would lose anything in an linux environment by not having this/a linux equivalent ? - Please note though as of now running non_gui from command line without running it as a daemon works 100% so probably not.

I'll go and test it now as a daemon and see if everything still works - thanks Simon.

[0x7464]

Hi Simon,

I've run through testing end-to-end as a daemon and am having no problems at all. I think we can just put this one down to user error.

Thanks so much for your help Simon.

[simonrob]

Excellent! It's great to hear you've got this all working now.

Just for completeness, in response to the notes in your comments:

• The non-GUI option is just a mode setting, and the same instance of the script can run in both modes. You shouldn't need to copy config files between two different locations; just run the script without the --no-gui option when you need to authorise an account.
• The launchctl and pyobjc messages you get when installing dependencies are just for information to let you know that these were not installed because you aren't running on macOS. You can ignore these these as those packages are not required on other platforms.

Thanks again for the idea for this new mode and the help testing it – I'm really glad you find this tool useful. I'll merge the non-GUI branch now and close this issue, but please feel free to open another if you run into any other problems.

[innovara]

I just want to leave a comment here for those who may take the content of the systemd unit file from this conversation.

I had to add Wants=network-online.target and more importantly After=network-online.target to make sure it would start after network is initialized. Otherwise it would fail to start since it couldn't bind to the port(s). It might have been related to the environment but it seems reasonable to have such setting.

So it just ends up being:

[Unit]
Description=Email OAuth 2.0 Proxy
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/path/to/folder/.env/bin/python3 /path/to/folder/emailproxy.py --no-gui
Restart=always

[Install]
WantedBy=multi-user.target