Skip to content
This repository has been archived by the owner on Nov 29, 2022. It is now read-only.

BlockingIOError with telnet #93

Closed
ciscohack opened this issue Jul 29, 2021 · 50 comments
Closed

BlockingIOError with telnet #93

ciscohack opened this issue Jul 29, 2021 · 50 comments
Labels
bug Something isn't working

Comments

@ciscohack
Copy link

Environment

  • OS: BigSur 11.5.1
  • Terminal: Hyperterm nightly
  • ChromaTerm: <0.7.0>

Description

Hi Team ,

I was working on one my router and left the device and telnet session open for few hour then i have seen this message on router telnet session and session freezed. first i thought virtual router has some issue then noticed the it's chromaterm script.

image

@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

I'm currently looking into this.

How do you use ChromaTerm? With ct ssh some-router or ssh some-router | ct?

@ciscohack
Copy link
Author

I have this config in my zshrc

ssh() { /usr/bin/ssh "$@" | ct; }
telnet() { /usr/local/telnet "$@" | ct; }

@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

Found the issue. It relates to stdout being in an unblocking state. More specifically, telnet is modifying the state of stdout to be unblocking and quick output it will cause this issue. ssh keeps stdout in the blocking state, which is why this issue doesn't occur more often. At any rate, I'll get it fixed.

I was able to replicate the issue with this:

import os
import sys
import threading

def spam():
    while True:
        print('hello' * 1000)

os.set_blocking(sys.stdout.fileno(), False)

t1 = threading.Thread(target=spam)
t2 = threading.Thread(target=spam)

t1.start(); t2.start()

Edit: also reproducible directly with ChromaTerm using:

import socket

s = socket.socket()
s.bind(('127.0.0.1', 50001))
s.listen()
c = s.accept()[0]

for _ in range(10):
    c.send(b'a' * 1000)

and then connecting using telnet 127.0.0.1 50001 | ct

@hSaria hSaria closed this as completed in e330cb0 Jul 29, 2021
@hSaria hSaria changed the title ChromaTerm Script Error BlockingIOError with telnet Jul 29, 2021
@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

v0.7.3 released with the fix. Thanks for raising this issue.

@ciscohack
Copy link
Author

wow super fast Thanks

@ciscohack
Copy link
Author

i ran pip3 install -U Chromaterm get some warning don't know what is this. one more issue i have noticed sometime chromaterm highlight partial keyword and later after start highlighting full keyword and same happen with IP address too.

image

@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

The above screenshot is an issue from pip. Install anything is going to generate these warnings. They'll eventually fix it (has happened before).

Regarding the highlighting, if you're referring to the text that you type, then it's an expected behaviour as ChromaTerm can't predict what you're about to type next and cannot go back and highlight what's already on screen.

However, if you're getting output from a command that is being inconsistently highlighted, then that may be a bug. If so, please open as issue with the details (I'd very much appreciate that).

@ciscohack
Copy link
Author

@hSaria when i run command show ip int brief and suppose i have 6 interface with ip. sometime chromaterm highlight all 6 and sometime it miss last entry and that is not expected. same way many other places i have seen inconsistency. I know while typing chromaterm can't guss automaticaly but after type complete match should be accurate

@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

I see. Is there high latency between your device and the destination? For instance, if there's a 100ms+ delay or you're using a console connection?

@ciscohack
Copy link
Author

nope i use normally either ssh or telnet. i have not seeing or noticed any delay in executing command so can't say delay is playing any role. command type and execute is fast enough

@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

I mention delay because ChromaTerm processes most of the output as soon as it receives it, except for the last chunk. It'll wait to see if there's anymore output in case the last chunk was cut short. The duration of this wait period is approximately 40ms, but it is cancelled if any data is received.

If there's more than 40ms between the last chunk of data that ChromaTerm has and the next chunk, then they will be processed independently. For instance, if ChromaTerm (or rather ssh) received 1.2. and then, after 200ms, it receives 3.4, they'll be processed separated, i.e. they will be matched against the highlight rules as if they were different messages, instead of being processed as 1.2.3.4.

Note that this delay may not always be the latency between your device and the destination, but could just be lag in the output being displayed from the destination itself, like if it was busy doing anything intensive in the background, even for a brief moment.

If you're up for it, we can change the wait duration for a couple of days so you can test to see if it helps, and if it does, I'll act accordingly. You can do that with:

sed -i -e 's/WAIT_FOR_SPLIT = .*/WAIT_FOR_SPLIT = 1 \/ 100/' /usr/local/lib/python3.9/site-packages/chromaterm/__main__.py

@ciscohack
Copy link
Author

Here is the issue :

image

and now next show command all okay

image

@hSaria
Copy link
Owner

hSaria commented Jul 29, 2021

Yep, that would be the processing of a jittery last chunk. If you do the following, and reconnect to the device to run your tests, the highlighting should be more consistent. If the workaround improves things, I can work on ways to fix the issue.

sed -i -e 's/WAIT_FOR_SPLIT = .*/WAIT_FOR_SPLIT = 1 \/ 100/' /usr/local/lib/python3.9/site-packages/chromaterm/__main__.py

You can undo the change by running pip3 install --force-reinstall chromaterm

@ciscohack
Copy link
Author

Yep, that would be the processing of a jittery last chunk. If you do the following, and reconnect to the device to run your tests, the highlighting should be more consistent. If the workaround improves things, I can work on ways to fix the issue.

sed -i -e 's/WAIT_FOR_SPLIT = .*/WAIT_FOR_SPLIT = 1 \/ 100/' /usr/local/lib/python3.9/site-packages/chromaterm/__main__.py

You can undo the change by running pip3 install --force-reinstall chromaterm

So should i first run the command in my terminal and after that connect the devices to test right? how do i know that above command applied the needy change in timeout.

I ran the above command in terminal then connected the devices but still i am seeing the inconsistency

@ciscohack
Copy link
Author

I can see setting applied but no effect. I mean behaviour is still inconsistent

image

@hSaria
Copy link
Owner

hSaria commented Jul 30, 2021 via email

@ciscohack
Copy link
Author

After made a change in chromaterm config. I closed the terminal completely and then reconnected issue. issue still there even after system restart.

@hSaria
Copy link
Owner

hSaria commented Jul 30, 2021 via email

@ciscohack
Copy link
Author

ciscohack commented Jul 30, 2021

Just tried new suggested value and so far it's working as expected matching the trigger . but need some optimal value which not cause any performance issue. Second sometime by typing Chromaterm random word to be highlighted then later fix. but over all this 1/10 solving the problem so far it seems.

It's happening with me on firewall and Routers. Server i do not check much and log reading mostly keywords get highlighted without any issue

@hSaria
Copy link
Owner

hSaria commented Jul 30, 2021 via email

@ciscohack
Copy link
Author

Well, at least the cause has been confirmed. Thanks for working with me on this. As for the optimal value, 1/256 was actually picked because it is shorter than the fastest key repetition interval (1/255 in some Linux distro) This is intentional to make sure key typing is responsive and doesn’t freeze when you hold a key down. While I think of a way to fix the problem, you should revert back to the default implementation.

On 30 Jul 2021, at 1:02 pm, ciscohack @.***> wrote:  Just tried new suggested value and so far it's working as expected matching the trigger . but need some optimal value which not cause any performance issue. Second sometime by typing Chromaterm random word to be highlighted then later fix. but over all this 1/10 solving the problem so far it seems — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

how to revert the changes. default is 1/256 should i use sed method?

@hSaria
Copy link
Owner

hSaria commented Jul 30, 2021 via email

@ciscohack
Copy link
Author

ciscohack commented Jul 30, 2021

sed is quick one 1/256 is default value right. second if i reinstall hope my chromaterm.yml file not get replaced.. I wish you get some workaround or fix for this problem

@hSaria
Copy link
Owner

hSaria commented Jul 30, 2021 via email

@ciscohack
Copy link
Author

The default is 1/256. And no, the reinstall doesn’t change your configuration. It simply downloads the module again.

On 30 Jul 2021, at 1:14 pm, ciscohack @.***> wrote:  sed is quick one 1/256 is default value right. second if i reinstall hope my chromaterm.yml file not get replaced — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

thanks i reverted the changes.. will wait for the fix. Thanks once again my friend

@ciscohack
Copy link
Author

I mention delay because ChromaTerm processes most of the output as soon as it receives it, except for the last chunk. It'll wait to see if there's anymore output in case the last chunk was cut short. The duration of this wait period is approximately 40ms, but it is cancelled if any data is received.

If there's more than 40ms between the last chunk of data that ChromaTerm has and the next chunk, then they will be processed independently. For instance, if ChromaTerm (or rather ssh) received 1.2. and then, after 200ms, it receives 3.4, they'll be processed separated, i.e. they will be matched against the highlight rules as if they were different messages, instead of being processed as 1.2.3.4.

Note that this delay may not always be the latency between your device and the destination, but could just be lag in the output being displayed from the destination itself, like if it was busy doing anything intensive in the background, even for a brief moment.

If you're up for it, we can change the wait duration for a couple of days so you can test to see if it helps, and if it does, I'll act accordingly. You can do that with:

sed -i -e 's/WAIT_FOR_SPLIT = .*/WAIT_FOR_SPLIT = 1 \/ 100/' /usr/local/lib/python3.9/site-packages/chromaterm/__main__.py

@hSaria Hello my friend how are you? Hey any plan to fix this issue in future release?

@hSaria
Copy link
Owner

hSaria commented Sep 6, 2021 via email

@ciscohack
Copy link
Author

@hSaria No my friend, you please take your time but lets have an optimal way to address. lets not break the code.

@hSaria
Copy link
Owner

hSaria commented Sep 6, 2021

@ciscohack I've done a bit of work on this. Do you mind testing?

You can download the development version with the following:

git clone https://github.com/hSaria/ChromaTerm.git
cd ChromaTerm
pip3 install .

Then use ChromaTerm as normal.

If there's an issue and you want to revert, you can do so with:

pip3 install --force-reinstall chromaterm

@ciscohack
Copy link
Author

@ciscohack I've done a bit of work on this. Do you mind testing?

You can download the development version with the following:

git clone https://github.com/hSaria/ChromaTerm.git
cd ChromaTerm
pip3 install .

Then use ChromaTerm as normal.

If there's an issue and you want to revert, you can do so with:

pip3 install --force-reinstall chromaterm

@hSaria i would love to test but can u please help me to know which path this file will be installed on macbook. i have 0.7.3 version already install so if above method override the older chromterm then it would be good or if it create new folder in different location then i would like to first delete older folder to avoid duplicate files

@hSaria
Copy link
Owner

hSaria commented Sep 6, 2021 via email

@ciscohack
Copy link
Author

Your configuration file will not be affected when you change the version. Even if you uninstall ChromaTerm, the configuration file will not be deleted or modified.

On 6 Sep 2021, at 4:09 pm, ciscohack @.***> wrote:  @ciscohack I've done a bit of work on this. Do you mind testing? You can download the development version with the following: git clone https://github.com/hSaria/ChromaTerm.git cd ChromaTerm pip3 install . Then use ChromaTerm as normal. If there's an issue and you want to revert, you can do so with: pip3 install --force-reinstall chromaterm @hSaria i would love to test but can u please help me to know which path this file will be installed on macbook. i have 0.7.3 version already install so if above method override the older chromterm then it would be good or if it create new folder in different location then i would like to first delete older folder to avoid duplicate files — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

no no i am not talking about the path location of chromaterm in macos

@hSaria
Copy link
Owner

hSaria commented Sep 6, 2021 via email

@ciscohack
Copy link
Author

You’re talking about the configuration file that keeps your highlight rules, right? That file will never be changed if you upgrade, downgrade, or even uninstall ChromaTerm. The config file is independent.

On 6 Sep 2021, at 4:27 pm, ciscohack @.> wrote:  Your configuration file will not be affected when you change the version. Even if you uninstall ChromaTerm, the configuration file will not be deleted or modified. … On 6 Sep 2021, at 4:09 pm, ciscohack @.> wrote:  @ciscohack I've done a bit of work on this. Do you mind testing? You can download the development version with the following: git clone https://github.com/hSaria/ChromaTerm.git cd ChromaTerm pip3 install . Then use ChromaTerm as normal. If there's an issue and you want to revert, you can do so with: pip3 install --force-reinstall chromaterm @hSaria i would love to test but can u please help me to know which path this file will be installed on macbook. i have 0.7.3 version already install so if above method override the older chromterm then it would be good or if it create new folder in different location then i would like to first delete older folder to avoid duplicate files — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. no no i am not talking about the path location of chromaterm in macos — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

no that i know i am taking about chromaterma folder where it keep the stuff when i download

@hSaria
Copy link
Owner

hSaria commented Sep 6, 2021 via email

@ciscohack
Copy link
Author

Anyway i have installed it

@ciscohack
Copy link
Author

@hSaria results looks so far good. will stay on this and play. Thanks once again bro

@hSaria
Copy link
Owner

hSaria commented Sep 6, 2021 via email

@hSaria
Copy link
Owner

hSaria commented Sep 8, 2021

v0.7.5 released. You can revert back to the normal version with pip3 install -U chromaterm.

Thanks a lot for your help on this one.

@ciscohack
Copy link
Author

v0.7.5 released. You can revert back to the normal version with pip3 install -U chromaterm.

Thanks a lot for your help on this one.

Thanks for the update. I just ran pip3 install -U chromaterm this and it saying version upgrade to 0.7.4 not 0.7.5 why?

@ciscohack
Copy link
Author

i think it's typo from your end version is 0.7.4 only

@hSaria
Copy link
Owner

hSaria commented Sep 8, 2021 via email

@ciscohack
Copy link
Author

thanks

@hSaria hSaria added the bug Something isn't working label Sep 12, 2021
@ciscohack
Copy link
Author

@hSaria How are you? Sorry again coming up with same issue and reporting late. I can see the issue we identified where chromaterm sometime either not highlight the keyword or highlight only partial character .. that issue is still in the code.

image

@hSaria
Copy link
Owner

hSaria commented Nov 19, 2021 via email

@ciscohack
Copy link
Author

Can you share your configuration file?

On 19 Nov 2021, at 1:26 pm, ciscohack @.***> wrote:  @hSaria How are you? Sorry again coming up with same issue and reporting late. I can see the issue we identified where chromaterm sometime either not highlight the keyword or highlight only partial character .. that issue is still in the code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

chromaterm.yml.zip

@hSaria
Copy link
Owner

hSaria commented Nov 19, 2021

@hSaria How are you? Sorry again coming up with same issue and reporting late. I can see the issue we identified where chromaterm sometime either not highlight the keyword or highlight only partial character .. that issue is still in the code.

image

I've ran your configuration file and couldn't recreate this. Does this happen often or is it a rare occurance?

@ciscohack
Copy link
Author

@hSaria How are you? Sorry again coming up with same issue and reporting late. I can see the issue we identified where chromaterm sometime either not highlight the keyword or highlight only partial character .. that issue is still in the code.
image

I've ran your configuration file and couldn't recreate this. Does this happen often or is it a rare occurance?

sometime work and sometime not..second if i wait for few second then trigger work fine.. it's like hit and miss
I use iterm2 nightly and hyperterm both i see this.. infact show ip int brief or long log file where it has to show lots of lines in those scenario it's always miss some triggers

@hSaria
Copy link
Owner

hSaria commented Nov 20, 2021 via email

@ciscohack
Copy link
Author

Okay Thanks for the input

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants