[Feature Request] Add support for Zulip

[vchrombie]

How about adding backend support for Zulip?

Zulip is another important chat and collaborative software, just like Rocket.Chat and Slack and it is open-source too.
https://anitab-org.zulipchat.com/

[vchrombie]

If the feature is ok, then I would like to work on its implementation.

[valeriocos]

Hi @vchrombie, thank you for proposing the inclusion of a new backend.

Zulip is a popular oss chat application, I believe that this inclusion will attract more people around grimoirelab, so +1 to your proposal.

[animeshk08]

Hi, @valeriocos I think including Zulip as a new backend is a great idea.

Also, I have found a lot of organizations using Gitter as a communication channel as well. Do you think we should include that as a backend as well? I would be interested in doing that. I can open a new issue and we can move this discussion there if it interests you :)

[valeriocos]

It sounds a good idea @animeshk08 , please open another issue to support gitter and start implementing the backend. Thanks!

[vchrombie]

Hi @valeriocos
Sorry for the delay. I started to work on this issue and posting a small update here.

Zulip is more or less like Slack. Each zulip chat server has different streams (like channels in slack) and also each stream has diff topics. I was thinking to target the streams (as parameter) just like how it was done in slack. WDYT?

I went through the API docs of Zulip, https://zulipchat.com/api/get-messages. You can use the API to extract the messages from a particular stream.

I will be working on this from now and send a draft PR soon.

Some of the channels where we can do the testing are

1. https://python.zulipchat.com
2. https://anitab-org.zulipchat.com

[valeriocos]

Hi @vchrombie ! no worries for the delay!

Zulip is more or less like Slack. Each zulip chat server has different streams (like channels in slack) and also each stream has diff topics. I was thinking to target the streams (as parameter) just like how it was done in slack. WDYT?

Yes, good idea!

I went through the API docs of Zulip, https://zulipchat.com/api/get-messages. You can use the API to extract the messages from a particular stream.

Thanks for the pointer! Have you checked if the narrow param can be used to fetch messages after a given date? This is needed to support incremental fetch operations. If there is no way to use a date, we can open a feature request at https://github.com/zulip/zulip to ask for it or implement an offset-based approach (param num_after, ref backend here).

I will be working on this from now and send a draft PR soon.

Great, thanks!

[vchrombie]

Hi @valeriocos

Thanks for the pointer! Have you checked if the narrow param can be used to fetch messages after a given date?

I checked the docs, https://python.zulipchat.com/api/construct-narrow.
They have a couple of filters but seems like none of them are of much use to us, except the stream filter.

This is needed to support incremental fetch operations. If there is no way to use a date, we can open a feature request at https://github.com/zulip/zulip to ask for it or implement an offset-based approach (param num_after, ref backend here).

I understand the requirement. I will open an issue and try to pull up discussion on it in the zulip/zulip repository and see if they are interested to introduce such feature. Let's decide based on that. WDYT?

[valeriocos]

Yes, totally agree! thanks @vchrombie for pushing this forward

[vchrombie]

Hi @valeriocos

I opened the issue in the zulip repository but didn't really receive much positive response from them.
I was thinking of starting the implementation with the offset-based approach which you have suggested.

WDYT?

[valeriocos]

Hi @vchrombie !

Ok, sure! Don't hesitate to write if you need any help!

[vchrombie]

I started working on it. I have a few questions already.

1. I went through the API docs. Coming to the authentication part, you need an email address and a token. It can be generated using a bot or a person. I would go for the bot email id and bot token. It is really simple to generate one.

Click on the ⚙️ >> Settings >> Your bots >> Add a new bot

So, the command for fetching messages from the importlib stream in python zulip chat will be like

perceval zulip 'https://python.zulipchat.com/' 'importlib' -e 'BOT_EMAIL_ADDRESS' -t 'BOT_API_KEY'

I just thought of confirming with you.

2. I understand that we need to prepare a URL path with the payloads and headers (if present) and use the fetch function to send a request and get the response.

The Zulip API is a little bit different. https://python.zulipchat.com/api/get-messages

curl -sSX GET -G https://python.zulipchat.com/api/v1/messages \
    -u BOT_EMAIL_ADDRESS:BOT_API_KEY \
    -d 'anchor=newest' \
    -d 'num_before=20' \
    -d 'num_after=0' \
    --data-urlencode narrow='[{"operand": "importlib", "operator": "stream"}]'

I am not sure how to implement this using payload. I went through StackOverflow and found one solution. You can find it's implementation here, zulip-script.py.

Do you have any comments or suggestions for this?

[valeriocos]

.. the command for fetching messages from the importlib stream in python zulip chat will be like

It should be fine. I understand that we cannot fetch messages with a standard email and user token, right?

I understand that we need to prepare a URL path with the payloads and headers...

We should try to use the library requests, since Perceval provides different mechanisms on top of it (sleep for rate, archiving, etc.). If we go for the use of subprocess, we will have to implement these mechanisms again.

Since Zulip has a Python client to fetch data from the API, could you check the code of the get_messages method (and related methods)? There, you should find examples of how to use requests to interact with that API.

Thanks!

[vchrombie]

Hi @valeriocos

It should be fine. I understand that we cannot fetch messages with a standard email and user token, right?

Do you mean the user email and token?

We can use

• BOT_EMAIL_ADDRESS (something we can create like [email protected]) & BOT_API_KEY
  or
• USER_EMAIL_ADDRESS (say personal email [email protected]) & USER_API_KEY.

I understand that we need to prepare a URL path with the payloads and headers...

We should try to use the library requests, since Perceval provides different mechanisms on top of it (sleep for rate, archiving, etc.). If we go for the use of subprocess, we will have to implement these mechanisms again.

Yes, even I was thinking about the same problem.

Since Zulip has a Python client to fetch data from the API, could you check the code of the get_messages method (and related methods)? There, you should find examples of how to use requests to interact with that API.

Yes, Zulip has a Python client which needs to install the zulip pip package. I will try that and let you know.
Thanks.

[valeriocos]

We can use

- BOT_EMAIL_ADDRESS (something we can create like [email protected]) & BOT_API_KEY
or
- USER_EMAIL_ADDRESS (say personal email [email protected]) & USER_API_KEY.

Ok, perfect!

Yes, Zulip has a Python client which needs to install the zulip pip package. I will try that and let you know.

Ok, perfect!

Thanks

[vchrombie]

Hi @valeriocos

Since Zulip has a Python client to fetch data from the API, could you check the code of the get_messages method (and related methods)? There, you should find examples of how to use requests to interact with that API.

Thanks for the help, it really helped.

curl -sSX GET -G https://python.zulipchat.com/api/v1/messages \
    -u BOT_EMAIL_ADDRESS:BOT_API_KEY \
    -d 'anchor=newest' \
    -d 'num_before=20' \
    -d 'num_after=0' \
    --data-urlencode narrow='[{"operand": "importlib", "operator": "stream"}]'

I understood that all the arguments are handles using requests. In fact a simple request for get_messages boils down to

GET https://python.zulipchat.com/api/v1/messages 15.0 {'params': {'anchor': 'newest', 'num_before': '2', 'num_after': '0', 'narrow': '[{"operator": "stream", "operand": "importlib"}]'}}

along with the required headers.

Now, I got a basic idea and I will be working on completing this.
Thanks.

[valeriocos]

Thanks for the updates @vchrombie

[vchrombie]

Hi @valeriocos
Just a small update.

As I have figured out how to use Zulip API using requests, I was thinking about how to perform the incremental fetch

If there is no way to use a date, we can open a feature request at https://github.com/zulip/zulip to ask for it or implement an offset-based approach (param num_after, ref backend here).

I tried your suggestion and it worked. I made a small script for it, zulip backend. Initially we can use the

'anchor': 'oldest'

for fetching the first 10 messages.

Later, we can use the message id of the latest fetched

'anchor': '159310824'

to fetch the next set of messages.

We can use the 'found_newest': True to check if we have reached the end of the list.

Now, I will work on how to integrate this with the perceval backend. Do you have any references where I can find help for this task?
Thanks.

[valeriocos]

Hi @vchrombie, thank you for the details.

You can check the backend for telegram: https://github.com/chaoss/grimoirelab-perceval/blob/master/perceval/backends/core/telegram.py . It is offset-based and I guess it shouldn't be too different from the one for zulip.

[vchrombie]

Adhering to the contributing guidelines (#incubating-repositories), this work is moved to a separate repository [1] and will be maintained over there for some time.

[1] Zulip Backend for Perceval: https://github.com/vchrombie/grimoirelab-perceval-zulip

Closing this issue.

Best,
Venu