Fix bug recurring events

[Community-Programmer]

What kind of change does this PR introduce?
Fixed the bug on creating recurring event only single event is created

Issue Number:

Fixes #1583

Did you add tests for your changes?
yes

Snapshots/Videos:

2024-01-17.23-14-33.mp4

If relevant, did you update the documentation?
No

Summary
Task-2

• Create Daily event recurring instance

Implementation Details:
Validation:

No other functionalities is affected by changes made in the createEvent mutation
All test files passed

Does this PR introduce a breaking change?
Yes

Have you read the contributing guide?
Yes

[github-actions]

Our Pull Request Approval Process

We have these basic policies to make the approval process smoother for our volunteer team.

Testing Your Code

Please make sure your code passes all tests. Our test code coverage system will fail if these conditions occur:

1. The overall code coverage drops below the target threshold of the repository
2. Any file in the pull request has code coverage levels below the repository threshold
3. Merge conflicts

The process helps maintain the overall reliability of the code base and is a prerequisite for getting your PR approved. Assigned reviewers regularly review the PR queue and tend to focus on PRs that are passing.

Reviewers

When your PR has been assigned reviewers contact them to get your code reviewed and approved via:

1. comments in this PR or
2. our slack channel

Reviewing Your Code

Your reviewer(s) will have the following roles:

1. arbitrators of future discussions with other contributors about the validity of your changes
2. point of contact for evaluating the validity of your work
3. person who verifies matching issues by others that should be closed.
4. person who gives general guidance in fixing your tests

CONTRIBUTING.md

Read our CONTRIBUTING.md file. Most importantly:

1. PRs with issues not assigned to you will be closed by the reviewer
2. Fix the first comment in the PR so that each issue listed automatically closes

Other

1. 🎯 Please be considerate of our volunteers' time. Contacting the person who assigned the reviewers is not advised unless they ask for your input. Do not @ the person who did the assignment otherwise.
2. Read the CONTRIBUTING.md file make

[palisadoes]

This is a Daily recurring event follow up PR to the Weekly calendar PR merged earlier today.

• Create weekly event recurring instance #1658

Please take a look:

1. @xoldyckk
2. @sumitra19jha
3. @DMills27
4. @JamaicanFriedChicken

[Community-Programmer]

@palisadoes
After this, can we have an issue on Talawa Admin? I would like to integrate these features with Talawa Admin
Thank you

[palisadoes]

Please review. This is a fix for the daily recurring calendar entry that was never implemented

1. @xoldyckk
2. @sumitra19jha
3. @DMills27
4. @JamaicanFriedChicken
5. @EshaanAgg

[palisadoes]

@palisadoes After this, can we have an issue on Talawa Admin? I would like to integrate these features with Talawa Admin Thank you

Yes, we'll need verification.

[EshaanAgg]

The code is functioning, but it certainly is not scalable. Even if a user enters a date too far into the future by mistake, this will blow up our database with events that we wouldn't be using anytime soon. We need to work on a dynamic strategy that only generates these events as and when required. You need to research how other large calendar based products like Google Calender, Taskade etc. store and query the same efficiently.

[EshaanAgg]

You certainly don't need to add all the functionality in one PR, but you need to first devise a strategy that accounts for all the future and edge use cases. Then we can work on incorporating each functionality in a step by step manner.

[Community-Programmer]

@EshaanAgg

Thank you for the thorough review and insightful suggestions. Your points on scalability and potential database issues were very valid. After conducting further research, I discovered that Google Calendar employs a similar approach by limiting event creation to 100 years into the future.

Can we follow the same?

By restricting event creation to a reasonable timeframe, we can mitigate the risk of database overload and ensure a more scalable solution.

[EshaanAgg]

Can you share the link for the same?
Also, even if Google does it, I don't feel the same is still feasible for us. They have dedicated SQL databases for the same, while most developers we cater to use the free Mongo Atlas cluster. We need to work on a strategy for the dynamic creation of events based on the pattern of their generation. Read this discussion and try to devise a strategy that works for us.

[palisadoes]

@EshaanAgg

1. We recently merged the equivalent for weekly recurring. 1. So we'll need to adjust that functionality to match the outcome of this discussion.

[EshaanAgg]

Don't stick to the link I shared. There are many other blog posts about the same. Please try to read as many as you can and come up with a strategy for efficient querying. Please also share all the links that guided your design when sharing, so that even we can vet them once.

[Community-Programmer]

@EshaanAgg

1. We recently merged the equivalent for weekly recurring. 1. So we'll need to adjust that functionality to match the outcome of this discussion.

1. We recently merged the equivalent for weekly recurring. 1. So we'll need to adjust that functionality to match the outcome of this discussion.

@palisadoes
I will make the necessary adjustments to the functionality for weekly recurring events to align with the outcome of this discussion in the same pull request.

[EshaanAgg]

@EshaanAgg

1. We recently merged the equivalent for weekly recurring. 1. So we'll need to adjust that functionality to match the outcome of this discussion.

I see. Either we should revert it, or we can let it be dormant. When this discussion is completed, we can update it's code too. The ideal solution once achieved should allow for all levels of flexibility in recuraance patterns without much additional configuration.

[Community-Programmer]

Don't stick to the link I shared. There are many other blog posts about the same. Please try to read as many as you can and come up with a strategy for efficient querying. Please also share all the links that guided your design when sharing, so that even we can vet them once.

@EshaanAgg
Okay, I'll conduct thorough research by reading various blog posts on efficient querying. I'll formulate a strategy based on the insights gathered. When sharing the design, I'll include all the relevant links so that we can collectively review and vet them. I'll keep you updated on my progress.

please have a look at this once
https://stackoverflow.com/questions/50204299/recurring-events-database-model

[Community-Programmer]

@EshaanAgg @palisadoes

We can follow various approaches for managing events:

• We can store cron expressions instead of storing each occurrence or follow a pattern-based recurrence. However, this may lead to a more complex structure and querying process.

In that case , updating a single occurrence within the recurring series becomes more challenging because MongoDB operates on the document level

• The main issue arises when we need to edit any recurrence of the event at any time it occurs, and track attendance for any type of event, etc.

I believe for this scenario, a denormalized structure is better for storing each occurrence separately. One approach could be to reduce constraints, such as allowing the creation of events for only a 10-year period.

what your views on it ?

[palisadoes]

Regarding the recurrence limits:

1. The limit cannot be solely date limited.
2. 100 years of monthly events is very different from 100 years of daily events.
3. There will need to be a sequence count limitation too that would depend on the literature you find.

I'm not a professional coder, so this should be taken into account for my next suggestion. I'd like to hear your feedback.

1. I think the denormalized approach is simpler to implement, though more of a data tax.
   1. We link each event in a series to a recurrence ID
   2. When a recurring event is edited you have the option of:
      1. Updating all event entries with the same ID
      2. Updating just this event in the series, by giving it a new ID and leaving the rest with the old ID.
      3. There will need to be a way to track historical IDs because:
         1. you may make a change in today's event (New ID Updated Read the docs #1) and save
         2. you change your mind, and make another change in today's event (New ID#2), but want it reflected in all future events too, but without an ID history it's impossible to do.
            1. A linked list of events could solve this too

This was the approach recommended here:

• https://stackoverflow.com/questions/50204299/recurring-events-database-model

You could also do a representation of a recurring event as a start date with an instance count tracker to save space. But that would be compute intensive when people scroll through calendars and the API has to generate recurring events in real time while simultaneously presenting non-recurring events.

In the issue discussion thread we thought it would be easier to make all events recurring, even one-time events (0 repetitions), as it would:

1. create standardized logic and;
2. facilitate a single API call.

The editing process would similarly need a historical tracker for the same reasons I mentioned previously.

[Community-Programmer]

Absolutely, I agree with your thoughts on the denormalized approach for managing recurring events. The idea of incorporating a sequence count limitation, especially for daily events over a 5-10 year period, seems like a sensible enhancement.I should have applied this approach previously but I didn't know about database constraints.

Treating every event as recurring not only simplifies the logic but also streamlines the codebase.
Moreover, the denormalized structure indeed ensures efficient updates and makes it feasible to track historical changes effectively. It's a well-rounded approach that addresses both practical considerations and maintainability.

Overall I think it can be a good call!
Need reviews from others also on this approach

[Community-Programmer]

@EshaanAgg @palisadoes

After conducting in-depth analysis and research, I have an approach to use a denormalized structure with minimal impact on the database, particularly for MongoDB, which operates at the document level. This strategy is crucial as it involves managing and tracking edits for each event.

Approach:

• Initially, we categorize all events as non-recurring, ONCE (0 repetition).

• For events with WEEKLY, MONTHLY, and YEARLY recurrence patterns, we set the default endDate as 2124-12-31, implying a duration of up to 100 years from the current year. For instance, considering monthly events, there can be a maximum of 5 weekly events in a week and 12 events in a year. Over 100 years, this results in approximately 12 * 5 * 100, totaling around 6000 events.

• For DAILY events, we allow users to create daily events for up to 10 years only, generating around 365 * 10, which amounts to 3650 events. Additionally, considering that the size of all events will be in kilobytes, this quantity of events is manageable. Users attempting to create daily events beyond the 10-year limit will receive an error message, advising them of the constraint.

By implementing this approach, we effectively limit the number of events stored, ensuring efficient data storage and query performance

Articles:

https://scullwm.medium.com/design-and-manage-recurring-events-fb43676e711a

I have also come across another method mentioned in the article that involves the exclusion of a particular event. However, this approach presents a challenge as it prevents us from tracking all events.

https://news.ycombinator.com/item?id=18477975

I would appreciate your input on this matter so that we can move forward and successfully implement this feature within the given time frame.

[palisadoes]

@Community-Programmer @EshaanAgg

The RFP (https://www.ietf.org/rfc/rfc2445.txt) referenced in the Medium article has the basics of what we were talking about in creating a recurring instance tracker.

RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4;UNTIL=19980404T070000Z

Recurring events have an RRULE entry inserted in the event and it looks like they use a linked list format too, as I was thinking. With a sequence of multiple recurring rules each with a specified start and end date.

The RRULE also allows you specify days of the week etc.

I even found this NPM package. Most of our work may be solved. Just use the schema it recommends.

• https://www.npmjs.com/package/rrule

I vote for the RFP approach. It's what iCal is based on.

[Community-Programmer]

Ok sounds good so we are going to store rrule string instead of creating instances for each event right?

[palisadoes]

1. It looks like it would simplify things a lot. You don't have to worry about the logic, it's already done. It generates all the dates automatically.
2. The RPF seems to explain how to keep track of successive modifications of recurring instances. When an event is changed, you'll need to modify all the future events previously created by the RRULE.
3. Let us know if this methodology is better and faster to implement than what you had before.

[Community-Programmer]

I think it will somehow help us to create recurring events logic and will give us recurrance dates.
main question is that how we are going to handle a single event in recurring series, if we want to change title, description or anything related to event?

[palisadoes]

In this case, couldn't you replace the existing document with a new one that points back to the original recurring instance?

[Community-Programmer]

Do we need to add exclusions for the day that is edited or updated, and should it point to the parent document?

We can certainly do that, but using rrule might have a significant impact on the overall project.

Here are some considerations:

• Firstly, we have the eventDashboard for every event, which involves adding projects and attendees.
• Additionally, we have more issues running such as adding volunteer groups and event agendas, making it even more complex.
• Tracking attendance for any type of event will also become difficult.
• If a user edits the event with the option to affect following events, we would need to add exclusions for all those events. This could lead to complex mongoose documents.

Implementing this will take some time, and we will need to make changes to several functions. There will be complex queries, and a majority of functionalities all over the project will be affected and need adjustments accordingly.

[xoldd]

dynamically generating the events that correspond to the same recurring event with identical data seems to be the better option

an instance of a recurring event that has been modified by the end user(an exception that has deviated from the identical data present in other instances corresponding to the same recurring event) should be created and stored seperately in another table

more info in this article:- https://vertabelo.com/blog/again-and-again-managing-recurring-events-in-a-data-model/

[Community-Programmer]

@xoldyckk @EshaanAgg @palisadoes

I am totally confused about which method to follow for recurring events creation,
also achieving dynamic event generation, editing each event, deleting events, and managing the event dashboard. Please guide me on how to proceed, the flow of development, as well as how events are rendered in the frontend. Additionally, provide insights into the necessary mutations, queries, models, and schemas. I really need your guidance on this

[palisadoes]

I agree with @EshaanAgg and @xoldyckk. If we dynamically generate and insert events beyond some global LIMIT, then we reduce DB bloat. It is also unlikely that there will be calendar view overload because the first person who views the event beyond the LIMIT will create the event, thereby relieving the need to do further computation by others viewing the same event.

The extra coding to do this shouldn't be excessive as RRULE will be doing most of the work.

Seeing the updated schema would assist.

[palisadoes]

@Community-Programmer, @xoldyckk, @EshaanAgg

This is my understanding.

1. As before:
   1. Track with an RRULE collection
   2. However, generate up to LIMIT denormalized events
2. New stuff:
   1. Generate additional denormalized events when someone wants to see beyond the previously generated LIMIT number of denormalized events, say at some future DATE. Possibly adding LIMIT denormalized events beyond DATE.

[palisadoes]

We can't do a pure generation on demand as the events in our series will often be uniquely modified. Possible cases include:

• New invitees
• New volunteers
• New agenda items

With this method, future generated events would copy all the characteristics of the most recently generated event, ie. the event at the LIMIT date.

[palisadoes]

@Community-Programmer is this clearer now?

[Community-Programmer]

@palisadoes @xoldyckk @EshaanAgg

This is the conclusion I have drawn from the conversation.
Please tell me if I am correct or not

WhiteBoard Link

[palisadoes]

1. So you are going with a generate events on demand without the LIMIT
2. How will you handle someone changing the date of just one event in the series with your schema?

[Community-Programmer]

1. So you are going with a generate events on demand without the LIMIT
2. How will you handle someone changing the date of just one event in the series with your schema?

@palisadoes

No, 'not on demand' means that, on event edit, only the event that is edited will be created and stored as a skipped event. This implies that we will not be generating any events until they are edited

[Community-Programmer]

Can you please explain the flow? I believe there might be some conflicts in our understanding of the solution. What exactly are you thinking?

[palisadoes]

1. Are skipped events the ones that are out of sequence?
   1. If that's the case then the name you have chosen isn't intuitive. Skipped events sounds like events that should be ignored or not attended.
   2. Or are they events that should not be in the RRULE series because another out of sequence event has replaced it? If that's the case, how is the new event tracked as part of the series for future updates?
2. Based on your flow, events that are queried will be created on demand if they don't previously exist. Is that correct?
3. You'll need logic to handle someone changing an event to be out of sequence and then changing it back to be in sequence.

[Community-Programmer]

yaa exactly skipped events are events that should not be in the RRULE series as they have been edited or modified so they will be stored as exclusion in the database and refer to the parent_event.

The flow is very simple only create event that is updated rest all will be generated using RRule and same will be returned to frontend for rendering calendar.

I am talking about below approach

[palisadoes]

This is the link we should reference:

1. https://news.ycombinator.com/item?id=18477975

From the discussion:

1. Their use of the exclusion terminology makes more sense, but I think exceptions is better. It looks like there are two types of exception events:
   1. Events in the series that must not be visible (as in deleted)
   2. Events in the series that have been edited in some way (some other flag)

This means that the RRULE object would need to include an event object to set the initial parameters to use for the series. This would need to be reflected in your schema.

If a user decides to change any parameter in the RRULE string from a point in time onward, how would that work? It would seem to be like this, which is validated by my Google Calendar tests

1. Edit the original RRULE
   1. Change the end dateTime
   2. Leave the exceptions unchanged or cleared?
2. Create an RRULEnew using:
   1. the point in time event as the seed
   2. an empty exceptions list

Google Calendar (Test: 1):

1. Created a recurring daily event (Title: Test Daily)
2. One week later I edited it to become a weekly event (Title: Test Weekly)
3. One month later I edited it again to become a monthly event (Title: Test Monthly)

Deleting all the weekly events, doesn't delete the daily or monthly. They are handled as completely different recurrences.

Google Calendar (Test: 2):

1. Created a recurring daily event (Title: Test Daily)
2. One week later I added somebody to the invitee list
3. I deleted the event, all instances, and all entries were deleted.

Google Calendar (Test: 3):

1. Created a recurring daily event (Title: Test Daily)
2. One week later I changed the title of the event
3. I deleted the event, all instances, and all entries were deleted.

It appears Google only creates a new recurrence instance when the recurrence definition changes.

This looks fairly straightforward to implement. Please post the updated schema.

[palisadoes]

@Community-Programmer

Have you seen this thread? You'll need to ensure that your solution works with the community version of MongoDB. If this is the case, why did your tests pass for the previous PR?

• https://thepalisadoes-dyb6419.slack.com/archives/CSWGQ70UQ/p1705924020861829

[Community-Programmer]

@palisadoes

As discussed earlier, the solution we're trying to implement at starting requires database replication for functioning of transactions. The error you encountered with the standalone instance is due to the absence of replication.

In the previous pull request, the tests passed because we were not starting a session. Transactions were not initiated, which is why the issue didn't arise.
Rest assured, the approach we're taking now does not rely on transactions, eliminating the need for them in our setup.
Also please revert the previous PR as we are using new approach now

[Community-Programmer]

This is the link we should reference:

1. https://news.ycombinator.com/item?id=18477975

From the discussion:

1. Their use of the exclusion terminology makes more sense, but I think exceptions is better. It looks like there are two types of exception events:

   1. Events in the series that must not be visible (as in deleted)
   2. Events in the series that have been edited in some way (some other flag)

This means that the RRULE object would need to include an event object to set the initial parameters to use for the series. This would need to be reflected in your schema.

If a user decides to change any parameter in the RRULE string from a point in time onward, how would that work? It would seem to be like this, which is validated by my Google Calendar tests

1. Edit the original RRULE

   1. Change the end dateTime
   2. Leave the exceptions unchanged or cleared?

2. Create an RRULEnew using:

   1. the point in time event as the seed
   2. an empty exceptions list

Google Calendar (Test: 1):

1. Created a recurring daily event (Title: Test Daily)
2. One week later I edited it to become a weekly event (Title: Test Weekly)
3. One month later I edited it again to become a monthly event (Title: Test Monthly)

Deleting all the weekly events, doesn't delete the daily or monthly. They are handled as completely different recurrences.

Google Calendar (Test: 2):

1. Created a recurring daily event (Title: Test Daily)
2. One week later I added somebody to the invitee list
3. I deleted the event, all instances, and all entries were deleted.

Google Calendar (Test: 3):

1. Created a recurring daily event (Title: Test Daily)
2. One week later I changed the title of the event
3. I deleted the event, all instances, and all entries were deleted.

It appears Google only creates a new recurrence instance when the recurrence definition changes.

This looks fairly straightforward to implement. Please post the updated schema.

I believe only minimal changes are needed in the schema. Adding just one field for the 'rrule' string and another for exception events should be sufficient to handle it.
And rest RRULE will manage as we have option to exclude a date

[palisadoes]

@palisadoes

As discussed earlier, the solution we're trying to implement at starting requires database replication for functioning of transactions. The error you encountered with the standalone instance is due to the absence of replication.

In the previous pull request, the tests passed because we were not starting a session. Transactions were not initiated, which is why the issue didn't arise. Rest assured, the approach we're taking now does not rely on transactions, eliminating the need for them in our setup. Also please revert the previous PR as we are using new approach now

The PR cannot be reverted it's too late. So you'll have to update those files in this PR

[palisadoes]

OK, let's proceed as we now have a way to minimize the data footprint while optimizing CPU load.

[palisadoes]

Adding for cross referencing

• Create weekly event recurring instance #1658

[Community-Programmer]

@palisadoes
As discussed earlier, the solution we're trying to implement at starting requires database replication for functioning of transactions. The error you encountered with the standalone instance is due to the absence of replication.
In the previous pull request, the tests passed because we were not starting a session. Transactions were not initiated, which is why the issue didn't arise. Rest assured, the approach we're taking now does not rely on transactions, eliminating the need for them in our setup. Also please revert the previous PR as we are using new approach now

The PR cannot be reverted it's too late. So you'll have to update those files in this PR

OK, I will make all the changes in this pull request only

[Community-Programmer]

OK, let's proceed as we now have a way to minimize the data footprint while optimizing CPU load.

@palisadoes @xoldyckk @EshaanAgg

OK, so I am proceeding now with my first steps:

• Updating the createEvent schema by adding a field for rruleString.
• Creating a query that will return all the events based on RRULE to the frontend.

Afterwards, I will focus on editing events and adding an exception for the edited and modified event

[palisadoes]

@Community-Programmer

1. We have discussed a lot of logic in this thread which will be useful for other developers.
2. Please add a detailed description of the workflow in the TypeDoc section of createEvent in src/resolvers/Mutation/createEvent.ts as part of this PR

[codecov]

Codecov Report

Attention: 85 lines in your changes are missing coverage. Please review.

Comparison is base (c0468a4) 98.17% compared to head (e6f92a8) 98.38%.
Report is 182 commits behind head on develop.

Files	Patch %	Lines
src/utilities/PII/decryption.ts	0.00%	17 Missing ⚠️
src/resolvers/Mutation/updateUserProfile.ts	70.90%	1 Missing and 15 partials ⚠️
src/resolvers/middleware/currentUserExists.ts	45.83%	13 Missing ⚠️
src/utilities/PII/encryption.ts	0.00%	13 Missing ⚠️
src/utilities/PII/isAuthorised.ts	0.00%	11 Missing ⚠️
src/resolvers/Mutation/createPost.ts	87.23%	6 Missing ⚠️
...tilities/encodedVideoStorage/uploadEncodedVideo.ts	96.29%	3 Missing ⚠️
src/resolvers/Mutation/removeAdvertisement.ts	92.85%	2 Missing ⚠️
...c/resolvers/Query/postsByOrganizationConnection.ts	33.33%	2 Missing ⚠️
src/resolvers/Mutation/createEvent.ts	99.16%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1712      +/-   ##
===========================================
+ Coverage    98.17%   98.38%   +0.20%     
===========================================
  Files          184      203      +19     
  Lines        10767    12473    +1706     
  Branches       835     1018     +183     
===========================================
+ Hits         10571    12272    +1701     
+ Misses         186      171      -15     
- Partials        10       30      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Community-Programmer]

Sorry,the force push was an unintended action, that resulted in the automatic closure of the pull request (PR). I understand the inconvenience and disruption this may have caused.

Thank you for your understanding