Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean data based on Interval/Pattern #22

Closed
mollle opened this issue Oct 30, 2024 · 1 comment · Fixed by #38
Closed

Clean data based on Interval/Pattern #22

mollle opened this issue Oct 30, 2024 · 1 comment · Fixed by #38
Assignees
@dh1542
Labels
component Related to a RTDIP component

Comments

@mollle
Copy link

mollle commented Oct 30, 2024

Example Data Timestamps:
00:00
01:00
01:10
02:00

If you expect the data to come in every minute, it would be nice to have an option to get rid of the entry at 01:10, but duplicate detection wont catch it.
Component would get either an interval (e.g. '1min') or a pattern [{'second': 0}, {'second': 13}, {'second': 49}] and a tolerance and allow only one entry in the time frame and delete all other data out of the time frame.
The logic is kind of similar to the identify missing data.
@dh1542 dh1542 self-assigned this Nov 4, 2024
dh1542 added a commit that referenced this issue Nov 4, 2024
@dh1542
#22: Initial commit
366360d 
Signed-off-by: Dominik Hoffmann <[email protected]>
@chris-1187 chris-1187 self-assigned this Nov 4, 2024
dh1542 added a commit that referenced this issue Nov 5, 2024
@dh1542
#22: Added boilerplate and test class
11c288b 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 5, 2024
@dh1542
#22: Updated description and added parameter
8bff78c 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 5, 2024
@dh1542
#22: implemented first steps of interval filtering
4f2a646 
Signed-off-by: Dominik Hoffmann <[email protected]>
@chris-1187 chris-1187 removed their assignment Nov 5, 2024
dh1542 added a commit that referenced this issue Nov 5, 2024
@dh1542
Merge branch 'develop' into #22-Clean-Data-Based-on-Intervall
e8b6500
@luccalb luccalb added the component Related to a RTDIP component label Nov 6, 2024
@luccalb luccalb changed the title [Component] Clean data based on Interval/Pattern Clean data based on Interval/Pattern Nov 6, 2024
dh1542 added a commit that referenced this issue Nov 6, 2024
@dh1542
#22: Returned dataframe. Need to pass also timestamp format.
3a41d10 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
Merge branch 'develop' into #22-Clean-Data-Based-on-Intervall
1f3218e
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Expanded test cases to cover all units and multiple intervals
396b49f 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Fixed logic and unit mismatches
9ec9b45 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Fixed uncaught exception
c034b31 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Added test cases for thrown exceptions
c4752c7 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Fixed exception print
0bcd73e 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Added years test case
f02cfb5 
Signed-off-by: Dominik Hoffmann <[email protected]>
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Implemented tolerance
5375d50 
Signed-off-by: Dominik Hoffmann <[email protected]>
@dh1542 dh1542 linked a pull request Nov 9, 2024 that will close this issue
#22 clean data based on intervall #38
Merged
dh1542 added a commit that referenced this issue Nov 9, 2024
@dh1542
#22: Linted/Formatted code + fixed import
9fd41b8 
Signed-off-by: Dominik Hoffmann <[email protected]>
@dh1542
Copy link

dh1542 commented Nov 9, 2024

Implemented logic according to description with interval value + unit and tolerance.

mollle added a commit that referenced this issue Nov 12, 2024
@mollle
Merge pull request #38 from amosproj/#22-Clean-Data-Based-on-Intervall
2ef20ee 
#22 clean data based on intervall
@github-project-automation github-project-automation bot moved this from Awaiting Review to Feature Archive in amos2024ws01-feature-board Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dh1542 dh1542

Labels
component Related to a RTDIP component
Projects
amos2024ws01-feature-board
Status: Feature Archive
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#22 clean data based on intervall amosproj/amos2024ws01-rtdip-data-quality-checker
4 participants
@luccalb @dh1542 @mollle @chris-1187