Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement time ago parser and improve localization handling #158

Merged
merged 8 commits into from
Nov 15, 2019
Merged

Implement time ago parser and improve localization handling #158

merged 8 commits into from
Nov 15, 2019

Conversation

mauriciocolli
Copy link
Contributor

@mauriciocolli mauriciocolli commented Apr 25, 2019 •
edited
Loading

Used some of the work that @wojcik-online did as the base, finishing the implementation of the time ago parser.

  • Time ago parsing for stream items where the service shows it (search, channel videos list)
  • Handle special cases for languages where the number is not shown
    • Hebrew, for example, shows "לפני שעתיים" for "2 hours ago". We cannot assume that it is 1 for those cases (popular among other languages), as it would be wrong in the bigger time units.
  • Rework the Downloader base implementation, allowing for more advanced things to be done
    • Get the response headers
    • Response code status and message
    • Possibility of making other type of requests (OPTIONS, HEAD, POST...)
  • Separate the localization from the content country (just like YouTube let's the user choose both).
  • Parse watching count in live streams items
  • Fix parsing of future videos (those with an option to set a reminder)

Edit: So I did see the PR #196 (related TeamNewPipe/NewPipe#2632), and as a result, no language parsing is needed anymore and english will be forced over others.

However, I think the language handling contained in this PR is still useful for NewPipe as we can now force languages easily and, in the future, help parse sites in other languages.

@mauriciocolli mauriciocolli mentioned this pull request Apr 25, 2019
Merged
@theScrabi
Copy link
Member

We need review and test on this. Someone willing to help?

@theScrabi theScrabi requested a review from yausername June 27, 2019 12:36
yausername
yausername suggested changes Jul 1, 2019
View reviewed changes
Copy link
Contributor

@yausername yausername left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good to me!

.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java Outdated Show resolved Hide resolved
@chetan00000007

This comment has been minimized.

Sign in to view
@mauriciocolli
Copy link
Contributor Author

I'll start to work on this again over the next few days (still have some catch up to do).

So I did see the PR #196 (related TeamNewPipe/NewPipe#2632), and as a result, no language parsing is needed anymore and english will be forced over others.

However, I think the language handling contained in this PR is still useful for NewPipe as we can now force languages easily and, in the future, help parse sites in other languages.

@TobiGr @Stypox @theScrabi

@Stypox
Copy link
Member

Stypox commented Oct 2, 2019

Yes, this is useful, who knows what the future will bring :-)

@TobiGr
Copy link
Contributor

TobiGr commented Oct 2, 2019

Hey @mauriciocolli, welcome back!

However, I think the language handling contained in this PR is still useful for NewPipe as we can now force languages easily and, in the future, help parse sites in other languages.

Yes, that functionality is a good addition. Please keep it.

@mauriciocolli mauriciocolli force-pushed the timeago-localization branch 2 times, most recently from 7a0bd92 to 6c5af3f Compare October 11, 2019 01:26
mauriciocolli added a commit to mauriciocolli/NewPipe that referenced this pull request Oct 11, 2019
@mauriciocolli
Use new Localization and Downloader implementations from extractor
f69ae24 
(TeamNewPipe/NewPipeExtractor#158)
mauriciocolli added a commit to mauriciocolli/NewPipe that referenced this pull request Oct 11, 2019
@mauriciocolli
Use new Localization and Downloader implementations from extractor
063bd33 
(TeamNewPipe/NewPipeExtractor#158)
mauriciocolli added a commit to mauriciocolli/NewPipe that referenced this pull request Oct 11, 2019
@mauriciocolli
Use new Localization and Downloader implementations from extractor
53f096c 
(TeamNewPipe/NewPipeExtractor#158)
@mauriciocolli mauriciocolli mentioned this pull request Oct 11, 2019
Merged
@mauriciocolli mauriciocolli force-pushed the timeago-localization branch 2 times, most recently from c99f44b to 49c736d Compare October 28, 2019 02:14
wojcik-online and others added 8 commits November 3, 2019 15:46
@wojcik-online @mauriciocolli
Base Implementation: Parse the upload date of StreamInfoItems
180836c 
In the format '2 days ago' (in English) on a YouTube channel page.
(Parser extensible to other pages.)
@mauriciocolli
Implement time ago parser and improve localization handling
3638f0e 
- Handle special cases for languages where the number is not shown
- Rework the Downloader base implementation, allowing for more
advanced things to be done
- Separate the localization from the content country (just like
YouTube let's the user choose both).
@mauriciocolli
[YouTube] Parse watching count in live streams items
d8280ce
@mauriciocolli
[YouTube] Fix parsing of video reminders
cb1e327
@mauriciocolli
Add instructions to quickly test changes in the extractor
b42bebd 
- Make use of the local Maven repository
@mauriciocolli
[YouTube] Force english as the only supported language (TeamNewPipe#196)
41b51ad
@mauriciocolli
Temp Commit: Ignore Test (Remove this before merge)
6ca4c89
@mauriciocolli
Introduce class that indicates when the time ago is an approximation
3d21ef5
@TobiGr
Copy link
Contributor

TobiGr commented Nov 9, 2019

That were a few lines to review :D Looks fantastic to me.

@TobiGr TobiGr merged commit 5c42034 into TeamNewPipe:dev Nov 15, 2019
@timfbfbfbfb timfbfbfbfb mentioned this pull request Nov 17, 2019
Closed
3 tasks
@mauriciocolli
Copy link
Contributor Author

@TobiGr Oh, the temp commit (6ca4c89) went through. Sorry for the late reply.

@TobiGr
Copy link
Contributor

TobiGr commented Nov 18, 2019

@mauriciocolli Oh yes. I reverted it in 0c6e2c8

@B0pol B0pol mentioned this pull request Feb 13, 2020
Open
3 tasks
@mauriciocolli mauriciocolli deleted the timeago-localization branch March 7, 2020 20:02
@AudricV AudricV mentioned this pull request May 3, 2023
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yausername yausername yausername requested changes

@TobiGr TobiGr TobiGr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@mauriciocolli @theScrabi @chetan00000007 @Stypox @TobiGr @yausername @wb9688 @wojcik-online