-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extensible web plugin #718
Comments
This sounds great. Thoughts:
|
Sounds great to me :) I agree with @asutherland that the current web API is not bad and a good starting point. I'm not sure we can design a new API from scratch without developing a web client at the same time to really identify that is needed. Here are my first thoughts :
I think i'll start hacking a client and see what else I actually need ! The revision-based replication API would certainly be a nice thing too, but I'm not sure how applies to web-based UI : you still have to store the data somewhere and imo local storage API are not meant for large amount of data, but I may be wrong. Yet it would be very useful to implement a native mobile client that would provide offline music access and statistics synchronization (like doubletwist airsync does it for example). A problem I see with replication though is that you have to re-implement the full query logic (which is very powerful !) in the client application. With that in mind I would really only use that approach for offline clients working on a limited subset of the collection. |
Thanks for the discussion, both of you! Here are a few brief thoughts: Backwards compatibilityGood points, @asutherland. The only consumers I know of are: Not exactly a herculean task to bring these up to speed. Maybe we can keep the web plugin around as legacy and build the new stuff (if we break compatibility) under a built-in Heavyweight UIs and syncThis is an important thing to keep in mind and a really hard problem. I agree that it makes sense to add this kind of functionality later on as it will be a significant design challenge in itself. Current API is mostly fineYes, also true. The warts I was referring to are just simple stuff: Mozilla's DataStore is an interesting pointer. I've also been intrigued by JSON API, which is an effort to standardize the boring boilerplate in simple APIs. It should theoretically make it easier to write clients by reusing a generic JSON API library. I also don't exactly know what to do with the query syntax. I've similarly considered encoding queries into crazy JSON objects, but I concur that this is probably overkill and not particularly helpful. One thing to consider, though, would be trying to at least write a specification of the query syntax—or a compatible parser/encoder in JavaScript or something—to prove to ourselves that it's sufficient for interoperability. (It's been designed ad-hoc along the way for human-legibility rather than machine manipulation.) |
Re: local storage API, I'm mainly referring to use of IndexedDB whose semantics are roughly equivalent to LevelDB's (and is indeed backed by LevelDB on Chrome; Firefox uses a SQLite implementation.) Browsers have permission-based quota mechanisms that do a good job of making sure the user opts into non-trivial data use. There is still some work (in at least Firefox) to make sure it's easy for the user to notice the data usage and clean it up. You definitely would not want to use the localStorage API in browsers for anything more than trivial data storage since it's a foot-gun that should be avoided at all costs. Re: duplicating the query API, I would not suggest heavy-weight clients try and duplicate the general beets query mechanism. As proposed, they could still use the query API, but they could then check their local cache (which need not be exhaustive and which can be kept up to date via the sync mechanism) to avoid issuing any redundant network requests while also not having to worry about stale data. Indices and fancier use-casesMainly what I meant is that for many fancier use-cases you are going to want to have pre-computed indices/views of data to have minimal latency. Looking at the current SQLite schema for the beets database there are not a lot of indices but even if there were, many complicated queries are going to end up doing a fair amount of computation either in SQLite or beets itself which will affect latency. The lessons around DataStore were basically that 1) you can't predict every index everyone will ever want (especially since the best ones are composite and may involve values computed by app logic) and 2) indexes aren't free to maintain from a performance or disk perspective. Beets could grow support for pre-computed queries that web UIs could either define in their minimal python extension definitions or that could be dynamically created on the fly. I definitely think it would be awesome for beets to gain this type of facility as arguably it most belongs inside beets. It's just easy to go down an ORM rabbit-hole here and end up complicating beets in a bad way, or just end up with people writing bad SQL that doesn't improve things. As an example of a good thing beets could do here in the long term, I always liked CouchDB's view model. There are docs on it at http://docs.couchdb.org/en/latest/couchapp/views/intro.html. The basic idea is it's pre-computed map-reduce and you don't have to worry cluttering up the items/albums records with indexing hints. I'm not sure I'd actually recommend CouchDB as a database backend, though. But as an example of how the browser can be good for experimentation here, http://pouchdb.com/ is a JS implementation of CouchDB. (I can't speak to its efficiency though.) Delta SyncImplementing this actually wouldn't be all that hard. The main thing needed is to add an "updated" column to "items" and "albums" alongside "added" that stores the timestamp of the last update to the record. The annoying thing is dealing with deleted albums and items. This requires one of the following: a bounded tombstone table, the client to just ask for the current set of all id's if it infers something might have been deleted, or a wide set of clever options like using merkle trees or probabilistic set stuff. I expect deletions to be rare so just checking the current set of valid id's seems workable. I'm happy to provide patches along these lines, too. Although I totally agree this is a separate issue that should be thought of as an enhancement. |
What do you mean by this, @sampsyo? |
Ah, that was unclear, sorry; I just meant adding a plugin hook for extending the web server by exposing new endpoints, etc. |
I really like the @asutherland 's idea of pre-computed queries. I totally agree that we need minimal latency for a Web API and some queries might be too slow with the current implementation. While there's certainly some possible optimization it would be very hard, as you mentioned, to guess which queries must be lighting-fast and custom pre-computed queries would be very useful I'm not sure I really get are you are planning to implement that though. Plugins would be able to request the core that an arbitrary query is pre-computed. Then the core, on each modification, would check which pre-computed queries might be affected by the change and run the query (probably in background) so that the result is already computed and available (it's result would probably be stored in the database). Is that wath you have in mind ? That would be great but must be used with caution as a bad plugin could slow everything down. I also thought of two additional functionalities that would be needed :
Response to paginated queries could also contain a link to the next and previous pages (with the same page size). This would also provide end-of-list detection. I'm not sure if every paginated request should return the total number of elements : it can be useful on the client side to know this in advance, but is not always convenient (or even possible) on the server side and could be optional.
|
FWIW, sorting and pagination are in the JSON API spec. So if we go with that as a template, we don't have to design the details. |
|
Ah, good point; I thought I had read something but apparently made it up. Thanks for the advice here. |
It's exciting to see this in development. This may be too high level to consider now, but anyway... Artist/Album/Track hierarchy is suitable (and good enough) for the majority of the users, I suppose. But it really is not enough for classical music listeners. Composer is always important (sometimes the most important) data that needs to be presented (or queried for). I would really like to see beets having a user-configurable hierarchy instead of a fixed one. Filter panes of WinAmp was a very good implementation of this approach. User has a bunch of pre-configured hierarchies that filters the music collection according to certain queries (e.g., one pane would list all tracks with "Classical" or "Baroque" Genre tag and list them with a Composer/Artist/Album/Track hierarchy). I don't know if this would be too heavy-weight, but this kind of flexibility is something I really miss among the server-client media libraries. (Next logical step from my point of view would be an MPRIS plugin for beets :)) |
One note on that (configurable hierarchies): beets has for some time shipped with a tiny |
Would love to help out with any UI (HTML/jQuery/CSS/Other frontend) stuff. Have loads of ideas for editing tags and fronting the db in browser. Sorry I can't be much help with the coding stuff, but am willing to learn. |
So what's the status on this? What's our first step? (am I perhaps missing where discussion is taking place?) |
Good question, @jjrh. I've broken this up into a few tickets that I think represent solid units of work:
If you're interested helping out, I suggest you add yourself to one or several of these tickets. (For example, @jonthomas83 may be interested in #738.) Let the good work begin! |
@sampsyo Sorry, what do you mean by "add myself" to the ticket? I don't see any way to do this in github. Do you mean 'subscribe' to it? |
Yep! Or just comment, which (a) subscribes you and (b) shows you in the "participants" thing on the side. |
Did I miss some further discussion or did people just get busy with other things too? I know Pierre Rust had started work ( https://github.com/PierreRust/beets/tree/web-api/beetsplug ) Is there a newer branch I should work from? |
I've started designing a new API for this hypothetical new plugin. The idea is to design a good REST API that should be usable by other projects as well. The docs and a reference implementation are currently hosted over in another repository: https://github.com/sampsyo/aura Please let me know if you're interested in collaborating on that piece of the puzzle. |
Adrian, this looks really exciting! Great work. From what I can understand GUI & UX helper here when you need him! :-) On 1 November 2014 21:53, Adrian Sampson [email protected] wrote:
|
Here's a little feedback of what I learned with a project of mine: The possible solutions are:
|
Neat! Whoosh looks nifty—I'll need to read more. Out of curiosity, can you elaborate more on where you ran into pain points with SQLite? For example, was pagination just the final straw, or was everything fine until you tried to add pagination? |
In fact, it's not related to paging, it's related to the use of |
Oh, right, that makes total sense—it's something that affects us already with ordinary, command-line beets queries. A real index for textual queries would be a nice thing to have someday, not just for web-based uses but for all beets interactions. |
Following discussion in #456 and on the mailing list here and here, it's time to get slightly more serious about overhauling the web plugin for extensibility.
Here's my wishlist:
/v2/
or something. The new API should be standards-worthy: clean and well-documented, potentially even for reuse by other tools.How does all that sound to everyone?
The text was updated successfully, but these errors were encountered: