-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bolt for Cloud Firestore #216
Comments
I bootstrapped a generator for firestore at wcandillon/firebase-bolt-compiler#18. Nothing working yet but just to let you know. The semantic of Bolt doesn't appear to match 100% the one of firestore but I'm trying see if the generator can work for small use cases. |
@wcandillon Awesome 👍 Yes, some changes will be needed to the semantic, for example: collection /users {
document /{userID} is User {
collection /tweets {
document /{tweet} is Tweet
}
}
}
type User {
name: String
}
type Tweet {
message: String,
postedAt: Date
} |
@SamyPesse I implemented a small use case with firestore and I think it can be done with Bolt, I will keep you posted. |
I believe that Cloud Firestore's rules system is supposed to be loosely modeled after Bolt, I don't know if we're looking to support anything here for Cloud Firestore. What does Bolt provide that the current rules language doesn't? |
@rockwotj Here are the main reasons we prefer Bolt:
For example, this is easy to read, understand and extend:
And this is not (and yet it's still a small example without write rules):
I'm a big fan of Firebase, and my company is currently investing in it. But I'm really worried about the focus you have on security.
A few firebase production applications that I checked out had obvious security flaws, it was possible to read/write almost everything in their database, I've contacted them to signal these flaws. I understand that pushing too much documentation/tooling about security to beginners who are building prototypes may push off some of them, but there should be a way for Firebase to help serous companies build reliable and secure applications on top of it. |
@SamyPesse I heard you loud and clear yesterday as for as bumping security concerns, I'm working with our Developer Relations and Documentation folks to improve things here for the Realtime Database (Better docs on best practices, examples of how to version and deploy rules across environments, etc). Firestore is also working on improving this, @mcdonamp can explain more on the efforts there. As for the rules example you gave, you can have functions in the new rules languages (see here), which I think will help make things more manageable/readable, but I do understand the appeal of having "types" at locations in your database. So you could make your example the following:
Thanks for your input! That's super helpful to hear. |
@rockwotj That was not me, That must have been Aaron, my co-founder :) Thank you for your answer! Let us know if you need some infos on how our team is using Firebase. |
@SamyPesse Ah sorry I never caught his name, and I saw the GitBook connection, so thought it was you! I love the feedback, it's not harsh, but I totally agree we need to do more in this area... stay tuned! If others have thoughts on this topic, I'd love to hear them as well. |
(BTW, I just found that we have a simulator here: https://github.com/firebase/bolt/blob/master/src/simulator.ts, which might be useful for some of those tests you've mentioned, it does require hitting a real firebase database however. I will add some documentation for this in the near future.) |
Currently, we have setup 800 unit tests using Jest and targaryen, tests don't require a real database and look like: expect({ uid: 'foo' }).canRead('/users/someone');
expect({ uid: 'foo' }).cannotRead('/users'); It works very well, but We've already setup a JS layer on top of the firebase database so that our application can be "database agnostic", and we can switch from the RTDB to Firestore as soon as we've updated the rules and the tests are passing. Our current setup The application code is ready, but adapting more than 1300 lines of Bolt to the Firestore security rules syntax without being able to run unit tests is an impossible task. Basically my biggest request for Firebase is not to support Firestore export for Bolt, but to release a JS assertion library for unit testing the new Firestore rules. |
Chiming in here to put my support behind a bolt compilation to Firestore Schema. Moreover, why not just support Bolt in the Firebase Console? I dread migrating security rules forward and am hesitant to invest in another schema language (Firestore) that is a step backwards from Bolt. Bitterness aside, Firestore Schema is a step in the right direction. Please keep up the good work evolving that language. |
@rockwotj I like the current security language but it would be great if you could provide support Type support the same way firebase bolt does. Providing us with with parsers and tools for the firestore security language would be great since we can use it to generate other code artefacts (as described here). The testing part as mentioned by @SamyPesse is critical too. I like firestore a lot ❤️ and I'm looking forward to see the updates on this side of things. |
(Product Manager for Rules here) First off, thanks for all the feedback. As @rockwotj said, please keep it coming--it's how we improve the product. To summarize the above, it sounds like there are a few requests to address Rules pain points:
I'll handle these in separate replies so we can reference them independently. |
@SamyPesse I can definitely see how types would be desirable for Firestore. Our preference has been to support this sort of feature in Firestore directly with the ability to import the types into the latest security rules so you could write the following:
Type information is super useful for code-generation and schema enforcement, and because it's a separate concern (often with a much different life-cycle from authorization), it was not included in the latest version of rules. We're working on the testing story. It needs a developer experience, and all of the public resources available to do testing are not documented. |
Better support for resource/schema definition, type validation I've spent a long time thinking about this, and the short answer is that I think we need to more clearly separate resource definition from validation and authorization (and possibly separate the latter two as well). I worry that it's too easy to break schema/validation/authorization when trying to change a different one. Especially on teams where one person may be responsible only for schema, while another may be responsible for authorization, we'll quickly run into visibility and access issues if they remain merged. There are a number of standards we can look to for defining resources in an API: As @TristonianJones said, there is the option to build this in to the language. I'd be curious if folks on this thread prefer the built-in version or a separated world. The worry here is that we end up with N different configs, and thus doing anything requires a ton of effort. Any feedback on separating this, or which config to use, would be appreciated. |
Better tooling to identify, reproduce, and fix Rules issues As @TristonianJones said, we're long overdue on better tooling here. I think there are three tools in particular that may be helpful here. Unit Testing: Unbeknownst to all (because it's not documented), we have an API endpoint that allows for unit testing Rules. We're working on documenting this, and if folks are interested in taking a look at it earlier, let me know and I can provide you early access docs to look at. We'd appreciate some feedback here. Eventually, we aim to build tests into the console, though we haven't invested a ton of UX love into this. Local (specifically offline) testing is a separate issue that we don't currently have a solution for yet, but I think we're laying the foundation for it (more on this in the next post). Simulation: Simulation can be thought of as a request to fetch appropriate data (either from Static Analysis: The main use we've seen is informing other views with information about what's covered in Rules. For example, imagine the data viewer being able to show which data is appropriately secured, if the types align, etc. We've also considered additional tools, such as canary deploys (run a new version of Rules on some % of requests and see if there's a difference in % of allows/denies, alert if this deviates significantly). If y'all have other thoughts, I'd love to hear them as well. |
Building a community around Rules We're already making moves to make the language more accessible to developers, including open-sourcing the spec for our expression language, with additional tooling that conforms to the spec forthcoming. We plan on open sourcing the Rules language spec and tooling as well. Working in the open should accelerate developer adoption and lead to a better dialog and evolution of tools. This will allow developers to start building static analysis tools, syntax highlighters, local test environments, etc., even if the core Firebase team isn't focused on building them at that moment. |
@SamyPesse Your assert thread jogged my memory and reminded me that the same folks (waves to @goldibex) also did a chai package for asserts: https://github.com/casetext/chai-fireproof
I bet that this could be hacked together with the unit test API we provide to steer more in the direction you're asking for. Eventually, with some of the other tools open sourced, local evaluation should also be possible. |
Bolt syntax is significantly easier for me to read and understand. While the Firestore rule syntax is a big improvement over the Realtime Database, it's not not as clean and concise as Bolt. I intend to use both databases in my project, and it would be great to have one, unified rule language. It sounds like Firebase is aware of most or all of these issues, but I just wanted to voice my thoughts. Thank you for these awesome libraries :) |
I'm addicted to Bolt. I can't imagine going back at this point. I haven't had a chance to look at Firestore yet, but hearing that there isn't a similar way to create rules means I'll be delaying that look. |
@PatrickMcD can you clarify what about Bolt it is you like? Syntax, features (like types), or something else? |
@mcdonamp, great responses / great work. I'm interested in those early access docs as I'm beginning to break ground on a new product that requires tight security. |
@willhlaw and others, here's a first draft of docs for using the unit test API. We know the experience using the Google API explorer is a little jank--we'd love feedback on how you'd like to see this better incorporated (e.g. UI, CLI, SDK support, etc.). Also would love info on the syntax (e.g. prefer YAML over JSON, etc.). |
@mcdonamp there is a piece that I think is missing from this discussion. Since Bolt is so similar to Typescript, all it takes is a few simple regex rules to parse the Bolt types into a Typescript file, to be used throughout a Typescript project. As you can imagine, having this kind of static type checking greatly increases the stability of software that relies on Firebase. I'm not sure I'd be willing to give up this kind of confidence, even for all the wonderful features that Firestore is promising. |
@bijoutrouvaille I believe this is what #216 (comment) is aiming towards. While Bolt types can easily translate to Typescript, OpenAPI and Proto/Thrift can generate clients for many different languages (e.g. swagger-codegen, This can potentially be used to keep types on clients and types in Rules in lock-step, though it adds significant overhead for developers as they're first getting started. Being able to go the other way (start writing an app, export those types to Rules) seems like it's more valuable for the path many Firebase developers go through. |
@mcdonamp my bad, I've never used these. If the trade-off is between supporting more languages at the cost of a sensible overhead and ability to export types from a select few, the first option seems preferable. If the "overhead" reads clearly in complex projects, installs without severe headache and is well-documented, then I'm rooting for it. It would be particularly nice to have typed clients, which seems at least feasible in the first option. Thanks for all your effort! |
@bijoutrouvaille not your fault at all--we likely would have picked one of those if there was a clear winner in the space. Unfortunately, many devs (including a majority of Firebase folks) aren't aware of these tools and thus they present an extra barrier to entry. I think we need a gradually typed system, where a developer can start building an app without learning a new syntax, but as they discover they need it (or want other benefits), they can add it on. |
@mcdonamp great to see a lot of thought is being put into how to best address this. Thanks for leading. I totally agree the validation and authorization should be separated. I've used/evaluated almost all the ones you listed in #216 (comment) Schema (API)
Personally I found that JSON Schema is a solid choice, at least for javascript web applications.
Hope this is helpful on the validation side. When it comes to Authorization, every project seems to be different in it's requirements on how to handle this. @TristonianJones proposal for authorization (#216 (comment)) looks pretty solid to me. Combine that with the ability to include a schema from the option of bolt / protocol buffers/ json schema would be really powerful. Alternatively a json schema plugin could be written to denote authorization.
|
I had similar thoughts for separating authorization from validation and got pointed to this thread. I'm a fan of JSON schema and have been using it to define my firestore schema. Using this library I've been able to output TS interfaces and have a small wrapper around the firestore API that types queries, docs, and collections. My current approach has shortcomings that I'd like to solve such as:
I was using a JSON schema library to generate validation functions from the schema file but there were additional validations I wanted to express like cross field constraints such as However from reading @willfarrell's post some of my issues might be solved with |
I have the start of a node npm package, https://www.npmjs.com/package/firestore-security-tests Hey @mcdonamp, it would be nice to be able to add a |
@willhlaw FYI you're going to want to delete the private key from that example (as well as delete the key in the Console and re-create a new private key). |
Not secret, just haven't had time to externalize this stuff in a sensible way. We are working to expose version history/rollback in the Console in the near future. Here's the TL;DR: of Rules... Firebase Rules is composed of two primitives:
This allows us to maintain a version history (e.g. Release naming also provides resource binding, and can creatively be used to support percent rollout or A/B testing. |
@mcdonamp, I realize that about the private key. I almost put a comment to that effect in the sample about why it was being disclosed. It's creds to a firebase project set up to help facilitate testing for firebase-security-store and would be fine with any abuse (not sure how Google/Firebase feels about that, though). The primary purpose is to reduce the friction needed for someone to try out the firebaserules api and to understand the firebaserules. |
I strongly recommend not making these keys public under any circumstances. Everyone is OK with sinking the cost of abuse until they get a $10k bill for mining bitcoins that aren't theirs ;) I'd instead recommend building a simple website and using Google Consumer OAuth instead, or providing instructions for downloading and providing a service account. This is also sort of the point of the API explorer (though I understand the difference between the API explorer and the client library). |
I've just published It has an API similar to @mcdonamp How can we test transactions (multiple set/update) ? A lot of our apps logic is build on top of transactions, and it doesn't seem testable yet. @mcdonamp When will you open source / publish the |
Nice, thanks for doing that! We should bring back the "third party" section on our site to make these libs more discoverable. Transaction support is hard, partly because I don't believe we yet support them in Rules (e.g. you can't write a rule that references data from different documents). We're working on the best design for this. The protoc plugin is in open source review now, but unlikely to make it out before the new year due to the approvers being on vacation. |
@mcdonamp Any news on open sourcing the protoc plugin ? But even if we could easily write rules, the impossibility to test batch sets is a deal breaker on our side (we have an important bunch of rules to ensure that some duplicated data are kept in sync and require a batch set). I'm quite surprised, it didn't come out as a key feature of the rules backend when Firestore was designing. I'd be interested in learning how other company keep a consistent FB database for their apps without using and testing batch sets. |
@SamyPesse The protoc plugin launched today (https://github.com/firebase/protobuf-rules-gen). Would love feedback on it :) Validating batch requests is coming shortly. Here's what we're planning:
Would love feedback on how this works, how you feel about the name |
@mcdonamp Awesome! We'll give it a try (hopefully there will be a release to download instead of compiling from sources 😁). I'm wondering if the I think the most common use case is to validate and operate on the "resulting" data, instead of the previous data. But |
@SamyPesse this afternoon I'm going to create some binary releases. Also some clarification about I do see that the bolt thing of making newData the default in writes and data the default in reads makes a lot of sense. However, it's a little hard to make that change now without breaking people. |
I've created binaries for Linux & Mac here: https://github.com/firebase/protobuf-rules-gen/releases/tag/v0.1.0 I'll document how to use the binaries tomorrow morning, if you're unfamiliar with protoc plugins (which I'm guessing most people are; there is an example usage script if you want to see an example now). If anyone needs Windows support please open an issue in that repo. Please give us feedback on that repo! |
@mcdonamp Where can we follow the progress on Firestore ? The releases notes don't seem updated: https://cloud.google.com/firestore/docs/release-notes |
@SamyPesse looks like Dan did an email blast to the google-cloud-firestore-discuss group with updates here. I'm also seeing some release notes on https://firebase.google.com/support/releases. I've asked why the Cloud site doesn't have any, so hopefully we'll get an answer; otherwise, check the Firebase site for the latest updates. (Also, sorry, have been on vacation most of the month, hence the delayed responses). |
Update here, |
Has there been any progress for a Firestore Bolt? |
No. What are the particular features you're interested in? |
I've written a bolt-like language parser for Firestore, called Fireward, some time ago. Its syntax is a combination of Typescript and Firestore rules, and it generates the Firestore rules and Typescript typings. I find it more convenient than the protobuf implementation. It has been out for about a year now, has collected over 60 stars on Github, and has had many kinks ironed out, largely thanks to active bug reporting by the community. https://github.com/bijoutrouvaille/fireward |
@bijoutrouvaille It looks awesome 👍 I'll take a look if we can use it for our product and contribute to the project. |
Groovy, @bijoutrouvaille I'll check and give feedback. Essentially making it robust to see specific rules and the applicable data is the major benefit @mcdonamp. Being less error-prone with security is particularly useful to the stakeholders, too. |
@bijoutrouvaille plz correct me if mistaken, but can I use js/ts expressions to cultivate security or is the ts use primarily on data validation bnefits? |
@34r7h I'm not sure what you mean by "cultivate security". Firestore uses a js-like syntax, and Fireward simply takes that. The types defined in the .ward file using the TS-like syntax convert into validations. For example:
will compile to the following firestore rules:
The TypeScript typings functionality (generated with the |
@bijoutrouvaille i see, thank you. by cultivate security i mean the ability to programmatically build rules using plain js features like loops and more expressive logic than the firestore rules currently allow (easily) at least |
@bijoutrouvaille would be nice to convert your Haskell => JavaScript so it can be more easily deployed in NPM. You can cross compile Haskell into JavaScript. Then its also cross platform. |
@tohagan Thank you for pointing me out to fireward 🙌🏻 |
Still on Realtime Database because I refuse to drop Bolt. Haven't seen anything that stands in quite so nicely for Firestore. |
I'm with you in this thought. Actually, I'm working on a type-safe version of the firebase javascript SDK(admin and client). I'll share here as soon as I launch. After that, I pretend to work on the continuation of the Firebase Bolt project, an alternative that supports Composed indexes(for rtdb), type, runtime validation, and database rules generation from a single file, data encryption, and data signature. It sounds good for you too? |
That would be a nice set of sugar for sure. Especially if code generated. |
Even if the new Cloud Firestore has a better rules language than the RTDB, it still looks limited to write easy to understand document types.
It'd be awesome to use the Bolt language to write Cloud Firestore rules.
The text was updated successfully, but these errors were encountered: