Roadmap for v1

[a8m]

List of features/changes we want to finish before releasing an official v1 version:

• PostgreSQL support - done
• Eager loading - proposal: eager loading support #91 - done
• User defined IDs - override id column #127 - done
• Hooks - entc/hooks: initial work for mutations and hooks #377 - done
• Bulk insert - done proposal: Bulk API #236
• OSS GraphQL integration - done (expected more work in the near future)
• Privacy layer - done (see entgo.io/docs/privacy)
• User defined types - WIP schema/field: add the option to use dialect specific types #277
  • Custom Go Type
  • Custom Dialect Type
  • "Other" type
  • Custom predicates (WIP)
• JSON operations
  • Predicates and filters
  • Append support
• Upsert and batch upsert (dialect specific)
• Query hooks (interceptors) - Hooks for Read Operations #833
• Better migration
  • offline and versioned migration
  • edge migration ({O2O, O2M} -> M2M).
• Multi-level caching (Experimental solution ariga/entcache)
• Cluster support (read replicas)
• Horizontal sharding
• Edge fields (metadata on edges)
• Cascading deletion - proposal: cascading deletion support #407
  • DDL support
  • Application-level support

Please note that this list is changing.

[winwisely99]

Can you elaborate on Cluster support (read replicas).
Sounds like queries go to the Read replicas and mutations to a master.
In which case you will need a CDC process ?

Eager to know this.

[a8m]

Sounds like queries go to the Read replicas and mutations to a master.

Exactly. We want to add a cluster-client option (leader and 1 or more read replicas).

In which case you will need a CDC process ?

Can you elaborate, I'm not sure I got your question.

[winwisely99]

CDC is short for Change Data Capture.
Implementations often use binlog parsing of the Master ( read write db), and detect the db changes and then pump them out over a stream so the Slave ( Read only DB) can update its data.

Here is a great example with cockroachdb.
https://www.cockroachlabs.com/docs/stable/change-data-capture.html

for MySQL thee is this: https://github.com/uber/storagetapper

for PostreSQLthere is this: https://github.com/samsarahq/thunder/blob/v0.5.0/livesql/binlog.go

[winwisely99]

I have a request for the Roadmap.

A change notification so that if a record changes the system tells me. Its basically CDC.
For me the use case is to make it easy to have Read only VIew models that might be a different structure from the READ / Write models. Sort of like what you do with CQRS.
For me i dont need any of the network code because i might want to use a different netowrk protocol.
But it woudl be good to describe the change feed using a Protobuf because Protobufs support Schema Evolution which is really important and often forgotten about. Also protobuf makes it easy to integreate other languges.

[3ks]

Do you have a translation plan for other languages ?

[sagikazarmark]

@winwisely99 I think the use of read replicas and clustering here means, that ent will support providing separate master and read connection details, but the actual replication should be done by the database cluster.

CQRS like systems usually build on domain operations, thus a change is actually detected by the domain layer, not the storage layer. If you want to detect changes, you can set up triggers or check the return value of the MySQL DML query. I hardly think this should be the job of an ORM, because it's either above or below it.

But that's my two cents only.

[Vliro]

Will there be support for GraphQL? I am working with a graph database that does not support Gremlin but rather GraphQL which means only an underlying driver would have to be written if I understand it correctly.

[a8m]

Do you have a translation plan for other languages ?

Not at the moment.

I think the use of read replicas and clustering here means, that ent will support providing separate master and read connection details, but the actual replication should be done by the database cluster.

Exactly.

Will there be support for GraphQL?

There are plans to add support for API layers, like GraphQL and REST (Swagger for example).
However, it's not in our roadmap to v1, because we see it as an extension to ent that can be done with an external plugin.

I am working with a graph database that does not support Gremlin but rather GraphQL which means only an underlying driver would have to be written if I understand it correctly.

About GraphQL as a storage driver - not part of v1 roadmap.

[gpopovic]

@a8m any eta for eager loading and hooks?

[a8m]

@a8m any eta for eager loading and hooks?

We are in the holiday season right now, and we'll get back to work on Oct 23rd (so I can't commit to an exact date). However, I can say that we plan to tackle immediately these 2 problems (eager-loading first). You are welcome to help us with the design of the eager-loading API (#91).

Right now, I'm sporadically working on adding the PostgreSQL support.

[manishrjain]

Would there be interest in support for Dgraph? We could help out if there’s interest (I’m the founder of Dgraph).

[a8m]

Hey @manishrjain, thanks for proposing your help.

Would there be interest in support for Dgraph?

Currently, we want to focus on our roadmap and don't want to commit to something we don't have the capacity for it (because it means maintaining and supporting a storage-driver we're not using nor have experience with).

However, @alexsn and I just discussed this morning on adding a public interface for pluggable storage-drivers. So, if someone is willing to take and maintain this, I'll be happy to help.

[hlubek]

It would be great to have support for UUIDs as IDs with explicit handling of the ID generation (instead of auto-generating the IDs)

[alexsn]

It would be great to have support for UUIDs as IDs with explicit handling of the ID generation (instead of auto-generating the IDs)

We do have plans to add support for custom ID generation but note that with universal IDs you're able to determine ent type by ID which you won't be able to do with UUIDs.

[smintz]

Does eager loading means using the practices of dataloader? If so, there's https://github.com/graph-gophers/dataloader that can be used.

As per hooks, I don't know what you have in mind but I saw a neat implementation by https://github.com/infobloxopen/protoc-gen-gorm

type EntUserWithBeforeCreate interface {
	EntUserBeforeCreate(context.Context, *EntUser) error
}
...
if prehook, ok := interface{}(m).(EntUserWithBeforeCreate); ok {
	if err = prehook.EntUserBeforeCreate(ctx, &user); err != nil {
		return to, err
	}
}

Hope this helps

[alexsn]

Does eager loading means using the practices of dataloader? If so, there's https://github.com/graph-gophers/dataloader that can be used.

dataloader can be used in combination with eager loading where you collect up a bunch of requested fields over some time period and run a single query to fetch all of those fields.

[Aligator77]

Will there be support for MSSql, Redis, Elasticsearch?

[a8m]

Will there be support for MSSql, Redis, Elasticsearch?

No, at least in the foreseeable future. However, as I mentioned above, we have plans to add a public interface for pluggable storage-drivers. So, if someone want to take and maintain it, I'll be happy to help.

[ansarizafar]

Please add geo types and location queries to roadmap.

[danf0rth]

Any plans to add ON DUPLICATE KEY UPDATE/ON CONFLICT DO UPDATE?

[ajwerner]

No, at least in the foreseeable future. However, as I mentioned above, we have plans to add a public interface for pluggable storage-drivers. So, if someone want to take and maintain it, I'll be happy to help.

Hey @a8m, I'm interested in seeing an interface for pluggable storage drivers. Do you have much vision on what this should look like? Would it be reasonable to first build out pluggable infrastructure for SQL dialects rather than something more general? If you're not opposed I'd like to open an issue to start discussing design requirements and a path forward.

[a8m]

Hey @ajwerner, and sorry for the delayed response.

Hey @a8m, I'm interested in seeing an interface for pluggable storage drivers. Do you have much vision on what this should look like?

I can share with you what I thought about - A "runtime" storage driver (in the templates), which generates code that expect for an interface that wraps a list of operations, like: CreateNode, UpdateNode, etc.
We wait with this, because we want to finish first all the critical parts in the codegen (like hooks).

Would it be reasonable to first build out pluggable infrastructure for SQL dialects rather than something more general?

Maybe. I'm open for this (I guess you talk about cockroachdb integration).

If you're not opposed I'd like to open an issue to start discussing design requirements and a path forward.

Let's open an issue and discuss about it there.

[a8m]

Hey @danf0rth! Yes, see #139

[danf0rth]

Sorry for so many requests, but i am currently using your lib and intend to continue to do so :) There is few things that i lack now.

Cursor

Is there are any plans to add cursor or iterator to query API?
I mean something like

pets, err := client.Pets.Query().Iterator(ctx) // instead of All(ctx)
if err != nil {
    // ...
}

for pets.Next() {
    pet, err := pets.Only() // not sure about naming, maybe pets.Get(), or pets.One()
    if err != nil {
        // ...
    }
}

If you need to load and process a lot of data from database, it is better to do this with cursor/iterator.

Alternative struct-initialized API

Is there are any plans to introduce alternative API for entity creation?

This is how to create user now:

a8m := client.User.
        Create().
        SetAge(30).
        SetName("a8m").
        AddGroups(hub, lab).
        SaveX(ctx)

This is how to create user with initialized struct:

a8m := &ent.User{
    Age: 30,
    Name: "a8m",
    Groups: []*ent.Group{
        hub, 
        lab,
    },
}
a8m = client.User.Create(a8m).SaveX(ctx)

This is not shorter, but when i writing with setter-like API, the IDE sometimes makes me crazy, advising me wrong completions. It is not problem of the lib, but maybe alternative API will be more suitable for community?

Also this maybe a step ahead to implement #236

var users = []*ent.User{ ... }
err := client.User.CreateMany(users).Save(ctx)

Any discuss appreciated.

[gedw99]

@winwisely99 I think the use of read replicas and clustering here means, that ent will support providing separate master and read connection details, but the actual replication should be done by the database cluster.

CQRS like systems usually build on domain operations, thus a change is actually detected by the domain layer, not the storage layer. If you want to detect changes, you can set up triggers or check the return value of the MySQL DML query. I hardly think this should be the job of an ORM, because it's either above or below it.

But that's my two cents only.

I know, but i prefer a pure SQL level of CQRS.

Like this:
https://materialize.com/
https://materialize.com/docs/sql/create-source/

so then your can migrate easily and then gen your Protobufs of the Ent Schema

[StevenACoffman]

Curious if Ent's OSS GraphQL roadmap includes Apollo Federation support (being federat_ed_). We are using Federation to compose multiple GraphQL services into one graph. Presently, we can use federated DGraph for a single service, but we also have some Federated Go services that are really just adaptors for relational databases (PostgreSQL). We haven't been able to adopt Ent and avoid the adapter service because we needed Apollo Federation support. Also curious if any Ent / DGraph combination has been tried or considered.

[giautm]

We haven't been able to adopt Ent and avoid the adapter service because we needed Apollo Federation support.

Hi, Ent use gqlgen to integration GraphQL feature, they also support Apollo Federation.

See: https://gqlgen.com/recipes/federation/

[thesobercoder]

@a8m Any chance for SQL Server support?

[agnel-foyernet]

Hi Ariel, we have been using the ent framework successfully, you did a great job, many thanks for that. we have a dependency on Edge fields (metadata on edges). when can we expect this change?

thanks
Agnel

[a8m]

Hey @agnel-foyernet, you can follow this feature on #1949

We're almost done with the new migration. MySQL and PostgreSQL are ready, and SQLite and MariaDB are in progress. Once it's done, we'll post a proper roadmap update with some of the most requested features in the issue tracker (new GraphQL integration, polymorphism, metadata on edges, and much more).

[efectn]

Logging support like https://gorm.io/docs/logger.html

[maaft]

@a8m Hi! Do you have any eta for the next roadmap update? :)

[a8m]

@a8m Hi! Do you have any eta for the next roadmap update? :)

Yes, tomorrow. We'll release v0.10 and update the next major features for v0.11 in a new blog post. cc @rotemtam

[maaft]

:-O Wow, that was totally unexpected! Great news! 🚀 🚀 🚀

[gedw99]

Fly.io have cluster support that is a generic pattern for databases.

https://fly.io/docs/reference/postgres/

they detect a write and then replay the request onto the writable db.

This pattern can be used for any database.

For replication they use stolen. That’s not database agnostic.

[maaft]

@a8m Can't wait for it! :D Will you comment here when it's published? :)

[sagikazarmark]

Congrats for the new release!

I was looking at the next items on the roadmap and noticed that "Cluster support (read replicas)" is not there. Any ideas when this could become a thing?

For large (even mid) scale applications this is a must and it doesn't feel like to be a very complicated thing (I could be absolutely wrong though). In other words: this could be a low hanging fruit. The alternative is using various proxy solutions, but I'm not sold on them just yet.

[gedw99]

@sagikazarmark

I don’t know if there is a design for it yet .

If you have a suggestion , put it forward

[sagikazarmark]

I think it was one of the first questions in this thread and was quickly clarified what it means. Don't know if any further details in terms of design help deciding it's place on the roadmap. I can certainly come up with some ideas for the interface if that helps.

[gedw99]

I don't know either. I liked the way fly.io solved their db scaling issue in a very pragmatic way.

https://fly.io/docs/getting-started/multi-region-databases/

https://www.youtube.com/watch?v=IqnZnFpxLjI

• jump to 12 minutes in...

[a8m]

Congrats for the new release!

Thanks @sagikazarmark

I was looking at the next items on the roadmap and noticed that "Cluster support (read replicas)" is not there. Any ideas when this could become a thing?

There is an example of how to implement a "cluster driver" in #1580 (comment). I also added it to our website.

I still think we should make it an official plugin/extension with additional features we created internally. We just didn't get to it yet.

[gedw99]

@a8m i had no idea this was done. both design and code.

Its not "ready" yet ?

Is there a repo anywhere that exercises this, so we can play around with it. ?

If feels like the Cache is where this "redirection" would be between primary ( write only) and secondary ( read only ) db's ?
https://github.com/ariga/entcache/blob/master/README.md#high-level-design

[sagikazarmark]

Thanks for the links @a8m! I'll take a look at them.

[StarpTech]

Add support for cockroachdb.

[crossworth]

Hello @StarpTech, someone on discord said that they are using CockroachDB without problems with Ent.

Feel free to join the Ent Discord https://discord.gg/qZmPgTE6RX

[StarpTech]

Hi @crossworth the difference between "it works" and support is huge. I'd like to see cockroachdb as part of the CI tests.

[StarpTech]

Please add support for Joins. The eager load mechanism is too limited.

Currently, it's not possible to filter on the dataset of an entity with multiple eager loaded edges. The filter capabilities are scoped to one specific edge at a time.

Imagine you would like to get all authors where posts OR books starts with the letter "a". Each author has two edges (1. Posts, 2. Books) We can't query authors with either a matched post or book because each edge is loaded in a separate database query.

This is a very fundamental feature. I'm curious what's the strategy of ent here? Did I forget something? 🤔

Updated: It is possible through the .hasX methods in the .Where clause of the starting node https://entgo.io/docs/traversals

[ilovesusu]

Do you have a plan for Horizontal sharding ?

[czyt]

hello,@a8m is there any plan to surpport document databases ,such as mongodb?

[zishiguo]

Do you have a plan for Horizontal sharding ?

[djensen47]

Hi, just wanted to say, Ent is fantastic! The quality speaks for itself in the shear volume of developers using it in production.

Therefore is super strange, and inconvenient, that ent does not have a 1.0 version.

Looking at the above list. These look like 1.1 or 2.0 features. We could easily have a 1.0 version today. All you have to do is change that number and send out a release. The problem with 0.11 is that 0.12 can be a breaking change so we all have to be super, extra careful when upgrading even if all we want is that small bug fix.

[wxmeng04]

Compared with new features, it's more meaningful to facilitate the usage and improve frequently used functionality.