Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: support an ALTER CHANGEFEED feature #75895

Closed
sherman-grewal opened this issue Feb 2, 2022 · 2 comments
Closed

cdc: support an ALTER CHANGEFEED feature #75895

sherman-grewal opened this issue Feb 2, 2022 · 2 comments
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-cdc

Comments

@sherman-grewal
Copy link
Contributor

sherman-grewal commented Feb 2, 2022 •
edited by exalate-issue-sync bot
Loading

Support an ALTER CHANGEFEED feature

Description

Currently, the process of altering an enterprise changefeed can be quite cumbersome. In particular, this process would consist of finding and saving the high-water timestamp of the changefeed that you would like to alter, cancelling the changefeed, and then creating a new changefeed starting at the high-water timestamp that we previously saved. This process is not only inefficient, but it is error-prone as well. As a result, altering a changefeed has become an annoyance for many clients.

Solution

We would like to address this issue by introducing a new SQL statement ALTER CHANGEFEED that would handle the process of altering a changefeed. By doing so, clients can avoid going through the process of altering a changefeed on their own, and rely on CRDB to carry out this task.

Jira issue: CRDB-12871

Epic CRDB-8670
@sherman-grewal sherman-grewal added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Feb 2, 2022
@sherman-grewal sherman-grewal self-assigned this Feb 2, 2022
@sherman-grewal sherman-grewal added the A-cdc Change Data Capture label Feb 2, 2022
@blathers-crl blathers-crl bot added the T-cdc label Feb 2, 2022
@blathers-crl
Copy link

blathers-crl bot commented Feb 2, 2022

cc @cockroachdb/cdc

@sherman-grewal sherman-grewal mentioned this issue Feb 2, 2022
Merged
@shermanCRL
Copy link
Contributor

shermanCRL commented Feb 3, 2022 •
edited
Loading

Based on a chat with @sherman-grewal, some scoping ideas. Sherman mentioned the complexities and considerations involved when the sink changes. I recommend doing this in phases:

  1. ALTER CHANGEFEEDS for updating targets. 🚢 it.
  2. ALTER CHANGEFEEDS for updating options. 🚢 it.
  3. ALTER CHANGEFEEDS for updating the sink but not the sink type. So changing a Kafka URL is fine, but switching to (e.g.) webhooks is unsupported. 🚢 it.
  4. ALTER CHANGEFEEDS allowing switching sink types.

We might even decide that the latter is uninteresting. How often is a user going to change a sink type? If so, they would probably have better clarity by just creating a new changefeed.

cc @amruss

sherman-grewal pushed a commit to sherman-grewal/cockroach that referenced this issue Feb 9, 2022
changefeedccl: support ALTER CHANGEFEED statement for
9d40c8e 
adding/dropping targets

References issue cockroachdb#75895

This PR introduces a new SQL statement called ALTER
CHANGEFEED, which allows users to add/drop targets
for an existing changefeed. Note that the changefeed
job must be paused in order for the alterations to
be applied to the changefeed. The syntax of the
statement is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an arbitrary number of ADD/
DROP commands in any order. Once a user
executes this statement, the targets will be
added/dropped, and the statement will return the
job id and the new job description of the changefeed
job. It is also important to note that a user cannot
drop all targets in a changefeed.

Example:
ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

Release note (enterprise change): Added support
for a new SQL statement called ALTER CHANGEFEED,
which allows users to add/drop targets for an
existing changefeed. The syntax of the statement
is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an abritrary number of ADD/DROP
commands in any order. The following is an example
of this statement:

ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

With this statement, users can avoid going through
the process of altering a changefeed on their own,
and rely on CRDB to carry out this task.
sherman-grewal pushed a commit to sherman-grewal/cockroach that referenced this issue Feb 9, 2022
changefeedccl: support ALTER CHANGEFEED statement for
132d8bf 
adding/dropping targets

References issue cockroachdb#75895

This PR introduces a new SQL statement called ALTER
CHANGEFEED, which allows users to add/drop targets
for an existing changefeed. Note that the changefeed
job must be paused in order for the alterations to
be applied to the changefeed. The syntax of the
statement is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an arbitrary number of ADD/
DROP commands in any order. Once a user
executes this statement, the targets will be
added/dropped, and the statement will return the
job id and the new job description of the changefeed
job. It is also important to note that a user cannot
drop all targets in a changefeed.

Example:
ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

Release note (enterprise change): Added support
for a new SQL statement called ALTER CHANGEFEED,
which allows users to add/drop targets for an
existing changefeed. The syntax of the statement
is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an abritrary number of ADD/DROP
commands in any order. The following is an example
of this statement:

ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

With this statement, users can avoid going through
the process of altering a changefeed on their own,
and rely on CRDB to carry out this task.
sherman-grewal pushed a commit to sherman-grewal/cockroach that referenced this issue Feb 10, 2022
changefeedccl: support ALTER CHANGEFEED statement for
e52c173 
adding/dropping targets

References issue cockroachdb#75895

This PR introduces a new SQL statement called ALTER
CHANGEFEED, which allows users to add/drop targets
for an existing changefeed. Note that the changefeed
job must be paused in order for the alterations to
be applied to the changefeed. The syntax of the
statement is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an arbitrary number of ADD/
DROP commands in any order. Once a user
executes this statement, the targets will be
added/dropped, and the statement will return the
job id and the new job description of the changefeed
job. It is also important to note that a user cannot
drop all targets in a changefeed.

Example:
ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

Release note (enterprise change): Added support
for a new SQL statement called ALTER CHANGEFEED,
which allows users to add/drop targets for an
existing changefeed. The syntax of the statement
is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an abritrary number of ADD/DROP
commands in any order. The following is an example
of this statement:

ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

With this statement, users can avoid going through
the process of altering a changefeed on their own,
and rely on CRDB to carry out this task.
sherman-grewal pushed a commit to sherman-grewal/cockroach that referenced this issue Feb 10, 2022
changefeedccl: support ALTER CHANGEFEED statement for
450088f 
adding/dropping targets

References issue cockroachdb#75895

This PR introduces a new SQL statement called ALTER
CHANGEFEED, which allows users to add/drop targets
for an existing changefeed. Note that the changefeed
job must be paused in order for the alterations to
be applied to the changefeed. The syntax of the
statement is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an arbitrary number of ADD/
DROP commands in any order. Once a user
executes this statement, the targets will be
added/dropped, and the statement will return the
job id and the new job description of the changefeed
job. It is also important to note that a user cannot
drop all targets in a changefeed.

Example:
ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

Release note (enterprise change): Added support
for a new SQL statement called ALTER CHANGEFEED,
which allows users to add/drop targets for an
existing changefeed. The syntax of the statement
is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an abritrary number of ADD/DROP
commands in any order. The following is an example
of this statement:

ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

With this statement, users can avoid going through
the process of altering a changefeed on their own,
and rely on CRDB to carry out this task.
@sherman-grewal sherman-grewal mentioned this issue Feb 15, 2022
Merged
craig bot pushed a commit that referenced this issue Feb 25, 2022
@rimadeodhar
Merge #76266 #76583
dab67a9 
76266: pprofui: Increase concurrency for profiles r=dhartunian a=rimadeodhar

In this PR, we increase the concurrency limit while
running performance profiles (e.g. heap, CPU) from
the Advanced Debug page within DB Console. Previously,
attempting to run performance profiling in parallel
for the same node would result in race condition causing
one of the profiles to overwrite the other. This would cause
"Profile not found: profile may have expired" errors.
The occurrence of these errors was exacerbated by the new
feature enabling running profiles on any nodes as it
increased the likelihood of race conditions for profiles
being run on a node at the same time.
By allowing atleast two profile runs at the same time
decreases the likelihood of one request overwriting
the other. This does not completely eliminate
the problem but will reduce the frequency of occurrence.
This PR also updates the error message returned when
a profile is not found to provide more details on
the potential causes and remediation steps.

Release note: None

76583: changefeedccl: allow users to alter changefeed options r=sherman-grewal a=sherman-grewal

changefeedccl: allow users to alter changefeed options
with the ALTER CHANGEFEED statement

References #75895

Currently, with the ALTER CHANGEFEED statement users
can only add or drop targets from an existing
changefeed. In this PR, we would like to extend this
functionality so that an user can edit and unset
the options of an existing changefeed as well.
The syntax of this addition is the following:

ALTER CHANGEFEED <job_id> SET \<options\> UNSET <opt_list>

Note that the <options> must follow the same syntax
that is used when creating a changefeed with options.
In particular, if you would like to set an option
that requires a value, you must write

SET opt = 'value'

On the other hand, if you would like to set an
option that requires no value, you must write

SET opt

Furthermore, this PR allows users to unset options.
This can be achieved by writing

UNSET <opt_list>

Where <opt_list> is a list of options that you
would like to unset. For example, if we would like
to unset the diff and resolved options for
changefeed 123, we would achieve this by writing

ALTER CHANGEFEED 123 UNSET diff, resolved

Release note (enterprise change): Added support to
the ALTER CHANGEFEED statement so that users can edit
and unset the options of an existing changefeed. The
syntax of this addition is the following:

ALTER CHANGEFEED <job_id> SET \<options\> UNSET <opt_list>

Co-authored-by: rimadeodhar <[email protected]>
Co-authored-by: Sherman Grewal <[email protected]>
@sherman-grewal sherman-grewal mentioned this issue Feb 25, 2022
Merged
craig bot pushed a commit that referenced this issue Feb 26, 2022
Merge #77043
c7b4a53 
77043: changefeedccl: allow users to alter the sink URI of an existing changefeed r=sherman-grewal a=sherman-grewal

changefeedccl: allow users to alter the sink URI of
an existing changefeed

References #75895

In this PR, we introduce the capability to alter
the sink URI of an existing changefeed. This
can be achieved by executing the following
statement:

ALTER CHANGEFEED <job_id> SET sink = '<sink_uri>'

Note that the sink type cannot be altered. That is,
the sink type must be the same type that was chosen
when the changefeed was initially created.

Release note (enterprise change): Users may now alter
the sink URI of an existing changefeed. This can be
achieved by executing the following statement:

ALTER CHANGEFEED <job_id> SET sink = '<sink_uri>'

Where the sink type of the new sink must match the
sink type of the old sink that was chosen at the
creation of the changefeed.

Co-authored-by: Sherman Grewal <[email protected]>
@HonoreDB HonoreDB mentioned this issue Feb 28, 2022
Open
@sherman-grewal sherman-grewal mentioned this issue Mar 2, 2022
Merged
RajivTS pushed a commit to RajivTS/cockroach that referenced this issue Mar 6, 2022
@RajivTS
changefeedccl: support ALTER CHANGEFEED statement for
43c5ad2 
adding/dropping targets

References issue cockroachdb#75895

This PR introduces a new SQL statement called ALTER
CHANGEFEED, which allows users to add/drop targets
for an existing changefeed. Note that the changefeed
job must be paused in order for the alterations to
be applied to the changefeed. The syntax of the
statement is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an arbitrary number of ADD/
DROP commands in any order. Once a user
executes this statement, the targets will be
added/dropped, and the statement will return the
job id and the new job description of the changefeed
job. It is also important to note that a user cannot
drop all targets in a changefeed.

Example:
ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

Release note (enterprise change): Added support
for a new SQL statement called ALTER CHANGEFEED,
which allows users to add/drop targets for an
existing changefeed. The syntax of the statement
is as follows:

ALTER CHANGEFEED <job_id> {{ADD|DROP} <targets>}...

Where there can be an abritrary number of ADD/DROP
commands in any order. The following is an example
of this statement:

ALTER CHANGEFEED 123 ADD foo,bar DROP baz;

With this statement, users can avoid going through
the process of altering a changefeed on their own,
and rely on CRDB to carry out this task.
craig bot pushed a commit that referenced this issue Mar 14, 2022
@jordanlewis @otan
@jaylim-crl @arulajmani
Merge #74877 #77263 #77567 #77711 #77734 #77776
49a24b5 
74877: sql: add support for MOVE cursor r=jordanlewis a=jordanlewis

Closes #77100.

Release note (sql change): add support for the MOVE command, which moves
a SQL cursor without fetching any rows from it. MOVE is identical to
FETCH, including in its arguments and syntax, except it doesn't return
any rows.

Release justification: low-risk update to new functionality

77263: changefeedccl: allow users to perform initial scans on newly added targets r=sherman-grewal a=sherman-grewal

changefeedccl: allow users to perform initial scans on
newly added targets in the ALTER CHANGEFEED statement

References #75895

Currently, when a new target is added to an existing
changefeed through the ALTER CHANGEFEED statement,
no initial scan is performed on the newly added
target. With this change, users can indicate that
they want an initial scan by executing the following
statement:

ALTER CHANGEFEED <job_id> ADD <targets> WITH initial_scan

Users can also explicitly ask for no initial scan by
replacing 'initial_scan' with 'no_initial_scan'.
The default behavior is to perform no initial scans
when new targets are added.

Release note (enterprise change): Users can now
perform initial scans on newly added targets
by executing the following statement:

ALTER CHANGEFEED <job_id> ADD <targets> WITH initial_scan

The default behavior is to perform no initial scans
on newly added targets, but users can explicitly
request this by replacing the 'initial_scan' with
'no_initial_scan'.

Release justification: low danger enhancement to newly
implemented functionality.

77567: ttljob: only add labels when `ttl_label_metrics` is set r=otan a=otan

Follow up to [this slack convo](https://cockroachlabs.slack.com/archives/C0168LW5THS/p1645556875126379).

Release justification: high benefit change to new stuff

Release note (sql change): TTL metrics are labelled by relation name if
`SET CLUSTER SETTING server.child_metrics.enabled=true;` is set and
the `ttl_label_metrics` storage parameter is set to true. This is to
prevent a potentially unbounded cardinality on TTL related metrics.

77711: changefeedccl, kv: Add few metrics r=miretskiy a=miretskiy

See commits for details.

Release Notes: None
Release justification: Low impact observability improvement

77734: ccl/sqlproxyccl: fix a forwarding race after evaluating a safe transfer point r=JeffSwenson a=jaylim-crl

Previously, there's a possibility where the forwarder will continue to forward
messages **after** we evaluate a safe transfer point. This isn't a problem in
the old design, and is only present in the updated design. In this bug, there
could be a case where a query gets sent right after evaluating the safe
transfer point because the processors' locks were released, leading to a
possible connection termination.

Consider the following messages sent by the client: Q1 -> Q2 (relies on Q1).
Right before Q1, the forwarder detects that we're in a safe transfer point.
When we release the lock, we forward Q1 to the server before suspending. It is
possible that the connection migration will succeed, but forwarding Q2 through
the new connection will fail. To avoid such protocol errors, we'll ensure that
we set suspendReq = true once we start the transferring process. That way,
there won't be such a race. We want to be sure that when we start sending
the transfer request message, we are in a safe transfer point to minimize
connection terminations.

Release justification: Low-risk sqlproxy only change, high priority bug fix.

Release note: None

77776: spanconfig: stop generating split points for views r=ajwerner a=arulajmani

The range corresponding to a view's descriptor ID contains no data,
and as such, should not be treated as a split point.

Fixes #77744

Release justification: bug fixes and low-risk updates to new functionality
Release note: None

Co-authored-by: Jordan Lewis <[email protected]>
Co-authored-by: Sherman Grewal <[email protected]>
Co-authored-by: Oliver Tan <[email protected]>
Co-authored-by: Yevgeniy Miretskiy <[email protected]>
Co-authored-by: Jay <[email protected]>
Co-authored-by: arulajmani <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-cdc Change Data Capture C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-cdc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sherman-grewal @miretskiy @shermanCRL