Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added self documentation to the KSQL config public config variables. #422

Merged
merged 10 commits into from
Nov 8, 2017
Merged

Added self documentation to the KSQL config public config variables. #422

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
db38492
Added self documentation to the KSQL config publis config variables.
hjafarpour Oct 27, 2017
17de98e
Added examples to the descriptions.
hjafarpour Oct 30, 2017
34f2231
Merge remote-tracking branch 'upstream/4.0.x' into KSQLConfig-selfdoc…
hjafarpour Oct 31, 2017
b155a65
Renamed some KSQL config props and used ConfigDef for populating the …
hjafarpour Nov 1, 2017
94cc2db
Merge remote-tracking branch 'upstream/4.0.x' into KSQLConfig-selfdoc…
hjafarpour Nov 2, 2017
02a2f32
Only update ksqlStreamConfigProps since ksqlConfigProps is populated …
hjafarpour Nov 2, 2017
b3438c6
More details on a config value doc.
hjafarpour Nov 3, 2017
5104828
Merge remote-tracking branch 'upstream/4.0.x' into KSQLConfig-selfdoc…
hjafarpour Nov 6, 2017
7026f77
Created a new KsqlConstants file to hold constants such as property n…
hjafarpour Nov 6, 2017
ae5689a
Minor config doc fix.
hjafarpour Nov 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 54 additions & 38 deletions ksql-common/src/main/java/io/confluent/ksql/util/KsqlConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,67 +57,83 @@ public class KsqlConfig extends AbstractConfig implements Cloneable {

public static final String
KSQL_SERVICE_ID_CONFIG = "ksql.service.id";
public static final ConfigDef.Type
KSQL_SERVICE_ID_TYPE = ConfigDef.Type.STRING;
public static final String
KSQL_SERVICE_ID_DEFAULT = "ksql_";
public static final ConfigDef.Importance
KSQL_SERVICE_ID_IMPORTANCE = ConfigDef.Importance.MEDIUM;
public static final String
KSQL_SERVICE_ID_DOC =
"Indicates the ID of the ksql service. It will be used as prefix for all KSQL queries in "
+ "this service.";

public static final String
KSQL_PERSISTENT_QUERY_NAME_PREFIX_CONFIG = "ksql.persistent.prefix";
public static final ConfigDef.Type
KSQL_PERSISTENT_QUERY_NAME_PREFIX_TYPE = ConfigDef.Type.STRING;
public static final String
KSQL_PERSISTENT_QUERY_NAME_PREFIX_DEFAULT = "query_";
public static final ConfigDef.Importance
KSQL_PERSISTENT_QUERY_NAME_PREFIX_IMPORTANCE = ConfigDef.Importance.MEDIUM;
public static final String
KSQL_PERSISTENT_QUERY_NAME_PREFIX_DOC =
"Second part of the prefix for persitent queries.";

public static final String
KSQL_TRANSIENT_QUERY_NAME_PREFIX_CONFIG = "ksql.transient.prefix";
public static final ConfigDef.Type
KSQL_TRANSIENT_QUERY_NAME_PREFIX_TYPE = ConfigDef.Type.STRING;
public static final String
KSQL_TRANSIENT_QUERY_NAME_PREFIX_DEFAULT = "transient_";
public static final ConfigDef.Importance
KSQL_TRANSIENT_QUERY_NAME_PREFIX_IMPORTANCE = ConfigDef.Importance.MEDIUM;
public static final String
KSQL_TRANSIENT_QUERY_NAME_PREFIX_DOC =
"Second part of the prefix for transient queries.";

public static final String
KSQL_TABLE_STATESTORE_NAME_SUFFIX_CONFIG = "ksql.statestore.suffix";
public static final ConfigDef.Type
KSQL_TABLE_STATESTORE_NAME_SUFFIX_TYPE = ConfigDef.Type.STRING;
public static final String
KSQL_TABLE_STATESTORE_NAME_SUFFIX_DEFAULT = "transient_";
public static final ConfigDef.Importance
KSQL_TABLE_STATESTORE_NAME_SUFFIX_IMPORTANCE = ConfigDef.Importance.MEDIUM;
public static final String
KSQL_TABLE_STATESTORE_NAME_SUFFIX_DOC =
"Suffix for state store names in Tables.";
KSQL_TABLE_STATESTORE_NAME_SUFFIX_DEFAULT = "_ksql_statestore";

public int defaultSinkNumberOfPartitions = 4;
public short defaultSinkNumberOfReplications = 1;
public static int defaultSinkNumberOfPartitions = 4;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be public, i.e., they are set as the defaults anyway

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily, changed them to private.

public static short defaultSinkNumberOfReplications = 1;
// TODO: Find out the best default value.
public long defaultSinkWindowChangeLogAdditionalRetention = 1000000;
public static long defaultSinkWindowChangeLogAdditionalRetention = 1000000;

public String defaultAutoOffsetRestConfig = "latest";
public long defaultCommitIntervalMsConfig = 2000;
public long defaultCacheMaxBytesBufferingConfig = 10000000;
public int defaultNumberOfStreamsThreads = 4;
public static String defaultAutoOffsetRestConfig = "latest";
public static long defaultCommitIntervalMsConfig = 2000;
public static long defaultCacheMaxBytesBufferingConfig = 10000000;
public static int defaultNumberOfStreamsThreads = 4;

Map<String, Object> ksqlConfigProps;
Map<String, Object> ksqlStreamConfigProps;

private static final ConfigDef CONFIG_DEF = new ConfigDef();
private static final ConfigDef CONFIG_DEF;

static {
CONFIG_DEF = new ConfigDef()
.define(KSQL_SERVICE_ID_CONFIG,
ConfigDef.Type.STRING,
KSQL_SERVICE_ID_DEFAULT,
ConfigDef.Importance.MEDIUM,
"Indicates the ID of the ksql service. It will be used as prefix for all KSQL queries in "
+ "this service.")

.define(KSQL_PERSISTENT_QUERY_NAME_PREFIX_CONFIG,
ConfigDef.Type.STRING,
KSQL_PERSISTENT_QUERY_NAME_PREFIX_DEFAULT,
ConfigDef.Importance.MEDIUM,
"Second part of the prefix for persitent queries.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps a description of what the final query name would look like?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an example.

.define(KSQL_TRANSIENT_QUERY_NAME_PREFIX_CONFIG,
ConfigDef.Type.STRING,
KSQL_TRANSIENT_QUERY_NAME_PREFIX_DEFAULT,
ConfigDef.Importance.MEDIUM,
"Second part of the prefix for transient queries.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an example.

.define(KSQL_TABLE_STATESTORE_NAME_SUFFIX_CONFIG,
ConfigDef.Type.STRING,
KSQL_TABLE_STATESTORE_NAME_SUFFIX_DEFAULT,
ConfigDef.Importance.MEDIUM,
"Suffix for state store names in Tables.")
.define(DEFAULT_SINK_NUMBER_OF_PARTITIONS,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why we have DEFAULT_SOME_PROPERTY and SOME_PROPERTY? We just define what the default value is here and then we can use SOME_PROPERTY everywhere. The ConfigDef takes care of providing the default value if none is supplied. So further down from line 141 etc you don't need to create a new copy of the Map and put the defaults in, they will already be there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is to let user define the default for a set of statements. For instance if user want to have the default value of partitions to be 10 for all of his/her queries they can set the default and just write their queries instead of having to specify it in each of their queries. On the other hand, if user wants to have a different value for a specific query he/she can set the value in the specific query syntax and it will be applied to only that one. User can change the default to another value and that value will be used as default partition number for all the queries after that.

ConfigDef.Type.INT,
defaultSinkNumberOfPartitions,
ConfigDef.Importance.MEDIUM,
"The default number of partitions for the topics created by KSQL.")
.define(DEFAULT_SINK_NUMBER_OF_REPLICATIONS,
ConfigDef.Type.SHORT,
defaultSinkNumberOfReplications,
ConfigDef.Importance.MEDIUM,
"The default number of replications for the topics created by KSQL."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replications -> replicas

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

)
.define(DEFAULT_SINK_WINDOW_CHANGE_LOG_ADDITIONAL_RETENTION,
ConfigDef.Type.LONG,
defaultSinkWindowChangeLogAdditionalRetention,
ConfigDef.Importance.MEDIUM,
"The default window change log additional retention time."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably want to drop the 'default' here. Also, it may make sense to expand the description. It doesn't make sense as is. Perhaps a sentence like 'The amount of time to retain window change logs'. Actually I am still not sure what that means Also, there should be a time unit in the name of the config and the description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, added the time unit and more details to the description.

)
;
}


public KsqlConfig(Map<?, ?> props) {
super(CONFIG_DEF, props);
Expand Down