doc change

docs/configuration/config.md

@@ -1,7 +1,6 @@
-# basic configs
+# Config Items
 > Here we use json with comments for documentation
 
-
 ```
 {
   // clickhouse configs, it's map[string]ClickHouse for multiple clickhouse
@@ -26,12 +25,32 @@
   "kafka": {
     "brokers": "127.0.0.1:9093",
 
-    // somethings about sasl
+    // SSL
+    "tls": {
+      "enable": false,
+      // Required. It's the CA certificate with which Kafka brokers certs be signed.
+      "caCertFiles": "/etc/security/ca-cert",
+      // Required if Kafka brokers require client authentication.
+      clientCertFile: "",
+      // Required if and only if ClientCertFile is present.
+      clientKeyFile: "",
+      // Whether disable broker FQDN verification. Set it to `true` if kafka-console-consumer.sh uses `ssl.endpoint.identification.algorithm=`.
+      "insecureSkipVerify": true
+    }
+
+    // SASL
     "sasl": {
       "enable": false,
-      "password": "",
+      // Mechanism is the name of the enabled SASL mechanism.
+      // Possible values: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI (defaults to PLAIN)
+      "mechanism": "PLAIN",
+      // Username is the authentication identity (authcid) to present for
+      // SASL/PLAIN or SASL/SCRAM authentication
       "username": "",
+      // Password for SASL/PLAIN or SASL/SCRAM authentication
+      "password": "",
       "gssapi": {
+        // authtype - 1. KRB5_USER_AUTH, 2. KRB5_KEYTAB_AUTH
         "authtype": 0,
         "keytabpath": "",
         "kerberosconfigpath": "",
@@ -77,6 +96,11 @@
     // "this columns will be excluded by insert SQL "
     "excludeColumns": []
 
+    // shardingKey is the column name to which sharding against
+    "shardingKey": "",
+    // shardingPolicy is `stripe,<interval>`(requires ShardingKey be numerical) or `hash`(requires ShardingKey be string)
+    "shardingPolicy": "",
+
     // batch size to insert into clickhouse
     "bufferSize": 90000,
     // min batch size to insert into clickhouse
@@ -93,4 +117,4 @@
   "logLevel": "debug"
   }
 }
-```
+```

docs/dev/introduction.md

@@ -37,7 +37,7 @@ Refers to [design](./design.md) for how it works.
 
 ## Configuration
 
-Refers to how [integration test](./go.test.sh) use the example config. Also refers to [code](./config/config.go) for all config items.
+Refers to how [integration test](https://github.com/housepower/clickhouse_sinker/blob/master/go.test.sh) use the example config. Also refers to [code](https://github.com/housepower/clickhouse_sinker/blob/master/config/config.go) for all config items.
 
 ### Kafka Encryption
 
@@ -75,13 +75,13 @@ An example kafka config:
 FYI. `kafka-console-consumer.sh` works as the following setup:
 
 ```
-$ cat config/SSL_NOAUTH_client.properties
+$ cat config/client_SSL_NOAUTH.properties
 security.protocol=SSL
 ssl.truststore.location=/etc/security/kafka.client.truststore.jks
 ssl.truststore.password=123456
 ssl.endpoint.identification.algorithm=
 
-$ bin/kafka-console-consumer.sh --bootstrap-server 192.168.31.64:9094 --topic sunshine --group test-consumer-group --from-beginning --consumer.config config/SSL_NOAUTH_client.properties
+$ bin/kafka-console-consumer.sh --bootstrap-server 192.168.31.64:9094 --topic sunshine --group test-consumer-group --from-beginning --consumer.config config/client_SSL_NOAUTH.properties
 ```
 
 Please follow [`Kafka SSL setup`](https://kafka.apache.org/documentation/#security_ssl). Use `-keyalg RSA` when you create the broker keystore, otherwise there will be no cipher suites in common between the keystore and those Golang supports. See [this](https://github.com/Shopify/sarama/issues/643#issuecomment-216839760) for reference.