Skip to content

Latest commit

 

History

History
354 lines (235 loc) · 12 KB

README.md

File metadata and controls

354 lines (235 loc) · 12 KB

Hercules Gate

Gate is used to transmit events from clients to Apache Kafka.

API methods

swagger documentation

Ping

Description: The method to ping service.

Method: GET

URL: /ping

Response codes:

Code Description
200 Successfully ping.

About

Description: The method to get service information.

Method: GET

URL: /about

Response codes:

Code Description
200 Successfully getting service information.

Response body:

Response body contains information about service:

applicationName - human readable application name
applicationId - robot readable application name
version - application version
commitId - commit id
environment - environment in which service is running (production, testing etc.)
zone - datacenter in which instance is located
hostName - server host name
instanceId - instance identifier

Status

Description: Readiness probe of the service.

Method: GET

URL: /status

Response codes:

Code Description
200 The service is ready to accept new requests.
410 The service should not accept new requests since it is shutting down.
503 The service is initializing.

Send

Description: The method to send event to Apache Kafka.

Method: POST

URL: /stream/send

Request headers

Authorization
Value should be with prefix "Hercules apiKey ".
The API Key with write access to the stream is specified.
Required

apiKey
The API Key with write access to the stream is specified.
Deprecated: use header Authorization instead.

Content-Type: application/octet-stream
Required

Content-Length
Required

Content-Encoding: lz4
If LZ4-compression is used.
Optional

Original-Content-Length
If Content-Encoding is used. Value MUST equal original content length (before compression) and must be lesser than 100 * 10^6 bytes.
Optional (required if Content-Encoding is used)

Query parameters:

stream - the name of stream. Required

Request body:

Events		Count, Event*
Count		Integer

Response codes:

Code Description
200 Successfully send data into stream.
400 Invalid request body or parameters (see response message for details).
401 apiKey is not valid or absent.
403 Access is denied (check if apiKey has appropriate access rights).
404 Source stream not found.
411 Content length must be specified.
413 Content length is too long.
415 Unsupported content encoding.
500 Internal error while processing request.
503 The gate is overloaded and request has been throttled.

Send Async

Description: The method to asynchronously send event to Apache Kafka.

Method: POST

URL: /stream/sendAsync

Request headers

Authorization
Value should be with prefix "Hercules apiKey ".
The API Key with write access to the stream is specified.
Required

apiKey
The API Key with write access to the stream is specified.
Deprecated: use header Authorization instead.

Content-Type: application/octet-stream
Required

Content-Length
Required

Content-Encoding: lz4
If LZ4-compression is used.
Optional

Original-Content-Length
If Content-Encoding is used. Value MUST equal original content length (before compression) and cannot be greater than 100 * 10^6 bytes.
Optional (required if Content-Encoding is used)

Query parameters:

stream - the name of stream. Required

Request body:

Events		Count, Event*
Count		Integer

Response codes:

Code Description
200 Successfully send data into stream.
400 Invalid request body or parameters (see response message for details).
401 apiKey is not valid or absent.
403 Access is denied (check if apiKey has appropriate access rights).
404 Source stream not found.
411 Content length must be specified.
413 Content length is too long.
415 Unsupported content encoding.
503 The gate is overloaded and request has been throttled.

Settings

Application is configured through properties file.

Main Application settings

application.host - server host, default value: 0.0.0.0

application.port - server port, default value: 8080

application.shutdown.timeout.ms - timeout to stop all application components, default value: 5000 ms

application.shutdown.grace.period.ms - period before stopping application components, default value: 0 ms

HTTP Server settings

HTTP Server binds on <host>:<port> are defined in Main Application settings.

http.server.maxContentLength - max Content-Length in POST-request

http.server.connection.threshold - maximum active http connections, default value: 100000

http.server.throttling.capacity - default value: 100000000

http.server.throttling.requestTimeout - timeout for request, which capacity throttling more then permissible, default value: 5000

http.server.readTimeout - Configure a read timeout for a socket, in milliseconds.

http.server.writeTimeout - Configure a write timeout for a socket, in milliseconds.

http.server.requestParseTimeout - The maximum allowed time of reading HTTP request in milliseconds. -1 or missing value disables this functionality.

http.server.idleTimeout - The idle timeout in milliseconds after which the channel will be closed. If the underlying channel already has a read or write timeout set the smaller of the two values will be used for read/write timeouts.

http.server.noRequestTimeout - The amount of time the connection can be idle with no current requests before it is closed.

Validation settings

validation.max.event.size - max size of Hercules event, value must be consistent with broker setting max.message.bytes, default value: 500000

Event sender settings

gate.event.sender.default.partitioner.type - type of the default partitioner (which is used if sharding key is absent). See possible values at ru.kontur.vostok.hercules.gate.PartitionerType, default value: KAFKA_DEFAULT

gate.event.sender.default.partitioner.batch.size - batch size for partitioner of type PartitionerType#BATCHED, default value: 16

Kafka Producer settings

Producer settings have base scope gate.event.sender.producer. See Producer's Config from Apache Kafka documentation. Main settings are presented below.

gate.event.sender.producer.bootstrap.servers

gate.event.sender.producer.acks

gate.event.sender.producer.batch.size

gate.event.sender.producer.linger.ms

gate.event.sender.producer.buffer.memory

gate.event.sender.producer.retries

gate.event.sender.producer.retry.backoff.ms

gate.event.sender.producer.metric.reporters

Send request processor settings

Sampling metrics settings

Gate supports sampling metrics. Settings for them have base scope gate.send.request.processor.metrics.sampling.

gate.send.request.processor.metrics.sampling.enable - enable sampling metrics if true. Default value: false.

gate.send.request.processor.metrics.sampling.request.data.size.bytes - sample requests if request data size in bytes is less or equal to this value. Default value: 1048576.

Apache Curator settings

See Apache Curator Config from Apache Curator documentation. Main settings are presented below.

curator.connectString - default value: localhost:2181

curator.connectionTimeout - default value: 10000

curator.sessionTimeout - default value: 30000

curator.retryPolicy.baseSleepTime - default value: 1000

curator.retryPolicy.maxRetries - default value: 5

curator.retryPolicy.maxSleepTime - default value: 8000

Graphite metrics reporter settings

metrics.graphite.server.addr - hostname of graphite instance to which metrics are sent, default value: localhost

metrics.graphite.server.port - port of graphite instance, default value: 2003

metrics.graphite.prefix - prefix added to metric name

metrics.period - the period with which metrics are sent to graphite, default value: 60

Application context settings

context.instance.id - id of instance

context.environment - id of environment

context.zone - id of zone

Service Discovery settings

sd.address - http address of service to register in service discovery. If no address is specified, then application settings http://<host>:<port> are used

sd.periodMs - period of beacon registration check in milliseconds, default value: 10000

Command line

java $JAVA_OPTS -jar hercules-gate.jar application.properties=file://path/to/file/application.properties

Also, ZooKeeper can be used as source of application.properties file:

zk://zk_host_1:port[,zk_host_2:port,...]/path/to/znode/application.properties

Quick start

Initialization

Gate uses Stream's metadata and auth rules from ZooKeeper. Thus, ZK should be configured by Hercules Init. See Hercules Init for details.

application.properties sample:

application.host=0.0.0.0
application.port=6306
application.shutdown.timeout.ms=5000
application.shutdown.grace.period.ms=0

http.server.maxContentLength=25165824
http.server.connection.threshold=100000
http.server.throttling.capacity=1073741824
http.server.throttling.requestTimeout=10000

validation.max.event.size=500000

gate.event.sender.producer.bootstrap.servers=localhost:9092
gate.event.sender.producer.acks=all
gate.event.sender.producer.batch.size=65536
gate.event.sender.producer.linger.ms=1
gate.event.sender.producer.buffer.memory=335544320
gate.event.sender.producer.retries=4
gate.event.sender.producer.retry.backoff.ms=250
gate.event.sender.producer.metric.reporters=ru.kontur.vostok.hercules.kafka.util.metrics.GraphiteReporter

gate.event.sender.default.partitioner.type=KAFKA_DEFAULT

gate.send.request.processor.metrics.sampling.enable=true
gate.send.request.processor.metrics.sampling.request.data.size.bytes=1048576

curator.connectString=localhost:2181
curator.connectionTimeout=10000
curator.sessionTimeout=30000
curator.retryPolicy.baseSleepTime=1000
curator.retryPolicy.maxRetries=3
curator.retryPolicy.maxSleepTime=3000

metrics.graphite.server.addr=localhost
metrics.graphite.server.port=2003
metrics.graphite.prefix=hercules
metrics.period=60

context.instance.id=1
context.environment=dev
context.zone=default

sd.address=http://localhost:6306
sd.periodMs=10000