[exporter/clickhouse] Upgrade to opentelemetry-collector-contrib v0.107.0 failed due to bug in clickhouse-go driver

[Fiery-Fenix]

Component(s)

exporter/clickhouse

What happened?

Description

In version 0.107.0 were introduced changes to logs table structure with removal of some redundant columns in table: #34203
Unfortunately due to bug in clickhouse-go driver Incorrect column list parsing for multiline INSERT statements exporter fails to insert records into logs table.
That happens because clickhouse-go driver doesn't honor list of columns in INSERT statement and is using list of columns in actual ClickHouse table.
It's affecting upgrading existing users for both with default schema and custom schema
Also this bug is a blocker for future schema enchantments, like #34245
Issue created mostly for users who experienced described error and for tracking progress of bug fixing in upstream

Steps to Reproduce

• Deploy opentelemetry-collector-contrib version <0.107.0 and let it create default schema and put some data into otel_logs table
• Upgrade opentelemetry-collector-contrib to version 0.107.0 and let it put some data into otel_logs table

Expected Result

New version should put records into otel_logs table

Actual Result

Error produced and records couldn't be inserted into otel_logs table:

Error = no more retries left: ExecContext:clickhouse [Append]: clickhouse: expected 17 arguments, got 15

Collector version

v0.107.0

Environment information

Environment

OS: "Ubuntu 22.04"

OpenTelemetry Collector configuration

exporters:
  clickhouse:                                                                                                                                                                                        
    async_insert: false
    create_schema: false
    database: otel
    endpoint: http://somehost:8123?dial_timeout=10s&compress=brdial_timeout=10s&compress=br
    username: someuser
    password: somepassword
    retry_on_failure:                                                                                                                    
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
      max_interval: 30s
    sending_queue:
      queue_size: 10000
    timeout: 5s
processors:
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25
receivers:
  filelog:
    include:
      - /var/log/*.log
    # parsing part skipped for simplicity
service:
  pipelines:
    logs:
      exporters:
        - clickhouse
      processors:
        - memory_limiter
        - batch
      receivers:
        - filelog
  # Other pipelines skipped

Log output

{"level":"error","msg":"Exporting failed. Dropping data.","ts":1723736161.3814555,"caller":"exporterhelper/queue_sender.go:92","kind":"exporter","data_type":"logs","name":"clickhouse","error":"no more retries left: ExecContext:clickhouse [Append]:  clickhouse: expected 17 arguments, got 15","stacktrace":"go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1\n\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:92\ngo.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume\n\tgo.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52\ngo.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1\n\tgo.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43","dropped_items":788}

### Additional context

_No response_

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SpencerTorres]

Thank you for finding this bug. I will correctly mark the open PR as a breaking change with some steps for proper migration path.

Let me know if you need any suggestions for migrating these tables. For now I would suggest creating the new tables with a suffix such as otel_logs_0, otel_logs_1, etc. as they are upgraded. You should then be able to query across them using one of the table merge functions

[SpencerTorres]

Not sure which fix would be better for you (quicker to resolve on your end), but we could modify the insert statement if the only issue is the newline tokens. I see your change on the clickhouse-go repo

[Fiery-Fenix]

Not sure which fix would be better for you (quicker to resolve on your end), but we could modify the insert statement if the only issue is the newline tokens. I see your change on the clickhouse-go repo

Right now - it's not so urgent issue for me, I can keep going with version 0.106.0 until bug will be fixed in clickhouse-go driver first and Collector will be released using fixed version of the driver

[Fiery-Fenix]

Validated that upgrading to version 0.108.0+ fixes this issue
Fix was implemented by dependency version update in #34752
Closing this issue