Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace string literal cannot escape $ next to a capture group #21462

Closed
darinspivey opened this issue Oct 9, 2024 · 7 comments · Fixed by #21467
Closed

replace string literal cannot escape $ next to a capture group #21462

darinspivey opened this issue Oct 9, 2024 · 7 comments · Fixed by #21467
Labels
type: bug A code related bug.

Comments

@darinspivey
Copy link

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

There seems to be some issues with escaping certain things within string literals, as dollar sign $ has multiple uses

  • Referencing ENV vars in config with ${VAR}
  • Referencing capture groups within a replace command (which must be escaped with $$ in conf files)
  • But mainly, escaping itself with another $

With the replace command, it is not possible to escape a $ immediately preceding a capture group, which is useful when dealing with price data. Consider the following replace_with string literals, assuming the price has been placed in a price capture group:

s'$$$price' --> error=Missing environment variable in config. name = "price"
s'$$$${price}' --> ${price}
s'$$$$price' --> $price
s'#$$price' --> #75 // correct

As an aside, a similar issue happens when trying to escape a single backslash within a string literal.

.nope = s'\' --> syntax error
.nope = s'\\' --> "nope": "\\\\"

This condition may be a separate issue, but it also has a workaround. You can split the string literal into parts and use concatenation with the string type. but it's messy. For example, s'endinbackslash' + "\\". However, this workaround does not work with replace_with because the concatenated string portion does not honor capture group names.

Configuration

[sources.source1]
  type = "demo_logs"
  format = "apache_common"

  [transforms.replace]
    type = "remap"
    inputs = ["source1"]
    source = '''
        . = replace("The prices are 12, 75, and 99", r'(?P<price>\d+)', s'$$$${price}')                                               
    '''

  [sinks.sink1]
  type = "console"
  inputs = [ "replace" ]
  encoding.codec = "json"

Version

vector 0.42.0 (aarch64-apple-darwin)

Debug Output

2024-10-09T14:19:06.627941Z DEBUG vector::topology::builder: Building new source. component=source1
2024-10-09T14:19:06.628196Z DEBUG vector::topology::builder: Building new transform. component=replace
2024-10-09T14:19:06.628647Z DEBUG vector::topology::builder: Building new sink. component=sink1
2024-10-09T14:19:06.628779Z  INFO vector::topology::running: Running healthchecks.
2024-10-09T14:19:06.628799Z DEBUG vector::topology::running: Connecting changed/added component(s).
2024-10-09T14:19:06.628807Z DEBUG vector::topology::running: Configuring outputs for source. component=source1
2024-10-09T14:19:06.628829Z DEBUG vector::topology::running: Configuring output for component. component=source1 output_id=None
2024-10-09T14:19:06.628833Z DEBUG vector::topology::running: Configuring outputs for transform. component=replace
2024-10-09T14:19:06.628837Z DEBUG vector::topology::running: Configuring output for component. component=replace output_id=None
2024-10-09T14:19:06.629417Z DEBUG vector::topology::running: Connecting inputs for transform. component=replace
2024-10-09T14:19:06.628873Z  INFO vector::topology::builder: Healthcheck passed.
2024-10-09T14:19:06.629434Z DEBUG vector::topology::running: Adding component input to fanout. component=replace fanout_id=source1
2024-10-09T14:19:06.629448Z DEBUG vector::topology::running: Connecting inputs for sink. component=sink1
2024-10-09T14:19:06.629453Z DEBUG vector::topology::running: Adding component input to fanout. component=sink1 fanout_id=replace
2024-10-09T14:19:06.629475Z DEBUG vector::topology::running: Spawning new source. key=source1
2024-10-09T14:19:06.629656Z DEBUG vector::topology::running: Spawning new transform. key=replace
2024-10-09T14:19:06.629675Z TRACE vector::topology::running: Spawning new sink. key=sink1
2024-10-09T14:19:06.629800Z  INFO vector: Vector has started. debug="false" version="0.42.0" arch="aarch64" revision=""
2024-10-09T14:19:06.629824Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-10-09T14:19:06.629757Z DEBUG source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::topology::builder: Source pump supervisor starting.
2024-10-09T14:19:06.629905Z DEBUG source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::topology::builder: Source pump starting.
2024-10-09T14:19:06.629897Z DEBUG source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::topology::builder: Source starting.
2024-10-09T14:19:06.629911Z DEBUG transform{component_kind="transform" component_id=replace component_type=remap}: vector::topology::builder: Synchronous transform starting.
2024-10-09T14:19:06.629828Z DEBUG sink{component_kind="sink" component_id=sink1 component_type=console}: vector::topology::builder: Sink starting.
2024-10-09T14:19:06.631101Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:06.631113Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:06.631113Z DEBUG sink{component_kind="sink" component_id=sink1 component_type=console}: vector::utilization: utilization=0.055863110216406664
2024-10-09T14:19:06.631105Z TRACE vector: Beep.
2024-10-09T14:19:06.631273Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=126
2024-10-09T14:19:06.631305Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:06.631309Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=234 output=_default
2024-10-09T14:19:06.631320Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "replace" })
2024-10-09T14:19:06.631333Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:06.631378Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:06.631386Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=234
2024-10-09T14:19:06.631507Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Processing control message outside of send: ControlMessage::Add(ComponentKey { id: "sink1" })
2024-10-09T14:19:06.631544Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:06.631550Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:06.631556Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:06.631693Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:06.631730Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
2024-10-09T14:19:06.631735Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:07.632354Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:07.632432Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:07.632405Z TRACE vector: Beep.
2024-10-09T14:19:07.632625Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=121
2024-10-09T14:19:07.632695Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:07.632710Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=229 output=_default
2024-10-09T14:19:07.632773Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:07.633864Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:07.633910Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=229
2024-10-09T14:19:07.634094Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:07.634643Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:07.634660Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:07.634705Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:07.635367Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:07.635387Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
2024-10-09T14:19:08.630906Z TRACE vector: Beep.
2024-10-09T14:19:08.631009Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:08.631044Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:08.631151Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=121
2024-10-09T14:19:08.631219Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:08.631234Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=229 output=_default
2024-10-09T14:19:08.631292Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:08.631960Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:08.632010Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=229
2024-10-09T14:19:08.632196Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:08.632710Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:08.632729Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:08.632772Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:08.633325Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:08.633346Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
2024-10-09T14:19:09.632322Z TRACE vector: Beep.
2024-10-09T14:19:09.632353Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:09.632397Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:09.632580Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=124
2024-10-09T14:19:09.632691Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:09.632711Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=232 output=_default
2024-10-09T14:19:09.632776Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:09.633252Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:09.633293Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=232
2024-10-09T14:19:09.633530Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:09.633936Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:09.633960Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:09.634003Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:09.634506Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:09.634517Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
2024-10-09T14:19:10.630694Z TRACE vector: Beep.
2024-10-09T14:19:10.630682Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:10.630783Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:10.631506Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=127
2024-10-09T14:19:10.631579Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:10.631595Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=235 output=_default
2024-10-09T14:19:10.631655Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:10.632194Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:10.632236Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=235
2024-10-09T14:19:10.632570Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:10.632943Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:10.632954Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:10.632982Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:10.633660Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
2024-10-09T14:19:10.633674Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:11.631930Z TRACE vector: Beep.
2024-10-09T14:19:11.631930Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:11.631957Z DEBUG sink{component_kind="sink" component_id=sink1 component_type=console}: vector::utilization: utilization=0.006068953275014809
2024-10-09T14:19:11.632026Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:11.632671Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=115
2024-10-09T14:19:11.632763Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:11.632780Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=223 output=_default
2024-10-09T14:19:11.632854Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:11.634004Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:11.634043Z DEBUG transform{component_kind="transform" component_id=replace component_type=remap}: vector::utilization: utilization=0.00046742602724192395
2024-10-09T14:19:11.634070Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=223
2024-10-09T14:19:11.635028Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:11.635046Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:11.635056Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:11.635429Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:11.635475Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:11.635484Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
2024-10-09T14:19:12.631158Z TRACE vector: Beep.
2024-10-09T14:19:12.631187Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:12.631264Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:12.631367Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=133
2024-10-09T14:19:12.631472Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:12.631492Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=241 output=_default
2024-10-09T14:19:12.631548Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:12.631575Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:12.632995Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=241
2024-10-09T14:19:12.633181Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:12.633201Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:12.633730Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:12.633774Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:12.633829Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:12.634313Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
^C2024-10-09T14:19:12.712252Z  INFO vector::signal: Signal received. signal="SIGINT"
2024-10-09T14:19:12.712338Z  INFO vector: Vector has stopped.
2024-10-09T14:19:12.713677Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="sink1, source1, replace" time_remaining="59 seconds left"
2024-10-09T14:19:13.631803Z TRACE vector: Beep.
2024-10-09T14:19:13.631803Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2024-10-09T14:19:13.631875Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::internal_events::demo_logs: Received one event.
2024-10-09T14:19:13.632035Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_received: Events received. count=1 byte_size=125
2024-10-09T14:19:13.632086Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:13.632094Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=233 output=_default
2024-10-09T14:19:13.632152Z DEBUG source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::topology::builder: Source finished normally.
2024-10-09T14:19:13.632608Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Processing control message inside of send: None
2024-10-09T14:19:13.632712Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:13.632731Z TRACE source{component_kind="source" component_id=source1 component_type=demo_logs}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:13.633063Z DEBUG source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::topology::builder: Source pump finished normally.
2024-10-09T14:19:13.633139Z DEBUG source{component_kind="source" component_id=source1 component_type=demo_logs}: vector::topology::builder: Source pump supervisor task finished normally.
2024-10-09T14:19:13.633170Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_received: Events received. count=1 byte_size=233
2024-10-09T14:19:13.633302Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Processing control message inside of send: None
2024-10-09T14:19:13.633340Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_buffers::topology::channel::limited_queue: Sent item.
2024-10-09T14:19:13.633348Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_core::fanout: Sent item to fanout.
2024-10-09T14:19:13.633355Z TRACE transform{component_kind="transform" component_id=replace component_type=remap}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=63 output=_default
2024-10-09T14:19:13.633690Z DEBUG transform{component_kind="transform" component_id=replace component_type=remap}: vector::topology::builder: Synchronous transform finished normally.
2024-10-09T14:19:13.633724Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_received: Events received. count=1 byte_size=63
2024-10-09T14:19:13.633765Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=61
{"message":"The prices are ${price}, ${price}, and ${price}"}
2024-10-09T14:19:13.633940Z TRACE sink{component_kind="sink" component_id=sink1 component_type=console}: vector_common::internal_event::bytes_sent: Bytes sent. byte_size=62 protocol=console
2024-10-09T14:19:13.633982Z DEBUG sink{component_kind="sink" component_id=sink1 component_type=console}: vector::topology::builder: Sink finished normally.

Example Data

The configuration provided will re-produce the problem with example data inline.

Additional Context

No response

References

There have been other issues regarding escaping $, but not this specifically. I did not look into the backslash condition since I found a workaround, although not ideal.

@darinspivey darinspivey added the type: bug A code related bug. label Oct 9, 2024
@jszwedko
Copy link
Member

jszwedko commented Oct 9, 2024

@darinspivey it should be:

. = replace("The prices are 12, 75, and 99", r'(?P<price>\d+)', "$$price")  

Had you already tried that? The $$ translates to $.

@jszwedko
Copy link
Member

jszwedko commented Oct 9, 2024

This is mentioned here: https://vector.dev/docs/reference/configuration/#escaping. Admittedly it is one of the trickier foot guns to be aware of when using capture groups.

@darinspivey
Copy link
Author

Had you already tried that? The $$ translates to $.

Yes, I tried that, but it's not the desired outcome. The docs say to escape it with $$ in a TOML configuration file (which, of course we're using for this pipeline). That references the capture group properly, but does not account for tacking on a preceding $ in the output.

The intention is to grab the numeric price, and add a static $ before it, which cannot be done, e.g. $75, not 75 (which is the original data).

@jszwedko
Copy link
Member

jszwedko commented Oct 9, 2024

Oh I see. You want a leading $. You will need a lot of escaping 😓 I think it would be:

. = replace("The prices are 12, 75, and 99", r'(?P<price>\d+)', "$$$$$$price")  

The reason you need so many $s is that the first pass, env var interpolation, will convert $$ to $, so VRL sees:

. = replace("The prices are 12, 75, and 99", r'(?P<price>\d+)', "$$$price")  

The regex crate also uses $$ for escaping a $ so this ends up replacing with a literal $ followed by the price capture group.

Does that make sense?

@darinspivey
Copy link
Author

My goodness! I stopped trying after 4 dollars wouldn't work thinking that it would just rinse/repeat the behavior with 4. This indeed does work, although I would suggest adding some examples such as this to the docs. It's not quite enough to read that $ has to become $$ to avoid ENV interpolation. That bit about the regex crate is super important, and might be useful context for the replace docs :)

As an aside, do you have a similar fix for the single backslash problem mentioned above? Seems very similar, although it could be a separate string literal issue.

Thanks, @jszwedko !

@jszwedko
Copy link
Member

jszwedko commented Oct 9, 2024

No worries! It is definitely confusing. I opened #21467 to try to better document it.

For getting a single backslash character in string literals s'', I think that might actually be a bug in the VRL parser. vectordotdev/vrl#30 seems to be related. Would you want to file a bug over here: https://github.com/vectordotdev/vrl/issues?

@darinspivey
Copy link
Author

Would you want to file a bug over here:

Absolutely. Thank you for the docs update on this issue--that's much clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A code related bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants