Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CSV parsing to allow quoted and escaped fields #734

Merged
merged 2 commits into from
Nov 25, 2022

Conversation

EricTendian
Copy link
Contributor

The RadioReference talkgroup CSV output looks like the following:

Decimal,Hex,Alpha Tag,Mode,Description,Tag,Category
1,001,",T,"Installation / Techs / Evaluation",Business,"Motorola Installation / Testing"
2,002,"Motorola Tech 2",D,"Installation / Techs / Evaluation",Business,"Motorola Installation / Testing"
3,003,"Motorola Tech 3",DE,"Installation / Techs / Evaluation",Business,"Motorola Installation / Testing"

However, when trunk-recorder parses the CSV, it does not strip out the quotes, which leads to some malformed entry errors, as well as the call JSON to have syntax errors like the following:

{
    "talkgroup_tag": ""DC Tac 2"",
    "talkgroup_description": ""Tac 2"",
    "talkgroup_group_tag": "Law Dispatch",
    "talkgroup_group": ""ISP District Chicago""
}

This PR fixes that error by switching to the Boost escaped_list_separator class, which is able to handle text being quoted and escaped. The previous class char_separator did not handle this.

Both comma-separated values and tab-separated values are still supported by this parser. I've done some testing to make sure everything still works.

Besides this, I made a tiny change to the Dockerfile to make compiling trunk-recorder use all available CPU cores for faster compilation.

@robotastic
Copy link
Owner

These are great changes - thanks @EricTendian

@robotastic robotastic merged commit 7e9aefe into robotastic:master Nov 25, 2022
@EricTendian EricTendian deleted the better-csv-parsing branch November 25, 2022 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants