Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quantitative: corpus-line option doesn't work #391

Closed
RedXanadu opened this issue Nov 5, 2024 · 1 comment · Fixed by #404
Closed

quantitative: corpus-line option doesn't work #391

RedXanadu opened this issue Nov 5, 2024 · 1 comment · Fixed by #404
Assignees

Comments

@RedXanadu
Copy link
Member

RedXanadu commented Nov 5, 2024

In quantitative mode, the option --corpus-line does not currently work.

For example, the following call:

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-line=5674 --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --debug --paranoia-level=2

should only test corpus payload number 5674. In practice, it loops through all payloads:

$ ./ftw quantitative --crs-path ~/.git/coreruleset --corpus-line=5674 --corpus-lang=eng --corpus-source=news --corpus-year=2020 --corpus-size=10K --debug --paranoia-level=2
5:20PM INF ⏳Running quantitative tests with 1 goroutines
5:20PM DBG Using paranoia level: 2
5:20PM DBG Preparing download of corpus file from https://downloads.wortschatz-leipzig.de/corpora/eng_news_2020_10K.tar.gz
5:20PM DBG filename eng_news_2020_10K-sentences.txt already exists
5:20PM DBG **> rule 920221 with payload 7 =>  chain#0: get chain#1: /get?uri_payload=“A 30% correction sounds scary right, but the Dow, S&P 500 and the Nasdaq rallied to nearly 30% in just one year last year.
5:20PM DBG **> rule 920220 with payload 7 =>  chain#0: /get?uri_payload=“A 30% correction sounds scary right, but the Dow, S&P 500 and the Nasdaq rallied to nearly 30% in just one year last year. chain#1: /get?uri_payload=%E2%80%9CA+30%25+correction+sounds+scary+right%2C+but+the+Dow%2C+S%26P+500+and+the+Nasdaq+rallied+to+nearly+30%25+in+just+one+year+last+year. chain#2: ?uri_payload=“A 30% correction sounds scary right, but the Dow, S&P 500 and the Nasdaq rallied to nearly 30% in just one year last year.
5:20PM DBG **> rule 920220 with payload 14 =>  chain#0: /get?uri_payload=Aave (AAVE) traded 5.4% lower against the dollar and now trades at $34.47 or 0.00250836 BTC. chain#1: /get?uri_payload=Aave+%28AAVE%29+traded+5.4%25+lower+against+the+dollar+and+now+trades+at+%2434.47+or+0.00250836+BTC. chain#2: ?uri_payload=Aave (AAVE) traded 5.4% lower against the dollar and now trades at $34.47 or 0.00250836 BTC.
5:20PM DBG **> rule 920221 with payload 14 =>  chain#0: get chain#1: /get?uri_payload=Aave (AAVE) traded 5.4% lower against the dollar and now trades at $34.47 or 0.00250836 BTC.
⋮

So, --corpus-line is not being respected.

@M4tteoP M4tteoP changed the title corpus-line option doesn't work quantitative: corpus-line option doesn't work Nov 18, 2024
@M4tteoP M4tteoP self-assigned this Nov 18, 2024
@M4tteoP
Copy link
Member

M4tteoP commented Nov 18, 2024

I have this mostly fixed, I will open a PR soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants