Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run linters with configurable concurrency #149

Merged
merged 4 commits into from
May 25, 2017

Conversation

sudo-suhas
Copy link
Collaborator

@sudo-suhas sudo-suhas commented Apr 25, 2017

Changelog

  • package.json - Add lodash.chunk, p-map dependency
  • Add file callChunkSize.js for bullet proof chunk size calculation
  • Update findBin to use calcChunkSize.js and return array of runnables
  • index.js
    • Add helper method for reading config option
    • Add config options subTaskConcurrency and chunkSize to be passed to runScript
  • runScript
    • Use p-map to run runnables returned from findBin with configured concurrency
    • Catch all errors, if any, combine and throw back to caller
  • Add calcChunkSize.spec.js for tests
  • Update findBin.spec.js
    • Always pass files as an array to findBin
    • Expect array of { bin, args } as returned value
  • Update runScript.spec.js to always pass files as an array to runScript

Why p-map

Although Listr supports a concurrency option, it is just a switch. We cannot control the number of concurrent things we want to run at a time. So I have used p-map for this. Avoids having to include an entire Promise library implementation.

TODO

  • Fix failing tests
  • Improve tests. Need to add new tests for chunked execution
  • Add documentation for subTaskConcurrency and chunkSize
  • Edge case handling is present for subTaskConcurrency in both index.js as well as runScript.js.
    Similarly for chunkSize in index.js and calcChunkSize.js. Is there a better way?
  • Figure out reasonable defaults for chunkSize and subTaskConcurrency
  • Fix babel config for regeneratorRuntime on node 4

Notes

The pathsToLint which is passed to runScript and subsequently to findBin should always be an array and the tests have been updated for this. I tested locally when only 1 file was staged and even in that case, it was passed as an array. Is there a situation where it will be passed as a string?

Do we need to validate option subTaskConcurrency? It is expected to be a number greater than or equal to 1.

Since we are splitting the lint task into chunks, if multiple chunks have errors, the error report will be separate. The PR already ensures that we don't terminate on first rejection. But the err report is slightly non ideal. Extreme case example(intentionally simulated):

Error report
> [email protected] lint-staged E:\Projects\repos\elastic-builder
> lint-staged

 > Running tasks for src/**/*.js
   × eslint --fix
     git add
 > Running tasks for test/**/*.js
   × eslint --fix
     git add
🚫 eslint --fix found some errors. Please fix them and try committing again.

E:\Projects\repos\elastic-builder\test\core-test\request-body-search.test.js
  1:1  error  File must be at most 300 lines long. It's 316 lines long  max-lines

E:\Projects\repos\elastic-builder\test\core-test\highlight.test.js
   1:1  error  File must be at most 300 lines long. It's 317 lines long    max-lines
  40:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  50:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  60:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  95:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline

✖ 6 problems (6 errors, 0 warnings)



� eslint --fix found some errors. Please fix them and try committing again.

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\missing-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\ip-range-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\histogram-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\histogram-aggregation-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\global-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\geo-hash-grid-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\geo-distance-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\filters-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\filter-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\diversified-sampler-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\date-range-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\date-histogram-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\children-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\bucket-aggregation-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

✖ 15 problems (15 errors, 0 warnings)



E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\sum-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\stats-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\serial-differencing-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\pipeline-aggregation-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\percentiles-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\moving-average-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\min-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\max-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\extended-stats-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\derivative-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\cumulative-sum-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\bucket-selector-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\bucket-script-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\pipeline-aggregations\avg-bucket-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\value-count-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\top-hits-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\sum-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\stats-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\scripted-metric-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\percentiles-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\percentile-ranks-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\min-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\metrics-aggregation-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\max-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\geo-centroid-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\geo-bounds-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\extended-stats-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\cardinality-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\metrics-aggregations\avg-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\matrix-aggregations\matrix-stats-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\matrix-aggregations\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\terms-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\terms-aggregation-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\significant-terms-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\sampler-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\reverse-nested-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\range-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\range-aggregation-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\aggregations\bucket-aggregations\nested-aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

✖ 42 problems (42 errors, 0 warnings)



E:\Projects\repos\elastic-builder\src\queries\term-level-queries\wildcard-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\value-term-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\type-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\terms-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\term-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\regexp-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\range-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\prefix-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\multi-term-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\ids-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\fuzzy-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\term-level-queries\exists-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\specialized-queries\script-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\specialized-queries\percolate-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\specialized-queries\more-like-this-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\specialized-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-within-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-term-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-or-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-not-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-near-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-multi-term-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-little-big-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-first-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-field-masing-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\span-containing-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\span-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\match-none-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\match-all-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\joining-queries\parent-id-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\joining-queries\nested-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\joining-queries\joining-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\joining-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\joining-queries\has-parent-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\joining-queries\has-child-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\helper.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\geo-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\geo-queries\geo-shape-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\geo-queries\geo-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

✖ 42 problems (42 errors, 0 warnings)



E:\Projects\repos\elastic-builder\src\queries\geo-queries\geo-polygon-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\geo-queries\geo-distance-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\geo-queries\geo-bounding-box-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\simple-query-string-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\query-string-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\query-string-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\multi-match-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\mono-field-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\match-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\match-phrase-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\match-phrase-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\match-phrase-prefix-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\full-text-query-base.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\full-text-queries\common-terms-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\score-functions\script-score-function.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\score-functions\score-function.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\score-functions\random-score-function.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\score-functions\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\score-functions\field-value-factor-function.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\score-functions\decay-score-function.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\function-score-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\dis-max-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\constant-score-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\boosting-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\queries\compound-queries\bool-query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\index.js
  4:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\util.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\sort.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\script.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\rescore.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\request-body-search.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\query.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\inner-hits.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\indexed-shape.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\index.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\highlight.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\geo-shape.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\geo-point.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\consts.js
  2:5  error  'unused' is defined but never used  no-unused-vars

E:\Projects\repos\elastic-builder\src\core\aggregation.js
  2:5  error  'unused' is defined but never used  no-unused-vars

✖ 42 problems (42 errors, 0 warnings)





npm ERR! Windows_NT 6.2.9200
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "lint-staged"
npm ERR! node v6.10.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] lint-staged: `lint-staged`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] lint-staged script 'lint-staged'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the elastic-builder package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     lint-staged
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs elastic-builder
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls elastic-builder
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     E:\Projects\repos\elastic-builder\npm-debug.log

Closes #147

@okonet
Copy link
Collaborator

okonet commented Apr 25, 2017

I've found SamVerschueren/listr#54 related to this. Should we instead submit a PR to it and use it here?

@sudo-suhas
Copy link
Collaborator Author

Depends on whether we want to expose the chunked execution to the user. Might be more noisy to see the task being executed in multiple chunks. I would prefer if the library hid that from the user. But I am okay with it either way. Is there any advantage of using listr for concurrency? Also, there is already an open PR for this - SamVerschueren/listr#56

@sudo-suhas
Copy link
Collaborator Author

Looks like listr now uses p-map for concurrency in version 0.12.0.

@okonet
Copy link
Collaborator

okonet commented May 2, 2017

@sudo-suhas cool! Does this mean we can leverage it and simplify the code of this package?

@sudo-suhas
Copy link
Collaborator Author

sudo-suhas commented May 2, 2017

That depends on a couple of things

  • Do we want to distinguish between concurrency of linters and concurrency of sub tasks in linters?
  • If we do make this distinction, should we hide the execution of sub tasks or show it via listr? Depending on this, we could remove the p-map dependency and use listr directly.

One things is certain, in order to not hit the OS limitation on windows, we definitely have to chunk the files. So the only changes, if any, would be in runScript.js. The rest of it, for calculating chunk size in calcChunkSize.js, generating runnables in findbin.js would still be needed.

@okonet
Copy link
Collaborator

okonet commented May 2, 2017

  1. Yes, we want to run "linters" i.e. glob sections in parallel but not tasks inside each linter step. It's is like this right now so not sure why it's related to this decision.
  2. I think showing this is okay but if the output is too noise I might reconsider. Can we try it? I'd keep the surface of this package as minimal as possible.

@sudo-suhas
Copy link
Collaborator Author

sudo-suhas commented May 2, 2017

@okonet I'll try to clear up the terminology so that we are both on the same page.
Let's assume the following lint-staged config:

{
  "lint-staged": {
    "src/**/*.js": [
      "eslint --fix",
      "git add"
    ],
    "test/**/*.js": [
      "eslint --fix",
      "git add"
    ]
  }
}

So, like you said, src/**/*.js is a glob pattern and right now, unless explicitly overridden, it runs in parallel with the test/**/*.js glob pattern. The tasks inside each glob pattern still run serially. I am not proposing any changes to this.

So why does this PR exist? Let us say in the glob pattern src/**/*.js, ~100 files were modified. So what actually happens is a eslint command is run with 100 file paths passed as arguments. This breaks on windows(command length). So I am trying to break up these 100 files into chunks and run those tasks(eslint commands) with configurable concurrency. So these multiple eslint tasks are what I am calling sub tasks. Hence the added config options for subTaskConcurrency and chunkSize.

I hope I got that across. If you want me to remove the p-map dependency and use listr instead, I can. But it would mean that apart from using listr in index.js, I would also have to use it in runScript.js.

@okonet
Copy link
Collaborator

okonet commented May 2, 2017

I got it! Thanks for the detailed explanation. Would it be possible to have a comparison of using listr options and this PR console outputs? Or can you explain what is the main difference would be between doing this internally (this PR) or using Listr's functionality?

@sudo-suhas
Copy link
Collaborator Author

I have already included the console output the PR currently outputs in the worst case(errors in multiple chunks). It is at the bottom of the PR description in a collapsed section. Note that if there are no errors, the console output will be exactly the same as the current master version. I'll try to do the changes for using listr to see how that looks as well.

@sudo-suhas
Copy link
Collaborator Author

sudo-suhas commented May 8, 2017

@okonet I am finding it difficult to implement this with Listr. In runScript.js, I need to attach then and catch to the promise being returned from the function. However, if I run the tasks without delegating to the run in index.js, it produces wrong output:

Console output
E:\Projects\repos\elastic-builder (master) ([email protected])
λ npm run lint-staged

> [email protected] lint-staged E:\Projects\repos\elastic-builder
> lint-staged

 √ Chunk 0
 √ Chunk 1
 √ Chunk 2
 √ Chunk 0
 √ Chunk 1
 √ Chunk 2
 × Chunk 0
 × Chunk 1
 √ Chunk 2
 √ Running tasks for src/**/*.js
 > Running tasks for test/**/*.js
   × eslint --fix
     git add
� eslint --fix found some errors. Please fix them and try committing again.

E:\Projects\repos\elastic-builder\test\queries-test\more-like-this-query.test.js
  29:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  47:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  57:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  67:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline

✖ 4 problems (4 errors, 0 warnings)



E:\Projects\repos\elastic-builder\test\core-test\request-body-search.test.js
  1:1  error  File must be at most 300 lines long. It's 316 lines long  max-lines

E:\Projects\repos\elastic-builder\test\core-test\highlight.test.js
   1:1  error  File must be at most 300 lines long. It's 318 lines long    max-lines
  40:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  50:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  60:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline
  95:9  error  Unexpected newline between object and [ of property access  no-unexpected-multiline

✖ 6 problems (6 errors, 0 warnings)

If I do delegate, then I can't print the success, error messages correctly.

code
'use strict'

const execa = require('execa')
const Listr = require('listr')

const findBin = require('./findBin')

module.exports = function runScript(commands, pathsToLint, packageJson, options) {
    const concurrent = options && options.subTaskConcurrency
        ? options.subTaskConcurrency
        : 4
    const renderer = options && options.renderer
        ? options.renderer
        : 'update'
    const lintersArray = Array.isArray(commands) ? commands : [commands]
    return lintersArray.map(linter => ({
        title: linter,
        task: () => {
            try {
                const tasks = findBin(linter, pathsToLint, packageJson, options)
                    .map((res, idx) => ({
                        title: `Chunk ${ idx }`,
                        task: () => {
                            const execaOptions = res.bin !== 'npm' && options && options.gitDir
                                ? { cwd: options.gitDir } : {}

                            return execa(res.bin, res.args, execaOptions)
                        }
                    }))

                return new Listr(tasks, {
                    concurrent,
                    renderer,
                    exitOnError: false
                })
            } catch (err) {
                throw err
            }
        }
    }))
}
What do you suggest I do?

Copy link
Collaborator

@okonet okonet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only concern (besides the comments I left) is that findBin looks rather very complex now. Do you think it would make more sense to refactor it in a way it only returns the bin and have the second module that is concerned about arguments?

src/index.js Outdated
@@ -27,10 +27,16 @@ cosmiconfig('lint-staged', {
// result.config is the parsed configuration object
// result.filepath is the path to the config file that was found
const config = result.config

const readConfigOption = (key, defaultValue) =>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the approach but it would be great this to be in a separate file and have tests. Then it can also be reused here: https://github.com/okonet/lint-staged/pull/149/files#diff-f7efc3e72253568bae00aa123c61fd95R41

src/runScript.js Outdated

module.exports = function runScript(commands, pathsToLint, packageJson, options) {
const concurrency = options && options.subTaskConcurrency
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is subTaskConcurrency is 0 it will use fallback value 4 I believe. Please use the readConfigOption function here as well.

@okonet
Copy link
Collaborator

okonet commented May 8, 2017

@sudo-suhas okay let's go with your solution but let's clean up a bit.

@sudo-suhas
Copy link
Collaborator Author

@okonet I am doing all the changes you requested(including findBin).

Regarding the chunkSize, I was thinking, speed is less important as compared to user friendly error reports. And the readability of error report degrades rapidly with smaller chunk size in the worst case scenario. This is because, smaller the chunk size, more the chunks and thereby, more fragmented error report outputs. You can see an example of this in the first comment(bottom, collapsed section). Additionally, the speed of execution with concurrency is debatable. We already might be running multiple glob patterns in parallel for CPU bound systems. Shall we calculate the maximum possible chunk size by default? Obviously, user can override if so desired.

@okonet
Copy link
Collaborator

okonet commented May 8, 2017

I'm not sure about the maximum chunk size. I'd leave it to you :)

@sudo-suhas
Copy link
Collaborator Author

@okonet If you can help me a little on how to fix the failing tests, we are just about done.

@okonet
Copy link
Collaborator

okonet commented May 8, 2017

Hmm, I honestly don't know how you should change the test. It seems it has to do with pmap.

@sudo-suhas
Copy link
Collaborator Author

sudo-suhas commented May 11, 2017

The problem is due to the mocks. When I run the tests, I get the following error message multiple times:

(node:13228) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: � test got an unexpected error.
Cannot read property 'catch' of undefined

This is originating from this line. Which means that the mocked execa does not return a Promise on being called. I tried looking at the jest docs but am unable to figure out how this should be fixed.

Another problem is that the test is called before the Promise returned by pMap is. I tried waiting for the promise from task() to resolve but that gave a weirder error:

No longer relevant
    // apologies for the dirty code
    it('should work with multiple commands', async () => {
        const res = runScript(['test', 'test2'], ['test.js'], packageJSON)
        expect(res.length).toBe(2)
        expect(res[0].title).toBe('test')
        expect(res[1].title).toBe('test2')

        expect(res[0].task()).toBeAPromise()
        try { await res[0].task() } catch (ignore) {}
        expect(mockFn.mock.calls.length).toEqual(1)
        expect(mockFn.mock.calls[0]).toEqual(
            ['npm', ['run', '--silent', 'test', '--', 'test.js'], {}]
        )
        expect(res[1].task()).toBeAPromise()
        try { await res[0].task() } catch (ignore) {}
        expect(mockFn.mock.calls.length).toEqual(2)
        expect(mockFn.mock.calls[1]).toEqual(
            ['npm', ['run', '--silent', 'test2', '--', 'test.js'], {}]
        )
    })
// Console output
  ● runScript › should work with multiple commands

    Expected 3 to equal 1

      at assert (node_modules\expect\lib\assert.js:29:9)
      at Expectation.toEqual (node_modules\expect\lib\Expectation.js:81:30)
      at _callee$ (test\runScript.spec.js:62:82)
      at tryCatch (node_modules\regenerator-runtime\runtime.js:64:40)
      at GeneratorFunctionPrototype.invoke [as _invoke] (node_modules\regenerator-runtime\runtime.js:299:22)
      at GeneratorFunctionPrototype.prototype.(anonymous function) [as throw] (node_modules\regenerator-runtime\runtime.js:116:21)
      at step (test\runScript.spec.js:8:641)
      at test\runScript.spec.js:8:840
      at process._tickCallback (internal\process\next_tick.js:103:7)

If I log the contents of mockFn.mock.calls

      [ [ 'npm', [ 'run', '--silent', 'test', '--', 'test.js' ], {} ],
        [ 'npm', [ 'run', '--silent', 'test', '--', 'test.js' ], {} ],
        [ 'npm', [ 'run', '--silent', 'test', '--', 'test.js' ], {} ] ]

EDIT: So now the tests are not failing. But the mock needs to be fixed Then I can remove the ugly hack for ignoring the error.

@codecov
Copy link

codecov bot commented May 11, 2017

Codecov Report

Merging #149 into master will increase coverage by 10.25%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #149       +/-   ##
=========================================
+ Coverage   89.74%   100%   +10.25%     
=========================================
  Files           3      5        +2     
  Lines          39     65       +26     
  Branches        5      8        +3     
=========================================
+ Hits           35     65       +30     
+ Misses          4      0        -4
Impacted Files Coverage Δ
src/runScript.js 100% <100%> (+25%) ⬆️
src/calcChunkSize.js 100% <100%> (ø)
src/findBin.js 100% <100%> (ø) ⬆️
src/readConfigOption.js 100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7283b7...4c38c35. Read the comment docs.

@sudo-suhas
Copy link
Collaborator Author

Had to rm -rf node_modules, npm install to reproduce error in node 4.
Using transform-runtime either with babel-preset-env or ["es2015", "stage-0"] works. Which would you prefer?

{
  "presets": [
    ["env", {
      "targets": {
        "node": "current"
      }
    }]
  ],
  "plugins": [
    ["transform-runtime", {
      "helpers": false,
      "polyfill": false,
      "regenerator": true
    }]
  ]
}

or

{
    "presets": ["es2015", "stage-0"],
    "plugins": [
        ["transform-runtime", {
        "helpers": false,
        "polyfill": false,
        "regenerator": true
        }]
    ]
}

@okonet
Copy link
Collaborator

okonet commented May 11, 2017

I've just merged #167 that should actually fix it AFAIK. Can you try to rebase on master and see if it works?

@okonet
Copy link
Collaborator

okonet commented May 11, 2017

I'm also doing a pretty big refactoring to be able to add more test coverage (using Jest snapshots for the stdout): #141. Do you think it would make sense to merge mine and resolve conflicts in this PR? Or merge yours and leave me with it? ;)

@sudo-suhas
Copy link
Collaborator Author

I've just merged #167 that should actually fix it AFAIK. Can you try to rebase on master and see if it works?

I did the rebase(not yet pushed) but both problems still exist

  • regeneratorRuntime is still a problem on node 4.8. Out of the two babel config changes that I had shared, which one would you prefer I use? babel-preset-env or ["es2015", "stage-0"]?
  • execa mock still has the issue with promise:
    TypeError: Cannot read property 'catch' of undefined
          at mapper (E:\Projects\repos\lint-staged\src\runScript.js:40:70)
          at Promise.resolve.then.el (E:\Projects\repos\lint-staged\node_modules\p-map\index.js:42:16)
          at process._tickCallback (internal/process/next_tick.js:103:7)
    
npm test ERR output
E:\Projects\repos\lint-staged (concurrent-lint) ([email protected])
λ npm test

> [email protected] test E:\Projects\repos\lint-staged
> jest --coverage

 PASS  test\findBin.spec.js
 PASS  test\generateTasks.spec.js
 FAIL  test\runScript.spec.js
  ● runScript › encountered a declaration exception

    ReferenceError: regeneratorRuntime is not defined

      at Suite.<anonymous> (test/runScript.spec.js:45:63)
      at Object.<anonymous> (test/runScript.spec.js:29:1)
      at emitTwo (events.js:87:13)
      at process.emit (events.js:172:7)
      at internal/child_process.js:721:12
      at nextTickCallbackWith0Args (node.js:489:9)
      at process._tickCallback (node.js:418:13)

 PASS  test\calcChunkSize.spec.js
 PASS  test\readConfigOption.spec.js

Test Suites: 1 failed, 4 passed, 5 total
Tests:       1 failed, 21 passed, 22 total
Snapshots:   0 total
Time:        3.689s, estimated 122s
Ran all test suites.
---------------------|----------|----------|----------|----------|----------------|
File                 |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |
---------------------|----------|----------|----------|----------|----------------|
All files            |       75 |    65.22 |     62.5 |    77.97 |                |
 calcChunkSize.js    |      100 |      100 |      100 |      100 |                |
 findBin.js          |      100 |      100 |      100 |      100 |                |
 generateTasks.js    |      100 |      100 |      100 |      100 |                |
 readConfigOption.js |      100 |      100 |      100 |      100 |                |
 runScript.js        |    51.52 |    11.11 |    33.33 |    55.17 |... 54,56,57,59 |
---------------------|----------|----------|----------|----------|----------------|
npm ERR! Test failed.  See above for more details.

E:\Projects\repos\lint-staged (concurrent-lint) ([email protected])
λ npm ls babel-jest
[email protected] E:\Projects\repos\lint-staged
└── [email protected]

Do you think it would make sense to merge mine and resolve conflicts in this PR? Or merge yours and leave me with it? ;)

I am going to defer to your better judgement here. I have only recently started making pull requests and am not entirely sure how we would go about either option. However, I would be more than happy to read up and do the needful like I did for rebase.

@okonet
Copy link
Collaborator

okonet commented May 12, 2017

Let's go with a more bullet proof babel-preset-env.

How can I be the help with execa? Here is the mock for it: https://github.com/okonet/lint-staged/blob/master/test/__mocks__/execa.js that is just an empty function. You might want to modify it to return a Promise instead.

@sudo-suhas
Copy link
Collaborator Author

@okonet I had tried modifying the mock before. I tried a few things again and here is the list of things which did not help:

jest.genMockFromModule('execa')

const execaMock = jest.fn()
// These still fail on .catch
// execaMock.mockImplementation(() => Promise.resolve(true))
// execaMock.mockReturnValue(Promise.resolve(true))

module.exports = execaMock
// also fails on .catch
jest.genMockFromModule('execa')

module.exports = jest.fn(() => Promise.resolve(true))

package.json - Add lodash.chunk, p-map dependency
calcChunkSize
  Add helper function for bullet proof chunk size calculation
  Add calcChunkSize.spec.js with tests
findBin
  Modify signature, remove paths from input
  Return {bin, args}, file paths should be appended before passing to execa
  Modify tests to match new signature
index
  Use readConfigOption for reading option concurrent
  Pass config to runScript options for reading subTaskConcurrency, chunkSize
readConfigOption
  Add generic helper function for reading config option
  Add readConfigOption.spec.js with tests
runScript
  Use p-map to run runnables returned from findBin with configured concurrency
  Use readConfigOption for reading config, subTaskConcurrency, chunkSize
  Default chunkSize to Number.MAX_SAFE_INTEGER to minimise number of chunks
  Default subTaskConcurrency to 2
  Use lodash.chunk to generate file path chunks
  Cache execaOptions and use Object.assign for cloning
  Combine findBin.args and file paths before passing to execa
  Update runScript.spec.js to always pass files as an array to `runScript`
Setup babel-preset-env + transform-runtime for async in node 4

Closes lint-staged#147

Closes lint-staged#147
@sudo-suhas
Copy link
Collaborator Author

@okonet I have rebased again and was able to get the tests and mock working thanks to the changes in the previous PR. Can you please review once more? I would really like to do the final changes(if any) and merge this in.

Copy link
Collaborator

@okonet okonet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! It would be great if you could remove the commented line. Otherwise ready to merge! Great work!

try {
await taskPromise
} catch (err) {
// expect(err).toBeInstanceOf(Error)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just got forgotten here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad..

}

return pMap(filePathChunks, mapper, { concurrency })
.catch((err) => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't covered with tests. Should we add one?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd have to add a mock right?

@okonet
Copy link
Collaborator

okonet commented May 24, 2017

@sudo-suhas great work! Thanks for taking time and implementing this.

@okonet
Copy link
Collaborator

okonet commented May 24, 2017

From the TODO there is still documentation bits that are missing. Could you please add them before I merge?

@sudo-suhas
Copy link
Collaborator Author

From the TODO there is still documentation bits that are missing. Could you please add them before I merge?

Is this ok?

  • chunkSize — Max allowed chunk size based on number of files for glob pattern. This is important on windows based systems to avoid command length limitations. See The command line is too long(windows) #147
  • subTaskConcurrency2 — Controls concurrency for processing chunks generated for each linter.

@okonet
Copy link
Collaborator

okonet commented May 24, 2017

@sudo-suhas looks good to me!

@sudo-suhas
Copy link
Collaborator Author

@okonet All done!

Copy link
Collaborator

@okonet okonet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work! Thanks!

@okonet okonet merged commit 79ad8b3 into lint-staged:master May 25, 2017
@sudo-suhas sudo-suhas deleted the concurrent-lint branch May 26, 2017 03:02
@okonet
Copy link
Collaborator

okonet commented Aug 22, 2017

@sudo-suhas it looks like the concurrency causing issues when used with git add: #225 (comment)

I can't remember all the details but can the default subTaskConcurrency be set to 1 to prevent this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

The command line is too long(windows)
2 participants