Long running build process for a very small file #4

d2s · 2017-07-30T10:54:16Z

While starting to create a new dictionary package (companies), I noticed that running build process takes a lot of time. In this case, I took a screenshot after 15 minutes, but the build process continued running until I closed it manually (after 16 minutes since starting).

Command used to build the dictionary:

npm run build

Dictionary file companies.txt contents are basically almost empty:

Apple

Just wondering if this is an expected behaviour from cspell-tools or have I found a bug?

The text was updated successfully, but these errors were encountered:

Jason3S · 2017-07-30T21:47:24Z

Sorry for the frustration.
There might be a bug in the file reader if the file only contains 1 line.

Try adding some more words and maybe some empty lines at the end.

It should only take a second or so to run for small files.

d2s · 2017-08-01T11:18:27Z

Tried again with a longer list (with 100+ items), but the same result.
cspell-tools took over 8 minutes, and still had not outputted anything when I decided to quit it.

Using the latest Node.js version v8.2.1 (installed with nvm).

Kind of difficult to debug. 😞

Jason3S · 2017-08-01T11:31:42Z

All these things should work.

What happens if you go to the php or python directories and do a npm run build?

This is what I get:

~/projects/cspell-dicts/php (master)$ npm run build

> [email protected] build /Users/jason/projects/cspell-dicts/php
> cspell-tools compile "php.txt" -o .

Compile:
 output: .
 compress: true
 files:
  php.txt 

Process "php.txt" to "php.txt.gz"
Complete.

It only takes a second or so.

Jason3S · 2017-08-01T11:35:33Z

If that doesn't work, try doing a pull request with your companies, then I can try to make it work.

By the way, cspell already has a companies dictionary. It would be wonderful if you include those as well.

d2s · 2017-08-01T11:43:23Z

That was the original idea: to move those to a separate package (so that it could be installed more easily).

About the php directory, this happened (on the first try).

➜  cspell-dicts git:(d2s-dictionary-companies) ✗ cd php 
➜  php git:(d2s-dictionary-companies) ✗ npm run build

> [email protected] build /home/d2s/git/GitHub/d2s/cspell-dicts/php
> cspell-tools compile "php.txt" -o .

sh: 1: cspell-tools: not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] build: `cspell-tools compile "php.txt" -o .`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/d2s/.npm/_logs/2017-08-01T11_34_50_390Z-debug.log

Full error log visible at a (temporary) Gist snippet.

After that, I realized that npm should be run to install the dependencies.

➜  php git:(d2s-dictionary-companies) ✗ npm install
npm notice created a lockfile as package-lock.json. You should commit this file.
added 40 packages in 1.988s

After that, I started the build process, but it was never-ending.
(After 2 minutes, decided to quit it.)

➜  php git:(d2s-dictionary-companies) ✗ npm run build   

> [email protected] build /home/d2s/git/GitHub/d2s/cspell-dicts/php
> cspell-tools compile "php.txt" -o .

^C%

Jason3S · 2017-08-01T11:45:25Z

Did you do "npm install" in the php directory?

d2s · 2017-08-01T11:46:31Z

Yeah, I did.

I also tried the same steps in the python directory, but even that failed.

Jason3S · 2017-08-01T11:46:49Z

Oops I needed to read further.

Jason3S · 2017-08-01T11:58:33Z

There is something else going on there. I just installed this from scratch on another machine and it worked just fine.

What happen if you do the following in php:

node_modules/.bin/cspell-tools -V

and

cspell-tools -V

d2s · 2017-08-01T12:21:17Z

The first command failed to work, as it did not return any output.
Had to quit it manually.

➜  php git:(d2s-dictionary-companies) ✗ node_modules/.bin/cspell-tools -V
^C

The second command was missing from the system.

➜  php git:(d2s-dictionary-companies) ✗ cspell-tools -V
zsh: command not found: cspell-tools

Jason3S · 2017-08-01T13:22:57Z

Would you attach the package-lock.json file? I would like to see what got downloaded. Something isn't right.

The first command should have run just fine.

d2s · 2017-08-01T14:15:02Z

Here we go:
https://gist.github.com/d2s/f807efe7b2b0189821b4d492cb41443d

Jason3S · 2017-08-01T14:23:14Z

Hmm, looks the same as mine.

Can you run:

/usr/bin/env node -v

and

node -v

d2s · 2017-08-01T14:59:43Z

➜  cspell-dicts git:(master) ✗ /usr/bin/env node -v
v8.2.1

➜  cspell-dicts git:(master) ✗ node -v
v8.2.1

➜  cspell-dicts git:(master) ✗ whereis node
node: /home/d2s/.nvm/versions/node/v8.2.1/bin/node

Jason3S · 2017-08-02T06:52:29Z

Hmm, I am still not sure what the cause might be.

What happens when you do this:

 node node_modules/.bin/cspell-tools

This is what I get:


  Usage: cspell-tools [options] [command]


  Options:

    -V, --version  output the version number
    -h, --help     output usage information


  Commands:

    compile [options] <src...>       compile words lists into simple dictionary files.
    compile-trie [options] <src...>  Compile words lists or Hunspell dictionary into trie files used by cspell.

d2s · 2017-08-02T11:01:43Z

First made sure that I have the latest Git commits,
and that npm install was run properly on the php directory.

➜  cspell-dicts git:(master) ✗ ./pull-changes-from-upstream.sh
...
➜  cspell-dicts git:(master) ✗ cd php

➜  php git:(master) ✗ ll 
total 264K
-rw-rw-r--  1 d2s d2s 1,3K heinä 29 23:23 cspell-ext.json
-rw-rw-r--  1 d2s d2s   63 heinä 29 23:23 install.js
-rw-rw-r--  1 d2s d2s 1,1K heinä 29 23:23 LICENSE
-rwxrwxr-x  1 d2s d2s   84 heinä 29 23:23 link.js
drwxrwxr-x 43 d2s d2s 4,0K elo    1 14:35 node_modules
-rw-rw-r--  1 d2s d2s  911 heinä 29 23:23 package.json
-rw-rw-r--  1 d2s d2s  11K elo    1 14:35 package-lock.json
-rw-rw-r--  1 d2s d2s 199K heinä 29 23:23 php.txt
-rw-rw-r--  1 d2s d2s  12K heinä 29 23:23 php.txt.gz
-rw-rw-r--  1 d2s d2s  725 heinä 29 23:23 README.md
-rw-rw-r--  1 d2s d2s   64 heinä 29 23:23 uninstall.js
-rwxrwxr-x  1 d2s d2s   85 heinä 29 23:23 unlink.js
-rw-rw-r--  1 d2s d2s 1,1K heinä 29 23:23 util.js

➜  php git:(master) ✗ npm install
up to date in 0.297s

Now that seems to work:

➜  php git:(master) ✗ node node_modules/.bin/cspell-tools

  Usage: cspell-tools [options] [command]


  Options:

    -V, --version  output the version number
    -h, --help     output usage information


  Commands:

    compile [options] <src...>       compile words lists into simple dictionary files.
    compile-trie [options] <src...>  Compile words lists or Hunspell dictionary into trie files used by cspell.

Also tried to run the previously mentioned command, and the output looks to be working.

➜  php git:(master) ✗ node node_modules/.bin/cspell-tools -V
1.4.1

➜  php git:(master) ✗ ls -al node_modules/.bin/cspell-tools
lrwxrwxrwx 1 d2s d2s 27 elo    1 14:35 node_modules/.bin/cspell-tools -> ../cspell-tools/dist/app.js

➜  php git:(master) ✗

Not really sure what would be the reason.
Might have something to do with environmental variables?

Edit:
Now this is confusing. Tried to run this, and it worked:

➜  php git:(master) ✗ node node_modules/.bin/cspell-tools compile \"php.txt\" -o .
Compile:
 output: .
 compress: true
 files:
  "php.txt" 


➜  php git:(master) ✗

Apparently the issue has to do with the way how the tool is run.

d2s · 2017-08-02T11:09:31Z

Also tried the companies directory, and it also seems to work, if run with node before the filename.

➜  companies git:(d2s-dictionary-companies) ✗ node node_modules/.bin/cspell-tools compile \"companies.txt\" -o .
Compile:
 output: .
 compress: true
 files:
  "companies.txt"

For the comparison, tried to run same without the added node command in the beginning.

➜  companies git:(d2s-dictionary-companies) ✗ node_modules/.bin/cspell-tools compile \"companies.txt\" -o . 
^C
➜  companies git:(d2s-dictionary-companies) ✗ ./node_modules/.bin/cspell-tools compile \"companies.txt\" -o .
^C

So the tool itself works properly, but the node scripts environment detection is somehow broken.

Jason3S · 2017-08-02T11:24:32Z

I think the issue might be with the shell command in the file. First line of node_modules/.bin/cspell-tools:

#!/usr/bin/env node --max_old_space_size=4096

This tells node to run cspell-tools with 4GB max memory. I am not sure why that causes issues with your configuration.

Doing:

node node_modules/.bin/cspell-tools

skips the step of setting the memory limit.

d2s · 2017-08-02T11:37:55Z

Apparently it is possible to add the parameter directly to the package.json file, according to this comment: npm/npm#12238 (comment)

I modified the package.json with this line:

  "scripts": {
    "build": "node --max_old_space_size=4096 node_modules/.bin/cspell-tools compile \"php.txt\" -o .",

Now it seems to build properly, and even the memory limit should be there.
Might not be the most optimal way to do things, but... 😕

➜  php git:(master) ✗ npm run build

> [email protected] build /home/d2s/git/GitHub/d2s/cspell-dicts/php
> node --max_old_space_size=4096 node_modules/.bin/cspell-tools compile "php.txt" -o .

Compile:
 output: .
 compress: true
 files:
  php.txt 


Process "php.txt" to "php.txt.gz"
Complete.

Jason3S · 2017-08-02T12:39:51Z

the --max_old_space_size setting is not necessary for small dictionaries. It was only needed for language dictionaries like English, Russian, etc.

Jason3S · 2017-08-02T12:48:27Z

I think it is the same issue as this one:

Looks like its ionic-team/ionic-app-scripts@b4c287a
In linux we can only use max. 1 arg here, the '=' sign in it makes it just hang here instead of returning an error.
Removing the param from the shebang makes it work for me.

Jason3S · 2017-08-02T12:49:28Z

I'll fix cspell-tools and remove the memory setter.

Jason3S · 2017-08-02T12:52:38Z

Can you check something for me?

edit node_modules/.bin/cspell-tools and change the first line to:

#!/usr/bin/env node "--max_old_space_size=4096"

I would like to know if that works on your setup.

d2s · 2017-08-02T13:11:22Z

I tried tried adding that to the cspell-tools file, but the quoted parameters fail work on my system.

➜  php git:(master) ✗ node_modules/.bin/cspell-tools compile \"php.txt\" -o . 
^C
➜  php git:(master) ✗ node_modules/.bin/cspell-tools compile \"php.txt\" -o .
^C

Jason3S · 2017-08-02T14:55:54Z

This seems to be a linux vs bsd issue.

Jason3S · 2017-08-02T15:13:17Z

I updated cspell-tools.
Can you

cd php
rm -rf node_modules
rm package-lock.json
npm install
node_modules/.bin/cspell-tools

This should work.

d2s · 2017-08-02T21:42:01Z

Works now, but I had to manually update the devDependencies of package.json files.

  "devDependencies": {
    "cspell-tools": "^1.4.2"
  },

Will create a Pull Request about those in a moment.

d2s · 2017-08-07T14:30:32Z

This should be fixed now, so closing the issue.

Jason3S · 2017-08-07T14:31:58Z

Thank you for your help on this one.

d2s mentioned this issue Aug 2, 2017

fix(package.json) update devDependencies #5

Merged

d2s closed this as completed Aug 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Long running build process for a very small file #4

Long running build process for a very small file #4

d2s commented Jul 30, 2017

Jason3S commented Jul 30, 2017 •

edited

Loading

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017 via email •

edited

Loading

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017 via email •

edited

Loading

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 2, 2017 •

edited

Loading

d2s commented Aug 2, 2017 •

edited

Loading

d2s commented Aug 2, 2017

Jason3S commented Aug 2, 2017

d2s commented Aug 2, 2017 •

edited

Loading

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017

d2s commented Aug 2, 2017 •

edited

Loading

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017 •

edited

Loading

d2s commented Aug 2, 2017

d2s commented Aug 7, 2017

Jason3S commented Aug 7, 2017

Long running build process for a very small file #4

Long running build process for a very small file #4

Comments

d2s commented Jul 30, 2017

Jason3S commented Jul 30, 2017 • edited Loading

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017 via email • edited Loading

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017 via email • edited Loading

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 1, 2017

d2s commented Aug 1, 2017

Jason3S commented Aug 2, 2017 • edited Loading

d2s commented Aug 2, 2017 • edited Loading

d2s commented Aug 2, 2017

Jason3S commented Aug 2, 2017

d2s commented Aug 2, 2017 • edited Loading

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017

d2s commented Aug 2, 2017 • edited Loading

Jason3S commented Aug 2, 2017

Jason3S commented Aug 2, 2017 • edited Loading

d2s commented Aug 2, 2017

d2s commented Aug 7, 2017

Jason3S commented Aug 7, 2017

Jason3S commented Jul 30, 2017 •

edited

Loading

Jason3S commented Aug 1, 2017 via email •

edited

Loading

Jason3S commented Aug 1, 2017 via email •

edited

Loading

Jason3S commented Aug 2, 2017 •

edited

Loading

d2s commented Aug 2, 2017 •

edited

Loading

d2s commented Aug 2, 2017 •

edited

Loading

d2s commented Aug 2, 2017 •

edited

Loading

Jason3S commented Aug 2, 2017 •

edited

Loading