add scoped packages

[rlidwka]

Would be in the next npm release, so we'll need to support that.

npm/npm#5239

[jasonrm]

Happened to be messing with this yesterday as well, although I couldn't find any really clean way to make it happen. I eventually settled on a string replace in a few files so that it would create the right files/folders and return the right url to the tarball.

[themasch]

👍 i'd love to see this feature in sinopia. this could save a lot of trouble

[mreinstein]

+1 I would think this is the primary use case of sinopia now that scoped packages are implemented.

[danieltdt]

The only thing to do is accept @org/ format on the package name?

[rusintez]

@jasonrm 10x

[themasch]

@danieltdt that should do it, I think. yes.

[MalucoMarinero]

+1 @rlidwka, I would be willing to put some work into a pull request on this unless it's being actively worked on already?

Had a quick look, and there's more than just accepting the package name, as it needs to fit into the flat file structure as well, want to chat about implementation at all?

[rlidwka]

I would be willing to put some work into a pull request on this unless it's being actively worked on already?

It's already there, in this commit I believe: a425c5e

I just published it as [email protected] with a lot of other changes, so be sure to try that out.

Had a quick look, and there's more than just accepting the package name, as it needs to fit into the flat file structure as well

I prefer to keep it structured like node_modules, so it won't be flat anymore. I guess search will stop working again. :(

[MalucoMarinero]

Noice, much obliged publishing that, works a treat.

[drewwells]

Installing scoped packages from 1.0.0-alpha does not work.
@knksmith57

npm i @rmn/package

generates url

http://npm.retailmenot.demo/@rmn/package/-/@rmn/package-1.0.0.tgz

but sinopia only works with the following url

http://npm.retailmenot.demo/@rmn/package/-/package-1.0.0.tgz

This will need aliasing to work properly with npm v2.1.8

[rlidwka]

oh crap, I guess it's called "alpha" for a reason. Reopening...

[pluma]

👍 would be awesome to have this working 😸

[knksmith57]

@drewwells heads up that your working URL is wrong (at least on my local box).

http://npm.retailmenot.demo/@rmn/package/-/package-1.0.0.tgz

still fails; however:

http://npm.retailmenot.demo/@rmn%2fpackage/-/package-1.0.0.tgz

works. It looks like the formula here is:

http://{base-url}/{scoped package name with "/" --> %2f}/-/{versioned tarball}.tgz

Still trying to work out a fix.

[knksmith57]

@drewwells 7a32cf2 fixes the issues we are having. It's not the most elegant solution however, after digging through the source for a few hours, I couldn't figure out a clean-er integration point.

I suspect @rlidwka would prefer a less hacked solution before he merges into master and we may want to rebuild our container based on my feature branch in the meantime.

[drewwells]

Can we build from this commit and get a working sinopia for our use case?
On Sun, Nov 30, 2014 at 4:14 PM Kyle Smith [email protected] wrote:

@drewwells https://github.com/drewwells 7a32cf2
7a32cf2
fixes the issues we are having. It's not the most elegant solution however,
after digging through the source for a few hours, I couldn't figure out a
clean-er integration point.

I suspect @rlidwka https://github.com/rlidwka would prefer a less
hacked solution before he merges into master and we may want to rebuild our
container based on my feature branch in the meantime.

—
Reply to this email directly or view it on GitHub
#104 (comment).

[knksmith57]

@drewwells definitely. just issued a pull request to your docker-sinopia repo that will do just that. check out drewwells/docker-sinopia#2 for the change.

[markstickley]

Just adding a +1; I'd really appreciate support for scoped packages! Sounds like you're almost there, but if there's anything I can do I'm happy to help.

[drewwells]

We're publishing and downloading scoped packages thanks to Kyle's fixes.

On Wed Dec 03 2014 at 8:28:23 AM Mark Stickley [email protected]
wrote:

Just adding a +1; I'd really appreciate support for scoped packages!
Sounds like you're almost there, but if there's anything I can do I'm happy
to help.

—
Reply to this email directly or view it on GitHub
#104 (comment).

[rlidwka]

Fixed in 06bb301 and published as [email protected].

For anyone, who is interested: sinopia is actually overwriting this url with its own, and that code didn't work correctly with scoped packages. Now it does. Also, cleaned up a bit of over-engineering there...

[rusintez]

👍

[markstickley]

Great stuff, thanks!

On 5 Dec 2014, at 07:31, vlspopov [email protected] wrote:

—
Reply to this email directly or view it on GitHub.

[drewwells]

💥 👍

[markstickley]

I just tried publishing a scoped package but got an error. It's quite likely that I'm doing something wrong but I'd appreciate some advice, please!

$ npm publish
npm ERR! publish Failed PUT 403
npm ERR! Darwin 13.4.0
npm ERR! argv "node" "/usr/local/bin/npm" "publish"
npm ERR! node v0.10.33
npm ERR! npm  v2.1.11
npm ERR! code E403

npm ERR! user marks not allowed to publish it : @bb%2fnpmscopedmodule
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <http://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/marks/workspace/jspm_poc/modules/npmscopedmodule/npm-debug.log

Here's the Sinopia output:

 http  <-- 403, user: marks, req: 'PUT /@bb%2fnpmscopedmodule', error: user marks not allowed to publish it

And the NPM debug log:

0 info it worked if it ends with ok
1 verbose cli [ 'node', '/usr/local/bin/npm', 'publish' ]
2 info using [email protected]
3 info using [email protected]
4 verbose publish [ '.' ]
5 silly cache add args [ '.', null ]
6 verbose cache add spec .
7 silly cache add parsed spec { raw: '.',
7 silly cache add   scope: null,
7 silly cache add   name: null,
7 silly cache add   rawSpec: '.',
7 silly cache add   spec: '/Users/marks/workspace/jspm_poc/modules/npmscopedmodule',
7 silly cache add   type: 'directory' }
8 verbose tar pack [ '/Users/marks/.npm/@scope/npmscopedmodule/1.0.0/package.tgz',
8 verbose tar pack   '/Users/marks/workspace/jspm_poc/modules/npmscopedmodule' ]
9 verbose tarball /Users/marks/.npm/@scope/npmscopedmodule/1.0.0/package.tgz
10 verbose folder /Users/marks/workspace/jspm_poc/modules/npmscopedmodule
11 info prepublish @scope/[email protected]
12 verbose addLocalTarball adding from inside cache /Users/marks/.npm/@scope/npmscopedmodule/1.0.0/package.tgz
13 silly cache afterAdd @scope/[email protected]
14 verbose afterAdd /Users/marks/.npm/@scope/npmscopedmodule/1.0.0/package/package.json not in flight; writing
15 verbose afterAdd /Users/marks/.npm/@scope/npmscopedmodule/1.0.0/package/package.json written
16 silly publish { name: '@scope/npmscopedmodule',
16 silly publish   version: '1.0.0',
16 silly publish   description: '',
16 silly publish   keywords: [ 'Foo', 'bar' ],
16 silly publish   license: 'ISC',
16 silly publish   author: { name: 'Mark Stickley', email: '[email protected]' },
16 silly publish   main: 'npmscopedmodule',
16 silly publish   publishConfig: { registry: 'http://npm.companyname.local' },
16 silly publish   scripts: { test: 'echo "Error: no test specified" && exit 1' },
16 silly publish   readme: '...',
16 silly publish   readmeFilename: 'README.md',
16 silly publish   _id: '@scope/[email protected]',
16 silly publish   _shasum: '219e081e4e26d0bc4e78a5e8875e3a3e3ecd6121',
16 silly publish   _from: '.' }
17 silly mapToRegistry name @scope/npmscopedmodule
18 silly mapToRegistry scope (from package name) @scope
19 verbose mapToRegistry no registry URL found in name for scope @scope
20 silly mapToRegistry using default registry
21 silly mapToRegistry registry http://npm.companyname.local
22 silly mapToRegistry uri http://npm.companyname.local/@scope%2fnpmscopedmodule
23 verbose publish registryBase http://npm.companyname.local
24 silly publish uploading /Users/marks/.npm/@scope/npmscopedmodule/1.0.0/package.tgz
25 verbose request uri http://npm.companyname.local/@scope%2fnpmscopedmodule
26 verbose request sending authorization for write operation
27 info attempt registry request try #1 at 15:25:55
28 verbose request using bearer token for auth
29 verbose request id 32605b7e10f63fac
30 http request PUT http://npm.companyname.local/@scope%2fnpmscopedmodule
31 http 403 http://npm.companyname.local/@scope%2fnpmscopedmodule
32 verbose headers { 'x-powered-by': 'Sinopia/1.0.0-alpha.3',
32 verbose headers   'content-type': 'application/json; charset=utf-8',
32 verbose headers   'content-length': '54',
32 verbose headers   vary: 'Accept-Encoding',
32 verbose headers   'x-status-cat': 'http://flic.kr/p/aV6jFK',
32 verbose headers   date: 'Fri, 05 Dec 2014 15:25:31 GMT',
32 verbose headers   connection: 'keep-alive' }
33 verbose request invalidating /Users/marks/.npm/npm.companyname.local/_40bb_252fnpmscopedmodule on PUT
34 error publish Failed PUT 403
35 verbose stack Error: user marks not allowed to publish it : @scope%2fnpmscopedmodule
35 verbose stack     at CachingRegistryClient.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:242:14)
35 verbose stack     at Request._callback (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:172:14)
35 verbose stack     at Request.self.callback (/usr/local/lib/node_modules/npm/node_modules/request/request.js:372:22)
35 verbose stack     at Request.emit (events.js:98:17)
35 verbose stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1317:14)
35 verbose stack     at Request.emit (events.js:117:20)
35 verbose stack     at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1265:12)
35 verbose stack     at IncomingMessage.emit (events.js:117:20)
35 verbose stack     at _stream_readable.js:943:16
35 verbose stack     at process._tickCallback (node.js:419:13)
36 verbose statusCode 403
37 verbose pkgid @scope%2fnpmscopedmodule
38 verbose cwd /Users/marks/workspace/jspm_poc/modules/npmscopedmodule
39 error Darwin 13.4.0
40 error argv "node" "/usr/local/bin/npm" "publish"
41 error node v0.10.33
42 error npm  v2.1.11
43 error code E403
44 error user marks not allowed to publish it : @scope%2fnpmscopedmodule
45 error If you need help, you may report this error at:
45 error     <http://github.com/npm/npm/issues>
46 verbose exit [ 1, true ]

[rlidwka]

@markstickley , what's your config file for sinopia?

The issue could be that * will not match @scope/npmscopedmodule.

[markstickley]

@rlidwka Yep, that was it: I needed to add

  '@*/*':
    # scoped packages
    allow_access: $all
    allow_publish: $authenticated

to my config as I had upgraded from a previous version without support for scoped packages. Thanks :)

[necccc]

If I'm not mistaken, this only works, if we put the @scope in the module name.

However there is a part in the npm-config for the scope value.

Furthermore, as described here, you can associate a scope with a registry at login, e.g.

npm login --registry=http://localhost:4873/ --scope=@myco

In my ~/.npmrc I see

@myco:registry=http://localhost:4873/

So in my head it should work like this:

• login/adduser to my private sinopia instance, adding --scope
• create my module package:
• name is just the name, without @myco

"name": "mymodule",

• publishConfig is something like

  "publishConfig": { 
    "scope": "@myco',
    "registry": "http://localhost:4873" 
  },

• run npm publish
• as the scope is set both in the package.json, and in my .npmrc, this is expected to be published as @myco/mymodule

Am I missing something?

[rlidwka]

name is just the name, without @myco

Name should include scope.

[necccc]

Then what's the difference between this, and naming all my modules something like myco-mymodule on the private registry?

[rlidwka]

The difference is that you could set up @myco/mymodule installed from private registry and public modules from the npmjs registry at the same time without changing npm settings.

Yes, it doesn't give much benefits with sinopia, because it can forward requests to npmjs anyway. But it has its uses.

[necccc]

I get it, on the usage side, it's clearly a benefit.

Although i hoped that the scope metadata would have a more serious role during publish.

[valmiRe]

Hi, does somebody has this working behind a nginx server with a custom path ?
I got a 404 error when I try to publish a scoped package with sinopia behind nginx, and it works if I publish directly to sinopia.

The problem is that nginx seems to "decode" the URI before passing it to the backend
I can see in the sinopia web log that the URI is not the same if I publish through nginx or directly

nginx :
 http  <-- 404, user: admin, req: 'PUT /@myco/mymodule', bytes: 4918/30
direct :
 http  <-- 201, user: admin, req: 'PUT /@myco%2fmymodule', bytes: 4829/34

I tried 2 different nginx configs, with the same result:

location /sinopia/ {
    proxy_pass http://localhost:4873;
    rewrite /sinopia/(.*) /$1 break;
}
location ~ /sinopia/?(?<url>.*) {
    proxy_pass http://127.0.0.1:4873/$url;
}

[drewwells]

We host ours via an nginx proxy. Perhaps try our docker container and see if that works better for you: https://registry.hub.docker.com/u/drewwells/sinopia/

[RnbWd]

We also host via an nginx proxy. https://registry.hub.docker.com/u/rnbwd/sinopia/ . I use a slightly different Dockerfile but also included some instructions on how to set it up with an nginx-proxy - https://github.com/jwilder/nginx-proxy

[valmiRe]

Thanks for the answers but I am able to run sinopia behind nginx, that's not the problem and I don't need docker for that.
My problem is to publish scoped package with sinopia behind nginx with a custom path
Like explained in this link, nginx re-normalize uri when you try to rewrite it. I've tried the 'map directive' solution provided in the previous link, but with no success for now

[rlidwka]

@valmiRe , this is how I managed to solve this:

map $request_uri $uri_without_prefix {
  ~^/sinopia/(?<rest_of_uri>.*) $rest_of_uri;
}

server {
  listen 80 default_server;
  location /sinopia/ {
    proxy_pass http://127.0.0.1:4873/$uri_without_prefix;
  }
}

I don't know what side effects I'll give, but it seems to work. The fact that you can't do it with rewrite sucks though.

[valmiRe]

thanks @rlidwka , this works fine, I had to add a line in the map directive, otherwise the web interface was broken (my browser try to get url like "/sinopia//-/static/main.css")

map $request_uri $uri_without_prefix {
      ~^/sinopia/(?<rest_of_uri>.*) $rest_of_uri;
      ~^/sinopia//(?<rest_of_uri>.*) $rest_of_uri;
}

[rlidwka]

@valmiRe and others who had that nginx issue: we added a workaround (#275, fde2321). I hope it solves this.

[RnbWd]

Sorry I should have mentioned something months ago (maybe I did and don't remember). But I forked the repo and added one line of code that was relevant to the docker nginx-proxy package:

  // for nginx proxy 
  if (process.env.VIRTUAL_HOST) app.enable('trust proxy')

in the index.js file - now docker just uses this repo here, which probably broke everyone's builds who were using my docker container with nginx. There were always solutions in the config file to handle some of the nginx stuff - but I apologize if I someone happened to be using my fork and the build broke.