Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Online config URL parsing #21

Merged
merged 9 commits into from
Jun 26, 2021
Merged

Online config URL parsing #21

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c100e22
Online config (SIP008) support
alalamav Oct 15, 2020
9538f43
dynamic config url string
alalamav Mar 22, 2021
f7e50cc
merge branch 'master' into alalama-ssconf
alalamav Mar 22, 2021
1fb2764
s/dynamic/online config; readonly properties
alalamav Apr 1, 2021
9a81716
use regex to replace protocol; don't encode ipv6 ulr
alalamav Apr 6, 2021
c829fc3
parse extras with URLSearchParams
alalamav Apr 6, 2021
b3f2f19
address review comments
alalamav Apr 12, 2021
c325f47
fix protocol validation; ConfigFetchParams s/url/location
alalamav Apr 15, 2021
fb81d34
decode tag once; add test for uri encoded tag
alalamav Apr 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions build/shadowsocks_config.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,10 @@ export declare const SIP002_URI: {
parse: (uri: string) => Config;
stringify: (config: Config) => string;
};
export interface OnlineConfig {
readonly url: string;
readonly extra: {[key: string]: string;};
}
export declare const SIP008_URI: {
PROTOCOL: string; validateProtocol: (uri: string) => void; parse: (uri: string) => OnlineConfig;
};
58 changes: 53 additions & 5 deletions build/shadowsocks_config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ var __extends = (this && this.__extends) || (function () {
};
})();
Object.defineProperty(exports, "__esModule", { value: true });
var js_base64_1 = require("js-base64");
var ipaddr = require("ipaddr.js");
var js_base64_1 = require('js-base64');
var punycode = require("punycode");
var url_1 = require('url');
// Custom error base class
var ShadowsocksConfigError = /** @class */ (function (_super) {
__extends(ShadowsocksConfigError, _super);
Expand Down Expand Up @@ -81,8 +82,8 @@ var Host = /** @class */ (function (_super) {
}
if (ipaddr.isValid(host)) {
var ip = ipaddr.parse(host);
_this.isIPv4 = ip.kind() == "ipv4";
_this.isIPv6 = ip.kind() == "ipv6";
_this.isIPv4 = ip.kind() === 'ipv4';
_this.isIPv6 = ip.kind() === 'ipv6';
// Previous versions of outline-ShadowsocksConfig only accept
// IPv6 in normalized (expanded) form, so we normalize the
// input here to ensure that access keys remain compatible.
Expand Down Expand Up @@ -298,8 +299,8 @@ exports.LEGACY_BASE64_URI = {
var data = method.data + ":" + password.data + "@" + host.data + ":" + port.data;
var b64EncodedData = js_base64_1.Base64.encode(data);
// Remove "=" padding
while (b64EncodedData.slice(-1) == "=") {
b64EncodedData = b64EncodedData.slice(0, -1);
while (b64EncodedData.slice(-1) === '=') {
b64EncodedData = b64EncodedData.slice(0, -1);
}
return "ss://" + b64EncodedData + hash;
},
Expand Down Expand Up @@ -363,3 +364,50 @@ exports.SIP002_URI = {
return "ss://" + userInfo + "@" + uriHost + ":" + port.data + "/" + queryString + hash;
},
};
// Ref: https://github.com/shadowsocks/shadowsocks-org/issues/89
exports.SIP008_URI = {
PROTOCOL: 'ssconf',
validateProtocol: function (uri) {
if (!uri || !uri.startsWith(exports.SIP008_URI.PROTOCOL)) {
throw new InvalidUri("URI must start with \"" + exports.SIP008_URI.PROTOCOL + "\"");
}
},
parse: function (uri) {
exports.SIP008_URI.validateProtocol(uri);
// URL parser for expedience, replacing the protocol "ssconf" with "https" to ensure correct
// results, otherwise browsers like Safari fail to parse it.
var inputForUrlParser = uri.replace(new RegExp('^' + exports.SIP008_URI.PROTOCOL), 'https');
// The built-in URL parser throws as desired when given URIs with invalid syntax.
var urlParserResult = new URL(inputForUrlParser);
// Use ValidatedConfigFields subclasses (Host, Port, Tag) to throw on validation failure.
var uriFormattedHost = urlParserResult.hostname;
var host;
try {
host = new Host(uriFormattedHost);
} catch (_) {
// Could be IPv6 host formatted with surrounding brackets, so try stripping first and last
// characters. If this throws, give up and let the exception propagate.
host = new Host(uriFormattedHost.substring(1, uriFormattedHost.length - 1));
}
// The default URL parser fails to recognize the default HTTPs port (443).
var port = new Port(urlParserResult.port || '443');
// Parse extra parameters from the tag, which has the format `#key0=val0;key1=val1...[;]`
var extra = {};
var tag = new Tag(decodeURIComponent(urlParserResult.hash.substring(1)));
// Convert tag to search parameters to leverage URLSearchParams parsing.
var params = new url_1.URLSearchParams(tag.data.replace(';', '&'));
params.forEach(function(value, key) {
if (!key) {
return;
}
extra[key] = value;
});
var config = {
// Build the access URL with the parsed parameters. Exclude the query string, as the spec
// recommends against it.
url: "https://" + uriFormattedHost + ":" + port.data + urlParserResult.pathname,
extra: extra
};
return config;
},
};
79 changes: 62 additions & 17 deletions src/shadowsocks_config.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.

import {
Host, Port, Method, Password, Tag, Config, makeConfig,
SHADOWSOCKS_URI, SIP002_URI, LEGACY_BASE64_URI, InvalidConfigField, InvalidUri,
} from './shadowsocks_config';
import {Base64} from 'js-base64';

import {Config, Host, InvalidConfigField, InvalidUri, LEGACY_BASE64_URI, makeConfig, Method, Password, Port, SHADOWSOCKS_URI, SIP002_URI, SIP008_URI, Tag,} from './shadowsocks_config';

describe('shadowsocks_config', () => {
describe('Config API', () => {
it('has expected shape', () => {
Expand Down Expand Up @@ -62,13 +60,12 @@ describe('shadowsocks_config', () => {
it('normalizes IPv6 address hosts', () => {
const testCases = [
// Canonical form
['::1', '0:0:0:0:0:0:0:1'],
['2001:db8::', '2001:db8:0:0:0:0:0:0'],
['::1', '0:0:0:0:0:0:0:1'], ['2001:db8::', '2001:db8:0:0:0:0:0:0'],
// Expanded form
['1:02:003:0004:005:06:7:08', '1:2:3:4:5:6:7:8'],
// IPv4-mapped form
['::ffff:192.0.2.128', '0:0:0:0:0:ffff:c000:280']
]
];
for (const [input, expanded] of testCases) {
const host = new Host(input);
expect(host.data).toEqual(expanded);
Expand Down Expand Up @@ -204,7 +201,6 @@ describe('shadowsocks_config', () => {
});

describe('URI serializer', () => {

it('can serialize a SIP002 URI', () => {
const config = makeConfig({
host: '192.168.100.1',
Expand All @@ -213,8 +209,8 @@ describe('shadowsocks_config', () => {
password: 'test',
tag: 'Foo Bar',
});
expect(SIP002_URI.stringify(config)).toEqual(
'ss://[email protected]:8888/#Foo%20Bar');
expect(SIP002_URI.stringify(config))
.toEqual('ss://[email protected]:8888/#Foo%20Bar');
});

it('can serialize a SIP002 URI with a non-latin password', () => {
Expand All @@ -225,8 +221,9 @@ describe('shadowsocks_config', () => {
password: '小洞不补大洞吃苦',
tag: 'Foo Bar',
});
expect(SIP002_URI.stringify(config)).toEqual(
'ss://[email protected]:8888/#Foo%20Bar');
expect(SIP002_URI.stringify(config))
.toEqual(
'ss://[email protected]:8888/#Foo%20Bar');
});

it('can serialize a SIP002 URI with IPv6 host', () => {
Expand All @@ -237,8 +234,9 @@ describe('shadowsocks_config', () => {
password: 'test',
tag: 'Foo Bar',
});
expect(SIP002_URI.stringify(config)).toEqual(
'ss://YWVzLTEyOC1nY206dGVzdA@[2001:0:ce49:7601:e866:efff:62c3:fffe]:8888/#Foo%20Bar');
expect(SIP002_URI.stringify(config))
.toEqual(
'ss://YWVzLTEyOC1nY206dGVzdA@[2001:0:ce49:7601:e866:efff:62c3:fffe]:8888/#Foo%20Bar');
});

it('can serialize a legacy base64 URI', () => {
Expand All @@ -261,8 +259,9 @@ describe('shadowsocks_config', () => {
password: '小洞不补大洞吃苦',
tag: 'Foo Bar',
});
expect(LEGACY_BASE64_URI.stringify(config)).toEqual(
'ss://YmYtY2ZiOuWwj+a0nuS4jeihpeWkp+a0nuWQg+iLpkAxOTIuMTY4LjEwMC4xOjg4ODg#Foo%20Bar');
expect(LEGACY_BASE64_URI.stringify(config))
.toEqual(
'ss://YmYtY2ZiOuWwj+a0nuS4jeihpeWkp+a0nuWQg+iLpkAxOTIuMTY4LjEwMC4xOjg4ODg#Foo%20Bar');
});
});

Expand Down Expand Up @@ -390,7 +389,8 @@ describe('shadowsocks_config', () => {
});

it('can parse a valid legacy base64 URI with a non-latin password', () => {
const input = 'ss://YmYtY2ZiOuWwj+a0nuS4jeihpeWkp+a0nuWQg+iLpkAxOTIuMTY4LjEwMC4xOjg4ODg#Foo%20Bar';
const input =
'ss://YmYtY2ZiOuWwj+a0nuS4jeihpeWkp+a0nuWQg+iLpkAxOTIuMTY4LjEwMC4xOjg4ODg#Foo%20Bar';
const config = SHADOWSOCKS_URI.parse(input);
expect(config.method.data).toEqual('bf-cfb');
expect(config.password.data).toEqual('小洞不补大洞吃苦');
Expand All @@ -408,4 +408,49 @@ describe('shadowsocks_config', () => {
expect(() => SHADOWSOCKS_URI.parse('ss://not-base64')).toThrowError(InvalidUri);
});
});

describe('SIP008', () => {
it('can parse a valid ssconf URI with domain name and extras', () => {
const input = encodeURI(
'ssconf://my.domain.com/secret/long/path#certFp=AA:BB:CC:DD:EE:FF;sni=https://sni.other.com');
const onlineConfig = SIP008_URI.parse(input);
expect(new URL(onlineConfig.url)).toEqual(new URL('https://my.domain.com/secret/long/path'));
expect(onlineConfig.extra.certFp).toEqual('AA:BB:CC:DD:EE:FF');
expect(onlineConfig.extra.sni).toEqual('https://sni.other.com');
});

it('can parse a valid ssconf URI with domain name and custom port', () => {
const input =
encodeURI('ssconf://my.domain.com:9090/secret/long/path#certFp=AA:BB:CC:DD:EE:FF;');
const onlineConfig = SIP008_URI.parse(input);
expect(new URL(onlineConfig.url))
.toEqual(new URL('https://my.domain.com:9090/secret/long/path'));
expect(onlineConfig.extra.certFp).toEqual('AA:BB:CC:DD:EE:FF');
});

it('can parse a valid ssconf URI with hostname and no path', () => {
const input = encodeURI('ssconf://my.domain.com');
const onlineConfig = SIP008_URI.parse(input);
expect(new URL(onlineConfig.url)).toEqual(new URL('https://my.domain.com'));
expect(onlineConfig.extra.certFp).toBeUndefined();
});

it('can parse a valid ssconf URI with IPv4 address', () => {
const input =
encodeURI('ssconf://1.2.3.4/secret/long/path#certFp=AA:BB:CC:DD:EE:FF;other=param');
const onlineConfig = SIP008_URI.parse(input);
expect(new URL(onlineConfig.url)).toEqual(new URL('https://1.2.3.4/secret/long/path'));
expect(onlineConfig.extra.certFp).toEqual('AA:BB:CC:DD:EE:FF');
});

it('can parse a valid ssconf URI with IPv6 address and custom port', () => {
// encodeURI encodes the IPv6 address brackets.
const input = `ssconf://[2001:0:ce49:7601:e866:efff:62c3:fffe]:8081/secret/long/path#certFp=${
encodeURIComponent('AA:BB:CC:DD:EE:FF')}`;
const onlineConfig = SIP008_URI.parse(input);
expect(new URL(onlineConfig.url))
.toEqual(new URL('https://[2001:0:ce49:7601:e866:efff:62c3:fffe]:8081/secret/long/path'));
expect(onlineConfig.extra.certFp).toEqual('AA:BB:CC:DD:EE:FF');
});
});
});
75 changes: 68 additions & 7 deletions src/shadowsocks_config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.

import * as ipaddr from 'ipaddr.js';
import {Base64} from 'js-base64';
import * as ipaddr from 'ipaddr.js'
import * as punycode from 'punycode';
import {URLSearchParams} from 'url';

// Custom error base class
export class ShadowsocksConfigError extends Error {
Expand Down Expand Up @@ -53,13 +54,13 @@ export class Host extends ValidatedConfigField {
host = host.data;
}
if (ipaddr.isValid(host)) {
const ip = ipaddr.parse(host)
this.isIPv4 = ip.kind() == "ipv4"
this.isIPv6 = ip.kind() == "ipv6"
const ip = ipaddr.parse(host);
this.isIPv4 = ip.kind() === 'ipv4';
this.isIPv6 = ip.kind() === 'ipv6';
// Previous versions of outline-ShadowsocksConfig only accept
// IPv6 in normalized (expanded) form, so we normalize the
// input here to ensure that access keys remain compatible.
host = ip.toNormalizedString()
host = ip.toNormalizedString();
} else {
host = punycode.toASCII(host) as string;
this.isHostname = Host.HOSTNAME_PATTERN.test(host);
Expand Down Expand Up @@ -274,8 +275,8 @@ export const LEGACY_BASE64_URI = {
const data = `${method.data}:${password.data}@${host.data}:${port.data}`;
let b64EncodedData = Base64.encode(data);
// Remove "=" padding
while (b64EncodedData.slice(-1) == "=") {
b64EncodedData = b64EncodedData.slice(0, -1)
while (b64EncodedData.slice(-1) === '=') {
b64EncodedData = b64EncodedData.slice(0, -1);
}
return `ss://${b64EncodedData}${hash}`;
},
Expand Down Expand Up @@ -338,3 +339,63 @@ export const SIP002_URI = {
return `ss://${userInfo}@${uriHost}:${port.data}/${queryString}${hash}`;
},
};

export interface OnlineConfig {
fortuna marked this conversation as resolved.
Show resolved Hide resolved
// URL endpoint to retrieve a Shadowsocks configuration.
readonly url: string;
fortuna marked this conversation as resolved.
Show resolved Hide resolved
// Any additional configuration (e.g. `certFp`, `httpMethod`, etc.) may be stored here.
readonly extra: {[key: string]: string};
fortuna marked this conversation as resolved.
Show resolved Hide resolved
}

// Ref: https://github.com/shadowsocks/shadowsocks-org/issues/89
export const SIP008_URI = {
fortuna marked this conversation as resolved.
Show resolved Hide resolved
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should call this SIP008_URI. The SIP0008 is a bad standard. It's focused on the config format, not the URL format. They are conflating many things there. And we are kind of making it our own format.

Why do you need this object? Can't you just expose the method? I'd rename parse as parseOnlineConfigUrl to get ConfigFetchParams.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b3f2f19

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SIP0008 is a bad standard. It's focused on the config format, not the URL format.

SIP008 is focused on the config format because all existing solutions widely used by providers in China are just some random JSON encoded in base64 and then served by a typical web server. SIP008 was intended to address the situation by providing a standard JSON document format that is flexible and not encoded in any unnecessary means like base64.

As for the URL format, most people I know who set up their own servers for such purposes already own at least one domain name. And it's pretty easy to obtain a TLS certificate with Let's Encrypt. But I do agree utilizing self-signed certificates can be better in some situations, and that's why I mentioned this PR when discussing with V2Fly developers on V2Ray's upcoming subscription protocol that's based on SIP008ext with a custom URL format.

And we are kind of making it our own format.

I'm open to amendments to SIP008. What Outline plans to implement can be useful to the community as well. We are working towards the same goal after all.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, I think I misspoke there. The standard meets a different need than the one in Outline, so it's bad for our use case.

  1. It doesn't address the need for the certificate fingerprint
  2. It doesn't address the need to handle redirects in a way that allows migration of the config location
  3. It doesn't address the need for a special URI protocol for better user experience
  4. The file format is overly complicated with many things we don't actually support. It includes thinks that are not config, like "bytes used". It has flaws like not knowing what can be safely ignored. Configuring services need more careful consideration, but we'd like to punt on that for now.

I've argued for all of those items in the SIP discussions. I'd like to implement what works for Outline, and then go back with it as a proof of concept.

PROTOCOL: 'ssconf',

validateProtocol: (uri: string) => {
if (!uri || !uri.startsWith(SIP008_URI.PROTOCOL)) {
fortuna marked this conversation as resolved.
Show resolved Hide resolved
throw new InvalidUri(`URI must start with "${SIP008_URI.PROTOCOL}"`);
}
},

parse: (uri: string): OnlineConfig => {
SIP008_URI.validateProtocol(uri);

// URL parser for expedience, replacing the protocol "ssconf" with "https" to ensure correct
// results, otherwise browsers like Safari fail to parse it.
const inputForUrlParser = uri.replace(new RegExp(`^${SIP008_URI.PROTOCOL}`), 'https');
fortuna marked this conversation as resolved.
Show resolved Hide resolved
// The built-in URL parser throws as desired when given URIs with invalid syntax.
const urlParserResult = new URL(inputForUrlParser);
// Use ValidatedConfigFields subclasses (Host, Port, Tag) to throw on validation failure.
const uriFormattedHost = urlParserResult.hostname;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do a regex replace like ...hostname.replace(/\[(.*)\]/, '$1') here then we don't need the try-catch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately that doesn't quite work because some implementations fail without the brackets. I don't remember for which browsers/webviews we added the try catch in SIP002, but I thought it would make sense to replicate it here.

let host: Host;
try {
host = new Host(uriFormattedHost);
} catch (_) {
// Could be IPv6 host formatted with surrounding brackets, so try stripping first and last
fortuna marked this conversation as resolved.
Show resolved Hide resolved
// characters. If this throws, give up and let the exception propagate.
host = new Host(uriFormattedHost.substring(1, uriFormattedHost.length - 1));
}
// The default URL parser fails to recognize the default HTTPs port (443).
const port = new Port(urlParserResult.port || '443');

// Parse extra parameters from the tag, which has the format `#key0=val0;key1=val1...[;]`
const extra = {} as {[key: string]: string};
alalamav marked this conversation as resolved.
Show resolved Hide resolved
const tag = new Tag(decodeURIComponent(urlParserResult.hash.substring(1)));
// Convert tag to search parameters to leverage URLSearchParams parsing.
const params = new URLSearchParams(tag.data.replace(';', '&'));
fortuna marked this conversation as resolved.
Show resolved Hide resolved
params.forEach((value, key) => {
if (!key) {
return;
}
extra[key] = value;
});

const config: OnlineConfig = {
// Build the access URL with the parsed parameters. Exclude the query string, as the spec
// recommends against it.
url: `https://${uriFormattedHost}:${port.data}${urlParserResult.pathname}`,
extra
};
return config;
},
};