diff --git a/docs/auth.md b/docs/auth.md index c697b624db..93d836b66b 100644 --- a/docs/auth.md +++ b/docs/auth.md @@ -1,5 +1,5 @@ netCDF Authorization Support -============================ +====================================== # netCDF Authorization Support {#Header} @@ -37,29 +37,92 @@ directly insert the username and the password into a url in this form. This username and password will be used if the server asks for authentication. Note that only simple password authentication is supported in this format. + Specifically note that [redirection-based](#REDIR) -authorization will not work with this because the username and password -will only be used on the initial request, not the redirection +authorization may work with this but it is a security risk. +This is because the username and password +may be sent to each server in the redirection chain. + +Note also that the `user:password` form may contain characters that must be +escaped. See the password escaping section to see +how to properly escape the user and password. ## RC File Authentication {#DODSRC} The netcdf library supports an _rc_ file mechanism to allow the passing of a number of parameters to libnetcdf and libcurl. +Locating the _rc_ file is a multi-step process. + +### Search Order The file must be called one of the following names: ".daprc" or ".dodsrc". If both ".daprc" and ".dodsrc" exist, then the ".daprc" file will take precedence. -The rc file is searched for first in the current directory -and then in the home directory (as defined by the HOME environment -variable). +It is strongly suggested that you pick one of the two names +and use it always. Otherwise you may observe unexpected results +when the netcdf-c library finds one that you did not intend. + +The search for an _rc_ file looks in the following places in this order. + +1. Check for the environment variable named _DAPRCFILE_. + This will specify the full path for the _rc_ file + (not just the containing directory). +2. Search the current working directory (`./`) looking + for (in order) .daprc or .dodsrc. +3. Search the HOME directory (`$HOME`) looking + for (in order) .daprc or .dodsrc. The HOME environment + variable is used to define the directory in which to search. + +It is strongly suggested that you pick a uniform location +and use it always. Otherwise you may observe unexpected results +when the netcdf-c library get an rc file you did not expect. + +### RC File Format The rc file format is a series of lines of the general form: []= -where the bracket-enclosed host:port is optional and will be discussed -subsequently. +where the bracket-enclosed host:port is optional. + +### URL Constrained RC File Entries + +Each line of the rc file can begin with +a host+port enclosed in square brackets. +The form is "host:port". +If the port is not specified +then the form is just "host". +The reason that more of the url is not used is that +libcurl's authorization grain is not any finer than host level. + +Examples. + + [remotetest.unidata.ucar.edu]HTTP.VERBOSE=1 + +or + + [fake.ucar.edu:9090]HTTP.VERBOSE=0 + +If the url request from, say, the _netcdf_open_ method +has a host+port matching one of the prefixes in the rc file, then +the corresponding entry will be used, otherwise ignored. +This means that an entry with a matching host+port will take +precedence over an entry without a host+port. + +For example, the URL + + http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc + +will have HTTP.VERBOSE set to 1 because its host matches the example above. + +Similarly, + + http://fake.ucar.edu:9090/dts/test.01 + +will have HTTP.VERBOSE set to 0 because its host+port matches the example above. + +## Authorization-Related Keys {#AUTHKEYS} The currently defined set of authorization-related keys are as follows. The second column is the affected curl_easy_setopt option(s), if any. @@ -71,12 +134,12 @@ The second column is the affected curl_easy_setopt option(s), if any. HTTP.SSL.CERTIFICATECURLOPT_SSLCERT HTTP.SSL.KEYCURLOPT_SSLKEY HTTP.SSL.KEYPASSWORDCURLOPT_KEYPASSWORD -HTTP.SSL.CAINFOCURLOPT_SSLCAINFO -HTTP.SSL.CAPATHCURLOPT_SSLCAPATH +HTTP.SSL.CAINFOCURLOPT_CAINFO +HTTP.SSL.CAPATHCURLOPT_CAPATH HTTP.SSL.VERIFYPEERCURLOPT_SSL_VERIFYPEER HTTP.SSL.VALIDATECURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST HTTP.CREDENTIALS.USERPASSWORDCURLOPT_USERPASSWORD -HTTP.NETRCN.A.Specify path of the .netrc file +HTTP.NETRCCURLOPT_NETRC,CURLOPT_NETRC_FILE ### Password Authentication @@ -86,7 +149,9 @@ HTTP.CREDENTIALS.USERPASSWORD can be used to set the simple password authentication. This is an alternative to setting it in the url. The value must be of the form "username:password". -See redirection authorization +See the password escaping section +to see how this value must escape certain characters. +Also see redirection authorization for important additional information. ### Cookie Jar @@ -129,6 +194,29 @@ specifies the absolute path of the .netrc file. See [redirection authorization](#REDIR) for information about using .netrc. +## Password Escaping {#USERPWDESCAPE} + +With current password rules, it is is not unlikely that the password +will contain characters that need to be escaped. Similarly, the user +may contain characters such as '@' that need to be escaped. To support this, +it is assumed that all occurrences of `user:password` use URL (i.e. %%XX) +escaping for at least the characters in the table below. + +The minimum set of characters that must be escaped depends on the location. +If the user+pwd is embedded in the URL, then '@' and ':' __must__ be escaped. +If the user+pwd is the value for +the HTTP.CREDENTIALS.USERPASSWORD key in the _rc_ file, then +':' __must__ be escaped. +Escaping should __not__ be used in the `.netrc` file. + +The relevant escape codes are as follows. + + + + +
CharacterEscaped Form
'@'%40
':'%3a
+Additional characters can be escaped if desired. + ## Redirection-Based Authentication {#REDIR} Some sites provide authentication by using a third party site @@ -145,16 +233,18 @@ using the _https_ protocol (note the use of _https_ instead of _http_). 4. URS sends a redirect (with authorization information) to send the client back to the SOI to actually obtain the data. -It turns out that libcurl uses the password in the `.daprc` -file (or from the url) -only for the initial connection. This causes problems because -the redirected connection is the one that actually requires the password. -This is where the `.netrc` file comes in. Libcurl will use `.netrc` for -the redirected connection. It is possible to cause libcurl to use -the `.daprc` password always, but this introduces a security hole -because it may send the initial user+pwd to the redirection site. -In summary, if you are using redirection, then you must create a `.netrc` -file to hold the password for the site to which the redirection is sent. +It turns out that libcurl, by default, uses the password in the +`.daprc` file (or from the url) for all connections that request +a password. This causes problems because only the the specific +redirected connection is the one that actually requires the password. +This is where the `.netrc` file comes in. Libcurl will use `.netrc` +for the redirected connection. It is possible to cause libcurl +to use the `.daprc` password always, but this introduces a +security hole because it may send the initial user+pwd to every +server in the redirection chain. +In summary, if you are using redirection, then you are +''strongly'' encouraged to create a `.netrc` file to hold the +password for the site to which the redirection is sent. The format of this `.netrc` file will contain lines that typically look like this. @@ -165,52 +255,19 @@ where the machine, mmmmmm, is the hostname of the machine to which the client is redirected for authorization, and the login and password are those needed to authenticate on that machine. -The `.netrc` file can be specified by +The location of the `.netrc` file can be specified by putting the following line in your `.daprc`/`.dodsrc` file. HTTP.NETRC= +If not specified, then libcurl will look first in the current +directory, and then in the HOME directory. + One final note. In using this, you MUST to specify a real file in the file system to act as the cookie jar file (HTTP.COOKIEJAR) so that the redirect site can properly pass back authorization information. -## URL Constrained RC File Entries {#URLCONS} - -Each line of the rc file can begin with -a host+port enclosed in square brackets. -The form is "host:port". -If the port is not specified -then the form is just "host". -The reason that more of the url is not used is that -libcurl's authorization grain is not any finer than host level. - -Examples. - - [remotetest.unidata.ucar.edu]HTTP.VERBOSE=1 - -or - - [fake.ucar.edu:9090]HTTP.VERBOSE=0 - -If the url request from, say, the _netcdf_open_ method -has a host+port matching one of the prefixes in the rc file, then -the corresponding entry will be used, otherwise ignored. -This means that an entry with a matching host+port will take -precedence over an entry without a host+port. - -For example, the URL - - http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc - -will have HTTP.VERBOSE set to 1 because its host matches the example above. - -Similarly, - - http://fake.ucar.edu:9090/dts/test.01 - -will have HTTP.VERBOSE set to 0 because its host+port matches the example above. - ## Client-Side Certificates {#CLIENTCERTS} Some systems, notably ESG (Earth System Grid), requires @@ -244,8 +301,8 @@ the code is definitive. HTTP.SSL.CERTIFICATECUROPT_SSLCERT HTTP.SSL.KEYCUROPT_SSLKEY HTTP.SSL.KEYPASSWORDCUROPT_KEYPASSWORD -HTTP.SSL.CAINFOCUROPT_SSLCAINFO -HTTP.SSL.CAPATHCUROPT_SSLCAPATH +HTTP.SSL.CAINFOCUROPT_CAINFO +HTTP.SSL.CAPATHCUROPT_CAPATH HTTP.SSL.VERIFYPEERCUROPT_SSL_VERIFYPEER HTTP.CREDENTIALS.USERPASSWORDCUROPT_USERPASSWORD HTTP.NETRCCURLOPT_NETRC,CURLOPT_NETRC_FILE diff --git a/include/ncuri.h b/include/ncuri.h index 62e27392b6..72fbe1a429 100644 --- a/include/ncuri.h +++ b/include/ncuri.h @@ -21,11 +21,11 @@ #define NCU_ECONSTRAINTS (11) /* Define flags to control what is included by ncuribuild*/ -#define NCURIPATH 1 -#define NCURIPWD 2 -#define NCURIQUERY 4 -#define NCURIFRAG 8 -#define NCURIENCODE 16 /* If output should be encoded */ +#define NCURIPATH 1 +#define NCURIPWD 2 +#define NCURIQUERY 4 +#define NCURIFRAG 8 +#define NCURIENCODE 16 /* If output should be encoded */ #define NCURIBASE (NCURIPWD|NCURIPATH) #define NCURISVC (NCURIQUERY|NCURIBASE) /* for sending to server */ #define NCURIALL (NCURIPATH|NCURIPWD|NCURIQUERY|NCURIFRAG) /* for rebuilding after changes */ @@ -81,9 +81,13 @@ extern const char* ncurilookup(NCURI*, const char* param); extern const char* ncuriquerylookup(NCURI*, const char* param); /* URL Encode/Decode */ -extern char* ncuriencode(char* s, char* allowable); extern char* ncuridecode(char* s); -extern char* ncuridecodeonly(char* s, char*); +/* Partial decode */ +extern char* ncuridecodepartial(char* s, const char* decodeset); +/* Encode using specified character set */ +extern char* ncuriencodeonly(char* s, char* allowable); +/* Encode user or pwd */ +extern char* ncuriencodeuserpwd(char* s); #if defined(_CPLUSPLUS_) || defined(__CPLUSPLUS__) || defined(__CPLUSPLUS) } diff --git a/libdap4/d4curlfunctions.c b/libdap4/d4curlfunctions.c index 4fba4adbe6..df198d410f 100644 --- a/libdap4/d4curlfunctions.c +++ b/libdap4/d4curlfunctions.c @@ -56,9 +56,11 @@ set_curlflag(NCD4INFO* state, int flag) { int ret = NC_NOERR; switch (flag) { - case CURLOPT_USERPWD: - if(state->curl->creds.userpwd != NULL) { - CHECK(state, CURLOPT_USERPWD, state->curl->creds.userpwd); + case CURLOPT_USERPWD: /* Do both user and pwd */ + if(state->curl->creds.user != NULL + && state->curl->creds.pwd != NULL) { + CHECK(state, CURLOPT_USERNAME, state->curl->creds.user); + CHECK(state, CURLOPT_PASSWORD, state->curl->creds.pwd); CHECK(state, CURLOPT_HTTPAUTH, (OPTARG)CURLAUTH_ANY); } break; @@ -107,8 +109,10 @@ set_curlflag(NCD4INFO* state, int flag) if(state->curl->proxy.host != NULL) { CHECK(state, CURLOPT_PROXY, state->curl->proxy.host); CHECK(state, CURLOPT_PROXYPORT, (OPTARG)(long)state->curl->proxy.port); - if(state->curl->proxy.userpwd) { - CHECK(state, CURLOPT_PROXYUSERPWD, state->curl->proxy.userpwd); + if(state->curl->proxy.user != NULL + && state->curl->proxy.pwd != NULL) { + CHECK(state, CURLOPT_PROXYUSERNAME, state->curl->proxy.user); + CHECK(state, CURLOPT_PROXYPASSWORD, state->curl->proxy.pwd); #ifdef CURLOPT_PROXYAUTH CHECK(state, CURLOPT_PROXYAUTH, (long)CURLAUTH_ANY); #endif @@ -264,6 +268,7 @@ NCD4_curl_protocols(NCD4globalstate* state) } +#if 0 /* "Inverse" of set_curlflag; Given a flag and value, it updates state. @@ -349,6 +354,7 @@ NCD4_set_curlstate(NCD4INFO* state, int flag, void* value) done: return THROW(ret); } +#endif void NCD4_curl_printerror(NCD4INFO* state) diff --git a/libdap4/d4curlfunctions.h b/libdap4/d4curlfunctions.h index 6dcb98e4d5..cadecf5e99 100644 --- a/libdap4/d4curlfunctions.h +++ b/libdap4/d4curlfunctions.h @@ -28,7 +28,6 @@ extern ncerror NCD4_set_flags_perfetch(NCD4INFO*); extern ncerror NCD4_set_flags_perlink(NCD4INFO*); extern ncerror NCD4_set_curlflag(NCD4INFO*,int); -extern ncerror NCD4_set_curlstate(NCD4INFO* state, int flag, void* value); extern void NCD4_curl_debug(NCD4INFO* state); diff --git a/libdap4/d4file.c b/libdap4/d4file.c index e7e7a254ea..058648adfe 100644 --- a/libdap4/d4file.c +++ b/libdap4/d4file.c @@ -327,8 +327,10 @@ freeCurl(NCD4curl* curl) nullfree(curl->ssl.cainfo); nullfree(curl->ssl.capath); nullfree(curl->proxy.host); - nullfree(curl->proxy.userpwd); - nullfree(curl->creds.userpwd); + nullfree(curl->proxy.user); + nullfree(curl->proxy.pwd); + nullfree(curl->creds.user); + nullfree(curl->creds.pwd); if(curl->curlflags.createdflags & COOKIECREATED) d4removecookies(curl->curlflags.cookiejar); nullfree(curl->curlflags.cookiejar); diff --git a/libdap4/d4rc.c b/libdap4/d4rc.c index 8cb7659abe..478809918e 100644 --- a/libdap4/d4rc.c +++ b/libdap4/d4rc.c @@ -16,14 +16,14 @@ #define MEMCHECK(x) if((x)==NULL) {goto nomem;} else {} /* Forward */ -static char* extract_credentials(NCURI*); static int rccompile(const char* path); static struct NCD4triple* rclocate(char* key, char* hostport); -static void rcorder(NClist* rc); +static NClist* rcorder(NClist* rc); static char* rcreadline(char**); static int rcsearch(const char* prefix, const char* rcname, char** pathp); static void rctrim(char* text); static int rcsetinfocurlflag(NCD4INFO*, const char* flag, const char* value); +static int parsecredentials(const char* userpwd, char** userp, char** pwdp); #ifdef D4DEBUG static void storedump(char* msg, NClist* triples); #endif @@ -80,30 +80,35 @@ rctrim(char* text) } } -/* Order the triples: put all those with urls first */ -static void +/* Order the triples: those with urls must be first, + but otherwise relative order does not matter. +*/ +static NClist* rcorder(NClist* rc) { int i,j; int len = nclistlength(rc); - if(rc == NULL || len == 0) return; + NClist* newrc = nclistnew(); + if(rc == NULL || len == 0) return newrc; + /* Two passes: 1) pull triples with host */ + for(i=0;ihost == NULL) continue; + nclistpush(newrc,ti); + } + /* pass 2 pull triples without host*/ for(i=0;ihost != NULL) continue; - for(j=i;jhost != NULL) {/*swap*/ - NCD4triple* t = ti; - nclistset(rc,i,tj); - nclistset(rc,j,t); - } - } + nclistpush(newrc,ti); } #ifdef D4DEBUG - storedump("reorder:",rc); + + storedump("reorder:",newrc); #endif -} + return newrc; +} /* Create a triple store from a file */ static int @@ -373,10 +378,15 @@ rcsetinfocurlflag(NCD4INFO* info, const char* flag, const char* value) #endif } - if(strcmp(flag,"HTTP.CREDENTIALS.USERPASSWORD")==0) { - nullfree(info->curl->creds.userpwd); - info->curl->creds.userpwd = strdup(value); - MEMCHECK(info->curl->creds.userpwd); + if(strcmp(flag,"HTTP.CREDENTIALS.USERNAME")==0) { + nullfree(info->curl->creds.user); + info->curl->creds.user = strdup(value); + MEMCHECK(info->curl->creds.user); + } + if(strcmp(flag,"HTTP.CREDENTIALS.PASSWORD")==0) { + nullfree(info->curl->creds.pwd); + info->curl->creds.pwd = strdup(value); + MEMCHECK(info->curl->creds.pwd); } done: @@ -390,9 +400,7 @@ int NCD4_rcprocess(NCD4INFO* info) { int ret = NC_NOERR; - char userpwd[NC_MAX_PATH]; char hostport[NC_MAX_PATH]; - char* url_userpwd = userpwd; /* WATCH OUT: points to previous variable */ char* url_hostport = hostport; /* WATCH OUT: points to previous variable */ NCURI* uri = info->uri; @@ -403,15 +411,12 @@ NCD4_rcprocess(NCD4INFO* info) /* Note, we still must do this function even if NCD4_globalstate->rc.ignore is set in order - to getinfo e.g. user:pwd from url + to getinfo e.g. host+port from url */ + url_hostport = NULL; if(uri != NULL) { - NCD4_userpwd(uri,url_userpwd,sizeof(userpwd)); NCD4_hostport(uri,url_hostport,sizeof(hostport)); - } else { - url_hostport = NULL; - url_userpwd = NULL; } rcsetinfocurlflag(info,"HTTP.DEFLATE", @@ -450,29 +455,38 @@ NCD4_rcprocess(NCD4INFO* info) NCD4_rclookup("HTTP.NETRC",url_hostport)); { /* Handle various cases for user + password */ /* First, see if the user+pwd was in the original url */ - char* userpwd = NULL; char* user = NULL; char* pwd = NULL; - if(url_userpwd != NULL) - userpwd = url_userpwd; - else { + if(uri->user != NULL && uri->password != NULL) { + user = uri->user; + pwd = uri->password; + } else { user = NCD4_rclookup("HTTP.CREDENTIALS.USER",url_hostport); pwd = NCD4_rclookup("HTTP.CREDENTIALS.PASSWORD",url_hostport); - userpwd = NCD4_rclookup("HTTP.CREDENTIALS.USERPASSWORD",url_hostport); } - if(userpwd == NULL && user != NULL && pwd != NULL) { - char creds[NC_MAX_PATH]; - strncpy(creds,user,sizeof(creds)); - strncat(creds,":",sizeof(creds)); - strncat(creds,pwd,sizeof(creds)); - rcsetinfocurlflag(info,"HTTP.USERPASSWORD",creds); - } else if(userpwd != NULL) - rcsetinfocurlflag(info,"HTTP.USERPASSWORD",userpwd); + if(user != NULL && pwd != NULL) { + user = strdup(user); /* so we can consistently reclaim */ + pwd = strdup(pwd); + } else { + /* Could not get user and pwd, so try USERPASSWORD */ + const char* userpwd = NCD4_rclookup("HTTP.CREDENTIALS.USERPASSWORD",url_hostport); + if(userpwd != NULL) { + ret = parsecredentials(userpwd,&user,&pwd); + if(ret) return ret; + } + } + rcsetinfocurlflag(info,"HTTP.USERNAME",user); + rcsetinfocurlflag(info,"HTTP.PASSWORD",pwd); + nullfree(user); + nullfree(pwd); } - return THROW(ret); } +/** + * (Internal) Locate a triple by property key and host+port (may be null or ""). + * If duplicate keys, first takes precedence. + */ static struct NCD4triple* rclocate(char* key, char* hostport) { @@ -503,6 +517,10 @@ rclocate(char* key, char* hostport) return (found?triple:NULL); } +/** + * Locate a triple by property key and host+port (may be null|"") + * If duplicate keys, first takes precedence. + */ char* NCD4_rclookup(char* key, char* hostport) { @@ -524,13 +542,16 @@ storedump(char* msg, NClist* triples) for(i=0;ihost == NULL || strlen(t->host)==0?"--":t->host),t->key,t->value); + + ((t->host == NULL || strlen(t->host)==0)?"--":t->host),t->key,t->value); + } fflush(stderr); } #endif /** + * Locate rc file by searching in directory prefix. * Prefix must end in '/' */ static @@ -592,7 +613,8 @@ NCD4_parseproxy(NCD4INFO* info, const char* surl) return THROW(NC_NOERR); /* nothing there*/ if(ncuriparse(surl,&uri) != NCU_OK) return THROW(NC_EURL); - info->curl->proxy.userpwd = extract_credentials(uri); + info->curl->proxy.user = uri->user; + info->curl->proxy.pwd = uri->password; info->curl->proxy.host = strdup(uri->host); if(uri->port != NULL) info->curl->proxy.port = atoi(uri->port); @@ -601,15 +623,32 @@ NCD4_parseproxy(NCD4INFO* info, const char* surl) return THROW(ret); } -/* Caller must free result_url */ -static char* -extract_credentials(NCURI* url) +/* +Given form user:pwd, parse into user and pwd +and do %xx unescaping +*/ +static int +parsecredentials(const char* userpwd, char** userp, char** pwdp) { - char tmp[NC_MAX_PATH]; - if(url->user == NULL || url->password == NULL) - return NULL; - NCD4_userpwd(url,tmp,sizeof(tmp)); - return strdup(tmp); + char* user = NULL; + char* pwd = NULL; + + if(userpwd == NULL) + return NC_EINVAL; + user = strdup(userpwd); + if(user == NULL) + return NC_ENOMEM; + pwd = strchr(user,':'); + if(pwd == NULL) + return NC_EINVAL; + *pwd = '\0'; + pwd++; + if(userp) + *userp = ncuridecode(user); + if(pwdp) + *pwdp = ncuridecode(pwd); + free(user); + return NC_NOERR; } int diff --git a/libdap4/d4util.c b/libdap4/d4util.c index 01f3fa72ad..59e6c65d76 100644 --- a/libdap4/d4util.c +++ b/libdap4/d4util.c @@ -408,6 +408,7 @@ NCD4_hostport(NCURI* uri, char* space, size_t len) } } +#if 0 void NCD4_userpwd(NCURI* uri, char* space, size_t len) { @@ -420,6 +421,7 @@ NCD4_userpwd(NCURI* uri, char* space, size_t len) } } } +#endif #ifdef BLOB void diff --git a/libdap4/ncd4.h b/libdap4/ncd4.h index 92b680e7a4..2c47e55367 100644 --- a/libdap4/ncd4.h +++ b/libdap4/ncd4.h @@ -129,7 +129,6 @@ extern int NCD4_getToplevelVars(NCD4meta* meta, NCD4node* group, NClist* topleve /* From d4util.c */ extern d4size_t NCD4_dimproduct(NCD4node* node); extern void NCD4_hostport(NCURI* uri, char* space, size_t len); -extern void NCD4_userpwd(NCURI* uri, char* space, size_t len); extern size_t NCD4_typesize(nc_type tid); extern int NCD4_isLittleEndian(void);/* Return 1 if this machine is little endian */ extern int NCD4_errorNC(int code, const int line, const char* file); diff --git a/libdap4/ncd4types.h b/libdap4/ncd4types.h index d77578a4f8..1043758810 100644 --- a/libdap4/ncd4types.h +++ b/libdap4/ncd4types.h @@ -325,10 +325,12 @@ struct NCD4curl { struct proxy { char *host; /*CURLOPT_PROXY*/ int port; /*CURLOPT_PROXYPORT*/ - char* userpwd; /*CURLOPT_PROXYUSERPWD*/ + char* user; /*CURLOPT_PROXYUSERNAME*/ + char* pwd; /*CURLOPT_PROXYPASSWORD*/ } proxy; struct credentials { - char *userpwd; /*CURLOPT_USERPWD*/ + char *user; /*CURLOPT_USERNAME*/ + char *pwd; /*CURLOPT_PASSWORD*/ } creds; }; diff --git a/libdispatch/drc.c b/libdispatch/drc.c index f47ef46665..b9530427e6 100644 --- a/libdispatch/drc.c +++ b/libdispatch/drc.c @@ -1,7 +1,7 @@ -/********************************************************************* - * Copyright 2016, UCAR/Unidata - * See netcdf/COPYRIGHT file for copying and redistribution conditions. - *********************************************************************/ +/* +Copyright (c) 1998-2017 University Corporation for Atmospheric Research/Unidata +See LICENSE.txt for license information. +*/ #include "config.h" #ifdef HAVE_UNISTD_H diff --git a/libdispatch/ncuri.c b/libdispatch/ncuri.c index 7acf16e841..fe80097e7a 100644 --- a/libdispatch/ncuri.c +++ b/libdispatch/ncuri.c @@ -63,6 +63,10 @@ static char* pathallow = static char* queryallow = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!#$&'()*+,-./:;=?@_~"; +/* user+pwd allow = path allow - "@:" */ +static char* userpwdallow = +"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!$&'()*+,-.;=_~?#/"; + #ifndef HAVE_STRNCMP #define strndup ncstrndup /* Not all systems have strndup, so provide one*/ @@ -345,8 +349,13 @@ ncuriparse(const char* uri0, NCURI** durip) /* save original uri */ duri->uri = strdup(uri0); duri->protocol = nulldup(tmp.protocol); - duri->user = nulldup(tmp.user); - duri->password = nulldup(tmp.password); + /* before saving, we need to decode the user+pwd */ + duri->user = NULL; + duri->password = NULL; + if(tmp.user != NULL) + duri->user = ncuridecode(tmp.user); + if(tmp.password != NULL) + duri->password = ncuridecode(tmp.password); duri->host = nulldup(tmp.host); duri->port = nulldup(tmp.port); if(tmp.path != NULL) { @@ -530,9 +539,14 @@ ncuribuild(NCURI* duri, const char* prefix, const char* suffix, int flags) ncbytescat(buf,"://"); /* this will produce file:///... */ if((flags & NCURIPWD) && duri->user != NULL && duri->password != NULL) { - ncbytescat(buf,duri->user); + /* The user and password must be encoded */ + char* encoded = ncuriencodeonly(duri->user,userpwdallow); + ncbytescat(buf,encoded); + nullfree(encoded); ncbytescat(buf,":"); - ncbytescat(buf,duri->password); + encoded = ncuriencodeonly(duri->password,userpwdallow); + ncbytescat(buf,encoded); + nullfree(encoded); ncbytescat(buf,"@"); } if(duri->host != NULL) ncbytescat(buf,duri->host); @@ -544,7 +558,7 @@ ncuribuild(NCURI* duri, const char* prefix, const char* suffix, int flags) if(duri->path == NULL) ncbytescat(buf,"/"); else if(encode) { - char* encoded = ncuriencode(duri->path,pathallow); + char* encoded = ncuriencodeonly(duri->path,pathallow); ncbytescat(buf,encoded); nullfree(encoded); } else @@ -566,7 +580,7 @@ ncuribuild(NCURI* duri, const char* prefix, const char* suffix, int flags) if(p[1] != NULL && strlen(p[1]) > 0) { ncbytescat(buf,"="); if(encode) { - char* encoded = ncuriencode(p[1],queryallow); + char* encoded = ncuriencodeonly(p[1],queryallow); ncbytescat(buf,encoded); nullfree(encoded); } else @@ -583,7 +597,7 @@ ncuribuild(NCURI* duri, const char* prefix, const char* suffix, int flags) if(p[1] != NULL && strlen(p[1]) > 0) { ncbytescat(buf,"="); if(encode) { - char* encoded = ncuriencode(p[1],queryallow); + char* encoded = ncuriencodeonly(p[1],queryallow); ncbytescat(buf,encoded); nullfree(encoded); } else @@ -720,8 +734,8 @@ static char* hexchars = "0123456789abcdefABCDEF"; static void toHex(unsigned int b, char hex[2]) { - hex[0] = hexchars[(b >> 4) & 0xff]; - hex[1] = hexchars[(b) & 0xff]; + hex[0] = hexchars[(b >> 4) & 0xf]; + hex[1] = hexchars[(b) & 0xf]; } @@ -734,6 +748,14 @@ fromHex(int c) return 0; } +/* +Support encode of user and password fields +*/ +char* +ncuriencodeuserpwd(char* s) +{ + return ncuriencodeonly(s,userpwdallow); +} /* Return a string representing encoding of input; caller must free; watch out: will encode whole string, so watch what you give it. @@ -741,7 +763,7 @@ fromHex(int c) */ char* -ncuriencode(char* s, char* allowable) +ncuriencodeonly(char* s, char* allowable) { size_t slen; char* encoded; @@ -760,12 +782,10 @@ ncuriencode(char* s, char* allowable) } else { /* search allowable */ int c2; - char* a = allowable; - while((c2=*a++)) { - if(c == c2) break; - } - if(c2) {*outptr++ = (char)c;} - else { + char* p = strchr(allowable,c); + if(p != NULL) { + *outptr++ = (char)c; + } else { char hex[2]; toHex(c,hex); *outptr++ = '%'; @@ -782,14 +802,43 @@ ncuriencode(char* s, char* allowable) char* ncuridecode(char* s) { - return ncuridecodeonly(s,NULL); + size_t slen; + char* decoded; + char* outptr; + char* inptr; + unsigned int c; + + if (s == NULL) return NULL; + + slen = strlen(s); + decoded = (char*)malloc(slen+1); /* Should be max we need */ + + outptr = decoded; + inptr = s; + while((c = (unsigned int)*inptr++)) { + if(c == '%') { + /* try to pull two hex more characters */ + if(inptr[0] != EOFCHAR && inptr[1] != EOFCHAR + && strchr(hexchars,inptr[0]) != NULL + && strchr(hexchars,inptr[1]) != NULL) { + /* test conversion */ + int xc = (fromHex(inptr[0]) << 4) | (fromHex(inptr[1])); + inptr += 2; /* decode it */ + c = (unsigned int)xc; + } + } + *outptr++ = (char)c; + } + *outptr = EOFCHAR; + return decoded; } -/* Return a string representing decoding of input only for specified - characters; caller must free +/* +Partially decode a string. Only characters in 'decodeset' +are decoded. Return decoded string; caller must free. */ char* -ncuridecodeonly(char* s, char* only) +ncuridecodepartial(char* s, const char* decodeset) { size_t slen; char* decoded; @@ -797,7 +846,7 @@ ncuridecodeonly(char* s, char* only) char* inptr; unsigned int c; - if (s == NULL) return NULL; + if (s == NULL || decodeset == NULL) return NULL; slen = strlen(s); decoded = (char*)malloc(slen+1); /* Should be max we need */ @@ -805,7 +854,7 @@ ncuridecodeonly(char* s, char* only) outptr = decoded; inptr = s; while((c = (unsigned int)*inptr++)) { - if(c == '+' && only != NULL && strchr(only,'+') != NULL) + if(c == '+' && strchr(decodeset,'+') != NULL) *outptr++ = ' '; else if(c == '%') { /* try to pull two hex more characters */ @@ -814,13 +863,14 @@ ncuridecodeonly(char* s, char* only) && strchr(hexchars,inptr[1]) != NULL) { /* test conversion */ int xc = (fromHex(inptr[0]) << 4) | (fromHex(inptr[1])); - if(only == NULL || strchr(only,xc) != NULL) { + if(strchr(decodeset,xc) != NULL) { inptr += 2; /* decode it */ c = (unsigned int)xc; - } + } } - } - *outptr++ = (char)c; + *outptr++ = (char)c; /* pass either the % or decoded char */ + } else /* Not a % char */ + *outptr++ = (char)c; } *outptr = EOFCHAR; return decoded; diff --git a/oc2/daplex.c b/oc2/daplex.c index 00bd0b7ef6..cf13282b15 100644 --- a/oc2/daplex.c +++ b/oc2/daplex.c @@ -11,6 +11,9 @@ #undef URLCVT /* NEVER turn this on */ +/* Do we %xx decode all or part of a DAP Identifer: see dapdecode() */ +#define DECODE_PARTIAL + #define DAP2ENCODE #ifdef DAP2ENCODE #define KEEPSLASH @@ -355,8 +358,8 @@ daplexcleanup(DAPlexstate** lexstatep) 1. if the encoded character is in fact a legal DAP2 character (alphanum+"_!~*'-\"") then it is decoded, otherwise not. */ -#ifndef DECODE_IDENTIFIERS -static char* decodelist = +#ifdef DECODE_PARTIAL +static char* decodeset = /* Specify which characters are decoded */ "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_!~*'-\""; #endif @@ -364,10 +367,10 @@ char* dapdecode(DAPlexstate* lexstate, char* name) { char* decoded = NULL; -#ifdef DECODE_IDENTIFIERS - decoded = ncuridecode(name); +#ifdef DECODE_PARTIAL + decoded = ncuridecodepartial(name,decodeset); /* Decode selected */ #else - decoded = ncuridecodeonly(name,decodelist); + decoded = ncuridecode(name); /* Decode everything */ #endif nclistpush(lexstate->reclaim,(void*)decoded); return decoded; diff --git a/oc2/occurlfunctions.c b/oc2/occurlfunctions.c index 952c7fa405..b721a3cf18 100644 --- a/oc2/occurlfunctions.c +++ b/oc2/occurlfunctions.c @@ -88,9 +88,10 @@ ocset_curlflag(OCstate* state, int flag) switch (flag) { - case CURLOPT_USERPWD: - if(state->creds.userpwd != NULL) { - CHECK(state, CURLOPT_USERPWD, state->creds.userpwd); + case CURLOPT_USERPWD: /* Does both user and pwd */ + if(state->creds.user != NULL && state->creds.pwd != NULL) { + CHECK(state, CURLOPT_USERNAME, state->creds.user); + CHECK(state, CURLOPT_PASSWORD, state->creds.pwd); CHECK(state, CURLOPT_HTTPAUTH, (OPTARG)CURLAUTH_ANY); } break; @@ -149,8 +150,9 @@ ocset_curlflag(OCstate* state, int flag) if(state->proxy.host != NULL) { CHECK(state, CURLOPT_PROXY, state->proxy.host); CHECK(state, CURLOPT_PROXYPORT, (OPTARG)(long)state->proxy.port); - if(state->proxy.userpwd) { - CHECK(state, CURLOPT_PROXYUSERPWD, state->proxy.userpwd); + if(state->proxy.user != NULL && state->proxy.pwd != NULL) { + CHECK(state, CURLOPT_PROXYUSERNAME, state->proxy.user); + CHECK(state, CURLOPT_PROXYPASSWORD, state->proxy.pwd); #ifdef CURLOPT_PROXYAUTH CHECK(state, CURLOPT_PROXYAUTH, (long)CURLAUTH_ANY); #endif @@ -334,6 +336,7 @@ oc_curl_protocols(struct OCGLOBALSTATE* state) } +#if 0 /* "Inverse" of ocset_curlflag; Given a flag and value, it updates state. @@ -346,9 +349,14 @@ ocset_curlstate(OCstate* state, int flag, void* value) switch (flag) { - case CURLOPT_USERPWD: - if(state->creds.userpwd != NULL) free(state->creds.userpwd); - state->creds.userpwd = strdup((char*)value); + case CURLOPT_USERNAME: + if(state->creds.user != NULL) free(state->creds.user); + state->creds.user = strdup((char*)value); + break; + + case CURLOPT_PASSWORD: + if(state->creds.pwd != NULL) free(state->creds.pwd); + state->creds.pwd = strdup((char*)value); break; case CURLOPT_COOKIEJAR: case CURLOPT_COOKIEFILE: @@ -437,3 +445,4 @@ ocset_curlstate(OCstate* state, int flag, void* value) done: return stat; } +#endif diff --git a/oc2/occurlfunctions.h b/oc2/occurlfunctions.h index bb99e6bc99..1be5999189 100644 --- a/oc2/occurlfunctions.h +++ b/oc2/occurlfunctions.h @@ -27,7 +27,6 @@ extern OCerror ocset_flags_perfetch(OCstate*); extern OCerror ocset_flags_perlink(OCstate*); extern OCerror ocset_curlflag(OCstate*,int); -extern OCerror ocset_curlstate(OCstate* state, int flag, void* value); extern void oc_curl_debug(OCstate* state); diff --git a/oc2/ocinternal.c b/oc2/ocinternal.c index ea3ea124be..9ca549d9ea 100644 --- a/oc2/ocinternal.c +++ b/oc2/ocinternal.c @@ -427,8 +427,10 @@ occlose(OCstate* state) ocfree(state->ssl.cainfo); ocfree(state->ssl.capath); ocfree(state->proxy.host); - ocfree(state->proxy.userpwd); - ocfree(state->creds.userpwd); + ocfree(state->proxy.user); + ocfree(state->proxy.pwd); + ocfree(state->creds.user); + ocfree(state->creds.pwd); if(state->curl != NULL) occurlclose(state->curl); ocfree(state); } diff --git a/oc2/ocinternal.h b/oc2/ocinternal.h index a954c3cb7c..b4995b333c 100644 --- a/oc2/ocinternal.h +++ b/oc2/ocinternal.h @@ -217,10 +217,12 @@ struct OCstate { struct OCproxy { char *host; /*CURLOPT_PROXY*/ int port; /*CURLOPT_PROXYPORT*/ - char* userpwd; /*CURLOPT_PROXYUSERPWD*/ + char* user; /*CURLOPT_PROXYUSERNAME*/ + char* pwd; /*CURLOPT_PROXYPASSWORD*/ } proxy; struct OCcredentials { - char *userpwd; /*CURLOPT_USERPWD*/ + char *user; /*CURLOPT_USERNAME*/ + char *pwd; /*CURLOPT_PASSWORD*/ } creds; void* usercurldata; long ddslastmodified; diff --git a/oc2/ocrc.c b/oc2/ocrc.c index d000e75d2a..d4ec4dcf66 100644 --- a/oc2/ocrc.c +++ b/oc2/ocrc.c @@ -26,7 +26,6 @@ static OCerror rc_search(const char* prefix, const char* rcfile, char** pathp); static int rcreadline(FILE* f, char* more, int morelen); static void rctrim(char* text); -static char* combinecredentials(const char* user, const char* pwd); static void storedump(char* msg, struct OCTriple*, int ntriples); @@ -48,17 +47,49 @@ occredentials_in_url(const char *url) return 0; } +/* +Given form user:pwd, parse into user and pwd +and do %xx unescaping +*/ static OCerror -ocextract_credentials(const char *url, char **userpwd, char **result_url) +parsecredentials(const char* userpwd, char** userp, char** pwdp) +{ + char* user = NULL; + char* pwd = NULL; + + if(userpwd == NULL) + return OC_EINVAL; + user = strdup(userpwd); + if(user == NULL) + return NC_ENOMEM; + pwd = strchr(user,':'); + if(pwd == NULL) + return OC_EINVAL; + *pwd = '\0'; + pwd++; + if(userp) + *userp = ncuridecode(user); + if(pwdp) + *pwdp = ncuridecode(pwd); + free(user); + return OC_NOERR; +} + +static OCerror +ocextract_credentials(const char *url, char **user, char** pwd, char **result_url) { NCURI* parsed = NULL; - if(ncuriparse(url,&parsed) != NCU_OK) + + if(url == NULL || ncuriparse(url,&parsed) != NCU_OK) return OCTHROW(OC_EBADURL); if(parsed->user != NULL || parsed->password == NULL) { ncurifree(parsed); return OCTHROW(OC_EBADURL); } - if(userpwd) *userpwd = combinecredentials(parsed->user,parsed->password); + if(user) + *user = parsed->user; + if(pwd) + *pwd = parsed->password; ncurifree(parsed); return OC_NOERR; } @@ -85,24 +116,40 @@ occombinehostport(const NCURI* uri) return hp; } +#if 0 +/* +Combine user and pwd into the user:pwd form. +Note that we must %xx escape the user and the pwd +*/ static char* combinecredentials(const char* user, const char* pwd) { int userPassSize; char *userPassword; + char *escapeduser = NULL; + char* escapedpwd = NULL; - if(user == NULL) user = ""; - if(pwd == NULL) pwd = ""; - - userPassSize = strlen(user) + strlen(pwd) + 2; + if(user == NULL || pwd == NULL) + return NULL; + + userPassSize = 3*strlen(user) + 3*strlen(pwd) + 2; /* times 3 for escapes */ userPassword = malloc(sizeof(char) * userPassSize); if (!userPassword) { nclog(NCLOGERR,"Out of Memory\n"); return NULL; } - occopycat(userPassword,userPassSize-1,3,user,":",pwd); + escapeduser = ncuriencodeuserpwd(user); + escapedpwd = ncuriencodeuserpwd(pwd); + if(escapeduser == NULL || escapedpwd == NULL) { + nclog(NCLOGERR,"Out of Memory\n"); + return NULL; + } + occopycat(userPassword,userPassSize-1,3,escapeduser,":",escapedpwd); + free(escapeduser); + free(escapedpwd); return userPassword; } +#endif static int rcreadline(FILE* f, char* more, int morelen) @@ -158,7 +205,7 @@ ocparseproxy(OCstate* state, char* v) return OC_NOERR; /* nothing there*/ if (occredentials_in_url(v)) { char *result_url = NULL; - ocextract_credentials(v, &state->proxy.userpwd, &result_url); + ocextract_credentials(v, &state->proxy.user, &state->proxy.pwd, &result_url); v = result_url; } /* allocating a bit more than likely needed ... */ @@ -208,7 +255,7 @@ ocparseproxy(OCstate* state, char* v) if (ocdebug > 1) { nclog(NCLOGNOTE,"host name: %s", state->proxy.host); #ifdef INSECURE - nclog(NCLOGNOTE,"user+pwd: %s", state->proxy.userpwd); + nclog(NCLOGNOTE,"user+pwd: %s+%s", state->proxy.user,state->proxy.pwd); #endif nclog(NCLOGNOTE,"port number: %d", state->proxy.port); } @@ -373,8 +420,8 @@ ocrc_load(void) /* locate the configuration files in the following order: 1. specified by set_rcfile 2. set by DAPRCFILE env variable - 3. '.' - 4. $HOME + 3. '.'/ + 4. $HOME/ */ if(ocglobalstate.rc.rcfile != NULL) { /* always use this */ path = strdup(ocglobalstate.rc.rcfile); @@ -416,7 +463,6 @@ ocrc_process(OCstate* state) OCerror stat = OC_NOERR; char* value = NULL; NCURI* uri = state->uri; - char* url_userpwd = NULL; char* url_hostport = NULL; if(!ocglobalstate.initialized) @@ -428,7 +474,6 @@ ocrc_process(OCstate* state) to getinfo e.g. user:pwd from url */ - url_userpwd = combinecredentials(uri->user,uri->password); url_hostport = occombinehostport(uri); if(url_hostport == NULL) return OC_ENOMEM; @@ -545,21 +590,28 @@ ocrc_process(OCstate* state) { /* Handle various cases for user + password */ /* First, see if the user+pwd was in the original url */ - char* userpwd = NULL; char* user = NULL; char* pwd = NULL; - if(url_userpwd != NULL) - userpwd = url_userpwd; - else { + if(uri->user != NULL && uri->password != NULL) { + user = uri->user; + pwd = uri->password; + } else { user = ocrc_lookup("HTTP.CREDENTIALS.USER",url_hostport); pwd = ocrc_lookup("HTTP.CREDENTIALS.PASSWORD",url_hostport); - userpwd = ocrc_lookup("HTTP.CREDENTIALS.USERPASSWORD",url_hostport); } - if(userpwd == NULL && user != NULL && pwd != NULL) { - userpwd = combinecredentials(user,pwd); - state->creds.userpwd = userpwd; - } else if(userpwd != NULL) - state->creds.userpwd = strdup(userpwd); + if(user != NULL && pwd != NULL) { + state->creds.user = strdup(user); + state->creds.pwd = strdup(pwd); + } else { + /* Could not get user and pwd, so try USERPASSWORD */ + const char* userpwd = ocrc_lookup("HTTP.CREDENTIALS.USERPASSWORD",url_hostport); + if(userpwd != NULL) { + stat = parsecredentials(userpwd,&user,&pwd); + if(stat) goto done; + state->creds.user = user; + state->creds.pwd = pwd; + } + } } done: @@ -624,7 +676,7 @@ storedump(char* msg, struct OCTriple* triples, int ntriples) if(ntriples < 0 ) ntriples= ocrc->ntriples; for(i=0;i