Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure table implementation #217

Merged
merged 73 commits into from
May 3, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
778e592
work in progress
MindFlavor Sep 20, 2015
3a2ca1f
half
MindFlavor Sep 21, 2015
fa48f7a
CreateTable working
MindFlavor Sep 21, 2015
bcbb55e
CreateTable working
MindFlavor Sep 21, 2015
cff76ea
halhway gettables
MindFlavor Sep 21, 2015
c1fc8cc
Get tables just to conver to []string
MindFlavor Sep 21, 2015
7790a43
gettables working
MindFlavor Sep 21, 2015
6066ff9
Still unable to do insertentity via JSON
MindFlavor Sep 21, 2015
ea24748
Reverted exec to standard
MindFlavor Sep 22, 2015
5ffabc4
Renouncing json inserttable
MindFlavor Sep 22, 2015
5f2c805
Working
MindFlavor Sep 22, 2015
7c25a4d
InsertEntity working
MindFlavor Sep 22, 2015
d810da5
InsertEntity working
MindFlavor Sep 22, 2015
1fbcae9
Corrected git merge mess :@
MindFlavor Sep 22, 2015
e14c38d
eliminated unused code on blob.go
MindFlavor Sep 22, 2015
b5c2e22
Unexported unusefule table.go elements
MindFlavor Sep 22, 2015
069d5b1
Completed InsertOrReplaceEntity
MindFlavor Sep 22, 2015
8e926a4
Completed InsertOrReplaceEntity
MindFlavor Sep 22, 2015
7504f79
Completed migration to TableEntry interface
MindFlavor Sep 23, 2015
4e5c254
Completed unmarshal from table
MindFlavor Sep 23, 2015
001e984
QueryEntites completed
MindFlavor Sep 23, 2015
2323b79
exported createSharedKeyLite func
MindFlavor Sep 23, 2015
2c7826a
ok
MindFlavor Sep 23, 2015
5be7f82
Reverted PutBlockList
MindFlavor Sep 23, 2015
8ce1ad0
merged from master
MindFlavor Sep 23, 2015
b4339d3
Implemented basic tests
MindFlavor Sep 23, 2015
de7a7c9
Query not working
MindFlavor Sep 23, 2015
3e0094f
Finally worked out getentities auth
MindFlavor Sep 24, 2015
401b9d3
Continuaton token working
MindFlavor Sep 24, 2015
8bd86be
Moved code to table_entries.go
MindFlavor Sep 24, 2015
61b55bc
Added delete table
MindFlavor Sep 24, 2015
cba0d70
To test update, merge, insertorreplac, insertofmerge
MindFlavor Sep 24, 2015
334d9e3
Added some tests
MindFlavor Sep 24, 2015
3367625
Added some tests
MindFlavor Sep 24, 2015
b6f6c50
Before remove of unused buf
MindFlavor Sep 24, 2015
fa2d011
Before remove of unused buf
MindFlavor Sep 24, 2015
27554ba
After continuation token check
MindFlavor Sep 24, 2015
833cf1e
ready for github
MindFlavor Sep 24, 2015
2d9dbdc
Azure table implementation
MindFlavor Sep 24, 2015
1d80355
Changed return for get from []* to *[]
MindFlavor Sep 26, 2015
fd3421a
Added documentation
MindFlavor Sep 26, 2015
515af2f
Changed return for get from []* to *[]
MindFlavor Sep 26, 2015
3db94ff
Changed return for get from []* to *[]
MindFlavor Sep 26, 2015
f17c583
Changed time chk.Equals to time.Unix() equals to avoid check for ptr …
MindFlavor Sep 26, 2015
5f16d10
Merge branch 'master' of github.com:Azure/azure-sdk-for-go into azure…
MindFlavor Sep 27, 2015
3a56505
corrected storage/client.go:410: arg resp.StatusCode for printf verb …
MindFlavor Sep 27, 2015
d013164
Corrected misplaced comments
MindFlavor Sep 27, 2015
de36cad
Unexported Client.createSharedKeyLiteTable
MindFlavor Sep 27, 2015
64dec93
Merge branch 'master' of github.com:Azure/azure-sdk-for-go into azure…
MindFlavor Sep 30, 2015
467f573
Merge branch 'master' of github.com:Azure/azure-sdk-for-go into azure…
MindFlavor Oct 6, 2015
4aec3f7
Merge remote-tracking branch 'origin/master' into azure_table
MindFlavor Mar 7, 2016
a136306
fixed QueryTableEntities url bug as per michaelbironneau fix
MindFlavor Mar 7, 2016
beef942
Merge branch 'azure_table' of github.com:MindFlavor/azure-sdk-for-go …
MindFlavor Mar 7, 2016
7ed2aca
test
MindFlavor Mar 24, 2016
bf2e3e6
remove testEquals
MindFlavor Mar 24, 2016
3bf9667
removed extra pointer in deserialize
MindFlavor Mar 24, 2016
ca4300d
hardened extractContToken
MindFlavor Mar 24, 2016
d7326d1
changed buffer allocation
MindFlavor Mar 24, 2016
3cc45a3
changed buffer allocation
MindFlavor Mar 24, 2016
4a2c867
removed unused log entry
MindFlavor Mar 24, 2016
f6c65e8
moved insert and merge methods in execTable
MindFlavor Mar 24, 2016
9a2c286
used switch key
MindFlavor Mar 24, 2016
d38280f
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-go in…
MindFlavor Mar 25, 2016
66ee7e0
simplified extractContinuationTokenFromHeaders
MindFlavor Mar 25, 2016
ae417ef
moved defer resp.body.Close() after err != nil check
MindFlavor Mar 25, 2016
927b5a7
Simplified execTable
MindFlavor Mar 25, 2016
322889e
renamed createSharedKeyLiteTable in createSharedKeyLite
MindFlavor Mar 25, 2016
dbd2e47
implemented ahmetalpbalkan fixes
MindFlavor Mar 25, 2016
5a04aa0
Removed error from TableEntity interface
MindFlavor Mar 25, 2016
045caf2
Removed error from TableEntity interface
MindFlavor Mar 25, 2016
e44729a
corrected if block ends with a return statement, so drop this else an…
MindFlavor Mar 25, 2016
474030a
Merge remote-tracking branch 'official/master' into changes
MindFlavor Mar 30, 2016
430d128
modified execInternalJSON to reuse HTTPClient from Client
MindFlavor Mar 31, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions storage/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ package storage
import (
"bytes"
"encoding/base64"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
Expand Down Expand Up @@ -54,6 +55,11 @@ type storageResponse struct {
body io.ReadCloser
}

type odataResponse struct {
storageResponse
odata odataErrorMessage
}

// AzureStorageServiceError contains fields of the error response from
// Azure Storage Service REST API. See https://msdn.microsoft.com/en-us/library/azure/dd179382.aspx
// Some fields might be specific to certain calls.
Expand All @@ -68,6 +74,20 @@ type AzureStorageServiceError struct {
RequestID string
}

type odataErrorMessageMessage struct {
Lang string `json:"lang"`
Value string `json:"value"`
}

type odataErrorMessageInternal struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MindFlavor do you ever use these error message/code fields in generating a human-readable error? perhaps we should return a typed-error that people can cast into and read these fields?

Code string `json:"code"`
Message odataErrorMessageMessage `json:"message"`
}

type odataErrorMessage struct {
Err odataErrorMessageInternal `json:"odata.error"`
}

// UnexpectedStatusCodeError is returned when a storage service responds with neither an error
// nor with an HTTP status code indicating success.
type UnexpectedStatusCodeError struct {
Expand Down Expand Up @@ -166,6 +186,12 @@ func (c Client) GetQueueService() QueueServiceClient {
return QueueServiceClient{c}
}

// GetTableService returns a TableServiceClient which can operate on the table
// service of the storage account.
func (c Client) GetTableService() TableServiceClient {
return TableServiceClient{c}
}

// GetFileService returns a FileServiceClient which can operate on the file
// service of the storage account.
func (c Client) GetFileService() FileServiceClient {
Expand Down Expand Up @@ -228,6 +254,22 @@ func (c Client) buildCanonicalizedHeader(headers map[string]string) string {
return ch
}

func (c Client) buildCanonicalizedResourceTable(uri string) (string, error) {
errMsg := "buildCanonicalizedResourceTable error: %s"
u, err := url.Parse(uri)
if err != nil {
return "", fmt.Errorf(errMsg, err.Error())
}

cr := "/" + c.accountName

if len(u.Path) > 0 {
cr += u.Path
}

return cr, nil
}

func (c Client) buildCanonicalizedResource(uri string) (string, error) {
errMsg := "buildCanonicalizedResource error: %s"
u, err := url.Parse(uri)
Expand All @@ -236,6 +278,7 @@ func (c Client) buildCanonicalizedResource(uri string) (string, error) {
}

cr := "/" + c.accountName

if len(u.Path) > 0 {
cr += u.Path
}
Expand Down Expand Up @@ -266,6 +309,7 @@ func (c Client) buildCanonicalizedResource(uri string) (string, error) {
}
}
}

return cr, nil
}

Expand Down Expand Up @@ -364,6 +408,70 @@ func (c Client) exec(verb, url string, headers map[string]string, body io.Reader
body: resp.Body}, nil
}

func (c Client) execInternalJSON(verb, url string, headers map[string]string, body io.Reader) (*odataResponse, error) {
req, err := http.NewRequest(verb, url, body)
for k, v := range headers {
req.Header.Add(k, v)
}

httpClient := c.HTTPClient
if httpClient == nil {
httpClient = http.DefaultClient
}

resp, err := httpClient.Do(req)
if err != nil {
return nil, err
}

respToRet := &odataResponse{}
respToRet.body = resp.Body
respToRet.statusCode = resp.StatusCode
respToRet.headers = resp.Header

statusCode := resp.StatusCode
if statusCode >= 400 && statusCode <= 505 {
var respBody []byte
respBody, err = readResponseBody(resp)
if err != nil {
return nil, err
}

if len(respBody) == 0 {
// no error in response body
err = fmt.Errorf("storage: service returned without a response body (%d)", resp.StatusCode)
return respToRet, err
}
// try unmarshal as odata.error json
err = json.Unmarshal(respBody, &respToRet.odata)
return respToRet, err
}

return respToRet, nil
}

func (c Client) createSharedKeyLite(url string, headers map[string]string) (string, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm going to switch everything to SharedKeyLite after merging. 😄

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm going to switch everything to SharedKeyLite after merging. 😄

can, err := c.buildCanonicalizedResourceTable(url)

if err != nil {
return "", err
}
strToSign := headers["x-ms-date"] + "\n" + can

hmac := c.computeHmac256(strToSign)
return fmt.Sprintf("SharedKeyLite %s:%s", c.accountName, hmac), nil
}

func (c Client) execTable(verb, url string, headers map[string]string, body io.Reader) (*odataResponse, error) {
var err error
headers["Authorization"], err = c.createSharedKeyLite(url, headers)
if err != nil {
return nil, err
}

return c.execInternalJSON(verb, url, headers, body)
}

func readResponseBody(resp *http.Response) ([]byte, error) {
defer resp.Body.Close()
out, err := ioutil.ReadAll(resp.Body)
Expand Down
129 changes: 129 additions & 0 deletions storage/table.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
package storage

import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"net/url"
)

// TableServiceClient contains operations for Microsoft Azure Table Storage
// Service.
type TableServiceClient struct {
client Client
}

// AzureTable is the typedef of the Azure Table name
type AzureTable string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most probably you don't need this typedef. Just use string for table names. :)


const (
tablesURIPath = "/Tables"
)

type createTableRequest struct {
TableName string `json:"TableName"`
}

func pathForTable(table AzureTable) string { return fmt.Sprintf("%s", table) }

func (c *TableServiceClient) getStandardHeaders() map[string]string {
return map[string]string{
"x-ms-version": "2015-02-21",
"x-ms-date": currentTimeRfc1123Formatted(),
"Accept": "application/json;odata=nometadata",
"Accept-Charset": "UTF-8",
"Content-Type": "application/json",
}
}

// QueryTables returns the tables created in the
// *TableServiceClient storage account.
func (c *TableServiceClient) QueryTables() ([]AzureTable, error) {
uri := c.client.getEndpoint(tableServiceName, tablesURIPath, url.Values{})

headers := c.getStandardHeaders()
headers["Content-Length"] = "0"

resp, err := c.client.execTable("GET", uri, headers, nil)
if err != nil {
return nil, err
}
defer resp.body.Close()

if err := checkRespCode(resp.statusCode, []int{http.StatusOK}); err != nil {
return nil, err
}

buf := new(bytes.Buffer)
buf.ReadFrom(resp.body)

var respArray queryTablesResponse
if err := json.Unmarshal(buf.Bytes(), &respArray); err != nil {
return nil, err
}

s := make([]AzureTable, len(respArray.TableName))
for i, elem := range respArray.TableName {
s[i] = AzureTable(elem.TableName)
}

return s, nil
}

// CreateTable creates the table given the specific
// name. This function fails if the name is not compliant
// with the specification or the tables already exists.
func (c *TableServiceClient) CreateTable(table AzureTable) error {
uri := c.client.getEndpoint(tableServiceName, tablesURIPath, url.Values{})

headers := c.getStandardHeaders()

req := createTableRequest{TableName: string(table)}
buf := new(bytes.Buffer)

if err := json.NewEncoder(buf).Encode(req); err != nil {
return err
}

headers["Content-Length"] = fmt.Sprintf("%d", buf.Len())

resp, err := c.client.execTable("POST", uri, headers, buf)

if err != nil {
return err
}
defer resp.body.Close()

if err := checkRespCode(resp.statusCode, []int{http.StatusCreated}); err != nil {
return err
}

return nil
}

// DeleteTable deletes the table given the specific
// name. This function fails if the table is not present.
// Be advised: DeleteTable deletes all the entries
// that may be present.
func (c *TableServiceClient) DeleteTable(table AzureTable) error {
uri := c.client.getEndpoint(tableServiceName, tablesURIPath, url.Values{})
uri += fmt.Sprintf("('%s')", string(table))

headers := c.getStandardHeaders()

headers["Content-Length"] = "0"

resp, err := c.client.execTable("DELETE", uri, headers, nil)

if err != nil {
return err
}
defer resp.body.Close()

if err := checkRespCode(resp.statusCode, []int{http.StatusNoContent}); err != nil {
return err

}
return nil
}
Loading