Bitbucket private repos

[afragen]

@jazzsequence I think I've finally gotten this figured out. Please test latest develop branch.

[afragen]

I'm still having issues with this. I can get notifications but actually updating isn't quite working. I've got a couple more ideas to try.

[afragen]

@pdclark, I'm at a loss. There was one time, unless I'm being delusional, when I thought I actually got a private repo to update. I can't get it done now. I do have http_request_args passing Basic Authentication headers correctly.

[nadimtuhin]

You are doing a wonderful job there... I will be finding a gap and try to solve this problem with you :)

[afragen]

Thanks @nadimtuhin I can use the help on this one. I'm at an impasse.

[afragen]

https://bitbucket.org/site/master/issue/1087/private-repository-public-downloads-bb-733

Justen Stepka
changed status to wontfix
We have decided that we will not be implementing public downloads for repos with private source. Our goal is to build a product where folks collaborate around writing source code and not build a distribution platform for binary files. There are several low cost services that are optimized for distributing binary files, such as Amazon S3 or Rackspace which we recommend.
Respectfully, Justen -- Bitbucket product manager
6/14/2014

It looks as if support for Bitbucket private repos is not going to be possible. Sorry.

@jazzsequence @pdclark

[pdclark]

Looks like that just says that for a repo that requires authentication, the zip will require authentication as well. Doesn't mean updates for private repos aren't possible; just that they have to be downloaded with an authorized user account.

[afragen]

But Paul I think they mean a logged in user account, not just authorize. We are both using Basic Auth and it's not working.

[pdclark]

Like it's giving you an access denied error?

If that's the case, then it may just mean authorizing with oAuth, which then puts it in the same class as private Github repos — difficult, but possible.

[afragen]

I think the only way to do Bitbucket private repos is with oAuth. They don't make an access token like GitHub and it doesn't look like they're going too.

Creating a Bitbucket oAuth token might be a whole project to itself.

On another note, Paul it's great to hear from you.

[pdclark]

Nice to be here! Sorry my new job has had me under water for months. Just released three WordPress sites for ted.com.

Agreed that oAuth certainly isn't trivial. There do tend to be a lot of libraries that make it much easier. Not sure about the access token. I know I write a Github oAuth downloader at one time, but can't remember if the access token for downloads was Github-specific or part of the oAuth spec (which does provide something called an access token).

[afragen]

I've done a lot more stuff here. You should take another look.

I seem to remember your oAuth stuff being a PR to @jkudish's plugin. I'll try to find it again.

[afragen]

Found the oAuth code and it looks very specific to GitHub. 😦

[pdclark]

Yes, but the oAuth process is pretty much the same for any site. Mostly just the URLs and data format change.

[afragen]

Might be a good project to build an oAuth plugin and pass the secret key and token back to the caller.

[afragen]

Maybe using https://oAuth.io would somehow work?

[pdclark]

Looks like a great service!!

On Sunday, June 22, 2014 at 10:27 PM, Andy Fragen wrote:

Maybe using http://oAuth.io would somehow work?

—
Reply to this email directly or view it on GitHub (#59 (comment)).

[afragen]

Hey Paul! I discovered something very interesting while searching Google. The following are headers of downloaded private and public Bitbucket repos. https://core.trac.wordpress.org/ticket/22913

Private repository
Array
(
    [headers] => Array
        (
            [server] => nginx/1.5.10
            [date] => Mon, 23 Jun 2014 18:17:33 GMT
            [content-type] => text/html; charset=utf-8
            [connection] => close
            [x-served-by] => app16
            [x-render-time] => 0.0786230564117
            [content-language] => en
            [x-static-version] => 2668896b5f12
            [expires] => Mon, 23 Jun 2014 18:17:33 GMT
            [vary] => Cookie, Accept-Language
            [last-modified] => Mon, 23 Jun 2014 18:17:33 GMT
            [x-version] => 2668896b5f12
            [x-request-count] => 392
            [cache-control] => max-age=0
            [x-frame-options] => SAMEORIGIN
            [set-cookie] => csrftoken=RySijN7Dcz21hKlSFtDAQXjc0TdpI1lX; expires=Mon, 22-Jun-2015 18:17:33 GMT; Max-Age=31449600; Path=/; secure
            [strict-transport-security] => max-age=31536000
            [x-content-type-options] => nosniff
            [content-encoding] => gzip
        )
    [body] => 37745: 3c21444f435459504520
    [response] => Array
        (
            [code] => 200
            [message] => OK
        )
)
Public repository
Array
(
    [headers] => Array
        (
            [server] => nginx/1.5.10
            [date] => Mon, 23 Jun 2014 18:17:33 GMT
            [content-type] => application/zip; charset=utf-8
            [content-length] => 1448
            [connection] => close
            [last-modified] => Mon, 23 Jun 2014 18:02:38 GMT
            [cache-control] => max-age=0
            [expires] => Mon, 23 Jun 2014 18:17:33 GMT
            [content-disposition] => attachment; filename=afragen-test-bitbucket-plugin-8a9805ead42c.zip
            [etag] => "53a86bbe-5a8"
            [accept-ranges] => bytes
        )
    [body] => 1448: 504b03040a0000000000
    [response] => Array
        (
            [code] => 200
            [message] => OK
        )
)

I'm betting the content-type disparity and/or other headers are what is causing the issue. Is there some filter to try and change it?

[pdclark]

These look like response headers, rather than request headers. Response
headers are up to the responding server... And it does look like private is
sending you HTML, not a ZIP. You can filter outgoing headers, but response
is just information about what your'e getting.

Paul Clark
pdclark.com

On Mon, Jun 23, 2014 at 11:50 AM, Andy Fragen [email protected]
wrote:

Hey Paul! I discovered something very interesting while searching Google.
The following are headers of downloaded private and public Bitbucket repos.
https://core.trac.wordpress.org/ticket/22913

Private repository
Array
(
[headers] => Array
(
[server] => nginx/1.5.10
[date] => Mon, 23 Jun 2014 18:17:33 GMT
[content-type] => text/html; charset=utf-8
[connection] => close
[x-served-by] => app16
[x-render-time] => 0.0786230564117
[content-language] => en
[x-static-version] => 2668896b5f12
[expires] => Mon, 23 Jun 2014 18:17:33 GMT
[vary] => Cookie, Accept-Language
[last-modified] => Mon, 23 Jun 2014 18:17:33 GMT
[x-version] => 2668896b5f12
[x-request-count] => 392
[cache-control] => max-age=0
[x-frame-options] => SAMEORIGIN
[set-cookie] => csrftoken=RySijN7Dcz21hKlSFtDAQXjc0TdpI1lX; expires=Mon, 22-Jun-2015 18:17:33 GMT; Max-Age=31449600; Path=/; secure
[strict-transport-security] => max-age=31536000
[x-content-type-options] => nosniff
[content-encoding] => gzip
)

[body] => 37745: 3c21444f435459504520
[response] => Array
    (
        [code] => 200
        [message] => OK
    )

)

Public repository

Array
(
[headers] => Array
(
[server] => nginx/1.5.10
[date] => Mon, 23 Jun 2014 18:17:33 GMT
[content-type] => application/zip; charset=utf-8
[content-length] => 1448
[connection] => close
[last-modified] => Mon, 23 Jun 2014 18:02:38 GMT
[cache-control] => max-age=0
[expires] => Mon, 23 Jun 2014 18:17:33 GMT
[content-disposition] => attachment; filename=afragen-test-bitbucket-plugin-8a9805ead42c.zip
[etag] => "53a86bbe-5a8"
[accept-ranges] => bytes
)

[body] => 1448: 504b03040a0000000000
[response] => Array
    (
        [code] => 200
        [message] => OK
    )

)

I'm betting the content-type disparity and/or other headers are what is
causing the issue. Is there some filter to try and change it?

Reply to this email directly or view it on GitHub
#59 (comment)
.

[afragen]

So it looks like the issue is that Bitbucket is sending the file incorrectly. It does, however, download a zip file on the desktop.

I wonder if these response headers are causing the error. If so, how do we fix?

[ckpicker]

I wonder if some of the authentication code from this project could be used here?

https://bitbucket.org/alixandru/bitbucket-sync/overview

[afragen]

Not sure the authentication is the issue. When I get back from islands I'll see if changing wp_get_remote to file_get_contents or curl for the update DL makes a difference.

[hostz-frank]

It may help you investigating this WP plugin: http://wordpress.org/plugins/wp-bucket/ - it is also available on BB and has a great wiki page: https://bitbucket.org/khosroblog/wp_bucket/wiki/Home

So this statement, while using the plugin's classes and after setup and confirmation of an OAuth consumer key, will return a zip file of a private master branch:
$WP_Bucket->download_stream("my_repo_slug", "my_username", "master");

Good luck!

[hostz-frank]

Sorry about my previous posting - the plugin won't help probably. OAuth always needs interaction with the requesting user... So I had to switch to GH meanwhile.

[afragen]

@hostz-frank it was an interesting plugin though I was reaching he same conclusion. Too bad.

[hostz-frank]

I was also looking for a plugin that could even update WP plugins and themes from Gitlab - but no luck.

Since all these platforms are accessible via Git itself - isn't it a reasonable way to think in another direction and let Git itself do the work (without any OAuth and API's) ... :-)

[afragen]

@hostz-frank I believe there's another issue re:Gitlab integration. Can we move this discussion there? I could also use some help on that as I don't use Gitlab.

[coderaaron]

Thanks for all your work on this, I was elated when I saw you'd figured this out.

I'm running into a problem where repo_meta isn't getting populated so the password option isn't being added to the settings page. I think it's due to the fact that BitBucket is returning a 401 status for $response = wp_remote_get( $this->get_api_url( $url ) ); on line 92 in class-bitbucket-api.php and thus we fail if ( $repo_api->get_remote_info( 'style.css' ) ) { on line 62 in class-theme-updater.php and $repo_api->get_repo_meta(); doesn't get called.

I can dump all my debug info or add you to the private repo, if either of those would help.

Thanks again!

[afragen]

@coderaaron let me think about how to ensure do that. I've spent a lot of time trying to make sure the extra stuff doesn't show up in settings page. I missed the part about what happens the first time a private repo is added.

[afragen]

@coderaaron update to latest develop branch and let me know if it works. I've set it up so it should display in the Settings until a password is entered if the Bitbucket repo is private. Thanks as I had to rework it for GitHub too.

Let me know if it's working.

[haleeben]

@afragen I've updated to version 3.0.7.9, and it got rid of the no private repo message on the settings page. I've added my bitbucket password and saved on the settings page, but I've gone into the database and see the github-update value is "a:1:{s:14:"github-updater";N;}".

[afragen]

I just pushed 3.0.7.10 which, I think, fixed an odd problem where update_site_option wasn't working.

You should be looking for database value github_updater.

I've been testing locally with v3.0.7.10 and it's working great. If anyone is having issues with this version please let me know. Also, tell me if it's working for you.

[haleeben]

Updated but still seeing no change in the database after saving settings :(

[afragen]

Can you email me a zip of the plugin?

[afragen]

@haleeben if you're testing on a multisite install the github_updater data is in wp_sitemeta on a single install the site options are located in wp_options

[afragen]

Sorry everyone. I just pushed to develop so that settings on single site installs should properly save settings.

[haleeben]

Thanks @afragen , working now...........awesome.

[afragen]

@haleeben thanks for confirming. This version was tagged and released.

[zachabernathy]

I tested this and it is working great!

Maybe this isn't the right issue to post this on, but is there any way to encode the password so that if someone gains access to the admin they don't have the keys to the kingdom for our Bitbucket account?

[afragen]

@zachthezman a new issue might be a better place, but the short answer is if we encode the password we have to decode password and all that coding/decoding will be present in the plugin. This would mean that any user would be able to discern the password. The plugin needs to have the access to the password.

I would recommend setting up individual accounts for each private repo you wish to mass distribute. Not many other ways around it.

[afragen]

Hey everyone, I've got a better idea that's now on develop. Now in the Settings page you enter your personal Bitbucket user/pass and check the private repositories. If you don't own the private repo you must have at least read privileges to authenticate. Now you can add clients to your private repos with read-only access and not expose your credentials to them.

[carlgrundberg]

Awsome, just tried out your plugin and was just about to ask about support for organization repos in bitbucket. The new settings with username / password works, thanks!

[afragen]

@carlgrundberg thanks for testing this. Out of curiosity, what privileges do your organization members hold?

[abrudtkuhl]

I was able to add a private Bitbucket repo owned by our organization. However when I push an update to Bitbucket and go to check the plugin - it never sees the updated version is available.

Another issue (and probably should be opened as another issue) is that it is not remembering I selected "private repository" on installation. I am having to go back in and re-check that after the authentication fails when re-loading the plugin page.

I am running version 4.2.0

[afragen]

Remote installation and updating occur through different mechanisms. The Settings page must be updated after a remote install of a private repo.

Did that solve your issue?

[abrudtkuhl]

no not solved. The main issue im having is that when i update a plugin, i
dont see the update available in the plugins page on WP.

i just thought it was weird i had to check "private repository" twice when
the second one should know i've already checked the first one.

Andy Brudtkuhl
@abrudtkuhl http://twitter.com/abrudtkuhl

On Fri, Mar 6, 2015 at 3:23 PM, Andy Fragen [email protected]
wrote:

Remote installation and updating occur through different mechanisms. The
Settings page must be updated after a remote install of a private repo.

Did that solve your issue?

—
Reply to this email directly or view it on GitHub
#59 (comment)
.

[afragen]

no not solved. The main issue im having is that when i update a plugin, i
dont see the update available in the plugins page on WP.

Is your Settings page correct?

i just thought it was weird i had to check "private repository" twice when
the second one should know i've already checked the first one.

The issue is that the private repo won't be listed in the Settings page until it is installed. There's no place to put the setting for a private repo because it didn't exist prior to installing the app.

I'll consider it a feature request. Maybe I can figure something out.

[afragen]

i just thought it was weird i had to check "private repository" twice when
the second one should know i've already checked the first one.

The issue is that the private repo won't be listed in the Settings page until it is installed. There's no place to put the setting for a private repo because it didn't exist prior to installing the app.

I'll consider it a feature request. Maybe I can figure something out.

86ac601

[afragen]

no not solved. The main issue im having is that when i update a plugin, i
dont see the update available in the plugins page on WP.

You might have to clear the transients by clicking Check Again on the update-core.php page.