Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kube play target does not work with private repositories #281

Closed
alexanderniebuhr opened this issue Dec 20, 2022 · 9 comments
Closed

kube play target does not work with private repositories #281

alexanderniebuhr opened this issue Dec 20, 2022 · 9 comments

Comments

@alexanderniebuhr
Copy link

pretty sure I followed the docs. The pat works for configReload but not for kube plays. Some logs:

2022-12-20T22:15:22.726Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:15:23.001Z        INFO    engine/config.go:208    Current config backup placed at /opt/mount/config-backup.yaml
2022-12-20T22:15:23.001Z        INFO    engine/config.go:215    Config updates found from url: https://raw.githubusercontent.com/alexanderniebuhr/lab/main/fetchit/config.yaml, will load new targets
2022-12-20T22:15:23.001Z        INFO    engine/config.go:63     Updated config processed, restarting with new targets
2022-12-20T22:15:23.006Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:15:23.308Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:15:23.308Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:20:00.007Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:20:00.478Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:20:00.478Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:20:00.625Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:26:02.836Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:26:03.167Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:26:03.167Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:26:03.167Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:26:03.302Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:28:42.383Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:28:42.640Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:28:42.640Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:28:42.641Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:28:42.771Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:28:42.829Z        INFO    engine/config.go:208    Current config backup placed at /opt/mount/config-backup.yaml
2022-12-20T22:28:42.830Z        INFO    engine/config.go:215    Config updates found from url: https://raw.githubusercontent.com/alexanderniebuhr/lab/main/fetchit/config.yaml, will load new targets
2022-12-20T22:28:42.830Z        INFO    engine/config.go:63     Updated config processed, restarting with new targets
2022-12-20T22:28:42.836Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:28:42.953Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:28:42.953Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
@cooktheryan cooktheryan mentioned this issue Dec 21, 2022
Merged
@cooktheryan
Copy link
Collaborator

@alexanderniebuhr let me test this tomorrow. This may be a documentation issue

@cooktheryan
Copy link
Collaborator

Can you try to use the example file examples/pat-testing-kube.yaml

Here is the output from my system

rcook  @ ~/go/src/github.com/redhat-et/fetchit
pat-kube-check└─ $ podman run -d --rm --name fetchit     -v fetchit-volume:/opt     -v $HOME/.fetchit:/opt/mount     -v /run/user/$(id -u)/podman//podman.sock:/run/podman/podman.sock     --security-opt label=disable     quay.io/fetchit/fetchit:latest
Trying to pull quay.io/fetchit/fetchit:latest...
Getting image source signatures
Copying blob 455cd3476656 done  
Copying blob eee1869219e1 done  
Copying blob 3e7f57f156fa done  
Copying blob c9ba6b2d6c27 done  
Copying config c1dbddc792 done  
Writing manifest to image destination
Storing signatures
22d1e3b57b4cb2e6edfa1822dacaf173c0e54e13b769a1040c1d1eef710cf461
rcook  @ ~/go/src/github.com/redhat-et/fetchit
pat-kube-check└─ $ podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-22T14:39:26.888Z	INFO	engine/fetchit.go:401	git clone https://github.com/containers/fetchit main --recursive
2022-12-22T14:39:31.475Z	INFO	engine/fetchit.go:365	Processing git target: https://github.com/containers/fetchit Method: kube Name: kube-ex
2022-12-22T14:39:31.612Z	INFO	engine/kube.go:90	Creating podman container from fetchit/examples/kube/1-pvc.yaml using kube method
2022-12-22T14:39:31.654Z	INFO	engine/kube.go:160	Created pods from spec in fetchit/examples/kube/1-pvc.yaml
2022-12-22T14:39:31.654Z	INFO	engine/kube.go:90	Creating podman container from fetchit/examples/kube/2-example.yaml using kube method
2022-12-22T14:39:38.441Z	INFO	engine/kube.go:160	Created pods from spec in fetchit/examples/kube/2-example.yaml
2022-12-22T14:39:38.441Z	INFO	engine/kube.go:90	Creating podman container from fetchit/examples/kube/3-example.yaml using kube method
rcook  @ ~/go/src/github.com/redhat-et/fetchit
pat-kube-check└─ $ cat /home/rcook/.fetchit/config.yaml 
# for this test, start with this config, then wait to be sure the
# targetConfigs from .fetchit/config.yaml are populated
# and for follow-up test, push a change to the config and confirm
# new targetConfigs are fetched & run
targetConfigs:
- url: https://github.com/containers/fetchit
  pat: MYSECRETPAT
  kube:
  - name: kube-ex
    targetPath: examples/kube
    schedule: "*/1 * * * *"
  branch: main

@alexanderniebuhr
Copy link
Author

alexanderniebuhr commented Dec 22, 2022 •
edited
Loading

@cooktheryan, I tried this and it does not work for private repositories in my case. (I tested it with the repo set to public for just a minute and it gave another error). I think it does not work for private repositories, because the fetch for commit data is called without pat. Therefore the url to the repo will return 404, because the repo is private.

private repo

[root@applications ~]# cat .fetchit/config.yaml 
targetConfigs:
- url: https://github.com/alexanderniebuhr/lab
  pat: ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXX
  kube:
  - name: kube-ex
    targetPath: apps/home-assistant.yaml
    schedule: "*/1 * * * *"
  branch: main
[root@applications ~]# podman run -d --rm --name fetchit -v fetchit-volume:/opt -v $HOME/.fetchit:/opt/mount -v /run/podman/podman.sock:/run/podman/podman.sock --privileged quay.io/fetchit/fetchit:latest
b10bf00918c1fef67f96c9148859cdaf24f879d13593762221ec84dbc8e40a0a
[root@applications ~]# podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-22T16:25:45.183Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:25:45.391Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: kube-ex
2022-12-22T16:25:45.391Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:25:45.515Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-22T16:26:00.004Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:26:00.145Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist

public repo

[root@applications ~]# podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-22T16:28:55.960Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:28:56.424Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: kube-ex
2022-12-22T16:28:56.564Z        ERROR   engine/kube.go:62       Error moving current to latest: Failed to apply changes: Error getting tree from hash ae71047c3889ea00b58f183e8b9feef6ce31aa16: Error getting sub tree at apps/home-assistant.yaml from commit at ae71047c3889ea00b58f183e8b9feef6ce31aa16 from repository lab: directory not found
2022-12-22T16:29:00.173Z        ERROR   engine/kube.go:62       Error moving current to latest: Failed to apply changes: Error getting tree from hash ae71047c3889ea00b58f183e8b9feef6ce31aa16: Error getting sub tree at apps/home-assistant.yaml from commit at ae71047c3889ea00b58f183e8b9feef6ce31aa16 from repository lab: directory not found

@alexanderniebuhr
Copy link
Author

Interestingly the PAT Token is not used. I am confused?

image

@cooktheryan
Copy link
Collaborator

There was an error in the docs and the test case after a merge happened that changed the structure. I pushed a fix #283 sorry about that. Ill try to look into the networking piece before break

@alexanderniebuhr
Copy link
Author

@cooktheryan is it supposed to work with the changed syntax and private repositories. I still can't fetch any commit data if the repository is private?

@cooktheryan
Copy link
Collaborator

Was just looking at it and ran a test locally. So I had classic token without repo access and I got the following.

podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-26T20:27:39.254Z	INFO	engine/fetchit.go:401	git clone https://github.com/cooktheryan/aap-rhacm main --recursive
2022-12-26T20:27:39.468Z	INFO	engine/fetchit.go:365	Processing git target: https://github.com/cooktheryan/aap-rhacm Method: kube Name: kube-ex
2022-12-26T20:27:39.468Z	INFO	engine/fetchit.go:401	git clone https://github.com/cooktheryan/aap-rhacm main --recursive
2022-12-26T20:27:39.538Z	ERROR	engine/kube.go:55	Error moving to current: Failed to get current commit: Error opening repository aap-rhacm to fetch current commit: repository does not exist

Then I enabled repo access for the PAT and then things cleared up

2022-12-26T20:28:00.010Z	INFO	engine/fetchit.go:401	git clone https://github.com/cooktheryan/aap-rhacm main --recursive
2022-12-26T20:28:00.417Z	INFO	engine/kube.go:90	Creating podman container from aap-rhacm/examples/kube/1-pvc.yaml using kube method
2022-12-26T20:28:00.527Z	INFO	engine/kube.go:160	Created pods from spec in aap-rhacm/examples/kube/1-pvc.yaml
2022-12-26T20:28:00.527Z	INFO	engine/kube.go:90	Creating podman container from aap-rhacm/examples/kube/2-example.yaml using kube method
2022-12-26T20:28:01.597Z	INFO	engine/kube.go:160	Created pods from spec in aap-rhacm/examples/kube/2-example.yaml
2022-12-26T20:28:01.597Z	INFO	engine/kube.go:90	Creating podman container from aap-rhacm/examples/kube/3-example.yaml using kube method
2022-12-26T20:28:03.500Z	INFO	engine/kube.go:160	Created pods from spec in aap-rhacm/examples/kube/3-example.yaml
2022-12-26T20:28:03.501Z	INFO	engine/common.go:97	Moved kube-ex from 000000000 to 5bb7fd06bc1279a68ec0137a283a544fab064113 for git target https://github.com/cooktheryan/aap-rhacm



2022-12-26T20:29:00.129Z	INFO	engine/common.go:99	No changes applied to git target aap-rhacm this run, kube currently at 5bb7fd06b
2022-12-26T20:30:00.131Z	INFO	engine/common.go:99	No changes applied to git target aap-rhacm this run, kube currently at 5bb7fd06b

image

@cooktheryan cooktheryan reopened this Dec 26, 2022
@cooktheryan
Copy link
Collaborator

If we can't get this worked out by the 3 Jan I would be willing to jump on a call. I think this could benefit the HA community

@alexanderniebuhr
Copy link
Author

alexanderniebuhr commented Dec 27, 2022 •
edited
Loading

Okay just regenerated a new token, now it works.. Even if the old token had all the permissions, as you could see in the screenshot. Maybe it was generated too long ago. Now I just need #280, to be able to use this for everything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cooktheryan @alexanderniebuhr