Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SharingNG: initial OCM support #8909

Merged
merged 2 commits into from
Jun 28, 2024
Merged

SharingNG: initial OCM support #8909

merged 2 commits into from
Jun 28, 2024

Conversation

butonic
Copy link
Member

@butonic butonic commented Apr 19, 2024

I just wanted to fix the CI for cs3org/reva#4630, but to debug I had to make graph support OCM so that I could use web to actually send invites, create shares and browse them ...

This is rudimentary, only supports creating and accepting shares. It is a start ...

To test this you have to set the necessary OCM flags:

					// open cloud mesh
					"FRONTEND_OCS_INCLUDE_OCM_SHAREES": "true",
					"FRONTEND_OCS_LIST_OCM_SHARES": "true",
					"FRONTEND_ENABLE_FEDERATED_SHARING_INCOMING": "true",
					"FRONTEND_ENABLE_FEDERATED_SHARING_OUTGOING": "true",
					"OCIS_ADD_RUN_SERVICES": "ocm",
					"OCM_OCM_PROVIDER_AUTHORIZER_PROVIDERS_FILE": "/path/to/ocis/ocmproviders.json",
					"GRAPH_INCLUDE_OCM_SHAREES": "true",
					"OCM_OCM_INVITE_MANAGER_INSECURE": "true",
					"OCM_OCM_SHARE_PROVIDER_INSECURE": "true",
					"OCM_OCM_STORAGE_PROVIDER_INSECURE": "true",

enable the ocm web app

{
    "server": "https://cloud.ocis.test",
    "theme": "https://cloud.ocis.test/themes/junidrive/theme.json",
    "openIdConnect": {
        "metadata_url": "https://cloud.ocis.test/.well-known/openid-configuration",
        "authority": "https://cloud.ocis.test",
        "client_id": "web",
        "response_type": "code",
        "scope": "openid profile email",
        "loadUserInfo": false
    },
    "apps": [
        "files",
        "search",
        "text-editor",
        "pdf-viewer",
        "external",
        "admin-settings",
        "ocm"                               // <--- this needs to be added
    ],
    "external_apps": [
        {
            "id": "preview",
            "path": "web-app-preview",
            "config": {
                "mimeTypes": [
                    "image/tiff",
                    "image/bmp",
                    "image/x-ms-bmp"
                ]
            }
        }
    ],
    "options": {
        "previewFileMimeTypes": [
            "image/gif",
            "image/png",
            "image/jpeg",
            "text/plain",
            "image/tiff",
            "image/bmp",
            "image/x-ms-bmp"
        ],
        "sharingRecipientsPerPage": 200,
        "sidebar": {
            "shares": {
                "showAllOnLoad": false
            }
        },
        "routing": {
            "idBased": true
        },
        "contextHelpersReadMore": true,
        "openLinksWithDefaultApp": true,
        "tokenStorageLocal": true
    }
}

and finally configure two ocm providers in the /path/to/ocis/ocmproviders.json file:

[
    {
        "name": "oCIS Test",
        "full_name": "oCIS Test provider",
        "organization": "oCIS",
        "domain": "cloud.ocis.test",
        "homepage": "https://ocis.test",
		"description": "oCIS Example cloud storage",
        "services": [
            {
                "endpoint": {
                    "type": {
                        "name": "OCM",
                        "description": "cloud.ocis.test Open Cloud Mesh API"
                    },
                    "name": "cloud.ocis.test - OCM API",
                    "path": "https://cloud.ocis.test/ocm/",
                    "is_monitored": true
                },
                "api_version": "0.0.1",
                "host": "http://cloud.ocis.test"
            },
			{
				"endpoint": {
					"type": {
						"name": "Webdav",
						"description": "cloud.ocis.test Webdav API"
					},
					"name": "cloud.ocis.test Example - Webdav API",
					"path": "https://cloud.ocis.test/dav/",
					"is_monitored": true
				},
				"api_version": "0.0.1",
				"host": "https://cloud.ocis.test/"
			}
        ]
    },
	{
        "name": "ownCloud Test",
        "full_name": "ownCloud Test provider",
        "organization": "ownCloud",
        "domain": "cloud.owncloud.test",
        "homepage": "https://owncloud.test",
		"description": "ownCloud Example cloud storage",
		"services": [
            {
                "endpoint": {
                    "type": {
                        "name": "OCM",
                        "description": "cloud.owncloud.test Open Cloud Mesh API"
                    },
                    "name": "cloud.owncloud.test - OCM API",
                    "path": "https://cloud.owncloud.test/ocm/",
                    "is_monitored": true
                },
                "api_version": "0.0.1",
                "host": "http://cloud.owncloud.test"
            },
			{
				"endpoint": {
					"type": {
						"name": "Webdav",
						"description": "cloud.owncloud.test Webdav API"
					},
					"name": "cloud.owncloud.test Example - Webdav API",
					"path": "https://cloud.owncloud.test/dav/",
					"is_monitored": true
				},
				"api_version": "0.0.1",
				"host": "https://cloud.owncloud.test/"
			}
		]
	}
]

copy this proiders file to both instances.

note: you MUST use two routable domains. localhost will only work if both instances are running on the same host on different ports.

This is a vscode config to run an additional instance with all the ports configured to use a different port than the default as well as a different registry:

			{
				"name": "cloud.owncloud.test",
				"type": "go",
				"request": "launch",
				//"buildFlags": "-race",
				"mode": "debug",
				"program": "${workspaceFolder:ocis}/ocis/cmd/ocis",
				"args": [
					"server"
				],
				"env": {

					"OCIS_RUNTIME_PORT": "10250",
					"MICRO_REGISTRY_ADDRESS": "127.0.0.1:10233",
					"OCIS_EVENTS_ENDPOINT": "127.0.0.1:10233",
					"PROXY_HTTP_ADDR": "0.0.0.0:10200",
					"OCIS_CACHE_STORE_NODES": "127.0.0.1:10233",
					"OCIS_PERSISTENT_STORE_NODES": "127.0.0.1:10233",
					"OCIS_LDAP_URI": "ldaps://localhost:10235",
					"OCIS_URL": "https://cloud.owncloud.test",
					"OCIS_BASE_DATA_PATH": "/tmp/ocis",

					"ANTIVIRUS_DEBUG_ADDR": "127.0.0.1:10277",
					"APP_PROVIDER_DEBUG_ADDR": "127.0.0.1:10165",
					"APP_PROVIDER_GRPC_ADDR": "127.0.0.1:10164",
					"APP_REGISTRY_DEBUG_ADDR":   "127.0.0.1:10243",
					"APP_REGISTRY_GRPC_ADDR":      "127.0.0.1:10242",
					"AUDIT_DEBUG_ADDR":   "127.0.0.1:10229",
					//"AUDIT_EVENTS_ENDPOINT":  "127.0.0.1:10233",
					"AUTH_BASIC_DEBUG_ADDR":   "127.0.0.1:10147",
					"AUTH_BASIC_GRPC_ADDR":      "127.0.0.1:10146",
					"AUTH_BEARER_DEBUG_ADDR":   "127.0.0.1:10149",
					"AUTH_BEARER_GRPC_ADDR":      "127.0.0.1:10148",
					"AUTH_MACHINE_DEBUG_ADDR":   "127.0.0.1:10167",
					"AUTH_MACHINE_GRPC_ADDR":      "127.0.0.1:10166",
					"AUTH_SERVICE_DEBUG_ADDR":   "127.0.0.1:10198",
					"AUTH_SERVICE_GRPC_ADDR":      "127.0.0.1:10199",
					"CLIENTLOG_DEBUG_ADDR":   "127.0.0.1:10260",
					//"CLIENTLOG_EVENTS_ENDPOINT":  "127.0.0.1:10233",
					"EVENTHISTORY_DEBUG_ADDR":   "127.0.0.1:10270",
					//"EVENTHISTORY_EVENTS_ENDPOINT":  "127.0.0.1:10233",
					//"EVENTHISTORY_GRPC_ADDR":      "127.0.0.1:0",
					"FRONTEND_DEBUG_ADDR": "127.0.0.1:10141",
					"FRONTEND_HTTP_ADDR": "127.0.0.1:10140",
					//"FRONTEND_EVENTS_ENDPOINT": "127.0.0.1:10233",
					"GATEWAY_DEBUG_ADDR":   "127.0.0.1:10143",
					"GATEWAY_GRPC_ADDR":      "127.0.0.1:10142",
					"GRAPH_DEBUG_ADDR": "127.0.0.1:10124",
					"GRAPH_HTTP_ADDR": "127.0.0.1:10120",
					//"GRAPH_LDAP_URI": "ldaps://localhost:10235", -> OCIS_LDAP_URI
					//"GRAPH_EVENTS_ENDPOINT":  "127.0.0.1:10233",
					"GROUPS_DEBUG_ADDR":   "127.0.0.1:10161",
					"GROUPS_GRPC_ADDR":      "127.0.0.1:10160",
					//"GROUPS_LDAP_URI": "ldaps://localhost:10235", -> OCIS_LDAP_URI
					"IDM_DEBUG_ADDR":   "127.0.0.1:10239",
					"IDM_LDAPS_ADDR":    "127.0.0.1:10235",
					"IDP_DEBUG_ADDR": "127.0.0.1:10134",
					"IDP_HTTP_ADDR": "127.0.0.1:10130",
					//"IDP_LDAP_URI": "ldaps://localhost:10235", -> OCIS_LDAP_URI
					//"INVITATIONS_DEBUG_ADDR": "127.0.0.1:0",
					//"INVITATIONS_HTTP_ADDR":"127.0.0.1:0",
					"NATS_DEBUG_ADDR": "127.0.0.1:10234",
					"NATS_NATS_PORT": "10233",
					"NOTIFICATIONS_DEBUG_ADDR":   "127.0.0.1:10174",
					//"NOTIFICATIONS_EVENTS_ENDPOINT":  "127.0.0.1:10233",
					"OCDAV_DEBUG_ADDR": "127.0.0.1:10163",
					//"OCDAV_HTTP_ADDR": "127.0.0.1:0",
					"OCM_DEBUG_ADDR":   "127.0.0.1:10281",
					"OCM_HTTP_ADDR":      "127.0.0.1:10280",
					"OCM_GRPC_ADDR":      "127.0.0.1:10282",
					"OCS_DEBUG_ADDR":   "127.0.0.1:10114",
					"OCS_HTTP_ADDR":      "127.0.0.1:10110",
					//"OCS_PRESIGNEDURL_SIGNING_KEYS_STORE_NODES": "127.0.0.1:10233",
					"POLICIES_DEBUG_ADDR":   "127.0.0.1:10129",
					"POLICIES_GRPC_ADDR":      "127.0.0.1:10125",
					//"POLICIES_EVENTS_ENDPOINT":  "127.0.0.1:10233",
					"POSTPROCESSING_DEBUG_ADDR":   "127.0.0.1:10255",
					//"POSTPROCESSING_EVENTS_ENDPOINT": "127.0.0.1:10233",
					//"POSTPROCESSING_STORE_NODES":    "127.0.0.1:10233",
					"PROXY_DEBUG_ADDR": "0.0.0.0:10205",
					//PROXY_OIDC_USERINFO_CACHE_STORE_NODES
					"SEARCH_DEBUG_ADDR":  "127.0.0.1:10224",
					"SEARCH_GRPC_ADDR":      "127.0.0.1:10220",
					// SEARCH_EVENTS_ENDPOINT
					// SEARCH_HTTP_ADDR does not exists, but grep found it? can be deleted
					"SETTINGS_DEBUG_ADDR": "127.0.0.1:10194",
					"SETTINGS_HTTP_ADDR": "127.0.0.1:10190",
					"SETTINGS_GRPC_ADDR":      "127.0.0.1:10191",
					// SETTINGS_CACHE_STORE_NODES
					"SHARING_DEBUG_ADDR":   "127.0.0.1:10151",
					"SHARING_GRPC_ADDR":      "127.0.0.1:10150",
					"SHARING_EVENTS_ENDPOINT":      "127.0.0.1:10233",
					"SSE_DEBUG_ADDR":  "127.0.0.1:10135",
					//"SSE_HTTP_ADDR":      "127.0.0.1:0",
					// SSE_EVENTS_ENDPOINT
					"STORAGE_PUBLICLINK_DEBUG_ADDR":   "127.0.0.1:10179",
					"STORAGE_PUBLICLINK_GRPC_ADDR":      "127.0.0.1:10178",
					"STORAGE_SHARES_DEBUG_ADDR":   "127.0.0.1:10156",
					"STORAGE_SHARES_GRPC_ADDR":      "127.0.0.1:10154",
					"STORAGE_SYSTEM_DEBUG_ADDR":   "127.0.0.1:10217",
					"STORAGE_SYSTEM_GRPC_ADDR":      "127.0.0.1:10215",
					"STORAGE_SYSTEM_HTTP_ADDR":      "127.0.0.1:10216",
					"STORAGE_SYSTEM_DATA_SERVER_URL":  "http://localhost:10216/data",
					// STORAGE_SYSTEM_CACHE_STORE_NODES
					"STORAGE_USERS_DEBUG_ADDR":   "127.0.0.1:10159",
					"STORAGE_USERS_GRPC_ADDR":      "127.0.0.1:10157",
					"STORAGE_USERS_HTTP_ADDR":      "127.0.0.1:10158",
					"STORAGE_USERS_DATA_SERVER_URL":           "http://localhost:10158/data",
					"STORAGE_USERS_DATA_GATEWAY_URL":          "https://localhost:10200/data",
					"STORAGE_USERS_GATEWAY_GRPC_ADDR":     "127.0.0.1:10142",
					//STORAGE_USERS_EVENTS_ENDPOINT
					//STORAGE_USERS_FILEMETADATA_CACHE_STORE_NODES
					//STORAGE_USERS_ID_CACHE_STORE_NODES			
					"THUMBNAILS_DEBUG_ADDR": "127.0.0.1:10189",
					"THUMBNAILS_GRPC_ADDR": "127.0.0.1:10185",
					"THUMBNAILS_HTTP_ADDR": "127.0.0.1:10186",
					"THUMBNAILS_DATA_ENDPOINT":        "http://127.0.0.1:10186/thumbnails/data",
					"USERLOG_DEBUG_ADDR":   "127.0.0.1:10210",
					// "USERLOG_EVENTS_ENDPOINT":  "127.0.0.1:10233", -> OCIS_EVENTS_ENDPOINT
					// "USERLOG_HTTP_ADDR":      "127.0.0.1:0", 
					"USERS_DEBUG_ADDR":   "127.0.0.1:10145",
					"USERS_GRPC_ADDR":      "127.0.0.1:10144",
					//"USERS_LDAP_URI":                      "ldaps://localhost:10235", -> OCIS_LDAP_URI
					// USERS_IDP_URL:                      "https://localhost:10200", -> OCIS_URL;OCIS_OIDC_ISSUER
					"WEB_DEBUG_ADDR": "127.0.0.1:10104",
					"WEB_HTTP_ADDR": "127.0.0.1:10100",
					"WEBDAV_DEBUG_ADDR": "127.0.0.1:10119",
					"WEBDAV_HTTP_ADDR": "127.0.0.1:10115",
					// OCIS_PUBLIC_URL FIXME has no WEBDAV_ variable???
					// WEBFINGER_*_ADDR defaults to "127.0.0.1:0", picking a random port
					
					// open cloud mesh
					"FRONTEND_OCS_INCLUDE_OCM_SHAREES": "true",
					"FRONTEND_OCS_LIST_OCM_SHARES": "true",
					"FRONTEND_ENABLE_FEDERATED_SHARING_INCOMING": "true",
					"FRONTEND_ENABLE_FEDERATED_SHARING_OUTGOING": "true",
					//"OCIS_ADD_RUN_SERVICES": "ocm",
					"OCM_OCM_PROVIDER_AUTHORIZER_PROVIDERS_FILE": "/home/jfd/Repositories/ocis/ocmproviders.json",
					"GRAPH_INCLUDE_OCM_SHAREES": "true",
					"OCM_OCM_INVITE_MANAGER_INSECURE": "true",
					"OCM_OCM_SHARE_PROVIDER_INSECURE": "true",
					"OCM_OCM_STORAGE_PROVIDER_INSECURE": "true",


					"IDP_ACCESS_TOKEN_EXPIRATION": "3600",
					"OCIS_ASYNC_UPLOADS": "true",
					"OCIS_SHARING_PUBLIC_SHARE_MUST_HAVE_PASSWORD": "false",
					"OCIS_LOG_LEVEL": "error",
					"OCIS_LOG_PRETTY": "true",
					"OCIS_LOG_COLOR": "true",
					// set insecure options because we don't have valid certificates in dev environments
					"OCIS_INSECURE": "true",
					// enable basic auth for dev setup so that we can use curl for testing
					"PROXY_ENABLE_BASIC_AUTH": "true",
					// demo users
					"IDM_CREATE_DEMO_USERS": "true",	
					/*
					 * Keep secrets and passwords in one block to allow easy uncommenting
					 */
					// user id of "admin", for user creation and admin role assignement
					"OCIS_ADMIN_USER_ID": "some-admin-user-id-0000-000000000000", // FIXME currently must have the length of a UUID, see reva/pkg/storage/utils/decomposedfs/spaces.go:228
					// admin user default password
					"IDM_ADMIN_PASSWORD": "admin",
					// system user
					"OCIS_SYSTEM_USER_ID": "some-system-user-id-000-000000000000", // FIXME currently must have the length of a UUID, see reva/pkg/storage/utils/decomposedfs/spaces.go:228
					"OCIS_SYSTEM_USER_API_KEY": "some-system-user-machine-auth-api-key",
					// set some hardcoded secrets
					"OCIS_JWT_SECRET": "some-ocis-jwt-secret",
					"OCIS_MACHINE_AUTH_API_KEY": "some-ocis-machine-auth-api-key",
					"OCIS_TRANSFER_SECRET": "some-ocis-transfer-secret",
					// idm ldap
					"IDM_SVC_PASSWORD": "some-ldap-idm-password",
					"GRAPH_LDAP_BIND_PASSWORD": "some-ldap-idm-password",
					// reva ldap
					"IDM_REVASVC_PASSWORD": "some-ldap-reva-password",
					"GROUPS_LDAP_BIND_PASSWORD": "some-ldap-reva-password",
					"USERS_LDAP_BIND_PASSWORD": "some-ldap-reva-password",
					"AUTH_BASIC_LDAP_BIND_PASSWORD": "some-ldap-reva-password",
					// idp ldap
					"IDM_IDPSVC_PASSWORD": "some-ldap-idp-password",
					"IDP_LDAP_BIND_PASSWORD": "some-ldap-idp-password",
					// storage users mount ID
					"GATEWAY_STORAGE_USERS_MOUNT_ID": "storage-users-2",
					"STORAGE_USERS_MOUNT_ID": "storage-users-2",
					// graph application ID
					"GRAPH_APPLICATION_ID": "application-2",

					// service accounts
					"OCIS_SERVICE_ACCOUNT_ID": "service-account-id",
					"OCIS_SERVICE_ACCOUNT_SECRET": "service-account-secret",

				}
			},

Copy link

update-docs bot commented Apr 19, 2024

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

@butonic butonic changed the title SharingNG: OCM support [WIP] SharingNG: OCM support Apr 19, 2024
@butonic butonic force-pushed the graph-ocm-fixes branch 2 times, most recently from d98e8d4 to 4e973f4 Compare April 26, 2024 10:52
@butonic butonic mentioned this pull request Apr 26, 2024
@butonic butonic changed the title [WIP] SharingNG: OCM support SharingNG: initial OCM support Apr 26, 2024
@butonic butonic marked this pull request as ready for review April 26, 2024 15:24
@butonic butonic self-assigned this Apr 26, 2024
@butonic butonic added the Category:Enhancement Add new functionality label Apr 26, 2024
Copy link

@butonic butonic force-pushed the graph-ocm-fixes branch 5 times, most recently from 56a3c21 to 8808fb0 Compare June 3, 2024 13:48
@ScharfViktor ScharfViktor mentioned this pull request Jun 6, 2024
2 tasks
@ScharfViktor
Copy link
Contributor

Steps:

  • run 2 ocis instance
  • in the ocis1 choose scienceMesh in the app switcher
  • generate invitation and copy invite token
  • in the ocis2 enter invite token and click accept invitation

Actual: 401 error. invitation not accepted
Screenshot 2024-06-06 at 15 43 29

log:
2024-06-06T15:34:14+02:00 ERR provider not trusted error="error verifying mesh provider" pkg=rhttp request-id=oC-performance-test-INTEL/K5kV6ZCX3j-000588 service=ocm traceid=de4cb63966b1b4745ff9ca5a642b0273

@ScharfViktor
Copy link
Contributor

Steps:

  • run 2 ocis instance
  • in the ocis1 choose scienceMesh in the app switcher
  • generate invitation and copy invite token
  • in the ocis2 enter invite token and click accept invitation

Actual: 401 error. invitation not accepted Screenshot 2024-06-06 at 15 43 29

log: 2024-06-06T15:34:14+02:00 ERR provider not trusted error="error verifying mesh provider" pkg=rhttp request-id=oC-performance-test-INTEL/K5kV6ZCX3j-000588 service=ocm traceid=de4cb63966b1b4745ff9ca5a642b0273

There was config issue. no issue

@butonic can we merge this PR?

butonic and others added 2 commits June 24, 2024 12:07
Signed-off-by: Jörn Friedrich Dreyer <[email protected]>
Signed-off-by: Christian Richter <[email protected]>
Copy link

@butonic
Copy link
Member Author

butonic commented Jun 24, 2024

Rebased. IMO we should merge this as is. It allows developers to actually set up and debug OCM shares. It will allow implementing E2E tests.

The commented code sections point out missing pieces in the CS3 API. I would merge this ASAP to iterate further.

@micbar micbar requested review from rhafer and fschade June 24, 2024 12:54
@micbar micbar requested review from kobergj and dragonchaser June 24, 2024 12:54
@butonic butonic merged commit dc68bbb into master Jun 28, 2024
4 checks passed
ownclouders pushed a commit that referenced this pull request Jun 28, 2024
@micbar micbar mentioned this pull request Jul 8, 2024
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category:Enhancement Add new functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants