Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing initial install options #248

Merged
merged 34 commits into from
Jul 15, 2022
Merged

Conversation

noahwsmith
Copy link
Contributor

This MR addresses the issues raised in #244 and other places regarding upstream changes breaking the make local_install_profile_demo command.

To test this MR, please check:

make local
make demo
make local_install_profile_demo

For bonus points I guess also make demo_install_profile_demo but I sort of think we should pull that out, I don't really see the use for it, honestly.

The install_profile commands will fail until this is accepted - it's the root cause of most of the issues. Islandora_defaults changed a storage type, and we're just now catching up to it, I guess...
Islandora-Devops/islandora_install_profile_demo#16

@DonRichards
Copy link
Member

Everything except make local worked.

 [notice] Performed install task: install_settings_form
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [notice] Performed install task: install_profile_modules
 [notice] Performed install task: install_profile_themes
 [notice] Performed install task: install_install_profile
 [notice] Performed install task: install_configure_form
 [notice] Performed install task: install_finished
 [success] Installation complete.
make[1]: Leaving directory '/home/don/github/isle-dc'

In PmCommands.php line 316:

  Unable to install modules search_api_solr_defaults, islandora_defaults due   
  to missing modules islandora_defaults.                                       

make: *** [Makefile:355: local] Error 1

Makefile Outdated Show resolved Hide resolved
@noahwsmith
Copy link
Contributor Author

2022/04/07 21:49:46 [error] 1080#1080: *10 FastCGI sent in stderr: "PHP message: Uncaught PHP Exception Drupal\Core\Config\UnmetDependenciesException: "Configuration objects provided by islandora_base_theme have unmet dependencies: block.block.islandora_base_theme_branding (islandora_install_profile_demo_core), block.block.islandora_base_theme_breadcrumbs (islandora_install_profile_demo_core, block_class), block.block.islandora_base_theme_disclaimer (fixed_block_content), field.storage.block_content.field_carapace_cb_copyright (field_permissions)" at /var/www/drupal/web/core/lib/Drupal/Core/Config/UnmetDependenciesException.php line 100" while reading response header from upstream, client: 172.26.0.3, server: drupal, request: "GET /admin/appearance/default?theme=islandora_base_theme&token=Jxm5tr4UYprzMZe97IIJv99SqXFrXugmhWiOBpdMwqQ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm7/php-fpm7.sock:", host: "islandora.traefik.me", referrer: "https://islandora.traefik.me/admin/appearance"

@noahwsmith
Copy link
Contributor Author

This is so convoluted to update.
I had to make Islandora-Devops/islandora-sandbox#16 to allow this to be pulled in
Islandora-Devops/islandora_install_profile_demo#17
And for every change to iipd, you have to push an update to the lockfile in sandbox to have it picked up.

Once this is approved, these two lines will have to be changed before accepting the 3 PRs together:
https://github.com/Islandora-Devops/islandora-sandbox/blob/isle-dc-pr-248/composer.json#L82 needs to be dev-main

And the custom branch needs to be removed from this line
https://github.com/Islandora-Devops/isle-dc/blob/fixing_initial_install_options/Makefile#L390

As per my previous statement, I pulled make demo_install_profile which appears to have no valid use case.

Testing: make local should do the "install islandora_defaults and good luck" thing, make demo should bring up the pre-baked demo, and make local-install-profile should create a local with the new sandbox experience.

I am getting a 502 bad gateway with Fedora, though, on at least make local-install-profile. Can anyone else replicate this?

@ysuarez
Copy link
Contributor

ysuarez commented Apr 11, 2022

@noahwsmith I pulled your latest changes (66bd09f) and just tried running on macOS Big Sur make local-install-profile and Drupal loads, but the homepage says some content is not found (green theme looks OK) and the cert is expired. I also get a "bad gateway" at http://islandora.traefik.me:8081/fcrepo/

Hope this help a bit.

@misilot
Copy link
Contributor

misilot commented Apr 11, 2022

Notes:

make-demo - returns me blue site with the cats. Though it looks like alpaca is choking on invalid certificates, and I am not sure you can change the domain for the demo site to be anything other than islandora.traefik.me. milliner_1 is also returning a bunch of errors, and those seem to be related to the 502 bad gateway with fedora.

Will try and test the others in a bit.

@ajstanley
Copy link
Contributor

testing make local-install-profile now.

@ajstanley
Copy link
Contributor

ajstanley commented Apr 11, 2022

Local install profile worked perfectly, in as far as completed the build without fatal errors. The certificate expired on the 8th, but other than that looks good. Taxonomy terms are all in place, solr is wired and responsive, and the page not found message disappears when you build a basic page.

But http://islandora.traefik.me:8081/fcrepo/ gives bad gateway whether from the browser or curled from inside the container.

The system will not allow an image to be uploaded, possibly related to

Flysystem: fedora
http://islandora.traefik.me:8081/fcrepo/rest/ returned 502

The upload directory fedora://2022-04 for the file field field_media_image could not be created or is not accessible. A newly uploaded file could not be saved in this directory as a consequence, and the upload was canceled.

Update: certs no longer a problem, but build is not usable without a Fedora connection

@misilot
Copy link
Contributor

misilot commented Apr 11, 2022

I did see the following when make local-install-profile

 [notice] Performed install task: install_select_language
 [notice] Performed install task: install_select_profile
 [notice] Performed install task: install_load_profile
 [error]  Error importing/updating configuration
 [error]  Error importing/updating configuration
 [notice] Performed install task: install_verify_requirements
 [notice] Performed install task: install_verify_database_ready
 [notice] Performed install task: install_base_system
 [notice] Performed install task: install_bootstrap_full
 [error]  The Flysystem driver is missing.
 [warning] Could not find required jsonld.settings to add default RDF namespaces.
 [warning] The "views_block:members-block_1" was not found
 [error]  Error getting JWT token for message. Check JWT Configuration.
 [error]  Error getting JWT token for message. Check JWT Configuration.
 [notice] Performed install task: install_profile_modules
 [notice] Performed install task: install_profile_themes

Same, I am seeing the 502's on there and the following errors:

activemq

activemq_1    | 2022-04-11 15:55:51,141 | INFO  | No Spring WebApplicationInitializer types detected on classpath | /api | main
activemq_1    |  INFO | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml
activemq_1    | 2022-04-11 15:55:51,796 | INFO  | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml | /api | main
activemq_1    |  WARN | Transport Connection to: tcp://172.24.0.8:55932 failed: java.io.EOFException
activemq_1    | 2022-04-11 15:56:35,062 | WARN  | Transport Connection to: tcp://172.24.0.8:55932 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///172.24.0.8:55932@61616

alpaca

alpaca_1      | 2022-04-11 16:08:08,818 | ERROR |  #21 - Multicast | DefaultErrorHandler              | 56 - org.apache.camel.camel-core - 2.20.4 | Failed delivery for (MessageId: ID-0c3aa8a29f6e-1649692572272-0-632 on ExchangeId: ID-0c3aa8a29f6e-1649692572272-0-633). Exhausted after delivery attempt: 11 caught: org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://milliner:8000/node/16eb6da8-ad10-45a7-bf9d-367b332675b8 with statusCode: 502. Processed by failure processor: FatalFallbackErrorHandler[Channel[Log(ca.islandora.alpaca.indexing.fcrepo.FcrepoIndexer)[Error indexing resource in fcrepo: ${exception.message}
alpaca_1      |
alpaca_1      | ${exception.stacktrace}]]]
alpaca_1      |
alpaca_1      | Message History
alpaca_1      | ---------------------------------------------------------------------------------------------------------------------------------------
alpaca_1      | RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
alpaca_1      | [FcrepoIndexerNode ] [FcrepoIndexerNode ] [broker://queue:islandora-indexing-fcrepo-content                              ] [     10185]
alpaca_1      | [FcrepoIndexerNode ] [unmarshal1        ] [unmarshal[org.apache.camel.model.dataformat.JsonDataFormat@7e6f7813]          ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setProperty1      ] [setProperty[event]                                                            ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setProperty2      ] [setProperty[uuid]                                                             ] [         1]
alpaca_1      | [FcrepoIndexerNode ] [setProperty3      ] [setProperty[jsonldUrl]                                                        ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setProperty4      ] [setProperty[fedoraBaseUrl]                                                    ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [log3              ] [log                                                                           ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [removeHeaders1    ] [removeHeaders[*]                                                              ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setHeader1        ] [setHeader[CamelHttpMethod]                                                    ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setHeader2        ] [setHeader[Content-Location]                                                   ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setHeader3        ] [setHeader[X-Islandora-Fedora-Endpoint]                                        ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [setBody1          ] [setBody[simple{Simple: ${null}}]                                              ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [multicast1        ] [multicast                                                                     ] [         0]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        14]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        17]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        29]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        11]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        22]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        11]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        17]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        17]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        10]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        11]
alpaca_1      | [FcrepoIndexerNode ] [toD1              ] [                                                                              ] [        12]
alpaca_1      | [FcrepoIndexerNode ] [log2              ] [log                                                                           ] [         2]
alpaca_1      |
alpaca_1      | Stacktrace
alpaca_1      | ---------------------------------------------------------------------------------------------------------------------------------------
alpaca_1      | org.apache.camel.http.common.HttpOperationFailedException: HTTP operation failed invoking http://milliner:8000/node/16eb6da8-ad10-45a7-bf9d-367b332675b8 with statusCode: 502
alpaca_1      | 	at org.apache.camel.component.http4.HttpProducer.populateHttpOperationFailedException(HttpProducer.java:311)[115:org.apache.camel.camel-http4:2.20.4]
alpaca_1      | 	at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:206)[115:org.apache.camel.camel-http4:2.20.4]
alpaca_1      | 	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:132)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:127)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:860)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:86)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:330)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:316)[56:org.apache.camel.camel-core:2.20.4]
alpaca_1      | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_302]
alpaca_1      | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_302]
alpaca_1      | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_302]
alpaca_1      | 	at java.lang.Thread.run(Thread.java:748)[:1.8.0_302]

fcrepo

fcrepo_1      | WARN 15:56:40.398 [main] (Database) Flyway upgrade recommended: MariaDB 10.6 is newer than this version of Flyway and support has not been tested. The latest supported version of MariaDB is 10.5.

mariadb

mariadb_1     | 2022-04-11 15:55:08 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_fcrepo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:55:11 6 [Warning] Aborted connection 6 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_matomo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:55:11 9 [Warning] Aborted connection 9 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_matomo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:55:12 12 [Warning] Aborted connection 12 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_matomo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:55:13 15 [Warning] Aborted connection 15 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_matomo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:55:13 18 [Warning] Aborted connection 18 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_matomo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:55:13 21 [Warning] Aborted connection 21 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_matomo_1.isle-dc_default' (This connection closed normally without authentication)
mariadb_1     | 2022-04-11 15:57:09 34 [Warning] Aborted connection 34 to db: 'unconnected' user: 'unauthenticated' host: 'isle-dc_drupal_1.isle-dc_default' (This connection closed normally without authentication)

milliner

milliner_1    | [2022-04-11 16:11:51] app.ERROR:  {"Exception":"[object] (RuntimeException(code: 502): Client error: `GET http://islandora2.ksulib.net:8081/fcrepo/rest/d6/a1/de/fe/d6a1defe-99ee-48af-8df6-2e0df416a838` resulted in a `502 Bad Gateway` response: Bad Gateway at /var/www/crayfish/Milliner/src/Service/MillinerService.php:207)"} []
milliner_1    | [2022-04-11 16:11:51] app.DEBUG: < 502 [] []
milliner_1    | 172.24.0.8 - - [11/Apr/2022:16:11:51 +0000] "POST /node/d6a1defe-99ee-48af-8df6-2e0df416a838 HTTP/1.1" 502 179 "-" "Apache-HttpClient/4.5.3 (Java/1.8.0_302)" "-"
milliner_1    | [2022-04-11 16:11:52] app.INFO: Matched route "{route}". {"route":"POST_node_uuid","route_parameters":{"_controller":"milliner.controller:saveNode","uuid":"d6a1defe-99ee-48af-8df6-2e0df416a838","_route":"POST_node_uuid"},"request_uri":"http://milliner:8000/node/d6a1defe-99ee-48af-8df6-2e0df416a838","method":"POST"} []
milliner_1    | [2022-04-11 16:11:52] app.DEBUG: > POST /node/d6a1defe-99ee-48af-8df6-2e0df416a838 [] []
milliner_1    | [2022-04-11 16:11:52] app.DEBUG: JSONLD URL: http://islandora2.ksulib.net/taxonomy/term/64?_format=jsonld [] []
milliner_1    | [2022-04-11 16:11:52] app.DEBUG: FEDORA ENDPOINT: http://islandora2.ksulib.net:8081/fcrepo/rest/ [] []
milliner_1    | [2022-04-11 16:11:52] app.DEBUG: DID NOT GET 404 [] []
milliner_1    | [2022-04-11 16:11:52] app.ERROR:  {"Exception":"[object] (RuntimeException(code: 502): Client error: `GET http://islandora2.ksulib.net:8081/fcrepo/rest/d6/a1/de/fe/d6a1defe-99ee-48af-8df6-2e0df416a838` resulted in a `502 Bad Gateway` response: Bad Gateway at /var/www/crayfish/Milliner/src/Service/MillinerService.php:207)"} []
milliner_1    | [2022-04-11 16:11:52] app.DEBUG: < 502 [] []
milliner_1    | 172.24.0.8 - - [11/Apr/2022:16:11:52 +0000] "POST /node/d6a1defe-99ee-48af-8df6-2e0df416a838 HTTP/1.1" 502 179 "-" "Apache-HttpClient/4.5.3 (Java/1.8.0_302)" "-

@noahwsmith
Copy link
Contributor Author

Looking at the issues backlog, I think this has fixed it. I've pulled it into this branch and retested with alpha-15. I'll also note that alpha-12 worked great as it was prior to this pull, so this is definitely the Fedora/proxy stuff moving around that's not letting it boot. #222

cc @dannylamb and @nigelgbanks who were working on that.

So I think this addresses a bunch of issues I'm hearing on the isle slack about alpha-15 not booting. I'll leave a note there too...

Any takers for testing this? Thanks for all the confirmations earlier...

@ysuarez
Copy link
Contributor

ysuarez commented Apr 11, 2022

@noahwsmith I was able to run make local-install-profile

Drupal loaded fine without errors. THere were no items that had been ingested, but I was able to ingest an image.

I also visited the Fedora rest page and I can see new objects created after my ingest.
http://islandora.traefik.me:8081/fcrepo/rest/

Fedora REST API page Screen Shot 2022-04-11 at , Apr 11, 2022 - 6 09 31 PM

I will try building the make local and `make demo' targets and report back.

UPDATE 1: I was able to run make demo and got the cats/dog content. I could also load Fedora's REST page http://islandora.traefik.me:8081/fcrepo/rest/

UPDATE 2: I was able to run make local and got the blue theme and no content loaded up (as expected). I could also load Fedora's REST page http://islandora.traefik.me:8081/fcrepo/rest/

@ajstanley
Copy link
Contributor

ajstanley commented Apr 11, 2022

Testing make local-install-profile again now.

Update:
Ran with the usual forgivable errors. The site is up with good certs. Was able to ingest images and display with OpenSeadragon.
http://islandora.traefik.me:8081/fcrepo/rest/ brings back good results.

I've got two prod boxes I'm going to build real fast before it changes its mind.

@ajstanley
Copy link
Contributor

Ugh. On a prod install we're back to this

In ExtensionList.php line 522:
                                                              
  The profile islandora_install_profile_demo does not exist.  

@ysuarez
Copy link
Contributor

ysuarez commented Apr 12, 2022

@noahwsmith I wanted to add more details about running make demo.

I was not able to create a "repository item." I was getting a white page with the following error “The website encountered an unexpected error. Please try again later.”

In the Drupal recent log messages section I saw this two errors...

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'content_translation_uid' in 'field list': INSERT INTO {node_field_data} (nid, vid, type, langcode, status, uid, title, created, changed, promote, sticky, default_langcode, revision_translation_affected, content_translation_source, content_translation_outdated, content_translation_uid, content_translation_status, content_translation_created, content_translation_changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12, :db_insert_placeholder_13, :db_insert_placeholder_14, :db_insert_placeholder_15, :db_insert_placeholder_16, :db_insert_placeholder_17, :db_insert_placeholder_18); Array ( [:db_insert_placeholder_0] => 40 [:db_insert_placeholder_1] => 70 [:db_insert_placeholder_2] => islandora_object [:db_insert_placeholder_3] => en [:db_insert_placeholder_4] => 1 [:db_insert_placeholder_5] => 1 [:db_insert_placeholder_6] => SIFT process [:db_insert_placeholder_7] => 1649771155 [:db_insert_placeholder_8] => 1649771253 [:db_insert_placeholder_9] => 1 [:db_insert_placeholder_10] => 0 [:db_insert_placeholder_11] => 1 [:db_insert_placeholder_12] => 1 [:db_insert_placeholder_13] => und [:db_insert_placeholder_14] => 0 [:db_insert_placeholder_15] => 1 [:db_insert_placeholder_16] => 1 [:db_insert_placeholder_17] => 1649771253 [:db_insert_placeholder_18] => 1649771253 ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->saveToSharedTables() (line 1047 of /var/www/drupal/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

and...

Drupal\Core\Entity\EntityStorageException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'content_translation_uid' in 'field list': INSERT INTO {node_field_data} (nid, vid, type, langcode, status, uid, title, created, changed, promote, sticky, default_langcode, revision_translation_affected, content_translation_source, content_translation_outdated, content_translation_uid, content_translation_status, content_translation_created, content_translation_changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12, :db_insert_placeholder_13, :db_insert_placeholder_14, :db_insert_placeholder_15, :db_insert_placeholder_16, :db_insert_placeholder_17, :db_insert_placeholder_18); Array ( [:db_insert_placeholder_0] => 40 [:db_insert_placeholder_1] => 70 [:db_insert_placeholder_2] => islandora_object [:db_insert_placeholder_3] => en [:db_insert_placeholder_4] => 1 [:db_insert_placeholder_5] => 1 [:db_insert_placeholder_6] => SIFT process [:db_insert_placeholder_7] => 1649771155 [:db_insert_placeholder_8] => 1649771253 [:db_insert_placeholder_9] => 1 [:db_insert_placeholder_10] => 0 [:db_insert_placeholder_11] => 1 [:db_insert_placeholder_12] => 1 [:db_insert_placeholder_13] => und [:db_insert_placeholder_14] => 0 [:db_insert_placeholder_15] => 1 [:db_insert_placeholder_16] => 1 [:db_insert_placeholder_17] => 1649771253 [:db_insert_placeholder_18] => 1649771253 ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 846 of /var/www/drupal/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

@ajstanley
Copy link
Contributor

ajstanley commented Apr 12, 2022

The offending lines in a fresh build on an Ubuntu box

grep: /var/www/drupal/web/sites/default/settings.php: No such file or directory

In Filesystem.php line 203:
                                                                          
  Failed to chmod file "/var/www/drupal/web/sites/default/settings.php". 

Complete success with a build on my Mac last night.

UPDATE!! I was able to build without changing anything. I'm going to blame some caching on my server.

@nigelgbanks nigelgbanks self-requested a review April 13, 2022 15:17
@noahwsmith
Copy link
Contributor Author

I was just talking to Nigel and he's going to test this now too. Sounds like make local-install-profile has tested pretty well (though Alan had a few hiccups), make local was universally good, and there may be an issue with make demo (which ironically was the only one I didn't touch, I think).

@misilot
Copy link
Contributor

misilot commented Apr 13, 2022

I did find the use case for make demo_install_profile. Which probably was the original thought of a way to generate https://sandbox.islandora.ca/ with content, as well as locally.

https://islandora.github.io/documentation/installation/docker-demo/

@nigelgbanks
Copy link
Contributor

Demo is a bit messed up, it initially looks good though, but I've ran into a few issues.

The search display doesn't return results, but I checked the solr index and the content has been indexed correctly I think the view might be a bit messed up.

Trying to create islandora_object or media entities results in the following errors around translation.

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'content_translation_uid' in 'field list': INSERT INTO {node_field_data} (nid, vid, type, langcode, status, uid, title, created, changed, promote, sticky, default_langcode, revision_translation_affected, content_translation_source, content_translation_outdated, content_translation_uid, content_translation_status, content_translation_created, content_translation_changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12, :db_insert_placeholder_13, :db_insert_placeholder_14, :db_insert_placeholder_15, :db_insert_placeholder_16, :db_insert_placeholder_17, :db_insert_placeholder_18); Array ( [:db_insert_placeholder_0] => 39 [:db_insert_placeholder_1] => 69 [:db_insert_placeholder_2] => islandora_object [:db_insert_placeholder_3] => en [:db_insert_placeholder_4] => 1 [:db_insert_placeholder_5] => 1 [:db_insert_placeholder_6] => Foobar [:db_insert_placeholder_7] => 1649881213 [:db_insert_placeholder_8] => 1649881274 [:db_insert_placeholder_9] => 1 [:db_insert_placeholder_10] => 0 [:db_insert_placeholder_11] => 1 [:db_insert_placeholder_12] => 1 [:db_insert_placeholder_13] => und [:db_insert_placeholder_14] => 0 [:db_insert_placeholder_15] => 1 [:db_insert_placeholder_16] => 1 [:db_insert_placeholder_17] => 1649881274 [:db_insert_placeholder_18] => 1649881274 ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->saveToSharedTables() (line 1047 of /var/www/drupal/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

@nigelgbanks
Copy link
Contributor

make demo_install_profile works great

@misilot
Copy link
Contributor

misilot commented Apr 13, 2022

make demo_install_profile works great

Ok, I thought it was mentioned it was being removed.

@nigelgbanks
Copy link
Contributor

@misilot I didn't know, I thought it was different than make local in that make local could be some custom local build not based on the install profile.

@nigelgbanks
Copy link
Contributor

make local works well enough it's a plain looking site and I've had to manual configure the display of media form elements but it works, I think that's enough for it?

It takes a long time to test on the mac due to the bind mount volumes and install is like 30-40 mins to bring up a fresh one.

@nigelgbanks
Copy link
Contributor

Going back to make demo

[error]  Content Browser requires the Masonry library. Download the newest release from
https://github.com/desandro/masonry/releases and place it in /libraries 

This makes sense cause it's not installed in isle-buildkit at one point it was, though that was before the move to sandbox. I recon if we update the commit there to match this it should work.

@nigelgbanks
Copy link
Contributor

Pushed some changes to update the commits to pull the latest, though make demo is still broken it hangs indefinitely on

docker-compose exec -T drupal with-contenv bash -lc 'drush --root /var/www/drupal/web -l ${DRUPAL_DEFAULT_SITE_URL} vbo-exec non_fedora_files emit_file_event --configuration="queue=islandora-indexing-fcrepo-file-external&event=Update"'

I'll dig in more tomorrow.

@nigelgbanks
Copy link
Contributor

drupal_1  | NOTICE: PHP message: PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /var/www/drupal/web/core/lib/Drupal/Core/Plugin/ContextAwarePluginTrait.php on line 184
drupal_1  | 2022/04/14 09:09:52 [error] 1056#1056: *32 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /var/www/drupal/web/core/lib/Drupal/Core/Plugin/ContextAwarePluginTrait.php on line 184" while reading response header from upstream, client: 172.20.0.13, server: drupal, request: "GET /node/23 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm7/php-fpm7.sock:", host: "islandora.traefik.me", referrer: "https://islandora.traefik.me/admin/content"

@nigelgbanks
Copy link
Contributor

nigelgbanks commented Apr 14, 2022

Looks like there is an infinite recursion in islandora_file_update, caused by an interaction with filehash. filehash goes to update a file entity's hash field on first access. This triggers islandora_file_update which triggers a file load which triggers filehash to do the update, etc, etc.

log.txt

In particular it's the executeDerivativeReactions non_fedora_files views bulk operation that is triggering the subsequent file load.

Manually generating the hash table gets around this issue, in attempting to test with new objects for this infinite loop, I've encountered a new issue.

Symfony\Component\HttpKernel\Exception\HttpException: No bytes were copied to public://2022-04/14-FITS File.xml in Drupal\islandora\MediaSource\MediaSourceService->updateFile() (line 206 of /var/www/drupal/web/modules/contrib/islandora/src/MediaSource/MediaSourceService.php).

The file does not exist.

/sites/default/files/2022-04/8-FITS%20File.xml

Also lots of these:

Symfony\Component\HttpKernel\Exception\HttpException: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'content_translation_uid' in 'field list': INSERT INTO "media_field_data" ("mid", "vid", "bundle", "langcode", "status", "uid", "name", "thumbnail__target_id", "thumbnail__alt", "thumbnail__title", "thumbnail__width", "thumbnail__height", "created", "changed", "default_langcode", "revision_translation_affected", "content_translation_source", "content_translation_outdated", "content_translation_uid", "content_translation_status", "content_translation_created", "content_translation_changed") VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9, :db_insert_placeholder_10, :db_insert_placeholder_11, :db_insert_placeholder_12, :db_insert_placeholder_13, :db_insert_placeholder_14, :db_insert_placeholder_15, :db_insert_placeholder_16, :db_insert_placeholder_17, :db_insert_placeholder_18, :db_insert_placeholder_19, :db_insert_placeholder_20, :db_insert_placeholder_21); Array ( [:db_insert_placeholder_0] => 122 [:db_insert_placeholder_1] => 122 [:db_insert_placeholder_2] => fits_technical_metadata [:db_insert_placeholder_3] => en [:db_insert_placeholder_4] => 1 [:db_insert_placeholder_5] => 1 [:db_insert_placeholder_6] => 33-FITS File.xml [:db_insert_placeholder_7] => 7 [:db_insert_placeholder_8] => [:db_insert_placeholder_9] => [:db_insert_placeholder_10] => 180 [:db_insert_placeholder_11] => 180 [:db_insert_placeholder_12] => 1588263654 [:db_insert_placeholder_13] => 1649930528 [:db_insert_placeholder_14] => 1 [:db_insert_placeholder_15] => 1 [:db_insert_placeholder_16] => und [:db_insert_placeholder_17] => 0 [:db_insert_placeholder_18] => [:db_insert_placeholder_19] => [:db_insert_placeholder_20] => [:db_insert_placeholder_21] => ) in Drupal\islandora\Controller\MediaSourceController->putToNode() (line 192 of /var/www/drupal/web/modules/contrib/islandora/src/Controller/MediaSourceController.php).

Field comes from a contrib module content_translation.

Attempting to run updb to hopefully add that column results in the following errors.

[error]  The directory //var/www/drupal/config/sync/ does not exist.
 
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [warning] array_key_exists() expects parameter 2 to be array, bool given ContentTranslationHandler.php:269
 [error]   (Currently using Missing or invalid modules The following modules are marked as installed in the core.extension 
configuration, but they are missing:
 * content_browser
 * embed
 * entity_browser
 * entity_embed
 * views_slideshow
 * views_slideshow_cycle

Review the  suggestions for resolving this incompatibility [1] to repair your 
installation, and then re-run update.php.

[1] https://www.drupal.org/docs/8/update/troubleshooting-database-updates
) 
 [error]   (Currently using Missing or invalid themes The following themes are marked as installed in the core.extension 
configuration, but they are missing:
 * at_core
 * uikit
 * bootstrap
 * solid

Review the  suggestions for resolving this incompatibility [1] to repair your 
installation, and then re-run update.php.

[1] https://www.drupal.org/docs/8/update/troubleshooting-database-updates
) 

I ignored the errors and ran it anyways, I did not resolve the issue.

In reading through the modules code when it goes to define and add the fields to an entity it does a check.

    if (!$this->hasAuthor()) {
      $definitions['content_translation_uid'] = BaseFieldDefinition::create('entity_reference')
        ->setLabel(t('Translation author'))
        ->setDescription(t('The author of this translation.'))
        ->setSetting('target_type', 'user')
        ->setSetting('handler', 'default')
        ->setRevisionable(TRUE)
        ->setDefaultValueCallback(static::class . '::getDefaultOwnerId')
        ->setTranslatable(TRUE);
    }

Which makes sense to avoid the extra field if not necessary. media does have an author so this field is never supposed to be added. Likely this is some problem caused by content_sync being used to import entities where-in this logic didn't exist before, hence the column is now missing.

No this does not seem to be the case (it's actually a core module, and this logic has not changed really in 7 years...)

More oddities:

Drupal\Core\Extension\Exception\UnknownExtensionException: The theme solid does not exist. in Drupal\Core\Extension\ExtensionList->getPathname() (line 522 of /var/www/drupal/web/core/lib/Drupal/Core/Extension/ExtensionList.php).

The default theme is not installed, attempting to install it causes a whitescreen.

Drupal\Core\Config\UnmetDependenciesException: Configuration objects provided by <em class="placeholder">islandora_base_theme</em> have unmet dependencies: <em class="placeholder">block.block.islandora_base_theme_branding (islandora_install_profile_demo_core), block.block.islandora_base_theme_breadcrumbs (islandora_install_profile_demo_core, block_class), block.block.islandora_base_theme_disclaimer (fixed_block_content)</em> in Drupal\Core\Config\UnmetDependenciesException::create() (line 100 of /var/www/drupal/web/core/lib/Drupal/Core/Config/UnmetDependenciesException.php).

@DonRichards
Copy link
Member

DonRichards commented Jul 14, 2022

Sorry, I think I misunderstood some of the previous comments. I think the make local is broken at the repo release level as Mark suggested and isn't a problem associated with this PR. JP2s derivatives seems to be working for me on the *install profile commands.

@noahwsmith

  1. make demo-install-profile <-- Builds and appears to be working fine.
  2. make local-install-profile <-- Builds and appears to be working fine.
  3. make demo <-- Broken build process.
  4. make local <-- Broke at the repository level and not this PR and can likely be ignored for this PR.

Mariadb is showing some errors as Access denied for user drupal_default. Other errors on MariaDB were aborted unauthenticated user from matomo, drupal, and fedora for the make demo & local commands.

@noahwsmith
Copy link
Contributor Author

Ok, so the task here/now is to rebase this and fix make demo? That sounds correct to me too - thanks for getting this back on track...

@DonRichards
Copy link
Member

@noahwsmith Yes. I just spoke with Danny and we hatched out a plan.

  1. Rebase
  2. Shift the make demo-install-profile local-install-profile to be the new default demo & local.
  3. Correct a small help comment on local-install-profile
  4. Remove the unnecessary build/demo-data/ files.
  5. Update any relative documentation
  6. Then reach out to a committer for a review

@DonRichards DonRichards force-pushed the fixing_initial_install_options branch from 6d2723c to 296e1a5 Compare July 15, 2022 15:09
@DonRichards
Copy link
Member

OK, complete. @ajstanley Would you be interested in testing? I removed the "demo-install-profile & local-install-profile" as commands. They replaced the default demo and local. To test just run make local then test the make demo commands. Derivative seems to work for me. My only issue I had and can be addressed in another PR is that "browse collections" doesn't display newly created collections because of a filter on /admin/structure/views/view/home_page_featured_collections/edit/block_2
If you remove the filter that looks for collection in term it should work.

@ajstanley
Copy link
Contributor

Building now -

@ajstanley
Copy link
Contributor

make local gave the usual

 [error]  The Flysystem driver is missing. 
 [warning] Could not find required jsonld.settings to add default RDF namespaces.
 [warning] The "views_block:members-block_1" was not found
 [error]  Error getting JWT token for message. Check JWT Configuration. 
 [error]  Error getting JWT token for message. Check JWT Configuration.

But worked (with the collection caveat mentioned above)
Similar results with make demo

Nice work all!

@DonRichards
Copy link
Member

@ajstanley Yeah, not sure why that's complaining but it does seem to be building correctly. If you're satisfied with this as well I'll leave it up to you to "approve & merge" because I contributed to the code.

@ajstanley ajstanley merged commit 5705b87 into development Jul 15, 2022
@noahwsmith
Copy link
Contributor Author

^ Thanks @DonRichards and @ajstanley for seeing this across the line...

@seniordev-ca
Copy link

make local gave the usual

 [error]  The Flysystem driver is missing. 
 [warning] Could not find required jsonld.settings to add default RDF namespaces.
 [warning] The "views_block:members-block_1" was not found
 [error]  Error getting JWT token for message. Check JWT Configuration. 
 [error]  Error getting JWT token for message. Check JWT Configuration.

But worked (with the collection caveat mentioned above) Similar results with make demo

Nice work all!

@ajstanley @DonRichards

make local gives me the error:

image

Is this totally fine? Because I am getting errors on /_flysystem/fedora/styles/large/fedora/..../ files when I am using my own Drupal repo. Isn't it related to above error? -> The Flysystem driver is missing.

@rosiel rosiel deleted the fixing_initial_install_options branch July 12, 2023 13:21
@rosiel
Copy link
Contributor

rosiel commented Jul 12, 2023

In the Starter Site, we have these explained: https://github.com/Islandora-Devops/islandora-starter-site#warningserrors-during-installation

An FYI, we're encouraging people to install the starter site rather than the install profile as the Install Profile is about to be deprecated. That would be the make starter option rather than make local.

Feel free to join the Slack workspace and post in support if you want help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.