CLDR-14399 move to OpenLiberty, add API server #936

srl295 · 2021-01-14T21:57:05Z

CLDR-14399 move to OpenLiberty, add API server

See CLDR-14213 - this doesn't remove Derby, but doesn't enable it in the server easily anymore.

srl295 · 2021-01-14T22:00:21Z

maybe @yumaoka and @JCEmmons will want to review someday if it looks familliar 🤣

srl295 · 2021-01-14T22:01:39Z

tools/cldr-code/src/main/java/org/unicode/cldr/util/FileReaders.java

@@ -78,6 +78,8 @@ public static String getRelativeFileName(Class<?> class1, String filename) {
            return resourceString.substring(5);
        } else if (resourceString.startsWith("jar:file:")) {
            return resourceString.substring(9);
+        } else if (resourceString.startsWith("wsjar:file:")) {


@yumaoka did i redo it right?

(context: ICU4J had the exact same bug and fix…)

srl295 · 2021-01-14T22:02:15Z

tools/cldr-apps/src/main/java/org/unicode/cldr/web/api/XPathAPI.java

+
+@Path("/xpath")
+public class XPathAPI {
+    @GET @Path("{hexId}")


@yumaoka surveytool gets dragged into the 2010's … :)

this API doesn't validate quite yet.

srl295 · 2021-01-15T20:54:51Z

tools/cldr-apps/src/main/liberty/config/server.xml

+		TO SET UP:
+		Create a "server.env" file next to this xml
+		(that is, tools/cldr-apps/src/main/liberty/config/server.env)
+		with the following:
+
+		MYSQL_USER=surveytool
+		MYSQL_PASSWORD=yourpassword
+		MYSQL_DB=cldrdb


Interim setup instructions…

srl295 · 2021-01-15T20:55:33Z

@btangmu this is now working w/ the DB

srl295 · 2021-01-15T20:56:10Z

I'll clean this up and get it ready to be a real draft PR…

jira-pull-request-webhook · 2021-01-15T22:41:23Z

Notice: the branch changed across the force-push!

tools/cldr-apps/README.md is different
tools/cldr-apps/src/main/java/org/unicode/cldr/web/api/PropertiesResource.java is no longer changed in the branch
tools/cldr-apps/src/main/liberty/config/server.env.sample is now changed in the branch
tools/cldr-apps/src/main/resources/api/swagger.yaml is no longer changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

srl295 · 2021-01-15T22:43:50Z

Still need to figure out the deployment mechanism (actions and such). so i guess this is still draft.

jira-pull-request-webhook · 2021-01-15T22:45:26Z

Notice: the branch changed across the force-push!

tools/cldr-apps/pom.xml is different
tools/cldr-apps/src/main/liberty/config/server.xml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-19T20:32:00Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

btangmu · 2021-01-19T20:36:49Z

I've quickly looked through the changes and LGTM though I certainly don't understand it all. Seems worthwhile. How should we test?

srl295 · 2021-01-19T20:32:28Z

tools/cldr-apps/src/main/java/org/unicode/cldr/web/api/XPathAPI.java

+
+@Path("/xpath")
+public class XPathAPI {
+    @GET @Path("{hexId}")


this API doesn't validate quite yet.

srl295 · 2021-01-19T23:39:09Z

tools/cldr-apps/src/main/liberty/config/server.env.sample

@@ -0,0 +1,4 @@
+# copy this to server.env and set the password properly


@btangmu if you copy this file to server.env (with appropriate DB changes) then you can start the server with mvn -pl cldr-apps liberty:dev and it will fire up a dev server on port localhost:9080 until you hit control-C ..

jira-pull-request-webhook · 2021-01-19T23:40:33Z

Notice: the branch changed across the force-push!

tools/scripts/ansible/openliberty-playbook.yml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

srl295 · 2021-01-19T23:40:47Z

I've quickly looked through the changes and LGTM though I certainly don't understand it all. Seems worthwhile. How should we test?

#936 (comment) says how to do a local test. we should walk through that sometime.

jira-pull-request-webhook · 2021-01-20T14:41:17Z

Notice: the branch changed across the force-push!

tools/scripts/ansible/openliberty-playbook.yml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-20T14:52:34Z

Notice: the branch changed across the force-push!

.gitignore is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-20T15:07:19Z

Notice: the branch changed across the force-push!

.github/workflows/maven.yml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-20T16:51:16Z

Notice: the branch changed across the force-push!

.github/workflows/maven.yml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-20T20:32:27Z

Notice: the branch changed across the force-push!

tools/scripts/ansible/cldr-apps-playbook.yml is different
tools/scripts/ansible/openliberty-playbook.yml is different
tools/scripts/ansible/server-playbook.yml is different
tools/scripts/ansible/templates/56-surveytool.j2 is now changed in the branch
tools/scripts/ansible/templates/deploy-sh.j2 is now changed in the branch
tools/scripts/ansible/vars/main.yml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-20T20:32:44Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

srl295 · 2021-01-20T20:40:29Z

This is ready to review.

should be deployable to cldr-smoke and production once ansible is re-run. (may need to iterate there)
for local command line test, see …/cldr-apps/README.md

btangmu · 2021-01-22T02:26:38Z

Following …/cldr-apps/README.md, I get:

[ERROR] .../cldr/tools/cldr-apps/src/main/java/org/unicode/cldr/web/ErrorSubtypes.java:[24,43] cannot find symbol
  symbol:   method toHTML(java.lang.String)
  location: class org.unicode.cldr.util.CLDRURLS
...
...
[INFO] [ERROR   ] CWWKF0032E: The com.ibm.websphere.appserver.javaee-jdbc-4.3 feature requires a minimum Java runtime environment version of JavaSE 11.
...

Does the "CLDRURLS.toHTML" error ring any bells? Is JavaSE 11 really required?

srl295 · 2021-01-22T14:15:25Z

Following …/cldr-apps/README.md, I get:

[ERROR] .../cldr/tools/cldr-apps/src/main/java/org/unicode/cldr/web/ErrorSubtypes.java:[24,43] cannot find symbol
  symbol:   method toHTML(java.lang.String)
  location: class org.unicode.cldr.util.CLDRURLS
...
...
Does the "CLDRURLS.toHTML" error ring any bells?

it sounds like possibly an out of date package. Can you Try running this first:

mvn --file=tools/pom.xml compile install package -DskipTests=true

The liberty:dev may only look at code within cldr-apps for rebuild.

[INFO] [ERROR ] CWWKF0032E: The com.ibm.websphere.appserver.javaee-jdbc-4.3 feature requires a minimum Java runtime environment version of JavaSE 11.
...
Is JavaSE 11 really required?

Hm, yes, I think it may be required. CLDR-14311 proposes a bump to 11 LTS.

easy way to add it to the command line or eclipse:

download .tar.gz (not installer) of JDK from https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=openj9
Maven command line: then you can set JAVA_HOME= to the home directory of the archive as unpacked
Eclipse: can add it in the JREs dialog

btangmu · 2021-01-22T14:57:12Z

I switched to java version "11.0.2"
Now I still get:

[ERROR] .../cldr/tools/cldr-apps/src/main/java/org/unicode/cldr/web/ErrorSubtypes.java:[24,43] cannot find symbol
  symbol:   method toHTML(java.lang.String)
  location: class org.unicode.cldr.util.CLDRURLS

and then

...
[INFO] Listening for transport dt_socket at address: 7777
[INFO] Launching cldr (Open Liberty 20.0.0.12/wlp-1.0.47.cl201220201111-0736) on Java HotSpot(TM) 64-Bit Server VM, version 11.0.2+9-LTS (en_US)
[INFO] [AUDIT   ] CWWKE0001I: The server cldr has been launched.
[INFO] CWWKM2010I: Searching for CWWKF0011I: in /Users/tbishop/Documents/WenlinDocs/Organizations/Unicode/CLDR_job/cldr/tools/cldr-apps/target/liberty/wlp/usr/servers/cldr/logs/messages.log. This search will timeout after 90 seconds.
[INFO] [AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://192.168.0.120:9080/openapi/ui/
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://192.168.0.120:9080/openapi/
[INFO] [ERROR   ] CWWKO1650E: Validation of the OpenAPI document produced the following error(s): 
[INFO]                                                                                                                
[INFO]  - Message: Required "paths" field is missing or is set to an invalid value, Location: #
[INFO] 
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://192.168.0.120:9080/cldr-apps/
[INFO] [AUDIT   ] CWWKZ0001I: Application cldr-apps started in 10.497 seconds.
[INFO] [AUDIT   ] CWWKF0012I: The server installed the following features: [el-3.0, jaxrs-2.1, jaxrsClient-2.1, jdbc-4.3, jndi-1.0, jsonp-1.1, jsp-2.3, mpConfig-1.4, mpOpenAPI-1.1, servlet-4.0].
[INFO] [AUDIT   ] CWWKF0011I: The cldr server is ready to run a smarter planet. The cldr server started in 17.536 seconds.
[INFO] CWWKM2015I: Match number: 1 is [1/22/21, 9:49:58:673 EST] 00000022 com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0011I: The cldr server is ready to run a smarter planet. The cldr server started in 17.536 seconds..
[INFO] [ERROR   ] SRVE0319E: For the [SurveyMain] servlet, org.unicode.cldr.web.SurveyMain servlet class was found, but a resource injection failure has occurred. java.lang.reflect.InvocationTargetException
[INFO] [ERROR   ] SRVE0276E: Error while initializing Servlet [SurveyMain]: javax.servlet.UnavailableException: SRVE0319E: For the [SurveyMain] servlet, org.unicode.cldr.web.SurveyMain servlet class was found, but a resource injection failure has occurred. java.lang.reflect.InvocationTargetException
[INFO] 	at com.ibm.ws.webcontainer.servlet.ServletWrapper$1.run(ServletWrapper.java:1582)
[INFO] 	at [internal classes]
[INFO] 
[INFO] ************************************************************************
[INFO] *    Liberty is running in dev mode.
[INFO] *        To run tests on demand, press Enter.
[INFO] *        To restart the server, type 'r' and press Enter.
[INFO] *        To stop the server and quit dev mode, press Ctrl-C or type 'q' and press Enter.
[INFO] *        
[INFO] *    Liberty server port information:
[INFO] *        Liberty server HTTP port: [ 9080 ]
[INFO] *        Liberty debug port: [ 7777 ]
[INFO] ************************************************************************
[INFO] [AUDIT   ] CWWKT0017I: Web application removed (default_host): http://192.168.0.120:9080/cldr-apps/
[INFO] [AUDIT   ] CWWKZ0009I: The application cldr-apps has stopped successfully.
[INFO] [WARNING ] [ TargetsTableImpl@3e1d4b10 ] ANNO_TARGETS_CORRUPT_CLASS [ org.unicode.cldr.web.SurveyVettingParticipation ] (java.lang.ArrayIndexOutOfBoundsException: Index 6 out of bounds for length 0
[INFO] 	at org.objectweb.asm.ClassReader.readShort(ClassReader.java:3573)
[INFO] 	at [internal classes]
[INFO]  : Index 6 out of bounds for length 0)
[INFO] [WARNING ] Corrupt class
[INFO] Index 6 out of bounds for length 0
.../cldr/tools/cldr-apps/src/main/java/org/unicode/cldr/web/ErrorSubtypes.java:24: error: cannot find symbol
        r.put("CLDR_SUBTYPE_URL", CLDRURLS.toHTML(SubtypeToURLMap.getDefaultUrl()));
                                          ^
  symbol:   method toHTML(String)
  location: class CLDRURLS
[INFO] [ERROR   ] CWWKO1650E: Validation of the OpenAPI document produced the following error(s): 
[INFO]                                                                                                                
[INFO]  - Message: Required "paths" field is missing or is set to an invalid value, Location: #
...

btangmu · 2021-01-22T14:59:24Z

Then I pasted into browser address bar: http://192.168.0.120:9080/cldr-apps/
Browser says:

HTTP Error Code:   500
--
Error Message:JSPG0049E: /index.jsp failed to compile : JSPG0091E: An error occurred at line: 60 in the file: /index.jspJSPG0093E: Generated servlet error from file: /index.jsp /Users/tbishop/Documents/WenlinDocs/Organizations/Unicode/CLDR_job/cldr/tools/cldr-apps/target/liberty/wlp/usr/servers/cldr/workarea/org.eclipse.osgi/211/data/temp/default_node/SMF_WebContainer/cldr-apps/cldr-apps/_index.java : 164 : StatisticsUtils cannot be resolved

srl295 · 2021-01-22T15:03:37Z

Then I pasted into browser address bar: http://192.168.0.120:9080/cldr-apps/
Browser says:

HTTP Error Code:   500
--
Error Message:JSPG0049E: /index.jsp failed to compile : JSPG0091E: An error occurred at line: 60 in the file: /index.jspJSPG0093E: Generated servlet error from file: /index.jsp /Users/tbishop/Documents/WenlinDocs/Organizations/Unicode/CLDR_job/cldr/tools/cldr-apps/target/liberty/wlp/usr/servers/cldr/workarea/org.eclipse.osgi/211/data/temp/default_node/SMF_WebContainer/cldr-apps/cldr-apps/_index.java : 164 : StatisticsUtils cannot be resolved

it sounds like it might just need a clean build… let's discuss on the call?

btangmu

I wasn't able to get it working in Eclipse. I think we need that before merging this to master.

btangmu · 2021-01-22T20:31:40Z

I tried installing this: https://github.com/OpenLiberty/open-liberty-tools/files/5363408/openlibertytools-20.0.0.9.v2020-10-09_1337-with-fix.zip

Referenced from here: OpenLiberty/open-liberty-tools#407

I tried to create a new server by right-clicking in the Servers tab, but Open Liberty isn't one of the options.

More links:
OpenLiberty/blogs#791
OpenLiberty/open-liberty-tools#410

srl295 · 2021-01-22T23:26:23Z

I did the same install.

…

On Fri, Jan 22, 2021 at 2:31 PM Tom Bishop ***@***.***> wrote: I tried installing this: https://github.com/OpenLiberty/open-liberty-tools/files/5363408/openlibertytools-20.0.0.9.v2020-10-09_1337-with-fix.zip Referenced from here: OpenLiberty/open-liberty-tools#407 <OpenLiberty/open-liberty-tools#407> I tried to create a new server by right-clicking in the Servers tab, but Open Liberty isn't one of the options. More links: OpenLiberty/blogs#791 <OpenLiberty/blogs#791> OpenLiberty/open-liberty-tools#410 <OpenLiberty/open-liberty-tools#410> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#936 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGQZM5XNSQVF4DSQYMGLLTS3HODXANCNFSM4WDELENQ> .

jira-pull-request-webhook · 2021-01-25T14:53:06Z

Notice: the branch changed across the force-push!

.gitignore is different
tools/cldr-apps/.settings/org.eclipse.wst.common.project.facet.core.xml is different
tools/cldr-apps/pom.xml is different
tools/cldr-code/.settings/org.eclipse.wst.common.project.facet.core.xml is now changed in the branch
tools/cldr-rdf/.settings/org.eclipse.wst.common.project.facet.core.xml is now changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2021-01-25T14:54:09Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

- deployment is now different, but the github action has been updated - for local dev: see cldr-apps/README.md

jira-pull-request-webhook · 2021-01-25T22:43:31Z

Notice: the branch changed across the force-push!

tools/cldr-apps/src/main/liberty/config/server.xml is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

btangmu

Since your last change adding "localconnector-1.0" to server.xml (I guess that's what made the difference, though I'm not sure), the server finally appears to be running semi-normally for me in debug mode in Eclipse

The java console does have various errors like:

[WARNING ] table cldr_review_hide_40 did exist.
[ERROR   ] J2CA0081E: Method cleanup failed while trying to execute method cleanup on ManagedConnection WSRdbManagedConnectionImpl@7f7753d9 from resource jdbc/SurveyTool. Caught exception: com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException: DSRA0080E: An exception was received by the Data Store Adapter. See original exception message: Cannot call 'cleanup' on a ManagedConnection while it is still in a transaction..
	at com.ibm.ws.rsadapter.impl.WSRdbManagedConnectionImpl.cleanupTransactions(WSRdbManagedConnectionImpl.java:3177)
	at [internal classes]
	at org.unicode.cldr.web.DBUtils.closeDBConnection(DBUtils.java:155)
	at org.unicode.cldr.web.DBUtils.close(DBUtils.java:950)
	at org.unicode.cldr.web.DBUtils.hasTable(DBUtils.java:479)
	at org.unicode.cldr.web.ReviewHide.createTable(ReviewHide.java:29)
	at org.unicode.cldr.web.UserRegistry.setupDB(UserRegistry.java:626)
	at org.unicode.cldr.web.UserRegistry.createRegistry(UserRegistry.java:586)

srl295 · 2021-01-26T13:42:47Z

@btangmu thanks for the thorough check, lets get this merged and then i'll work on the items mentioned above

srl295 · 2021-01-26T15:01:51Z

update-ansible-log.txt

here's the ansible log of updating smoketest…

srl295 self-assigned this Jan 14, 2021

srl295 commented Jan 14, 2021

View reviewed changes

srl295 commented Jan 15, 2021

View reviewed changes

srl295 force-pushed the cldr-14399-api branch from 0150da1 to 8006220 Compare January 15, 2021 22:41

srl295 requested review from btangmu and yumaoka January 15, 2021 22:43

srl295 marked this pull request as ready for review January 15, 2021 22:43

srl295 changed the title ~~CLDR-14399 scary API PR~~ CLDR-14399 move to OpenLiberty, add API server Jan 15, 2021

srl295 marked this pull request as draft January 15, 2021 22:44

srl295 requested a review from JCEmmons January 15, 2021 22:44

srl295 force-pushed the cldr-14399-api branch from 8006220 to b5655a2 Compare January 15, 2021 22:45

srl295 force-pushed the cldr-14399-api branch from 09bfdfc to 0b13d2f Compare January 19, 2021 20:31

srl295 commented Jan 19, 2021

View reviewed changes

srl295 force-pushed the cldr-14399-api branch from 0b13d2f to a2e2664 Compare January 19, 2021 23:40

srl295 force-pushed the cldr-14399-api branch from a2e2664 to f614e8d Compare January 20, 2021 14:41

srl295 force-pushed the cldr-14399-api branch from f614e8d to 52d2e8a Compare January 20, 2021 14:52

srl295 force-pushed the cldr-14399-api branch from 52d2e8a to 13c7068 Compare January 20, 2021 15:07

srl295 force-pushed the cldr-14399-api branch from 13c7068 to 941e7a6 Compare January 20, 2021 16:51

srl295 force-pushed the cldr-14399-api branch from 941e7a6 to 4a4529d Compare January 20, 2021 20:32

srl295 force-pushed the cldr-14399-api branch from 4a4529d to 0405eb7 Compare January 20, 2021 20:32

srl295 marked this pull request as ready for review January 20, 2021 20:33

btangmu requested changes Jan 22, 2021

View reviewed changes

srl295 force-pushed the cldr-14399-api branch from 0405eb7 to 30c24b5 Compare January 25, 2021 14:53

srl295 force-pushed the cldr-14399-api branch from 30c24b5 to b3fdd1e Compare January 25, 2021 14:54

CLDR-14399 move to OpenLiberty, add API server

07014bf

- deployment is now different, but the github action has been updated - for local dev: see cldr-apps/README.md

srl295 force-pushed the cldr-14399-api branch from b3fdd1e to 07014bf Compare January 25, 2021 22:43

btangmu self-requested a review January 26, 2021 02:04

btangmu approved these changes Jan 26, 2021

View reviewed changes

srl295 merged commit c1befe3 into unicode-org:master Jan 26, 2021

srl295 deleted the cldr-14399-api branch January 26, 2021 14:58

		@@ -0,0 +1,4 @@
		# copy this to server.env and set the password properly

CLDR-14399 move to OpenLiberty, add API server #936

CLDR-14399 move to OpenLiberty, add API server #936

Conversation

srl295 commented Jan 14, 2021 • edited Loading

srl295 commented Jan 14, 2021 • edited Loading

srl295 Jan 14, 2021

Choose a reason for hiding this comment

srl295 Jan 14, 2021

Choose a reason for hiding this comment

srl295 Jan 19, 2021

Choose a reason for hiding this comment

srl295 Jan 15, 2021

Choose a reason for hiding this comment

srl295 commented Jan 15, 2021

srl295 commented Jan 15, 2021

jira-pull-request-webhook bot commented Jan 15, 2021

srl295 commented Jan 15, 2021

jira-pull-request-webhook bot commented Jan 15, 2021

jira-pull-request-webhook bot commented Jan 19, 2021

btangmu commented Jan 19, 2021

srl295 Jan 19, 2021

Choose a reason for hiding this comment

srl295 Jan 19, 2021

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented Jan 19, 2021

srl295 commented Jan 19, 2021

jira-pull-request-webhook bot commented Jan 20, 2021

jira-pull-request-webhook bot commented Jan 20, 2021

jira-pull-request-webhook bot commented Jan 20, 2021

jira-pull-request-webhook bot commented Jan 20, 2021

jira-pull-request-webhook bot commented Jan 20, 2021

jira-pull-request-webhook bot commented Jan 20, 2021

srl295 commented Jan 20, 2021

btangmu commented Jan 22, 2021

srl295 commented Jan 22, 2021 • edited Loading

btangmu commented Jan 22, 2021

btangmu commented Jan 22, 2021 • edited Loading

srl295 commented Jan 22, 2021

btangmu left a comment

Choose a reason for hiding this comment

btangmu commented Jan 22, 2021

srl295 commented Jan 22, 2021 via email

jira-pull-request-webhook bot commented Jan 25, 2021

jira-pull-request-webhook bot commented Jan 25, 2021

jira-pull-request-webhook bot commented Jan 25, 2021

btangmu left a comment

Choose a reason for hiding this comment

srl295 commented Jan 26, 2021

srl295 commented Jan 26, 2021

srl295 commented Jan 14, 2021 •

edited

Loading

srl295 commented Jan 14, 2021 •

edited

Loading

srl295 commented Jan 22, 2021 •

edited

Loading

btangmu commented Jan 22, 2021 •

edited

Loading