Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Multi-Database Support][h2] add h2 init sql #4775

Closed
wants to merge 1 commit into from
Closed

[Multi-Database Support][h2] add h2 init sql #4775

wants to merge 1 commit into from

Conversation

shoothzj
Copy link
Member

@shoothzj shoothzj commented Mar 5, 2023 •
edited
Loading

What's the purpose of this PR

add h2 init sql to complete h2 support.

Brief changelog

  • add h2 init sql when profile is h2
  • refresh ServerConfig immediately when write in to h2.

How to local quick start h2

configservice

-Dapollo_profile=github,database-discovery
-Dspring.profiles.group.github=h2
-Dspring.datasource.url=jdbc:h2:mem:testdb;DATABASE_TO_UPPER=FALSE
-Dspring.datasource.username=sa
-Dspring.datasource.password=

adminservice

-Dapollo_profile=github,database-discovery
-Dspring.profiles.group.github=h2
-Dspring.datasource.url=jdbc:h2:mem:testdb;DATABASE_TO_UPPER=FALSE
-Dspring.datasource.username=sa
-Dspring.datasource.password=

portal

-Dapollo_profile=github,database-discovery
-Dspring.profiles.group.github=h2
-Dspring.datasource.url=jdbc:h2:mem:testdb;DATABASE_TO_UPPER=FALSE
-Dspring.datasource.username=sa
-Dspring.datasource.password=
-Ddev_meta=http://localhost:8080/

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Read the Contributing Guide before making this pull request.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Write necessary unit tests to verify the code.
  • Run mvn clean test to make sure this pull request doesn't break anything.
  • Update the CHANGES log.

@codecov
Copy link

codecov bot commented Mar 5, 2023 •
edited
Loading

Codecov Report

Merging #4775 (5e0a1c3) into master (2f311ae) will decrease coverage by 0.05%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #4775      +/-   ##
============================================
- Coverage     47.26%   47.21%   -0.05%     
  Complexity     1657     1657              
============================================
  Files           346      348       +2     
  Lines         10683    10693      +10     
  Branches       1063     1063              
============================================
  Hits           5049     5049              
- Misses         5328     5338      +10     
  Partials        306      306
Impacted Files Coverage Δ
...m/ctrip/framework/apollo/biz/entity/AccessKey.java 58.33% <ø> (ø)
...a/com/ctrip/framework/apollo/biz/entity/Audit.java 52.94% <ø> (ø)
...com/ctrip/framework/apollo/biz/entity/Cluster.java 58.82% <ø> (ø)
.../com/ctrip/framework/apollo/biz/entity/Commit.java 5.55% <ø> (ø)
...p/framework/apollo/biz/entity/GrayReleaseRule.java 100.00% <ø> (ø)
...om/ctrip/framework/apollo/biz/entity/Instance.java 43.24% <ø> (ø)
...ip/framework/apollo/biz/entity/InstanceConfig.java 71.11% <ø> (ø)
...va/com/ctrip/framework/apollo/biz/entity/Item.java 86.36% <ø> (ø)
...m/ctrip/framework/apollo/biz/entity/Namespace.java 61.11% <ø> (ø)
...rip/framework/apollo/biz/entity/NamespaceLock.java 25.00% <ø> (ø)
... and 27 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

nobodyiam
nobodyiam reviewed Mar 6, 2023
View reviewed changes
Copy link
Member

@nobodyiam nobodyiam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe h2 support could make the quick start much easier, would you please help to add some instructions on how to enable h2 mode? e.g. in the quick start section(zh, en)

apollo-biz/src/main/java/com/ctrip/framework/apollo/biz/entity/ServerConfig.java Outdated Show resolved Hide resolved
apollo-biz/src/main/java/com/ctrip/framework/apollo/biz/service/BizDBPropertySource.java Show resolved Hide resolved
@shoothzj
Copy link
Member Author

shoothzj commented Mar 6, 2023

@nobodyiam h2's support is working. I will submit a doc PR this week.

nobodyiam
nobodyiam reviewed Mar 7, 2023
View reviewed changes
Copy link
Member

@nobodyiam nobodyiam left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, the login function doesn't work in my laptop...The startup options are as follows:

-Dapollo_profile=github,auth
-Dserver.port=8070
-Dspring.profiles.group.github=h2
-Dspring.datasource.url=jdbc:h2:mem:testdb
-Dspring.datasource.username=sa
-Dspring.datasource.password=
-Dapollo.portal.envs=dev
-Ddev_meta=http://localhost:8080
2023-03-07 08:35:36.145 ERROR 7850 --- [nio-8070-exec-4] w.a.UsernamePasswordAuthenticationFilter : An internal error occurred while trying to authenticate the user.

org.springframework.security.authentication.InternalAuthenticationServiceException: PreparedStatementCallback; bad SQL grammar [select Username,Password,Enabled from `Users` where Username = ?]; nested exception is org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "USERS" not found (candidates are: "Users"); SQL statement:
select Username,Password,Enabled from `Users` where Username = ? [42103-214]
	at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:109)
	at org.springframework.security.authentication.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:133)
	at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:182)
	at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:201)
	at org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter.attemptAuthentication(UsernamePasswordAuthenticationFilter.java:85)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
	at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142)
	at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select Username,Password,Enabled from `Users` where Username = ?]; nested exception is org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "USERS" not found (candidates are: "Users"); SQL statement:
select Username,Password,Enabled from `Users` where Username = ? [42103-214]
	at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:93)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79)
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815)
	at org.springframework.security.provisioning.JdbcUserDetailsManager.loadUsersByUsername(JdbcUserDetailsManager.java:177)
	at org.springframework.security.core.userdetails.jdbc.JdbcDaoImpl.loadUserByUsername(JdbcDaoImpl.java:184)
	at org.springframework.security.authentication.dao.DaoAuthenticationProvider.retrieveUser(DaoAuthenticationProvider.java:94)
	... 68 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "USERS" not found (candidates are: "Users"); SQL statement:
select Username,Password,Enabled from `Users` where Username = ? [42103-214]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:477)
	at org.h2.message.DbException.get(DbException.java:223)
	at org.h2.command.Parser.getTableOrViewNotFoundDbException(Parser.java:8401)
	at org.h2.command.Parser.getTableOrViewNotFoundDbException(Parser.java:8369)
	at org.h2.command.Parser.readTableOrView(Parser.java:8358)
	at org.h2.command.Parser.readTablePrimary(Parser.java:1863)
	at org.h2.command.Parser.readTableReference(Parser.java:2334)
	at org.h2.command.Parser.parseSelectFromPart(Parser.java:2772)
	at org.h2.command.Parser.parseSelect(Parser.java:2878)
	at org.h2.command.Parser.parseQueryPrimary(Parser.java:2762)
	at org.h2.command.Parser.parseQueryTerm(Parser.java:2633)
	at org.h2.command.Parser.parseQueryExpressionBody(Parser.java:2612)
	at org.h2.command.Parser.parseQueryExpressionBodyAndEndOfQuery(Parser.java:2605)
	at org.h2.command.Parser.parseQueryExpression(Parser.java:2598)
	at org.h2.command.Parser.parseQuery(Parser.java:2567)
	at org.h2.command.Parser.parsePrepared(Parser.java:724)
	at org.h2.command.Parser.parse(Parser.java:689)
	at org.h2.command.Parser.parse(Parser.java:661)
	at org.h2.command.Parser.prepareCommand(Parser.java:569)
	at org.h2.engine.SessionLocal.prepareLocal(SessionLocal.java:631)
	at org.h2.engine.SessionLocal.prepareCommand(SessionLocal.java:554)
	at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1116)
	at org.h2.jdbc.JdbcPreparedStatement.<init>(JdbcPreparedStatement.java:92)
	at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:288)
	at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)
	at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
	at org.springframework.jdbc.core.JdbcTemplate$SimplePreparedStatementCreator.createPreparedStatement(JdbcTemplate.java:1645)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:649)
	... 75 common frames omitted

...c/main/java/com/ctrip/framework/apollo/configservice/service/jpa/h2/ConfigH2InitService.java
public void runSqlScript() throws Exception {
Resource resource = new ClassPathResource("jpa/init.h2.sql");
DatabasePopulatorUtils.execute(new ResourceDatabasePopulator(resource), dataSource);
dbPropertySource.refresh();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't work to manually refresh it as some configs are only loaded once in the startup phase, e.g. portalSupportedEnvs.
To make it more compatible, it's better to load the data in an early stage.
I tried to use the jpa initialization mechanism(data.sql) but got some error saying the table serverconfig not found. There might be some configuration issue but I think this is a direction worth exploring.

Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "ServerConfig" not found (this database is empty); SQL statement:
INSERT INTO "ServerConfig" ("Key", "Cluster", "Value", "Comment", "DataChange_CreatedBy", "DataChange_CreatedTime") VALUES ('eureka.service.url', 'default', 'http://localhost:8080/eureka/', 'Eureka服务Url，多个service以英文逗号分隔', 'default', '1970-01-01 00:00:00'), ('namespace.lock.switch', 'default', 'false', '一次发布只能有一个人修改开关', 'default', '1970-01-01 00:00:00'), ('item.key.length.limit', 'default', '128', 'item key 最大长度限制', 'default', '1970-01-01 00:00:00'), ('item.value.length.limit', 'default', '20000', 'item value最大长度限制', 'default', '1970-01-01 00:00:00'), ('config-service.cache

image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nobodyiam How could we control profile when using data.sql?

@stale
Copy link

stale bot commented Apr 6, 2023

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 14 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

@stale stale bot added the stale label Apr 6, 2023
@nobodyiam nobodyiam removed the stale label Apr 7, 2023
@shoothzj
Copy link
Member Author

shoothzj commented May 6, 2023

closed by #4851

@shoothzj shoothzj closed this May 6, 2023
@shoothzj shoothzj deleted the h2-init-sql branch May 6, 2023 01:03
@github-actions github-actions bot locked and limited conversation to collaborators May 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@nobodyiam nobodyiam nobodyiam left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shoothzj @nobodyiam