Flyway migrations CRC32 checksum different after upgrading to 1.2.12 (from 1.2.11)

[nmapx]

There are multiple projects that rely on this library. In my case it is Flyway migrations that is using it to calculate checksum. 2 days ago it started producing different values for CRC32 what caused app crash on startup. It seems your recent release is not backwards compatible and shouldn't be a patch. Please verify it.

[madler]

Please provide more information and/or links to information on the issue you are referring to. This sounds serious, but there is nothing actionable in your comment.

[nmapx]

No problem. I'm using Spring Boot along with Flyway. By default it's validating migrations by calculating checksum based on the files. Migrations that were already executed have checksum stored in the database. This values are different then the ones calculated after migrating to recent zlib=1.2.12

2022-03-30 17:52:38,983 [main]  ERROR SpringApplication Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Migration checksum mismatch for migration version 20210507103222
-> Applied to database : 1878526119
-> Resolved locally    : 2013283769. Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 20210511103222
-> Applied to database : -584965974
-> Resolved locally    : 747715409. Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 20210527103222
-> Applied to database : 801009297
-> Resolved locally    : -556823002. Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 20210916160901
-> Applied to database : -1749342395
-> Resolved locally    : -333971783. Either revert the changes to the migration, or run repair to update the schema history.
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1786)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1154)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:908)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332)
        at ...
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:567)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation

[nmapx]

Flyway is using following code to calculate CRC32
https://github.com/flyway/flyway/blob/1fc8cff5356f9a151ea72e6d216a5586d4a03b31/flyway-core/src/main/java/org/flywaydb/core/internal/resolver/ChecksumCalculator.java

    private static int calculateChecksumForResource(LoadableResource resource) {
        final CRC32 crc32 = new CRC32();

        BufferedReader bufferedReader = null;
        try {
            bufferedReader = new BufferedReader(resource.read(), 4096);
            String line = bufferedReader.readLine();

            if (line != null) {
                line = BomFilter.FilterBomFromString(line);
                do {
                    //noinspection Since15
                    crc32.update(line.getBytes(StandardCharsets.UTF_8));
                } while ((line = bufferedReader.readLine()) != null);
            }
        } catch (IOException e) {
            throw new FlywayException("Unable to calculate checksum of " + resource.getFilename() + "\r\n" + e.getMessage(), e);
        } finally {
            IOUtils.close(bufferedReader);
        }

        return (int) crc32.getValue();
    }

[madler]

This commit should remedy the incorrect CRC inputs: ec3df00 . Please let me know if it does.

[nmapx]

It would require some effort to test it on my side before you actually release it. I trust you if you are confident enough. If you need any more details in order to test it locally I can provide it, just tell me what you need.

[madler]

What code is saying "Migration checksum mismatch"?

[nmapx]

https://github.com/flyway/flyway/blob/21519baaf861ff06d0b5aa87271793707964edf5/flyway-core/src/main/java/org/flywaydb/core/internal/info/MigrationInfoImpl.java#L394

This one i guess. If you follow the createMismatchMessage method you will find this
https://github.com/flyway/flyway/blob/21519baaf861ff06d0b5aa87271793707964edf5/flyway-core/src/main/java/org/flywaydb/core/internal/info/MigrationInfoImpl.java#L437

and

https://github.com/flyway/flyway/blob/21519baaf861ff06d0b5aa87271793707964edf5/flyway-core/src/main/java/org/flywaydb/core/internal/info/MigrationInfoImpl.java#L403

[nmapx]

Any progress here?

[demitriusbelai]

I confirm ec3df00 fixes the issue. I was having problem build Spring Boot project too:

[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.6.6:repackage (repackage) on project unespfc-servicedesk: invalid entry CRC (expected 0x6be9f308 but got 0x7dfa3146)

https://stackoverflow.com/questions/71698867/invalid-entry-crc-in-spring-boot
https://bugs.archlinux.org/task/74371

[phiresky]

This also seems to affect google closure compiler and thus breaks emscripten:

$ google-closure-compiler
java.util.zip.ZipException: invalid entry CRC (expected 0x4e1f14a4 but got 0xb1e0eb5b)
	at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:410)
	at java.util.zip.ZipInputStream.read(ZipInputStream.java:199)
	at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:143)
	at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:121)
	at com.google.javascript.jscomp.AbstractCommandLineRunner.getBuiltinExterns(AbstractCommandLineRunner.java:481)
	at com.google.javascript.jscomp.CommandLineRunner.createExterns(CommandLineRunner.java:2083)
	at com.google.javascript.jscomp.AbstractCommandLineRunner.doRun(AbstractCommandLineRunner.java:1168)
	at com.google.javascript.jscomp.AbstractCommandLineRunner.run(AbstractCommandLineRunner.java:532)
	at com.google.javascript.jscomp.CommandLineRunner.main(CommandLineRunner.java:2241)

[phiresky]

I can confirm that installing zlib from the develop branch fixes this issue and allows me to start google-closure-compiler.

It would be great if you could release a new version since this seems to affect multiple unrelated things.

[madler]

Thank you for the verification. There will be a new release. However all instances of zlib 1.2.12 impacting CRC calculations should be reported as bugs in those applications. The problem is caused by invalid CRC-32 values being giving to zlib's CRC function.

[demitriusbelai]

@madler I am not a C expert but I have taken a look at the java implementation and I do not realise the problem. There is no crc32(0L, Z_NULL, 0); but it will just return 0 and 0 is the default inital value for integer in Java. The int in Java (jint in C code) is a 32 bits signed integer and it is converted to long signed at the end of crc32 computation. Using signed 32 bits integer can be a problem?

I do not know why it occur only on my computer with i7 870. It do not occur on my i5 7600k. Can it be a CPU related behaviour?

Thank you in advance.

[LanceAndersen]

Thank you for the verification. There will be a new release. However all instances of zlib 1.2.12 impacting CRC calculations should be reported as bugs in those applications. The problem is caused by invalid CRC-32 values being giving to zlib's CRC function.

Hi Mark,

Wanted to ask, based on your comment above, do you have a timeframe that you were thinking for a patch release to 1.2.12 or perhaps a 1.2.13 release?