batch-reset creates an empty record at the end when writing and many empty records with encode marcxml

[TobiasNx]

See: TobiasNx/metafacture_workflows@64ba3e8

Running the workflow with Metafix-Runner 1.0.0

When using the flux module batch-reset and set the batch size to "1" (| batch-reset(batchsize="1"). MF creates an empty record after the last transfromation.

batch-reset should not output empty records.

e.g.:

infile
| open-file
| decode-xml
| handle-generic-xml
| fix(FLUX_DIR + "test.fix")
| batch-reset(batchsize="1")
| encode-xml
| write(FLUX_DIR + "test-output-${i}.xml")
;

or

infile
| open-file
| decode-xml
| handle-generic-xml
| fix(FLUX_DIR + "test.fix")
| batch-reset(batchsize="1")
| encode-yaml
| write(FLUX_DIR + "test-output-${i}.xml")
;

[TobiasNx]

After fixing #525 some strange behavior occures when using the combination of batch-reset AND encode-marcxml:

With encode-marcxml it creates an additional empty record with every single record.

See: https://gitlab.com/oersi/oersi-marc/-/commit/2abe46321aefc650fe85e5a88a43eba0add3b649

It must have to do with the combination of batch-reset error AND the changes done for encode-marcxml

With Metafix Runner 1.1.1

directory
| read-dir
| open-file
| as-records
| decode-json
| fix(FLUX_DIR + "oersiToMarc.fix", *)
| encode-marcxml
| write(FLUX_DIR + "test-output-${i}.xml")
;

[dr0i]

I think here are two issues mixed - one of them is no issue at all ( s. a) & b) ).
·
a) IMO it's ok - when explicitly resetting the stream - that a new file is opened (it's stated exactly like this in

metafacture-core/metafacture-io/src/test/java/org/metafacture/io/ObjectFileWriterTest.java

Line 103 in f5cc9dc

writer.resetStream(); // increments count, starts new file

.

b) You use batch-reset in combination with write-files to determine how many records should be in one file before another file will be created (which is renamed by incrementing the number i). If you use batchsize=1 it's always so that a) happens, i.e. the latest file will be empty. If you choose $countOfRecordsInInput modulo $batchsize > 0 you will not have an empty file at the end (s.

metafacture-core/metafacture-flowcontrol/src/main/java/org/metafacture/flowcontrol/StreamBatchResetter.java

Line 90 in f5cc9dc

recordCount %= batchSize;

)

c) Not https://github.com/metafacture/metafacture-core/pull/532/files changed the behaviour. It has indeed something to do with encode-marcxml, it's a consequence of #527 where a wrapper is used, where the wrapper also calls the resetStream() and thus two files were being created every time the stream was resetted. Fixed with 04a6312.

I've also discovered that when an empty process is triggered in ObjectFileWriter
a linebreak was nonetheless inserted. So there were sometimes (modulo!) files with one byte (the line break). This is fixed in 16b9349.

Another bug surfaced: if a record was empty a footer was written. This is fixed in 0ea6d23.

A good way to test this is the following flux (which is useful with CLI, not for playground because of the written files which you cannot see in that web app):

"http://lobid.org/download/marcXml-8-records.xml"
| open-http(accept="application/xml")
| decode-xml
| handle-marcxml
| batch-reset(batchsize="5")
| encode-marcxml
| write("test-output-${i}.xml")
;

[blackwinter]

it's a consequence of #527 where a wrapper is used

You mean #524, right? The wrapper was introduced in #539, not in #538.

[TobiasNx]

@dr0i i tested it with: https://github.com/TobiasNx/notWorkingFlux/blob/f8c3f5167e0ab254e7efd8498785ba1036cef822/batchResetEmptyFiles/batchResetEmptyFile.flux

See:
TobiasNx/notWorkingFlux@f8c3f51

The additional empty records are not processed anymore. Which is good :)
But I am not sure if I understood your comment correctly.

a) -> empty record at the end is created because of the resetting, right? So the empty record at the end is intentional?

b) what is the difference between write and write-file?
Anyway, if you choose batchsize > 0 is no guarantee that you wont get a empty file at the end. you get it always if the number of entering records is dividable by the batchsize. In my example commit above you see that 10 records go in and batchsize is 5. So batchsize is > 1. But we still get an empty record at the end because 10 is dividable by 5 so it creates 2 files with each 5 records and an empty one at the end. (Your test scenario got 8 records in but batchsize is 5, 8 is not dividable by 5 therefore no empty record.)

Not solved as stated.

c) is fixed which is great.

I've also discovered that when an empty process is triggered in ObjectFileWriter a linebreak was nonetheless inserted. So there were sometimes (modulo!) files with one byte (the line break). This is fixed in 16b9349.

Looks goo. There seems to be a linebreak between records too. That is not new but I dont know if we need it too?
But it does not create any problems therefore no priority: https://github.com/TobiasNx/notWorkingFlux/blob/f8c3f5167e0ab254e7efd8498785ba1036cef822/batchResetEmptyFiles/output/test-output-0.xml#L215

Another bug surfaced: if a record was empty a footer was written. This is fixed in 0ea6d23.

Looks good the footer in empty record is now deleted.

[dr0i]

a) -> empty record at the end is created because of the resetting, right? So the empty record at the end is intentional?

No. A new file is intentionally opened. If it's filled with data or not depends on the data, the resetsize etc.

b) ...Not solved as stated....

sure it is. What you said in your words is what mod does. And this is intentionally.

c) There seems to be a linebreak between records too. That is not new but I dont know if we need it too?

you may want to open a new issue