Osmosis efficiency problem - suggestion

[openstreetmap-trac]

Reporter: bemysocialdate[at]gmail.com
[Submitted to the original trac issue database at 8.04am, Wednesday, 30th August 2017]

I'm trying to make some more efficient pbf reader, and i had a chance to look at the osmosis one.
Although it's not reaching any major release version, but i have spotted two bugs:

1, evaluating DenseNode has got a programming error:
private void processNodes(DenseNodes nodes, PbfFieldDecoder fieldDecoder)

Looking at the second cycle, with keysValuesIterator unnecessary loops has been introduced.

The main problem in the file it's massive.

If the keysValuesIterator loop would be removed, and a ComplexEntityData would be introduced, osmosis would speed up 5 times.

ComplexEntityData
{
entities: List - CommonEntityData does not hold any tag in this case.
tags : List
}

2, Even if osmosis using some sort of Threading, it's not using all the cores of my processor.

I have refactored the whole thing using RandomAccessFile, and LinkedBlockingQueue for task separation.

Using
ExecutorService executorService = Executors.newFixedThreadPool(workers); inside the PbfReader.run method is a shame.

it should be:

int workers = 4
PbfReader pbfReader = new PbfReader();
ExecutorService executorService = Executors.newFixedThreadPool();
executorService.submit(pbfReader);

I have attached my refactored code:

Some of the classes are superfluous, as previously i didn't know about pbf, and i have recreated some of them, but realized later on.

The example is using all the 4 cores in my CPU, loading a huge map file, which i think on a modern computer is ok.

My purpose was not to recreate the osmosis that's why i haven't finished this work. I had to make some exercise to jump to node.

[openstreetmap-trac]

Author: bemysocialdate[at]gmail.com
[Added to the original trac issue at 6.43pm, Saturday, 2nd September 2017]

First point is irrelevant anymore, as i have misread the code.

I think it's simply an unreadable code. Now i know there is a delimiter between tags of nodes.