wrong coordinate in LTSM ocr mode and Japanese

[hoangtocdo90]

Hi all
i'm using Tesseract for get each char with Coordinate in image . I'm using ResultIterator with OCR MODE =2 (LTSM) and language = jpn.

tesseract::ResultIterator* ri = api->GetIterator();
int index_char = 0;
vector char_iterators;
do {
char *value = ri->GetUTF8Text(tesseract::RIL_SYMBOL);
//unknown value to space
if (value == nullptr || value == "")value = " ";
float conf = ri->Confidence(tesseract::RIL_SYMBOL);
ri->BoundingBox(tesseract::RIL_SYMBOL, &left, &top, &right, &bottom);
index_char++;
} while (ri->Next(tesseract::RIL_SYMBOL));
api->ClearAdaptiveClassifier();

Here is my program log and input image . You can see in り character i got wrong Coordinate . I tested using tsv and hocr but it's give me same result. Still wrong Coordinate .

Char value = こ left= 15 top = 14 right = 51 bottom = 51 conf = 99
Char value = ん left= 64 top = 9 right = 112 bottom = 54 conf = 99
Char value = ば left= 122 top = 5 right = 171 bottom = 54 conf = 99
Char value = ん left= 176 top = 9 right = 224 bottom = 54 conf = 99
Char value = は left= 234 top = 9 right = 281 bottom = 54 conf = 99
Char value = こ left= 295 top = 14 right = 331 bottom = 51 conf = 99
Char value = ん left= 344 top = 9 right = 392 bottom = 54 conf = 99
Char value = ば left= 402 top = 5 right = 445 bottom = 54 conf = 99
Char value = ん left= 456 top = 9 right = 497 bottom = 54 conf = 99
Char value = は left= 514 top = 9 right = 561 bottom = 54 conf = 99
Char value = ご left= 15 top = 79 right = 58 bottom = 126 conf = 99
Char value = 飯 left= 62 top = 80 right = 113 bottom = 130 conf = 99
Char value = 大 left= 120 top = 80 right = 225 bottom = 130 conf = 99
Char value = 盛 left= 242 top = 83 right = 260 bottom = 130 conf = 99
Char value = り left= 2328 top = 1616 right = 2328 bottom = 1616 conf = 99
Char value = 。 left= 289 top = 116 right = 305 bottom = 131 conf = 99

And one more question . I'm try to and fonts in jpn data but may be i must re train from scratch. But i don't know actrually my jpn tessseract data (i'm downloaded from tessdata repository) how to make this?
I'm try download data from langdata repository make image from jpn.traintext and train it by using tesstrain.sh and Jtessboxeditor . But i got low accurary than i download from repository. Some body can tell me extractly how to make it!
Sorry for my bad english

[kandaman]

i got a same problem. i am using jpn.traindata.
i tried RIL_SYMBOL, RIL_WORD. RIL_SYMBOL is better.
A critical problem is ---- the recgnized character is so good but the position is too bad.
i need the pair of image and character, don't you?
If you have new information pls tell me.

thanks

[hoangtocdo90]

I'm temple fix this by using this way
I'm using RIL_SYMBOL. in my case the wrong Coordinate usually appear in a end of lines or end of block
res_it->IsAtFinalElement(RIL_TEXTLINE, RIL_WORD)
res_it->IsAtFinalElement(RIL_PARA, RIL_WORD)
res_it->IsAtFinalElement(RIL_BLOCK, RIL_WORD)
when you get a wrong Coordinate you can predict a new coordinate by using the backforward of ResultIterator coordinate

[amitdo]

Char value = こ left= 15 top = 14 right = 51 bottom = 51 conf = 99
Char value = ん left= 64 top = 9 right = 112 bottom = 54 conf = 99
Char value = ば left= 122 top = 5 right = 171 bottom = 54 conf = 99
Char value = ん left= 176 top = 9 right = 224 bottom = 54 conf = 99
Char value = は left= 234 top = 9 right = 281 bottom = 54 conf = 99
Char value = こ left= 295 top = 14 right = 331 bottom = 51 conf = 99
Char value = ん left= 344 top = 9 right = 392 bottom = 54 conf = 99
Char value = ば left= 402 top = 5 right = 445 bottom = 54 conf = 99
Char value = ん left= 456 top = 9 right = 497 bottom = 54 conf = 99
Char value = は left= 514 top = 9 right = 561 bottom = 54 conf = 99
Char value = ご left= 15 top = 79 right = 58 bottom = 126 conf = 99
Char value = 飯 left= 62 top = 80 right = 113 bottom = 130 conf = 99
Char value = 大 left= 120 top = 80 right = 225 bottom = 130 conf = 99
Char value = 盛 left= 242 top = 83 right = 260 bottom = 130 conf = 99
Char value = り left= 2328 top = 1616 right = 2328 bottom = 1616 conf = 99
Char value = 。 left= 289 top = 116 right = 305 bottom = 131 conf = 99

Strange. It looks like a bug.

[Shreeshrii]

Please check if this is fixed by the latest set of commits by Ray.

jpn-1.txt
jpn-1.tsv.txt

level	page_num	block_num	par_num	line_num	word_num	left	top	width	height	conf	text
1	1	0	0	0	0	0	0	2550	470	-1	
2	1	1	0	0	0	104	96	2286	348	-1	
3	1	1	1	0	0	111	96	546	49	-1	
4	1	1	1	1	0	111	96	546	49	-1	
5	1	1	1	1	1	111	105	36	37	96	こ
5	1	1	1	1	2	160	100	48	45	96	ん
5	1	1	1	1	3	218	96	49	49	96	ば
5	1	1	1	1	4	272	100	48	45	96	ん
5	1	1	1	1	5	330	100	47	45	96	は
5	1	1	1	1	6	391	105	36	37	95	こ
5	1	1	1	1	7	440	100	48	45	96	ん
5	1	1	1	1	8	498	96	49	49	96	ば
5	1	1	1	1	9	552	100	48	45	96	ん
5	1	1	1	1	10	610	100	47	45	95	は
3	1	1	2	0	0	111	170	962	52	-1	
4	1	1	2	1	0	111	170	962	52	-1	
5	1	1	2	1	1	111	170	43	47	96	ご
5	1	1	2	1	2	158	171	107	50	95	飯
5	1	1	2	1	3	271	171	50	50	96	大
5	1	1	2	1	4	338	174	29	47	96	盛
5	1	1	2	1	5	0	0	2550	470	96	り
5	1	1	2	1	6	385	207	16	15	96	。
5	1	1	2	1	7	439	172	123	50	93	今
5	1	1	2	1	8	0	0	2550	470	95	年
5	1	1	2	1	9	567	171	65	51	96	は
5	1	1	2	1	10	624	173	87	48	96	初
5	1	1	2	1	11	0	0	2550	470	95	め
5	1	1	2	1	12	722	178	43	41	96	て
5	1	1	2	1	13	776	171	48	50	94	恋
5	1	1	2	1	14	832	173	101	48	96	人
5	1	1	2	1	15	944	171	48	50	96	出
5	1	1	2	1	16	1001	173	26	46	96	来
5	1	1	2	1	17	1021	191	25	28	96	た
5	1	1	2	1	18	1057	207	16	15	96	。
3	1	1	3	0	0	106	245	2284	126	-1	
4	1	1	3	1	0	106	245	2284	51	-1	

[hoangtocdo90]

Thank sir

[hoangtocdo90]

5 1 1 2 1 1 111 170 43 47 96 ご
5 1 1 2 1 2 158 171 107 50 95 飯
5 1 1 2 1 3 271 171 50 50 96 大
5 1 1 2 1 4 338 174 29 47 96 盛
5 1 1 2 1 5 0 0 2550 470 96 り
2 1 10 624 173 87 48 96 初
5 1 1 2 1 11 0 0 2550 470 95 め
5 1 1 2 1 12 722 178 43 41 96 て

Please check this . Still wrong coordinate in り and め character

[GHamrouni]

We are still able to reproduce it in the Arabic language in LSTM mode.
Most BBoxes are correct but there are some boxes that contain valid text and wrong coordinates (the region contained in the bbox is empty).

[SimonTheBaptist]

I'm getting the same behavior for Thai language in LSTM - BoundingBox() often returns the whole image size.
The image size was 400, 266. Here is a small portion of some results [X1, Y1; X2, Y2].
(As a side note, I'm using RIL_WORD, but it seems to behave like RIL_SYMBOL, I'm not sure why).

'ร' - Confidence: 94.3645 [0, 0; 400, 266]
'ม' - Confidence: 95.7061 [19, 68; 33, 77]
'า' - Confidence: 96.9703 [0, 0; 400, 266]
'ส' - Confidence: 96.976 [35, 67; 50, 77]

[wanghaisheng]

@amitdo sir could you show me where to get more info about how tesseract analyze input image to get the Coordinate of words/character and then recognize them through LSTM or old method and last combine the ocr result word with the coordinate ?

[amitdo]

@wanghaisheng

See here:
https://github.com/tesseract-ocr/tesseract/blob/master/lstm/recodebeam.cpp
Search for 'box', 'xcoords', 'blob'

[aniseddali]

This happened to me also in Arabic language. Here is an example that reproduce the problem.

struct OcrResult
{
	std::string text;
	cv::Rect box;
};
int main(int argc, char *argv[])
{
	tesseract::TessBaseAPI tesseract ;
	tesseract.Init("./data/tessdata/", "ara", tesseract::OcrEngineMode::OEM_LSTM_ONLY);
	tesseract.SetPageSegMode(tesseract::PageSegMode::PSM_SINGLE_WORD);
	PIX *patch_pix = pixRead(argv[1]);
	tesseract.SetImage(patch_pix);
	tesseract.Recognize(0);
	std::vector<OcrResult> ocrResults;
	tesseract::ResultIterator *ri = tesseract.GetIterator();
	tesseract::PageIteratorLevel level = tesseract::RIL_WORD ;
	if (ri != 0)
	{
		do
		{
			char *word = ri->GetUTF8Text(level);
			int left, top, right, bottom;
			if (ri->BoundingBox(level, &left, &top, &right, &bottom))
			{
				OcrResult res;
				res.box = cv::Rect(left,top,right - left,bottom - top);
				res.text = std::string(word);
				ocrResults.push_back(res);
			}
			delete[] word;
		} while (ri->Next(level));
	}
	cv::Mat image = cv::imread(argv[1]);
	cv::Mat DrawingImg = image.clone();
	for(int i=0;i<ocrResults.size();i++){
		cv::Rect rect = ocrResults[i].box; 
		cv::rectangle(DrawingImg, rect, cv::Scalar(255, 0, 0), 1);
		std::cout<<ocrResults[i].text<<std::endl;
		cv::imshow("DrawingImg",DrawingImg);
		cv::waitKey();
	}
}

I used OpenCV to draw boxes.
This image contains two arabic words. The recognition is correct for both words.
But the box position of the first word is wrong. (the word in the right)
The box is matching some noise on the top of the image.

And this is the original image.

[lpatruno]

I'm also getting this bug for english text, though I can't provide the data files as they contain PII.

[atuyosi]

I got same issue on beta.4 with jpn.traineddata.

In my case, the image size(width, height) and the invalid coordinate value are correlated.
Even with the same letter,　It's cause incorrect results depending on the position in the image.

$ tesseract -l jpn  'images/sample/test-jpn_01.jpg' stdout tsv | grep 596
Warning. Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 167
1	1	0	0	0	0	0	0	596	118	-1
5	1	1	1	1	10	0	0	596	118	92	、
5	1	1	1	2	4	0	0	596	118	92	字
5	1	1	1	2	10	0	0	596	118	97	て
5	1	1	1	2	15	0	0	596	118	93	す

My test image size is 596x118. The same letter appears multiple times(ex. '字', 'て'), but the value of boundigbox is wrong only once.

FYI, In the above image, recognition of the character '日' incorrect by jpn.traineddata( traineddata_fast).

[amitdo]

Same issue as #1192