Input data

[greav]

Hello,
Could you provide your input data for the model to reproduce the results or at least the input data format so that I can try the model on my custom dataset

[ziodos]

Hello,
I couldn't find the exact input data to train the model on the icdar dataset , can you provide explanations for it ?
thanks.

[vsymbol]

Hello,
Could you provide your input data for the model to reproduce the results or at least the input data format so that I can try the model on my custom dataset

The project is refreshed with all history removed. All programs are runnable expect that the data example is not uploaded.

You may infer the correct data format from the data_loader_json.py file. Pull request is welcomed for making the project runnable out of the box. I'll let you know when the original data format can be provided, otherwise please feel free to create a pull request.

[4kssoft]

Please tell me if this is the correct data format?

Format:

file_name.json

{
  "global_attributes": {
    "file_id": "$file_name"
  },
  "fields":[
    {
      "field_name": "$class_name",
      "key_id": [],
      "key_text": [],
      "value_id": [$word_id],
      "value_text": "$word_text"
    },...
  ],

  "text_boxes":[
    {
      "id": $word_id,
      "bbox": [$word_x_min, $word_y_min, $word_x_max, $word_y_max],
      "text": "$word_text"
    },...
  ]
}

Example:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1],
      "value_text": "sample1"
    },
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [2],
      "value_text": "sample2"
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": "sample3"
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

Or maybe the correct format should look like this
Example 2:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1, 2],
      "value_text": ["sample1", "sample2"]
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": ["sample3"]
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

[langheran]

Hello @vsymbol , http://52.193.30.103 seems to be down. Could you provide the updated link?

[varshaneya]

Hello @vsymbol , http://52.193.30.103 seems to be down. Could you provide the updated link?

I too am unable to reach 52.193.30.103 even via ping. Can you confirm if this is up?

[4kssoft]

Sample data file #8 (comment)

[vsymbol]

Hello @vsymbol , http://52.193.30.103 seems to be down. Could you provide the updated link?

I too am unable to reach 52.193.30.103 even via ping. Can you confirm if this is up?

Hi varshaneya, the link is down.

[samhita-alla]

Hi,

Can you please provide details on how to generate and test the model? There are a whole lot of files and command line arguments to be given. Can you please update README as to how this model has to be trained and tested?

[Ibmaria]

Hello @vsymbol
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

[4kssoft]

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

[Ibmaria]

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

[4kssoft]

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

I use own software for labeling documents (https://www.youtube.com/watch?v=1okRMNxC0ec)

[Ibmaria]

@4kssoft
Thanks you !

[gandalf012]

@4kssoft

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

I use own software for labeling documents (https://www.youtube.com/watch?v=1okRMNxC0ec)

@4kssoft Thanks for sharing. How can i access you tool ?

[4kssoft]

@4kssoft

@4kssoft
Thanks a lot for sharing how to train the model . however how engine(api) did you use to get the boxes coordinates from the images ?
Thanks

I use own software for labeling documents (https://www.youtube.com/watch?v=1okRMNxC0ec)

@4kssoft Thanks for sharing. How can i access you tool ?

This is a beta version for now. I plan to publish this software, but not as open source

[varshaneya]

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

Could you please provide the ckpt file CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt and the invoice dataset that you had used for training?

[sathvikask0]

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

@4kssoft if possible please provide the pretrained model that you are using!

And guys for the annotation with bounding boxes please look into this link, might be useful :
Tesseract OCR: Text localization and detection

[Hrishkesh]

@4kssoft Hi I have my own data and extracted text using OCR tesseract and got the position of each word, can i know how to get in the format you showed an example in your repository for sample pdf file Faktura1.pdf_0.json how to get in this format and i need in the format you done can you let me know ???

[Neelesh1121]

@4kssoft Thanks for your suggestions, I have generated my own training datasets and i am able to train the model, but I am not getting what should be the input format to predict the result. If you know what modification it requires to get the result please just inform us.

[4kssoft]

Hello all

@4kssoft if possible please provide the pretrained model that you are using!

@sathvikask0
Sorry but unfortunately I cannot share my model

@4kssoft Hi I have my own data and extracted text using OCR tesseract and got the position of each word, can i know how to > get in the format you showed an example in your repository for sample pdf file Faktura1.pdf_0.json how to get in this format > > and i need in the format you done can you let me know ???

@Hrishkesh
as I wrote #7 (comment) earlier, I use my own tool to annotate documents, I also have ready functions for exporting training data to various models.
I'm planning to publish a beta version of my solution soon

@4kssoft Thanks for your suggestions, I have generated my own training datasets and i am able to train the model, but I am >not getting what should be the input format to predict the result. If you know what modification it requires to get the result ?>please just inform us.

@Neelesh1121
The format is the same as for training. Look at the https://github.com/vsymbol/CUTIE/blob/master/main_evaluate_json.py script

[vishal7894]

everytime i try to use main_evaluate_json.py I get this error

@4kssoft @samhita-alla @vsymbol
Can anyone please help

2 root error(s) found.
(0) Not found: Key feature_fuser/biases not found in checkpoint
[[node save/RestoreV2 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Not found: Key feature_fuser/biases not found in checkpoint
[[node save/RestoreV2 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save/RestoreV2/_49]]
0 successful operations.
0 derived errors ignored.

[n0ct4li]

@Ibmaria
Hello 4kssoft,
Please how do you get this format ?.I have no idea .Can you explain me please?
Thanks in advance

To get the format. I analyzed this file https://github.com/vsymbol/CUTIE/blob/master/data_loader_json.py

@samhita-alla

I'm training a model with these parameters:

python main_train_json.py \ --doc_path 'invoice_data/' \ --save_prefix 'INVOICE' \ --test_path '' \ --embedding_file '' \ --ckpt_path 'graph/' \ --ckpt_file 'CUTIE_highresolution_8x_d20000c9(r80c80_iter_40000.ckpt' \ --tokenize True \ --update_dict True \ --dict_path 'dict/' \ --rows_segment 72 \ --cols_segment 72 \ --augment_strategy 1 \ --positional_mapping_strategy 1 \ --rows_target 64 \ --cols_target 64 \ --rows_ulimit 80 \ --fill_bbox False \ --data_augmentation_extra True \ --data_augmentation_dropout 1 \ --data_augmentation_extra_rows 16 \ --data_augmentation_extra_cols 16 \ --batch_size 32 \ --iterations 40000 \ --lr_decay_step 13000 \ --learning_rate 0.0001 \ --lr_decay_factor 0.1 \ --hard_negative_ratio 3 \ --use_ghm 0 \ --ghm_bins 30 \ --ghm_momentum 0 \ --log_path 'log/' \ --log_disp_step 100 \ --log_save_step 100 \ --validation_step 100 \ --test_step 400 \ --ckpt_save_step 50 \ --embedding_size 128 \ --weight_decay 0.0005 \ --eps 1e-6

@4kssoft Do you generate your own dictionnary? I don't really understand the part "Generate your own dictionary with main_build_dict.py / main_data_tokenizer.py". Can you explain how to apply this process on own dataset? Thanks

Also, to what the ckpt_path argument refers to?

[Karthik1904]

Hello @vsymbol

can you please give brief about how to generate the texts and corresponding bounding boxes & manually labelling each text and their bounding box

Which tools we have use for manually labelling

[mohammedayub44]

@4kssoft Thanks for the labeling video. Does your software export in the format required by CUTIE (json template you provided) or you have to run explicit post processing ?
In the json example you provided, what does "key_id" and "key_value" represent ? all of them look empty.

[vsymbol]

Hello @vsymbol

can you please give brief about how to generate the texts and corresponding bounding boxes & manually labelling each text and their bounding box

Which tools we have use for manually labelling

Apply any OCR tool that help you detecting and recognizing words in the scanned document image.
For example, refer to what @4kssoft has done to the document image and generated a .json file with position and text of the image.

https://github.com/4kssoft/CUTIE/blob/master/invoice_data/Faktura1.pdf_0.json

[hhien]

@Hrishkesh , @sathvikask0, @Karthik1904
Guys, I have written a simple file to run Tesseract ocr and output a json file in the format as in invoice_data/ example:
https://github.com/hhien/tesseract_applications.git

[shrivastavapankajj]

Did someone able to train and test the model? I couldn't find how to predict on new data .

[hhien]

Hi, if you have ground truth data in a different format, you can simply read that data and fill in these field_names, otherwise, you have to fill it manually. The script I wrote only does Teserract OCR on the image and output in the format that CUTIE needs.

…

On Thu, Feb 11, 2021 at 5:30 PM ywsyws ***@***.***> wrote: @Hrishkesh <https://github.com/Hrishkesh> , @sathvikask0 <https://github.com/sathvikask0>, @Karthik1904 <https://github.com/Karthik1904> Guys, I have written a simple file to run Tesseract ocr and output a json file in the format as in invoice_data/ example: https://github.com/hhien/tesseract_applications.git @hhien <https://github.com/hhien> Thank you so much for the script. one question: It doesn't fill in the values of each field_name. Did you manuelly fill it up (which I doubt)? or you did another script to do it? Thank you for your help! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7ZO655LL2P4OYVT4F42F3S6QAYRANCNFSM4KYRDGHA> .

[fj0n]

Did someone able to train and test the model? I couldn't find how to predict on new data .

I'm struggling with it. So far I was able to create the .json-files, with the solution of hhien's code.

I anyone succeeded, I'm thankful for any recommendation on how to train the model.

[darsh169]

Please tell me if this is the correct data format?

Format:

file_name.json

{
  "global_attributes": {
    "file_id": "$file_name"
  },
  "fields":[
    {
      "field_name": "$class_name",
      "key_id": [],
      "key_text": [],
      "value_id": [$word_id],
      "value_text": "$word_text"
    },...
  ],

  "text_boxes":[
    {
      "id": $word_id,
      "bbox": [$word_x_min, $word_y_min, $word_x_max, $word_y_max],
      "text": "$word_text"
    },...
  ]
}

Example:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1],
      "value_text": "sample1"
    },
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [2],
      "value_text": "sample2"
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": "sample3"
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

Or maybe the correct format should look like this
Example 2:

file1.json

{
  "global_attributes": {
    "file_id": "file1.jpg"
  },
  "fields":[
    {
      "field_name": "class1",
      "key_id": [],
      "key_text": [],
      "value_id": [1, 2],
      "value_text": ["sample1", "sample2"]
    },
    {
      "field_name": "class2",
      "key_id": [],
      "key_text": [],
      "value_id": [3],
      "value_text": ["sample3"]
    }
  ],

  "text_boxes":[
    {
      "id": 1,
      "bbox": [10, 10, 50, 20],
      "text": "sample1"
    },
    {
      "id": 2,
      "bbox": [55, 10, 100, 20],
      "text": "sample2"
    },
    {
      "id": 3,
      "bbox": [50, 30, 100, 40],
      "text": "sample3"
    }
  ]
}

what are bbox entries? x1,y1,widht,height? or x1,y1(top left),x2,y2(bottom right)

[gibotsgithub]

i have created the json files in the required format. i have 400 invoices data. the main_train_json.py gets killed because it utilises all the RAM. has anyone faced this issue? I have 16 gb of ram.

[Ajithbalakrishnan]

Anyone pls share the inference script?