Discussion for GSOC'22

[ashwani-rathee]

Hey guys!! I am Ashwani Rathee and I wish to participate in GSOC'22 with JuliaImages ecosystem community

Currently the project that interests me the most is the camera calibration target detection:

Camera calibration involves determining a camera's intrinsic parameters from a series of images
of a so-called "calibration target". Knowledge of the intrinsic parameters facilitates three-dimensional
reconstruction from images or video. The most frequently used calibration target is a checkerboard
pattern. A key step in camera calibration involves automatically detecting the checkerboard and
identifying landmarks such as the corners of each checkerboard square.

This project will implement a recent automatic checkerboard detection and feature extraction algorithm.

Example algorithm:

Y. Yan, P. Yang, L. Yan, J. Wan, Y. Sun, and K. Tansey, “Automatic checkerboard detection for 
camera calibration using self-correlation,” Journal of Electronic Imaging, vol. 27, no. 03, p. 1, May 2018.

Implementation of above mentioned algorithm( it doesn't include final outlier refinement and self correlation
method is slightly different too) is available here:

https://github.com/ashwani-rathee/CameraCalibrations.jl

with code of particular algorithm here:
https://github.com/ashwani-rathee/CameraCalibrations.jl/blob/9e46e8047d2f7ae3878bb3222f594cdcf81a66e0/src/checkerboard.jl#L231

In above algorithm, harris method and use of imcorner slows down the algorithm(around 22ms, due to
calculation of gradients which is expensive) but results are much better compared to use of fastcorner
in place of harris's method. It would be great if we can implement below mentioned algorithm for camera
calibrations to provide standard support for camera calibration in Julia.

• Zhang method(had been working on this and point 2 but still WIP) for intrinsic/extrinsic parameters
• 3d reconstruction

Other projects that I would love to discuss:

- Package like pyautogui for julia: provide access display, keyboard, mouse access and other utilies with cross os support

I have some rudimentary implementation for screenshots, keyboard and mouse data,control in grabscreen.jl using CXXwrap for to X system and quite intersting as I see it 
https://github.com/ashwani-rathee/GrabScreen.jl

- QRcode project is quite interesting too and I have worked on qrcodes using zbar for android flutter developement 
and have used QRcode for data storage(music pieces, quite cool to compress classic piano pieces using compressed 
midi into qrcode with very small space, and recreate music with very rudimentary data)

I would love to know who all will be available for mentoring and what projects interest them,
so I can see projects and contribute accordingly

Update:

I just noticed that the camera calibrations project got removed too sadly recently. I think with given changes and johnny being available for benchmark project,benchmark project is also equally interesting and I have experience with using cpp wrap and cwrap, exif wrapper project is a pretty doable project.

[zygmuntszpak]

Hi Ashwani,

Thank you very much for your interest. Regrettably, I don't have the bandwidth currently to be involved in the GSoC`22 and so the project you were most interested in got pulled. It would indeed be fantastic if you could still be involved in the benchmark project.

[timholy]

I too have other commitments (big plans in base Julia itself), so don't have the bandwidth to mentor a JuliaImages project.

WRT your pyautogui idea, I would recommend GtkObservables as a place to start. Gtk itself has gotten a lot of really great work lately and finally seems to be maturing into a top-tier GUI platform for Julia. GtkObservables makes it pretty simple to make "Julian" GTK-based GUIs.

[ashwani-rathee]

Thank you zygmunt and tim for your responses!! After initial discussion with @johnnychen94, we want to approach
and work on a IO based long project which would involve providing support for 2 things:

• EXIF Viewer
• Support for GIF and JPEG 2000 in ImageIO.jl

We discussed whether we should write the code natively or wrap c packages but eventually came to the conclusion
that wrapping c packages is better for now as we won't have to workout all the details of these standards ourselves.

We want to wrap these libraries written in c:

• libexif : for EXIF viewer to provide read/update/write operations
• giflib : for GIF standard
• openjpeg : for JPEG 2000 standard

These seem to be the standard mature libraries in c, although we might need a discussion on the library choices.

All these are currently available as jll in yggdrasil:

• https://github.com/JuliaPackaging/Yggdrasil/tree/master/L/libexif
• https://github.com/JuliaPackaging/Yggdrasil/tree/master/G/Giflib
• https://github.com/JuliaPackaging/Yggdrasil/tree/master/O/OpenJpeg

We might need to update libexif for a newer version, but having them already available through binary builders.jl helps
as I see it.

In past week, I was trying to figure the EXIF viewer project by wrapping libexif and having been succesful in wrapping
through both clang.jl and cbinding.jl and have been able to read/update operations.
@Gnimuc thankfully agreed to co-mentor the project and he would be great help with bunch of questions that would
most definitely arise while trying to figure out the inconspicous segmentation faults.

Wrapper generator:
https://gist.github.com/ashwani-rathee/eec65034ed5c582bbdf9f513bc9542b0

Test code:

julia>include("LibExif.jl")
Main.LibExif

julia> ed_ptr = LibExif.exif_data_new_from_file("test.jpg")
Ptr{Main.LibExif._ExifData} @0x0000000000c9fb50

julia> ed = unsafe_load(ed_ptr)
Main.LibExif._ExifData((Ptr{Main.LibExif._ExifContent} @0x0000000000a45f00, Ptr{Main.LibExif._ExifContent} @0x0000000000abc660, Ptr{Main.LibExif._ExifContent} @0x0000000000dbc100, Ptr{Main.LibExif._ExifContent} @0x000000000107aec0, Ptr{Main.LibExif._ExifContent} @0x0000000000d8b270), Ptr{UInt8} @0x0000000000000000, 0x00000000, Ptr{Main.LibExif._ExifDataPrivate} @0x0000000000a80b20)

julia> content_ptr = ed.ifd[1]
Ptr{Main.LibExif._ExifContent} @0x0000000000a45f00

julia> function read_exifcontent(content_ptr)
           data = unsafe_load(content_ptr)
           count = Int(data.count)
           res = unsafe_wrap(Array, data.entries, count)
           result = Dict()
           for i in 1:count
               str = Vector{Cuchar}(undef, 1024);
               LibExif.exif_entry_get_value(Ref(unsafe_load(res[i])), str, length(str))
               result[string(unsafe_load(res[i]).tag)] = rstrip(String(str), '\0')
           end
           return result
       end
read_exifcontent (generic function with 1 method)

julia> read_exifcontent(content_ptr)
Dict{Any, Any} with 9 entries:
  "EXIF_TAG_DOCUMENT_NAME"   => "apple"
  "EXIF_TAG_SOFTWARE"        => "asdd"
  "EXIF_TAG_COPYRIGHT"       => "adfasd (Photographer) - [None] (Editor)"
  "EXIF_TAG_Y_RESOLUTION"    => "72"
  "EXIF_TAG_ORIENTATION"     => "Bottom-right"
  "EXIF_TAG_ARTIST"          => "Ashwani"
  "EXIF_TAG_RESOLUTION_UNIT" => "Inch"
  "EXIF_TAG_MODEL"           => "asd"
  "EXIF_TAG_X_RESOLUTION"    => "72"

julia> function readtag(filepath,  tag)
           ed_ptr = LibExif.exif_data_new_from_file(filepath)
           ed = unsafe_load(ed_ptr)
           content_ptr = ed.ifd[1]
           make_ptr = LibExif.exif_content_get_entry(content_ptr, tag)
           str = Vector{Cuchar}(undef, 1024);
           LibExif.exif_entry_get_value(make_ptr, str, length(str))
           return rstrip(String(str), '\0')
       end
readtag (generic function with 1 method)

julia> readtag("test.jpg", LibExif.EXIF_TAG_ARTIST)
"Ashwani"

Github repo while figuring out the project: https://github.com/ashwani-rathee/io-project

I'll update in this discussion when I'll have more to add but its great to know we have some direction and helps to figure out clear plan of action and output expectations. And I look forward to a good discussion how to best approach the project.

[johnnychen94]

I've long wanted to eventually replace ImageMagick with ImageIO so I'm fully supportive of this IO project. There are some assumptions and consensus on ImageIO:

• ImageIO is a thin wrapper with all real functionality implemented in separate packages(e.g., TiffImages, PNGFiles). This is because we want to give common users a simple answer "use ImageIO and FileIO", and we also want to give advanced users the flexibility to use only specific format backends.
• Thus the expectation of this project, if selected, should be three independent packages: EXIFViewer, GIFImages, and JPEG2000. The names can be different. After each format is supported, we register it in both ImageIO and FileIO. (e.g., registry(jpeg): add JpegTurbo and ImageIO JuliaIO/FileIO.jl#357)
• benchmarks tests are required and we should be at least as fast as ImageMagick; we want to ensure that we get positive results by replacing ImageMagick with your new packages.
• cross-platform support: at least support platforms that Julia declares tier-1.
• we want to ensure small dependencies. The essential dependencies are ColorTypes, FixedPointNumbers, and maybe OffsetArrays. You might want to use ImageCore for convenience but we should try to avoid using anything "larger" than it.

[johnnychen94]

Also to @IanButterworth @tlnagy in case you're interested.

[tlnagy]

Seems reasonable. I don't know much about the EXIF standard, but it sounds identical to how I have to read the TIFF tag format in TiffImages.jl, not sure if it's possible to extract that logic to have a pure Julia implementation. Just a thought.

[ashwani-rathee]

Hey guys!!, time for some updates:

Exif subproject

---

Code source: https://github.com/ashwani-rathee/io-project/blob/main/exifviewer/

General wrapper and its generators are in gen/ by name LibExif.jl:
https://github.com/ashwani-rathee/io-project/blob/main/exifviewer/gen/LibExif.jl

For read operations, functions that would most probably will be reworked for the final exifviewer.jl in src/ are here:
https://github.com/ashwani-rathee/io-project/blob/main/exifviewer/gen/read.jl

Which intend to provide support for:

• Read all the exifdata and dump it in dicts
• Read specific tag/tags and return dicts with them(johnny is also interested in support like imfino in matlab
  which we can be done with this too)

For write, it's a bit complex with what I know currently, so I would keep the scope slightly limited by currently
saying that it might be only possible to support tiff and jpeg write right now. I need more time to figure this one out.

• https://gist.github.com/ashwani-rathee/a884d9c211b1d7d683cef2cd83daf94f write example which is mostly
  a translation of https://github.com/libexif/libexif/blob/master/contrib/examples/write-exif.c

ImageIO.jl file support subprojects:

---

There are two file formats we are interested in supporting i.e. GIF, JPEG 2000. We want to write wrapper for
GIFLIB and OPENJPEG to support encoding and decoding of these files mainly.

There is one common thing about these two projects, that they both intend to focus on command line utils
which makes our job slightly difficult to write the wrappers. Because Giflib_jll and OpenJpeg_jll that are available
through yggdrasil provide support for headers of the library but these libraries have couple of conversion util
programs that are provided separately through giflib_tools like this one:
http://giflib.sourceforge.net/gif2rgb.html
which provide RGB2GIF and GIF2RGB functions which basically make our jobs relatively easier by avoiding
gory details(see this) of decoding and encoding (other than some decode parameters perhaps which can be
passed as keyword arguments from Julian side).
From my point of view in the case of GIFs, we are mainly interested in receiving a vector of multiple vector{uints8}
with vector{uints8} being RGBRGB or GGG(for gray) and then just reinterpret it like here:
https://github.com/ashwani-rathee/GrabScreen.jl/blob/62ec3008e470ba570d224eb6f67810bbd6baa69e/src/GrabScreen.jl#L44

For OpenJpeg, it is the same issue as they provide a majority of decoding and encoding functions in
https://github.com/uclouvain/openjpeg/blob/master/src/bin/jp2/opj_decompress.c
https://github.com/uclouvain/openjpeg/blob/master/src/bin/jp2/opj_compress.c
which has a lot of code that makes our job easier but it isn't available through current jlls.

I could rewrite everything in Julia that these separate programs provide again through their clang.jl wrappers:
LibGif.jl: https://github.com/ashwani-rathee/io-project/blob/main/gifwrapper/gen/LibGif.jl
LibOpenJpeg.jl: https://github.com/ashwani-rathee/io-project/tree/main/jpeg2000wrapper/gen
like johnny did for JpegTurbo.jl(here )

or I could create two c libraries libgifextra and libopenjpegextra like https://github.com/vchuravy/LLVMExtra.
Yupei and Johnny both suggested to rewrite/wrapper c libraries is a better option to provide the extra functions. I think
this would be relatively faster too than ccalls again and again like its done in JpegTurbo.jl and would be less tiresome
to translate code.

• https://github.com/ashwani-rathee/io-project/tree/main/gifwrapper/gen/libgifextra
• https://github.com/ashwani-rathee/io-project/tree/main/jpeg2000wrapper/libopenjpegextra

These both c libraries basically scrape/reuse code from those command line programs, although I needed to
rewrite a bunch of functions for openjpeg part to stick to get the RGB buffer to easily pass to Julian side.
I tried to write a proper makefile to generate the necessary files but the process of making jlls from them is still WIP.

Usage of these two libraries in c is here which mainly tests decode from file to rgb buffer and encode from rgb buffer to file:

• https://github.com/ashwani-rathee/io-project/blob/main/gifwrapper/gen/libgifextra/notes/main.c
• https://github.com/ashwani-rathee/io-project/blob/main/jpeg2000wrapper/libopenjpegextra/notes/example.c

Their corresponding tests in Julia that I am still working on:

• https://github.com/ashwani-rathee/io-project/blob/main/gifwrapper/gen/libgifextratest.jl
• https://github.com/ashwani-rathee/io-project/blob/main/jpeg2000wrapper/libjpeg2ktest.jl

We are close to proper concept proofs but still WIP, I wanted to share the updates and perhaps give more
time to the proposal right now. I am reasonably comfortable with all 3 libraries now and I think this project is doable
and I could finish it.

Now issues that concern me that I haven't figured out yet:

• How to make jll from those c libraries, provide it through yggdrasil, licensing
• Writing/Encoding back to file is troublesome and write in exif, gif(for multiple images case) is still not exactly
  working
• How to write it all in a way that would be maintainable and figure out the finer details like colorspaces,
  keyword arguments for encoding and decoding etc

I know Johnny and you guys would be concerned about proper tests, docs, benchmarks, and CIs and
writing good code at the end of the day. I am also trying to figure all that out but yeah I have in some way used/
done all those things in Julia.

I'll keep track of the proposal here:
https://docs.google.com/document/d/1AiSmMWJXdjeBs01HCiDy2KmR426oDoUd6LKiLOg_WLY/edit?usp=sharing

Although it's still rather rudimentary, I could use constructive feedback to figure out a good plan which if things go
well will help with executing the whole project well.