There is going to be a very large update comming and I mneed some bug testers.

[kdschlosser]

This update is really large. It completely changes how the bus drivers work. It is probably going to break things so I will need to work through the issues. I just need people to test it out.

For those that do not know I had spent some time really looking at the RGB bus driver in search of a way to get better performance and also add software rotation without taking a huge performance hit for it. For the people running a dual core ESP32 I did find a solution. That solution involves running the RGB bus driver from the second core. This was tricky to do to keep things synchronized but I did get it to work properly. Fast forward a few months and I picked up a display that using an I8080 bus and as it turns out the display IC doesn't support hardware rotation. I did not know that IC's from busses other than the RGB bus would need to have software rotation available....

This problem gets more complicated because of needing to perform a byte swap when using RGB565 with some of the display IC's. It would be less than ideal to iterate over the buffer 2 times.

I had to pull the code from the RGB driver and rewrite it so the code isn't bus specific. Then I had to add in the RGB565 byte swapping, And just because I am a glutton for punishment I decided to add dithering when using RGB565 as well. The dithering is a runtime option that is able to be turned on and off at any time while the program is running. So if you are displaying an image that has some banding issues in the colors you can turn it on when that image is being displayed and then turn it off when the image is not being displayed. One of the things that really shows the banding is gradients. The dithering option smooths all of that out..

RGB565 gradient, no dithering...

Same image with dithering..

It's a night and day difference... That is the reason why I decided to add it.

I am going to need some people willing to test things out for me. If you are up for it let me know.

[mark-hoy]

I''ll try it tomorrow on my Mac mini and Esp32 Elecrow 7" panel.

[spienz]

Hy,

i can do this i've you wan't me. I have a second display ready for testing.

https://www.elecrow.com/wiki/esp32-display-502727-intelligent-touch-screen-wi-fi26ble-800480-hmi-display.html

Regards

Sebastian

[kdschlosser]

Not yet folks. It will be in a few days. I am still making changes...

[0ut4t1m3]

I'm up for some testing. I love the look of the dithering, what sort of performance hit are you expecting when it's enabled?

[kdschlosser]

It depends. If you are using RGB565 byte swapping then the performance hit will be very minimal. the same thing applies if you are using an RGB bus... I don't foresee that much of a performance hit when using a dual core ESP32. This is because the second core is being used when doing the dithering. This will allow the core that LVGL is running on to continue working. I am in the process of tweaking the code so there will not be any kind of s stall while the copying is taking place.

There is a cost and that comes on the form of memory use. 2 partial buffers will be made for LVGL and those buffers can reside in internal memory. And 2 more buffers are made which will reside in DMA memory in PSRAM. if the RGB Bus is being used those 2 buffers will be full buffers and any other buffer they will be partial. This will allow LVGL to fill one of the partial buffers while the other partial buffer while it is being dithered, swapped and rotated. this is not something that can be done in place which is the reason why the other 2 buffers are needed. whole one of those other buffers is transmitting using DMA memory the second one if having the data copied into it while performing the dither, rotation and byte swap. Those things are all done using a single iteration over the buffer data to keep the performance up. There are no conditionals that are tested during the iteration. What I mean by that is this kind of code doesn't exist.

for y in range(height):
    for x in range(width):
        if dither:
            ...
        if byte_swap:
            ...
        if rotate:
            ...

The code is not written in that manner as the testing for the conditionals would slow the process down big time. At the cost of making the program a couple hundred bytes larger I had to write iterative code that tests the conditionals prior to entering the loop..

if dither and byte_swap and rotation:
    for y in range(height):
        for x in range(width):
            # code that does all 3
elif dither and byte_swap:
    for y in range(height):
        for x in range(width):
            # code that does dither and byte_swap
elif dither and rotation:
    for y in range(height):
        for x in range(width):
            # code that does dither and rotation
elif byte_swap and rotation:
    for y in range(height):
        for x in range(width):
            # code that does rotation and byte_swap
elif byte_swap:
    for y in range(height):
        for x in range(width):
            # code that does byte_swap            
elif rotation:
    for y in range(height):
        for x in range(width):
            # code that does rotation
elif dither:
    for y in range(height):
        for x in range(width):
            # code that does dither
else:
    # perform original transfer

It's a bit more complex than this and it is written in C code. The above is easier to understand and see that the conditional is tested prior to the loop instead of being tested on each and every single pixel. You have to think of the number of pixels we are dealing with. The display I have on my desk has a resolution of 1280 x 400 which is 512,000 pixels. testing for 3 conditionals that many times is a lot when doing a full screen update. even if we deal with partial buffers that time spent adds up very fast. so by testing for the conditionals prior to the iteration occurring I am able to optimize the code.

The dithering code that I am using is super fast. there is no complex math involved. It is done using bit manipulation some addition and collecting values from arrays that are allocated when dithering is turned on and freed when dithering is turned off. The Floyd-Steinberg algorithm is typically the go to code to perform RGB565 dithering. The code I am using is faster. There is a small loss of quality compared to the Floyd-Steinberg method which is OK due to the size of the displays that are being used, the human eye is not able to detect the differences because of the physical size of the pixels.

Here is an example of the Floyd-Steinberg method written in Python using the Python Imaging Library and Numpy.

import numpy as np
from PIL import Image

img_name = '1665_Girl_with_a_Pearl_Earring.jpg'

img = Image.open(img_name)

width, height = img.size
new_width = 400
new_height = int(height * new_width / width)
img = img.resize((new_width, new_height), Image.ANTIALIAS)

p = np.linspace(0, 1, nc)
p = np.array(list(product(p,p,p)))

def get_new_val(old_val):
    idx = np.argmin(np.sum((old_val[None,:] - p)**2, axis=1))
    return p[idx]

def fs_dither(img, nc):
    arr = np.array(img, dtype=float) / 255

    for ir in range(new_height):
        for ic in range(new_width):
            old_val = arr[ir, ic].copy()
            new_val = get_new_val(old_val, nc)
            arr[ir, ic] = new_val
            err = old_val - new_val
            if ic < new_width - 1:
                arr[ir, ic+1] += err * 7/16
            if ir < new_height - 1:
                if ic > 0:
                    arr[ir+1, ic-1] += err * 3/16
                arr[ir+1, ic] += err * 5/16
                if ic < new_width - 1:
                    arr[ir+1, ic+1] += err / 16

    carr = np.array(arr/np.max(arr, axis=(0,1)) * 255, dtype=np.uint8)
    return Image.fromarray(carr)


def palette_reduce(img, nc):
    arr = np.array(img, dtype=float) / 255
    arr = get_new_val(arr, nc)

    carr = np.array(arr/np.max(arr) * 255, dtype=np.uint8)
    return Image.fromarray(carr)

for nc in (2, 3, 4, 8, 16):
    print('nc =', nc)
    dim = fs_dither(img, nc)
    dim.save('dimg-{}.jpg'.format(nc))
    rim = palette_reduce(img, nc)
    rim.save('rimg-{}.jpg'.format(nc))

and here is an example using my way....

from PIL import Image

red_thresh = [
    1, 7, 3, 5, 0, 8, 2, 6,
    7, 1, 5, 3, 8, 0, 6, 2,
    3, 5, 0, 8, 2, 6, 1, 7,
    5, 3, 8, 0, 6, 2, 7, 1,
    0, 8, 2, 6, 1, 7, 3, 5,
    8, 0, 6, 2, 7, 1, 5, 3,
    2, 6, 1, 7, 3, 5, 0, 8,
    6, 2, 7, 1, 5, 3, 8, 0
]

grn_thresh = [
    1, 3, 2, 2, 3, 1, 2, 2,
    2, 2, 0, 4, 2, 2, 4, 0,
    3, 1, 2, 2, 1, 3, 2, 2,
    2, 2, 4, 0, 2, 2, 0, 4,
    1, 3, 2, 2, 3, 1, 2, 2,
    2, 2, 0, 4, 2, 2, 4, 0,
    3, 1, 2, 2, 1, 3, 2, 2,
    2, 2, 4, 0, 2, 2, 0, 4
]

blu_thresh = [
    5, 3, 8, 0, 6, 2, 7, 1,
    3, 5, 0, 8, 2, 6, 1, 7,
    8, 0, 6, 2, 7, 1, 5, 3,
    0, 8, 2, 6, 1, 7, 3, 5,
    6, 2, 7, 1, 5, 3, 8, 0,
    2, 6, 1, 7, 3, 5, 0, 8,
    7, 1, 5, 3, 8, 0, 6, 2,
    1, 7, 3, 5, 0, 8, 2, 6
]

img = Image.open(r"some_image.jpg").convert('RGB')

width, height = img.size

for y in range(height):
    for x in range(width):
        r, g, b = img.getpixel((x, y))
        treshold_id = ((y & 7) << 3) + (x & 7)

        r = ((r + red_thresh[treshold_id]) >> 3) << 3
        g = ((g + grn_thresh[treshold_id]) >> 2) << 2
        b = ((b + blu_thresh[treshold_id]) >> 3) << 3
        
        img.putpixel((x, y), (r, g, b))

img.show()

once again these are in Python so it is easier to see the simplicity involved. when written in C code there is actually less involved and I am able to perform the copy and dither, copy, dither and byte swap or the copy, dither, byte swap and rotation in a smaller number of operations. The single biggest advantage is there is no division math which is really slow and no conditionals need to be checked in the loop.

[kdschlosser]

If you want to play around with it and compare the results here is some code you can use on CPython to do that...
put a path to an image file in the IMAGE_PATH variable and then run the script. It will show you what the image looks like as RGB565 and also as dithered RGB565. It will work on any image that is not indexed. PNG. BMP, TIFF, GIF, JPG, etc.... The path needs to be absolute to ensure it is going to work.

IMAGE_PATH = 'absolute/path/to/image.png'

try:
    from PIL import Image
except ImportError:
    print('You need to install PIL\n"pip3 install pillow"\n\n')
    raise

red_thresh = [
    1, 7, 3, 5, 0, 8, 2, 6,
    7, 1, 5, 3, 8, 0, 6, 2,
    3, 5, 0, 8, 2, 6, 1, 7,
    5, 3, 8, 0, 6, 2, 7, 1,
    0, 8, 2, 6, 1, 7, 3, 5,
    8, 0, 6, 2, 7, 1, 5, 3,
    2, 6, 1, 7, 3, 5, 0, 8,
    6, 2, 7, 1, 5, 3, 8, 0
]

grn_thresh = [
    1, 3, 2, 2, 3, 1, 2, 2,
    2, 2, 0, 4, 2, 2, 4, 0,
    3, 1, 2, 2, 1, 3, 2, 2,
    2, 2, 4, 0, 2, 2, 0, 4,
    1, 3, 2, 2, 3, 1, 2, 2,
    2, 2, 0, 4, 2, 2, 4, 0,
    3, 1, 2, 2, 1, 3, 2, 2,
    2, 2, 4, 0, 2, 2, 0, 4
]

blu_thresh = [
    5, 3, 8, 0, 6, 2, 7, 1,
    3, 5, 0, 8, 2, 6, 1, 7,
    8, 0, 6, 2, 7, 1, 5, 3,
    0, 8, 2, 6, 1, 7, 3, 5,
    6, 2, 7, 1, 5, 3, 8, 0,
    2, 6, 1, 7, 3, 5, 0, 8,
    7, 1, 5, 3, 8, 0, 6, 2,
    1, 7, 3, 5, 0, 8, 2, 6
]

img = Image.open(IMAGE_PATH).convert('RGB')
rgb565_image = Image.new('RGB', (img.size[0], img.size[1]))

width, height = img.size

for y in range(height):
    for x in range(width):
        r, g, b = img.getpixel((x, y))

        rgb565_image.putpixel((x, y), ((r >> 3) << 3, (g >> 2) << 2, (b >> 3) << 3))
        
        treshold_id = ((y & 7) << 3) + (x & 7)

        r = ((r + red_thresh[treshold_id]) >> 3) << 3
        g = ((g + grn_thresh[treshold_id]) >> 2) << 2
        b = ((b + blu_thresh[treshold_id]) >> 3) << 3

        img.putpixel((x, y), (r, g, b))


rgb565_image.show()
img.show()

[kdschlosser]

OK so I have pushed the last batch of changes. As I have said I am not expecting this to run properly right out of the gate.
Grab a fresh clone of the repo and then check the branch sw_rotate_for_all_bus_new..

build arguments are the same as before so nothing changed there.

The first big change is how the frame buffer is created. It is a separate entity from the bus drivers. Meaning you no longer need to create the bus and then create the frame buffer. The frame buffer is able to be created before the bus gets created. to create frame buffers you use lcd_bus.framebuffer passing the size and the capability as usual. The returned instance is no longer a memoryview object, instead it is a bytearray. The buffer that gets created is going to be the buffer that gets passed to LVGL. There is no need to create the buffer in DMA memory. it is wise to place it in internal ram if possible as this is going to allow LVGL to render faster. Depending on the type of bus that is being used and depending on if you are byte swapping RGB565, dithering RGB565 or performing a software rotation is going to dictate of 2 more buffers of the same size get created. These buffers will reside in DMA memory and they will be located in SPIRAM. They will be the same size as the original buffers unless you are using an RGB bus and then they will be full sized frame buffers and not partials.

give it a test run using the RGB565 byte swap for now and lets go from there..

[kdschlosser]

can you add --debug to your build command so we can get some debugging output, That may shed some light on where it is crashing. The function that is listed is quite large and zeroing in on where the problem could be would be really helpful.

[0ut4t1m3]

Okay done, here's what I get before the crash:

D (28304) intr_alloc: Connected src 42 to int 5 (cpu 1)
pclk_hz=13000000
hsync_pulse_width=4
hsync_back_porch=8
hsync_front_porch=8
vsync_pulse_width=4
vsync_back_porch=8
vsync_front_porch=8
hsync_idle_low=0
vsync_idle_low=0
de_idle_high=0
pclk_active_neg=1
pclk_idle_high=0
clk_src=6
hsync_gpio_num=46
vsync_gpio_num=3
de_gpio_num=5
pclk_gpio_num=7
data_gpio_nums[0]=14
data_gpio_nums[1]=38
data_gpio_nums[2]=18
data_gpio_nums[3]=17
data_gpio_nums[4]=10
data_gpio_nums[5]=39
data_gpio_nums[6]=0
data_gpio_nums[7]=45
data_gpio_nums[8]=48
data_gpio_nums[9]=47
data_gpio_nums[10]=21
data_gpio_nums[11]=1
data_gpio_nums[12]=2
data_gpio_nums[13]=42
data_gpio_nums[14]=41
data_gpio_nums[15]=40
sram_trans_align=8
psram_trans_align=64
refresh_on_demand=0
fb_in_psram=0
double_fb=0
data_width=16
h_res=800
v_res=480
bits_per_pixel=16
rgb565_byte_swap=0
mp_lcd_sw_rotate_task - STARTED

Nothing after the rotate call I'm afraid, still the same invalid pointer on decode.

LVGL MicroPython 
IDF version : c9763f62
Machine     : Generic ESP32S3 module with Octal-SPIRAM with ESP32S3

Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x42004a24: e0ed9b81 f01d0008 7d008136
Core  0 register dump:
PC      : 0x42004a2a  PS      : 0x00060c30  A0      : 0x00000000  A1      : 0x3fcc7590  
A2      : 0x3c2782b0  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x00000000  
A6      : 0x00000000  A7      : 0x3fcc7590  A8      : 0x82004750  A9      : 0x3fcc7560  
A10     : 0x3c2783dc  A11     : 0x3c1d0acc  A12     : 0x3fca4b58  A13     : 0x00000000  
A14     : 0x3fca4b50  A15     : 0x3fcc7560  SAR     : 0x00000018  EXCCAUSE: 0x00000000  
EXCVADDR: 0x00000000  LBEG    : 0x400554b9  LEND    : 0x400554dd  LCOUNT  : 0x821c9a87  


Backtrace: 0x42004a27:0x3fcc7590   <- Decodes to: 0x42004a27: mp_lcd_sw_rotate_task at ??:?




ELF file SHA256: 8e97ecfab

[kdschlosser]

Where are you seeing invalid pointer?

[0ut4t1m3]

Maybe a poor phrasing on my behalf - I assume it's supposed to be showing a line number instead of ??:?

[kdschlosser]

I just pushed a commit. Update and try it again. keep the --debug build argument.

[0ut4t1m3]

Ok no more reboot, I'm getting something a bit more useful now:

D (52224) intr_alloc: Connected src 42 to int 5 (cpu 1)
pclk_hz=13000000
hsync_pulse_width=4
hsync_back_porch=8
hsync_front_porch=8
vsync_pulse_width=4
vsync_back_porch=8
vsync_front_porch=8
hsync_idle_low=0
vsync_idle_low=0
de_idle_high=0
pclk_active_neg=1
pclk_idle_high=0
clk_src=6
hsync_gpio_num=46
vsync_gpio_num=3
de_gpio_num=5
pclk_gpio_num=7
data_gpio_nums[0]=14
data_gpio_nums[1]=38
data_gpio_nums[2]=18
data_gpio_nums[3]=17
data_gpio_nums[4]=10
data_gpio_nums[5]=39
data_gpio_nums[6]=0
data_gpio_nums[7]=45
data_gpio_nums[8]=48
data_gpio_nums[9]=47
data_gpio_nums[10]=21
data_gpio_nums[11]=1
data_gpio_nums[12]=2
data_gpio_nums[13]=42
data_gpio_nums[14]=41
data_gpio_nums[15]=40
sram_trans_align=8
psram_trans_align=64
refresh_on_demand=0
fb_in_psram=0
double_fb=0
data_width=16
h_res=800
v_res=480
bits_per_pixel=16
rgb565_byte_swap=0
Traceback (most recent call last):
  File "<stdin>", line 36, in <module>
  File "rgb_display.py", line 40, in __init__
  File "display_driver_framework.py", line 245, in __init__
  File "display_driver_framework.py", line 258, in _init_bus
MemoryError: Not enough memory to allocate frame buffers

[kdschlosser]

I sorted most of those issues out in the library. The build script does a lot of in place modifications to the core MicroPython files and previously those changes would stay between builds. I updated the code so it makes a backup of the original file and then modifies the existing one. If the build completes successfully or not the updated files get replaced with the backups. This keeps things so they are original so old changes to files don't cause issues when I make changes. So pulling the updates in most cases is OK to do now.

[0ut4t1m3]

okay no probs, I've changed it to

fb1 = lcd_bus.framebuffer(32768, lcd_bus.MEMORY_INTERNAL)
fb2 = lcd_bus.framebuffer(32768, lcd_bus.MEMORY_INTERNAL)

import lvgl as lv
import rgb_display
display = rgb_display.RGBDisplay(
    data_bus=bus,
    frame_buffer1=fb1,
    frame_buffer2=fb2,
    display_width=800,
    display_height=480,
    color_space=lv.COLOR_FORMAT.RGB565,
    )

Still no output, it's definitely pushing pixels though, the screen goes noticeably darker and no glitches just pure black.

[kdschlosser]

I am going to call it a night. It's 2:30 AM where I am at and I have not been to bed in over 48 hours. so I am a little bit tired.

[kdschlosser]

making some headway tho. I do appreciate you for helping with this. All of my MCU's and displays are packed up in boxes because I am in the middle of doing a remodel of my office/workshop. things are strewn about all over the house too so even trying to locate the box the stuff is in would take me a while to do.

[0ut4t1m3]

Not a problem mate, happy to help where I can, have a good night.

[mark-hoy]

I can verify that the new branch builds on MacOS.

$ git clone https://github.com/lvgl-micropython/lvgl_micropython
$ cd lvgl_micropython
$ python3 make.py macOS DISPLAY=sdl_display INDEV=sdl_pointer

$ git branch -v -a
$ git switch -c sw_rotate_for_all_bus_new
$ python3 make.py macOS DISPLAY=sdl_display INDEV=sdl_pointer

When using a simple gui app on Mac, there is an error in the new code that didn't exist in the previous code. I'm not familiar enough with the task_handler.py to debug this further other than that it's a change that seemed to work before.

$X
Traceback (most recent call last):
  File "task_handler.py", line 136, in _task_handler
  File "gui.py", line 865, in gcode_draw_event_cb
AttributeError: 'lv_draw_task_t' object has no attribute 'draw_dsc'

The error is from the following line:

def gcode_draw_event_cb(e):
    obj = e.get_target_obj()
    task = e.get_draw_task()
    dscb = task.draw_dsc. # THIS LINE
    base_dsc = lv.draw_dsc_base_t.__cast__(task.draw_dsc)
    # If the cells are drawn...           

[mark-hoy]

Hmm.. the error I just showed was sometime between my Dec 6 build and the current build. Might not be related to your newest branch changes.