Small refactor (#221)

* Set variable only when SMALL_PAGES is 1

* Fix mmap print

* Example of arch separation

* Typo fixes

* Fix memory initialization issue

* Update tests and fix documentation

* For now the tests page size is fixed at 0

* Typo fixes in documentation

* Update map_framebuffer to avail of the earlier initialization of pmm

* Minor updates to memory documentation

build/Common.mk

@@ -43,7 +43,7 @@ TESTFLAGS := -std=gnu99 \
         -I src/include/kernel/arch/common/mem \
         -I src/include/sys \
         -I src/include/utils \
-        -DSMALL_PAGES=$(SMALL_PAGES) \
+        -DSMALL_PAGES=0 \
         -D_TEST_=1
 
 PRJ_FOLDERS := src

docs/kernel/Initialization.md

@@ -37,11 +37,13 @@ The sequence of component that are intialized (refer to `src/main.c`):
 * Load the PSF font from memory
 * Basic System Initialization:
     - Parse the multiboot information received from the bootloader.
-    - Parse the mmap and initialize
-    - Initia physical memory manager, marking the pmm areas as busy and setup the hhdm.
-    - Initialize the physical memory manager, marking the area in the mmap as already taken.
+    - Parse the mmap and compute physical memory available.
+    - Initialize physical memory manager in the following order:
+        * Prepare the higher half direct map, it will avail of anonymous memory not used after the end of the kernel.
+        * Initialize the physical memory bitmap
+        * Parse and mark as busy the memory map.
     - Validate and parse the SDT tables
-* Finish mapping the Framebuffer (there is a potential bug here, need to chek what i do while mapping it)
+* Finish mapping the Framebuffer
 * Initialize the kernel VMM
 * Initialize the kernel heap
 * Initialize the apic

docs/kernel/MemoryManagement.md

@@ -30,7 +30,7 @@ What they do will be detailed in the next sections.
 
 #### mmap_parse()
 
-The first function, is very simple and it's main purpose is to initialize the global variables that will contain the pointer to the `memory map` and the total number of entries in it.
+This function, is very simple and it's main purpose is to initialize the global variables that will contain the pointer to the `memory map` and the total number of entries in it.
 
 The memory map is used to understand what parts of the physical memory are reserved and what are available to the kernel.
 
@@ -39,7 +39,7 @@ The memory map is used to understand what parts of the physical memory are reser
 
 Before explaining what `pmm_setup` does, let's see first an issue that arised and how we decided to fix it.
 
-So one of the problems, especially if we use 4k pages (it was not present using 2M pages) is that we need to map a lot of memory to cater for all the structures needed by the PMM, but also by the VMM, and everything else. When the kernel is loaded we usually have some space already mapped just after it's end, but it is pure coincidence, and if we eventually reach an unmapped part of memory, without considering the potential `#PF`, it requires the kernel to map that new region but that can be a problem on early stages, let' s see why.
+So one of the problems, especially if we use 4k pages (it was not present using 2M pages) is that we need to map a lot of memory to cater for all the structures needed by the PMM, but also by the VMM, and everything else. When the kernel is loaded we usually have some space already mapped just after its end, but it is pure coincidence, and we can eventually reach an unmapped part of memory, causing a `#PF`, the kernel needs to map that new region in advance, but that can be a problem on early stages, let' s see why.
 
 On many architectures paging is usually obtained using a `multi-level`  approach, where a portion of a address is just an entry into a specific table, and the value is the address of the next level table, until we reach the last part of the address that is the offset within the address contained on the last-level entry table (we don' t cover the details here, if interested in learning more this is explained in the [Paging](https://github.com/dreamportdev/Osdev-Notes/blob/master/04_Memory_Management/03_Paging.md) chapter of the _Osdev Notes_).
 
@@ -60,24 +60,24 @@ char *pointer_to_variable = (char *) hhdm_get_variable(physical_address);
 
 And we will get access to the content of the `physical_address` variable using it's *higher half representative*
 
-And again we face the problem that the _HHDM_ needs to be initialized, and to do it we still need the PMM, but if we could be able to initialize the `hhdm`  before everyhing else in the PMM it will provide us two important new information:
+And again we face the problem that the _HHDM_ needs to be initialized, and to do it we still need the PMM, but if we could be able to initialize the `hhdm`  before everyhing else it will provide us two important new information:
 
-* The first is that once the hhdm is initialized we are sure that for the rest of initialization of any memory management part, we don't neet to map anything else, because we can access all the memory we want already.
-* The `hhdm` will never change once initailized, yeah the content of the memory can changes, but the mapping will always be the same. So there will nevery be the need to free any address within the hhdm.
+* The first is that once the hhdm is initialized we are sure that for the rest of initialization of any memory management part, we don't need to map anything else, because we can already access all the memory we want.
+* The `hhdm` will never change once initailized, yeah the content of the memory can change, but the mapping will always be the same. So there will never be the need to free any address within the hhdm.
 
-Another important thing that we already know is that usually any memory right after the kernel is usually free, and if it is not is because something is loaded there, and we should be able to tell it by parsing some data structures provided by grub (the memory map, and the modules). At this point we can safely assume that unless the addresses above the end of the kernel are reserved in the mmap, or used by some module (and that can be easily achieved by parsing the tables already present in memory), that area can be used.
+Another important thing that we already know is that usually any memory right after the kernel is free, and if it is not is because something is loaded there, we should be able to tell it by parsing some data structures provided by grub (the memory map, and the modules). At this point we can safely assume that unless the addresses above the end of the kernel are reserved in the mmap, or used by some module, that area can be used.
 
-And here is the solution, until the memory maanger is not initialized, we will keep returning addresses that are just after the kernel for creating the page tables required for the `hhdm` Initialization (again with the exclusion of the parts that we already knows are marked as used by the bios/bootloader).
+And here is the solution, until the memory manager is not initialized, we will keep returning addresses that are just after the kernel for creating the page tables required for the `hhdm` Initialization (again with the exclusion of the parts that we already knows are marked as used by the bios/bootloader).
 
 Once the _hhdm_  is initialized, we proceed with the PMM Initialization, but now reassured that we will not encounter any unexpected memory mapping, because we will use the memory map just initialized.
 
 #### pmm_setup()
 
-The first thing that the `pmm_setup()` function is setting the start address of the `anon memory`  used for allocations of the page tables until the pmm is fully initialized and functional.
+The first thing that the `pmm_setup()` function does is setting the start address of the `anon memory`  used for allocations of the page tables until the pmm is fully initialized and functional.
 
 These addresses are tracked in `anon_memory_loc` (for the virtual address to be returned) and `anon_physical_memory_loc` (for the physical counterpart).
 
-Then it calls the `hhdm_map_physical_memory()` function, thata prepare the mememory map inthe higher half. After this point we are able to access the whole physical memory using it's `hhdm` represenation. To clarify, a phyiscal address hhdm representation is given by a simple formula:
+Then it calls the `hhdm_map_physical_memory()` function, thata prepare the memory map in the higher half. After this point we are able to access the whole physical memory using it's `hhdm` represenation. To clarify, a phyiscal address hhdm representation is given by a simple formula:
 
 ```c
 hhdm_var_addres = var_address_phys + HHDM_OFFSET
@@ -90,7 +90,7 @@ At this point is possible to initalize in the following order:
 * the physical memory bitmap: that keeps track of the physical memory that is allocated and free
 * then mark as reserved in the bitmap the memory used for the bitmap itself
 
-Before returning this function set the status of the `pmm_initialized` variable as `true`, meaning that from now on, any page table allocation request from now on will pass through the pmm, instead of using the `anonymous allocation`.
+Before returning this function set the status of the `pmm_initialized` variable as `true`, meaning that from now on, any page table allocation request from will pass through the `pmm`, instead of using the `anonymous allocation`.
 
 This is also the end of the physical memory initialization.
 
@@ -113,9 +113,9 @@ Paging is provided with fixed size paging (only one size of page is supported at
 
 It avails of `x86_64`paging mechanism.
 
-### Higher Hald Direct Map (HHDM)
+### Higher Half Direct Map (HHDM)
 
-An hhdm is provided to the kernel as convenience.
+An _hhdm_ is provided to the kernel as convenience.
 
 ## Virtual Memory  Manager
 
@@ -127,4 +127,15 @@ Currently only the allocation of virtual memory is implemented. There is no `vmm
 
 This is the kernel heap, this is used by the kernel when it needs to allocate resources.
 
+## Memory Organization
+
+Below MMIO_HIGHER_HALF_ADDRESS_OFFSET the address space is reserved by user space memory.
+
+* `MMIO_HIGHER_HALF_ADDRESS_OFFSET = 0xFFFF800000000000`
+    - This address mark the start of the address space reserved for allocating MMIO devices
+    - The MMIO address space size is: `0x280000000` (defined in `MMIO_RESERVED_SPACE_SIZE`)
+* `HIGHER_HALF_ADDRESS_OFFSET = (MMIO_HIGHER_HALF_ADDRESS_OFFSET + MMIO_RESERVED_SPACE_SIZE) = 0xFFFF800280000000`
+    - This addres is the start of the address space that will be used by the kernel while in supervisor mode.
+* `KERNEL_VIRTUAL_ADDR =  0xFFFFFFFF80000000`
+    - From that address, we hae the original mapping of the kernel, in the higher half of memory.
 

src/asm/boot.s

@@ -30,6 +30,8 @@ global multiboot_framebuffer_data
 global multiboot_mmap_data
 global multiboot_basic_meminfo
 global multiboot_acpi_info
+global multiboot_tag_start
+global multiboot_tag_end
 global read_multiboot
 global gdt64
 global stack
@@ -160,6 +162,7 @@ kernel_jumper:
 
     ;.bss section should be already 0  at least on unix and windows systems
     ;no need to initialize
+    mov [multiboot_tag_start], rax
 
 read_multiboot:
     ;Check if the tag is needed by the kernel, if yes store its address
@@ -225,6 +228,8 @@ read_multiboot:
         ; && multiboot_tag.size == 8?
         cmp dword [rax + multiboot_tag.size], 8
         jne read_multiboot
+        add rax, multiboot_tag.size
+        mov qword [multiboot_tag_end], rax
 
     mov rax, higher_half
     jmp rax
@@ -268,6 +273,10 @@ fbb_pt_tables:
 align 4096
 end_of_mapped_memory:
     resq 1
+multiboot_tag_end:
+    resq 1
+multiboot_tag_start:
+    resq 1
 multiboot_framebuffer_data:
     resb 8
 multiboot_mmap_data:

src/include/kernel/framebuffer.h

@@ -27,6 +27,11 @@ typedef struct framebuffer_info {
 extern _fb_window_t framebuffer_main_window;
 extern _fb_window_t framebuffer_logo_area;
 
+extern size_t cur_fb_line;
+extern framebuffer_info framebuffer_data;
+extern uint32_t number_of_lines;
+extern _fb_window_t *logo_area_ptr;
+
 void _fb_putchar(char symbol, size_t cx, size_t cy, uint32_t fg, uint32_t bg);
 void _fb_printStrAt(const char *string, size_t cx, size_t cy, uint32_t fg, uint32_t bg);
 void _fb_printStr(const char *string, uint32_t fg, uint32_t bg);
@@ -35,7 +40,7 @@ void _fb_put_pixel(uint32_t, uint32_t, uint32_t);
 
 uint32_t _fb_get_pixel(uint32_t x, uint32_t y);
 
-/*void map_framebuffer(struct multiboot_tag_framebuffer *);*/
+void* map_framebuffer(struct framebuffer_info fbdata);
 void set_fb_data(struct multiboot_tag_framebuffer *);
 void _fb_printStrAndNumber(const char*, uint64_t, uint32_t, uint32_t);
 void _fb_printStrAndNumberAt(const char*, uint64_t, size_t, size_t, uint32_t, uint32_t);

src/kernel/arch/x86_64/framebuffer/framebuffer.c

@@ -0,0 +1,66 @@
+#include <framebuffer.h>
+#include <stacktrace.h>
+#include <bitmap.h>
+#include <hh_direct_map.h>
+#include <framebuffer.h>
+#include <logging.h>
+#include <numbers.h>
+#include <pmm.h>
+#include <psf.h>
+#include <stdio.h>
+#include <video.h>
+#include <vm.h>
+#include <vmm.h>
+#include <vmm_mapping.h>
+
+extern uint64_t p4_table[];
+extern uint64_t p3_table_hh[];
+extern uint64_t p2_table[];
+#if SMALL_PAGES == 1
+extern uint64_t pt_tables[];
+#endif
+
+/*struct framebuffer_info framebuffer_data;*/
+
+void* map_framebuffer(framebuffer_info fbdata) {
+    uint64_t address_to_map = (uint64_t) fbdata.phys_address;
+    uint64_t virtual_address_start = ensure_address_in_higher_half(address_to_map, VM_TYPE_MMIO);
+    uint64_t virtual_address = virtual_address_start;
+    uint64_t upper_address_to_map = address_to_map + fbdata.memory_size;
+    pretty_logf(Verbose, "Preparing framebuffer: phys_addr: 0x%x, virtual_address: 0x%x - Fb size: 0x%x ", address_to_map, virtual_address, fbdata.memory_size);
+    while ( address_to_map < upper_address_to_map) {
+        map_phys_to_virt_addr((void*)address_to_map, (void*)virtual_address, VMM_FLAGS_PRESENT | VMM_FLAGS_WRITE_ENABLE);
+        address_to_map += PAGE_SIZE_IN_BYTES;
+        virtual_address += PAGE_SIZE_IN_BYTES;
+    }
+    pretty_logf(Verbose, "Framebuffer mapping end at 0x%x - virtual end: 0x%x", address_to_map, virtual_address);
+    return (void *) virtual_address_start;
+}
+
+
+void set_fb_data(struct multiboot_tag_framebuffer *fbtag){
+    //FRAMEBUFFER_MEM = (void*)(uint64_t)fbtag->common.framebuffer_addr;
+#if USE_FRAMEBUFFER == 1
+    //framebuffer_data.address = (void*)(uint64_t)_FRAMEBUFFER_MEM_START;
+    //framebuffer_data.address = hhdm_get_variable((uintptr_t) (fbtag->common.framebuffer_addr));
+    framebuffer_data.pitch = fbtag->common.framebuffer_pitch;
+    framebuffer_data.bpp = fbtag->common.framebuffer_bpp;
+    framebuffer_data.memory_size = fbtag->common.framebuffer_pitch * fbtag->common.framebuffer_height;
+    framebuffer_data.width = fbtag->common.framebuffer_width;
+    framebuffer_data.height = fbtag->common.framebuffer_height;
+    framebuffer_data.phys_address = fbtag->common.framebuffer_addr;
+
+    number_of_lines = 0;
+
+    map_framebuffer(framebuffer_data);
+    framebuffer_data.address = (void*)map_framebuffer(framebuffer_data);
+    cur_fb_line = 0;
+    framebuffer_main_window.x_orig = 0;
+    framebuffer_main_window.y_orig = 0;
+    framebuffer_main_window.width = framebuffer_data.width;
+    framebuffer_main_window.height = framebuffer_data.height;
+    logo_area_ptr = NULL;
+
+#endif
+}
+

src/kernel/arch/x86_64/mem/vmm_mapping.c

@@ -4,6 +4,7 @@
 #include <pmm.h>
 #include <vm.h>
 #include <vmm.h>
+#include <vmm_mapping.h>
 
 void *map_phys_to_virt_addr_hh(void* physical_address, void* address, size_t flags, uint64_t *pml4_root) {
 

src/kernel/framebuffer/framebuffer.c

@@ -1,4 +1,3 @@
-#include <multiboot.h>
 #include <framebuffer.h>
 #include <hh_direct_map.h>
 #include <pmm.h>
@@ -7,9 +6,7 @@
 #include <vm.h>
 #include <stdio.h>
 #include <numbers.h>
-//#ifdef DEBUG - This will be uncommented when the framebuffer library will be completed
 #include <qemu.h>
-//#endif
 #include <video.h>
 #include <dreamcatcher.h>
 #include <logging.h>
@@ -18,7 +15,11 @@ extern void *cur_framebuffer_pos;
 extern uint64_t p4_table[];
 extern uint64_t p3_table_hh[];
 extern uint64_t p2_table[];
+
+#if SMALL_PAGES == 1
 extern uint64_t pt_tables[];
+#endif
+
 extern uint8_t psf_font_version;
 
 uint32_t FRAMEBUFFER_PITCH;
@@ -27,7 +28,7 @@ uint8_t FRAMEBUFFER_BPP = 0;
 uint32_t FRAMEBUFFER_MEMORY_SIZE = 0;
 uint32_t FRAMEBUFFER_WIDTH;
 uint32_t FRAMEBUFFER_HEIGHT;
-struct framebuffer_info framebuffer_data;
+framebuffer_info framebuffer_data;
 
 size_t cur_fb_line;
 uint32_t number_of_lines;
@@ -36,92 +37,6 @@ _fb_window_t framebuffer_main_window;
 _fb_window_t framebuffer_logo_area;
 _fb_window_t *logo_area_ptr;
 
-void map_framebuffer(struct framebuffer_info fbdata) {
-    uint32_t fb_entries = fbdata.memory_size / PAGE_SIZE_IN_BYTES;
-    pretty_logf(Verbose, "Fbdata size: 0x%x", fbdata.memory_size);
-
-    uint64_t phys_address = (uint64_t) fbdata.phys_address;
-
-    uint32_t pd = PD_ENTRY(_FRAMEBUFFER_MEM_START);
-    uint32_t pdpr = PDPR_ENTRY(_FRAMEBUFFER_MEM_START);
-    uint32_t pml4 = PML4_ENTRY(_FRAMEBUFFER_MEM_START);
-#if SMALL_PAGES == 1
-    uint32_t fb_pd_entries = fb_entries / VM_PAGES_PER_TABLE;
-    //uint32_t pt = PT_ENTRY(_FRAMEBUFFER_MEM_START);
-#endif
-
-    if(p4_table[pml4] == 0x00l || p3_table_hh[pdpr] == 0x00l){
-        pretty_log(Verbose, "PANIC - PML4 or PDPR Empty - not supported for now\n");
-        asm("hlt");
-    }
-
-#if SMALL_PAGES == 1
-    uint64_t *current_page_table = pt_tables;
-    for(uint32_t i = 0; i <= fb_pd_entries; i++){
-        bool newly_allocated = false;
-        // Probably should be safer to rely on the direct map if possible?
-        if(p2_table[pd] == 0x00){
-            uint64_t *new_table = pmm_prepare_new_pagetable();
-            p2_table[pd] = (uint64_t)new_table | (PRESENT_BIT | WRITE_BIT);
-            uint64_t *new_table_hhdm = hhdm_get_variable((uintptr_t)new_table);
-            current_page_table = new_table_hhdm;
-            clean_new_table((uint64_t *)new_table_hhdm);
-            newly_allocated = true;
-        }
-        for(int j=0; j < VM_PAGES_PER_TABLE && fb_entries > 0; j++){
-            if(newly_allocated == false){
-            } else {
-                current_page_table[j] = phys_address + (((VM_PAGES_PER_TABLE * i) + j) * PAGE_SIZE_IN_BYTES) | PAGE_ENTRY_FLAGS;
-            }
-            fb_entries--;
-        }
-        newly_allocated = false;
-        pd++;
-    }
-#elif SMALL_PAGES == 0
-    uint32_t fb_entries_mod =  fbdata.memory_size % PAGE_SIZE_IN_BYTES;
-    if(fb_entries_mod != 0){
-        fb_entries++;
-    }
-    for(int j=0; fb_entries > 0; j++){
-        fb_entries--;
-        if( (p2_table[pd+j] < phys_address
-                || p2_table[pd+j] > (phys_address + fbdata.memory_size) )
-                || p2_table[pd+j] == 0x00l ) {
-                p2_table[pd+j] = (phys_address + (j * PAGE_SIZE_IN_BYTES)) | PAGE_ENTRY_FLAGS;
-        }
-    }
-
-
-#endif
-}
-
-
-void set_fb_data(struct multiboot_tag_framebuffer *fbtag){
-    //FRAMEBUFFER_MEM = (void*)(uint64_t)fbtag->common.framebuffer_addr;
-#if USE_FRAMEBUFFER == 1
-    framebuffer_data.address = (void*)(uint64_t)_FRAMEBUFFER_MEM_START;
-    //framebuffer_data.address = hhdm_get_variable((uintptr_t) (fbtag->common.framebuffer_addr));
-    framebuffer_data.pitch = fbtag->common.framebuffer_pitch;
-    framebuffer_data.bpp = fbtag->common.framebuffer_bpp;
-    framebuffer_data.memory_size = fbtag->common.framebuffer_pitch * fbtag->common.framebuffer_height;
-    framebuffer_data.width = fbtag->common.framebuffer_width;
-    framebuffer_data.height = fbtag->common.framebuffer_height;
-    framebuffer_data.phys_address = fbtag->common.framebuffer_addr;
-
-    number_of_lines = 0;
-
-    map_framebuffer(framebuffer_data);
-    cur_fb_line = 0;
-    framebuffer_main_window.x_orig = 0;
-    framebuffer_main_window.y_orig = 0;
-    framebuffer_main_window.width = framebuffer_data.width;
-    framebuffer_main_window.height = framebuffer_data.height;
-    logo_area_ptr = NULL;
-
-#endif
-}
-
 void _fb_putchar(char symbol, size_t cx, size_t cy, uint32_t fg, uint32_t bg){
     uint8_t *framebuffer = (uint8_t *) framebuffer_data.address;
     uint32_t pitch = framebuffer_data.pitch;