How to?

[mixMugz]

Not fully understand how to use? =)

[Gjanosh61]

Hi Maksim first of all thank you for all this. Oh yeah ... I just can't get the hang of the process. I only got to the first level ... of running the first scripts.
I have no idea how to build quadkeys .... how to go on ... very frustrating.

I did some terrain mesh for FSX / P3D also with the ALOS PALSAR DEM source at 12,50m. But as you already said here you have to work only with DEM it is not a terrain mesh.
I can also use MICRODEM to save a Tiff file in BIL format ... and then thick fog.

[muumimorko]

I'm reworking the whole dem-cgl generation to be more intuitive and better functionalized.
Ideally the user has to only input latitude and longitude, select source files, set few parameters, check visually if covered area is OK and press go. Also comes with better instructions.

Until then, if you can't understand it, it is not your fault, but mine.

[mixMugz]

I'm reworking the whole dem-cgl generation to be more intuitive and better functionalized.
Ideally the user has to only input latitude and longitude, select source files, set few parameters, check visually if covered area is OK and press go. Also comes with better instructions.

Until then, if you can't understand it, it is not your fault, but mine.

Thanks! I'm very appreciative.

[Gjanosh61]

What can I say, You are very courteous and helpful. Thank you! I am very grateful,

[JpPereti]

Hi muumimorko
First of all, a big thank you for sharing you work which I am very interested in.
As I can see, you posted new stuff some hours ago about CGL header and others; I presume it means that you are going deeper in the CGL structure, but I can hardly realise what it really means in human language ;)
I did some high-res mesh development for FSX/P3D but that didn't involve Python scripts in which I'm a total newbie.
Best regards
JpP

[daghb]

Looking forward to seeing more of the process. I have a bunch of 1, 5 and 10m DEM files to process, but must I change format from a GeoTIFF?

[JpPereti]

Hi muumimorko
You said "I'm reworking the whole dem-cgl generation to be more intuitive and better functionalized."
I am wondering if something like a meshbgl_to_meshcgl tool would be feasible ?
Regards,
JpP

[muumimorko]

Looking forward to seeing more of the process. I have a bunch of 1, 5 and 10m DEM files to process, but must I change format from a GeoTIFF?

Not necessary. Current version uses Blue Marble Global Mapper, and it can process "anything". I'm working on moving away from Global Mapper to QGIS, because of GM license cost, 550$, although affordable in GIS world, is still a large sum of money.
QGIS also has good file type support (including GeoTIFF).

What comes to higher resolution DEMs, I believe now that at the moment it is not possible to load higher than level 12 from disk. (Means 40 meter resolution on the equator, better on higher latitudes) Reasons for that being:

1. The game ships with level 6 CGLs and those are "replaced" by loading custom CGLs from addon folders.
2. DEM CGL can have only 6 levels. That is because it seems DEM CGL can contain maximum of 4096 tiles on a "level", so if level 0 has 1 tile, level 1 4, level 2 16, level 3 64, level 4 256, level 5 1024 and level 6 4096.

The game does load level 8 CGLs from Azure, there is https involved, but the game "obeys" master certificate and windows proxy setting, so it seems "injection" there is possible :) That would maybe make level 14 possible (10 m on the equator). Also would allow loading textured TINs for extreme resolution.

Hi muumimorko
You said "I'm reworking the whole dem-cgl generation to be more intuitive and better functionalized."
I am wondering if something like a meshbgl_to_meshcgl tool would be feasible ?
Regards,
JpP

I am not familiar with BGLs, but if they can be "unpacked" to raw elevation data, it should be easy to process to cgl. Would it make sense to do so, instead of processing from original data, though?

[JpPereti]

Thanks a lot muumimorko for giving us such valuable explanations; in fact, very different from FSX/P3D; I don't know the way (if any) for extracting elevation data from FSX bgl's, I am only able to generate them from GeoTiff's with the resampler in the SDK. Should you need a sample file, I would be glad to send you some piece of that work.
As for the TINs, would they be considered by the game as simple objects like any building and no more as a terrain (mesh) structure?
Regards,
JpP

[JpPereti]

Would it make sense to do so, instead of processing from original data, though?
Only for people having a lot of custom mesh bgl's :o)
JpP

[muumimorko]

As for the TINs, would they be considered by the game as simple objects like any building and no more as a terrain (mesh) structure?

TINs are used for photogrammetry areas in game. So also terrain.
Opened issue for TINs: #2

[Gjanosh61]

Hello muumimorko,
Despite everything, I still do not grasp the flow of the current.
I really appreciate your willingness to upgrade to QGIS.

As for my learning curve of this _CGLTools
Is there an order to run each scripts? I suppose so.

In 1_GM_create_tilescripts.py ... I replaced the Latitude and Longitude values ​​by taking them from a source file (s05e011.tif). Now starting with py tilescript_000 up to _16.gms are generated (I assume they are the Global Mapper files, which I have not installed)

if not padded, edges will go to zero elevation

px, py = bingtile.LatLongToPixelXY (-4.241032, 11.787552.6)
tx, ty = bingtile.PixelXYToTileXY (px, py)

What I understand, that based on the coordinates entered above a quadkeys listing is generated

........
300010333313
300010333320
300010333321
300010333322
300010333323
300010333330
300010333331
300010333332
300010333333
4096

Then before starting the 2_gm_create_tiles.py

Run scripts generated in previous file

files = glob.glob ("tilescript_006.gms")

but nothing happens.

I wanted to do this overview to show how far I have come in the tunnel ... and that I still can't see the light.

[muumimorko]

Is there an order to run each scripts? I suppose so.

Yes, scripts should be run 1...4.

I assume they are the Global Mapper files, which I have not installed

It is mandatory to to continue, before QGIS is supported.

and that I still can't see the light.

There will be light

New version, no QGIS yet, will be much simpler to use:

After that comes QGIS.

[Gjanosh61]

Hello @muumimorko, Happy New Year!

Well, what I was missing in your workflow was that I had to create a catalog file with Global Mapper (I don't have much experience with GM).

a) Created the .gmc catalog file

b) After entering the geographical coordinates LatLon (-4.5930, 12.7525) in files 1_, 3_ and 4_

if not padded, edges will go to zero elevation

px, py = bingtile.LatLongToPixelXY (-4.5930, 12.7525.6)
tx, ty = bingtile.PixelXYToTileXY (px, py)
qkeybase = bingtile.TileXYToQuadKey (tx, ty, 6)

script.write (
'IMPORT FILENAME = J: \ MeshAngola_QGIS \ MESH_ETM12 \ my.gmc ELEV_UNITS = METERS ELEV_SCALE = 1 VOID_ELEV = 0.0 \ n')

I start the script 1_ tutto Ok

Script start 2_ the process starts after a few minutes the script crashes without showing any errors. I have to force close

Start scripts 3_ and 4 _

A 16 Mbyte dem010.cgl file is created in the "300" folder, which corresponds to level 3 of the world quadkey.

Upload the "creator-exampledem" folder to Community.

layout.json

{
"content": [
{
"path": "CGL / 300 / dem010.cgl",
"size": 16394412,
"date": 132541684221955346
},
{
"path": "ContentInfo / creator-exampledem / Thumbnail.jpg",
"size": 14667,
"date": 132526016300000000
}
]
}

I start Msfs2020 but I don't see any difference in altitude between the default scenario and the additional DEM.

[henrikx]

Is there an order to run each scripts? I suppose so.

Yes, scripts should be run 1...4.

I assume they are the Global Mapper files, which I have not installed

It is mandatory to to continue, before QGIS is supported.

and that I still can't see the light.

There will be light

New version, no QGIS yet, will be much simpler to use:

After that comes QGIS.

Any progress on new version?

Frankly, even with programming experience I can not really work out how to use this. E.g. what is "target area"? is it the coordinates of the upper left corner of the DEM and if so, which DEM file (if you have multiple)?

[muumimorko]

New version is up.
Needs one to install OSGeo4W (express desktop install is OK, specifically only GDAL and python environment are needed).
And python modules "pyshp" and "click".
Also wrote some instructions on "how-to": https://github.com/muumimorko/MSFS2020_CGLTools/blob/main/docs/tut/DEM-CGL.md

If there is further questions, please ask.
Next step to easier use will be moving to QGIS base system with a GUI.

In other news, there is GDAL-based heightmap generation script "genHeightmaps.py", which is able to generate BGL heightmaps. No docs for it, but it might give inspiration :) To be integrated with QGIS based "thing".

[henrikx]

New version is up.
Needs one to install OSGeo4W (express desktop install is OK, specifically only GDAL and python environment are needed).
And python modules "pyshp" and "click".
Also wrote some instructions on "how-to": https://github.com/muumimorko/MSFS2020_CGLTools/blob/main/docs/tut/DEM-CGL.md

If there is further questions, please ask.
Next step to easier use will be moving to QGIS base system with a GUI.

In other news, there is GDAL-based heightmap generation script "genHeightmaps.py", which is able to generate BGL heightmaps. No docs for it, but it might give inspiration :) To be integrated with QGIS based "thing".

The work done here is nothing short of amazing. Thank you for your incredible effort to the flightsim community! I will be sure to generate some heightmaps for Norway soon.

Only thing that confuses me:

Edit Options

DEM input files is a list of source data, that is corrected to EGM2008, all specified files will be loaded to Global Mapper. First item in the list will be at the bottom, and last item at the top. So input order should be from lowest quality to highest quality.

Can you simply put in the path to the map catalog here?

[muumimorko]

Can you simply put in the path to the map catalog here?

Yes.
DEMInputFiles=[r'C:\path\to_undulated_catalog.gmc']

[henrikx]

What can I say? Except it works indeed. There's just one problem. MSFS will according to debug tile download DEM data from Azure instead of loading terrain from disk, so to use the CGL, offline mode must be enabled. Do you know if there's a way around this?

[muumimorko]

No known way to "override" Azure, if there is higher resolution data in the cloud, the game will load it and ignore local data. And I agree that offline mode is not really an option.

[muumimorko]

Just to let you know, I've successful generated and loaded a level 13 CGL! <3

Compiling lvl14 now. Releasing when I get few details ironed out.

[henrikx]

Very nice!

Does this mean we can generate higher (or same) LODs as Azure and therefore override Azure?

[muumimorko]

Does this mean we can generate higher (or same) LODs as Azure and therefore override Azure?

Yes :)
However, spiking issue gets worse as more detail is added:

This is I believe because the pyramid generation parameters do not match to what the game uses.
(also they don't match inside the pyramid generation function :D)

[henrikx]

That's pretty good news.

Will you be able to make the pyramid generation parameters match the game's parameters to fix spiking?

[Gjanosh61]

Finally everything is fine ... the workflow is now very simple ... thank you very much !!!
I would like to ask a question, I apologize if this may be trivial, how can I see if the new cgl have been loaded in the sim?
I tried to do this by comparing the value on GM and that on the Sim. My source DEM data are ALOS Palsar at 12.50 meters.

In GM we see the altitude of 1440,094 meters. ALOS Palsar file n44e034.tif has been translated into EGM2008. In MSFS2020 the altitude is 1388.289 mt / 4554.753 ft. The difference in altitude is 52 meters (lower altitude in MSFS).

[Gjanosh61]

Other matter, this only happens to me in quadkey 120330 (dem330.cgl). When I move away from that position it disappears .... what could be influencing this artifact?

[muumimorko]

Will you be able to make the pyramid generation parameters match the game's parameters to fix spiking?

I don't know. Maybe it could be figured out by correlating input data and it's result in game. (Response test to known signal)

The difference in altitude is 52 meters (lower altitude in MSFS).

Same issue, kind of. Because pyramid generation algorithm does not match what the game does in inverse when it loads the terrain, the elevation values will not be absolutely correct, especially where there are large altitude changes.

what could be influencing this artifact?

Does it go away if you go close to it?
If yes, it could be: https://flightsim.to/file/5167/scenery-glitch-fixes-for-update-1-12-13-0
If not, please send me a copy of that file so I can take a look.

[Gjanosh61]

Yes, when I get close they disappear (the shadow remains on the surface anyway). I downloaded scenery-glitch-fixes-for-update-1-12-13-0 and replaced the four default "vecnXXX.cgl" (H: \ MSFS_2020 \ Official \ OneStore \ fs-base-cgl \ CGL \ 120 ) with the update mod (vecn321, vecn330, vecn 323 and vecn 332). The result does not change, the artifacts are still visible.

On my Google Drive you can find the _temp folder (I send you the relative link with a private message on "FSDeveloper")

The Mesh_ETM12 folder contains the source files .tif ALOS Palsar while the _msl.tif files have been translated with egm208.tiff, as the DEM included in an ASF's ALOS PALSAR RTC product was converted from the orthometric height of the source DEM to ellipsoid height using the ASF MapReady geoid_adjust tool.

The MiG-ETM12m-crimea-lvl12_EGM2008 folder
has CGL files produced with the .tif ALOS Palsar source translated with egm08_25.gtx

The MiG-ETM12m-crimea-lvl12_ALOS_MD folder
has CGL files produced with source .tif files translated with EGM2008-WGS84.tif

The MiG-ETM12m-crimea-lvl12_ALOS folder has the produced CGL files
with files .tiff ALOS Palsar RTC

https://geographiclib.sourceforge.io/cgi-bin/GeoidEval?input=44.41102+33.51939&option=Submit