Command line export with Linux headless always re-imports PNG resources #27963

jamie-pate · 2019-04-12T20:21:06Z

Godot version:
3.1

OS/device including version:
linux_headless

Issue description:

When run with --export the headless version will re-import all resources, causing the export to take 20x longer than before.

~/bin/Godot_v3.1-stable_linux_headless.64 --export Linux/X11 Kinematic\ Character\ 3D.x86_64 
ERROR: instance: Class 'EditorSettings' can only be instantiated by editor.
   At: core/class_db.cpp:518.
ERROR: poll: /home/jpate/.config/godot/editor_settings-3.tres:3 - Parse Error: Can't create sub resource of type: EditorSettings
   At: scene/resources/resource_format_text.cpp:561.
ERROR: load: Condition ' err != OK ' is true. returned: RES()
   At: core/io/resource_loader.cpp:208.
ERROR: _load: Failed loading resource: /home/jpate/.config/godot/editor_settings-3.tres
   At: core/io/resource_loader.cpp:285.
WARNING: create: Could not open config file.
   At: editor/editor_settings.cpp:872.
reimport: begin: (Re)Importing Assets steps: 2
	reimport: step 0: purple_wood.png
	reimport: step 1: white_wood.png
savepack: begin: Packing steps: 102
	savepack: step 2: Storing File: res://cubelib.res
	savepack: step 14: Storing File: res://cubio.gdc
	savepack: step 27: Storing File: res://follow_camera.gdc
	savepack: step 39: Storing File: res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.stex
	savepack: step 39: Storing File: res://icon.png.import
	savepack: step 52: Storing File: res://level.scn
	savepack: step 64: Storing File: res://.import/purple_wood.png-ae65a206e7a59edf759728c3bad04e56.s3tc.stex
	savepack: step 64: Storing File: res://purple_wood.png.import
	savepack: step 77: Storing File: res://purplecube.scn
	savepack: step 89: Storing File: res://.import/white_wood.png-6895acd60ce97b4315494d2be377c357.s3tc.stex
	savepack: step 89: Storing File: res://white_wood.png.import
	savepack: step 102: Storing File: res://cubio.gd.remap
	savepack: step 102: Storing File: res://follow_camera.gd.remap
	savepack: step 102: Storing File: res://icon.png
	savepack: step 102: Storing File: res://project.binary
savepack: end
reimport: end
ERROR: ~List: Condition ' _first != __null ' is true.
   At: ./core/self_list.h:111.
ERROR: cleanup: There are still MemoryPool allocs in use at exit!
   At: core/pool_vector.cpp:70.

Steps to reproduce:
Download attached modified version of kinematic_character demo
extract

cd kinematic_character_export/kinematic_character
path/to/Godot_v3.1-stable_linux_headless.64 --export Linux/X11 Kinematic\ Character\ 3D.x86_64

Minimal reproduction project:
kinematic_character_export.zip

The text was updated successfully, but these errors were encountered:

jamie-pate · 2019-04-12T20:35:46Z

looks like it's getting tripped up here

Can we warn on invalid import settings? It doesn't seem to reimport them when i use the non-headless version.

editor_file_system.cpp:349

if (!ResourceFormatImporter::get_singleton()->are_import_settings_valid(p_path)) {
		//reimport settings are not valid, reimport
		return true;
}

Also, it only seems to happen for my .png files

Looks like this change:
f669ebe

if (!metadata.has("vram_texture")) {

but purple_wood.png.import
contains

[remap]
importer="texture"
type="StreamTexture"
path.s3tc="res://.import/purple_wood.png-ae65a206e7a59edf759728c3bad04e56.s3tc.stex"
path.etc2="res://.import/purple_wood.png-ae65a206e7a59edf759728c3bad04e56.etc2.stex"
metadata={
"imported_formats": [ "s3tc", "etc2" ],
"vram_texture": true
}

It looks like it's not recognizing path.s3tc as a valid format for the platform?
ResourceFormatImporter::_get_path_and_type returns ERR_FILE_CORRUPT

VSG::storage->has_os_feature() always returns false
I assume in the linux non-headless build this would work for p_feature == 's3tc'?

static bool has_server_feature_callback(const String &p_feature) {

	if (VisualServer::get_singleton()) {
		if (VisualServer::get_singleton()->has_os_feature(p_feature)) {
			return true;
		}
	}

	return false;
}

#0  RasterizerStorageDummy::has_os_feature (this=0x555559eed1b0, p_feature=...) at ./drivers/dummy/rasterizer_dummy.h:728
#1  0x0000555558176720 in VisualServerRaster::has_os_feature (this=0x555559ee3740, p_feature=...) at servers/visual/visual_server_raster.cpp:179
#2  0x000055555804c94b in has_server_feature_callback (p_feature=...) at servers/register_server_types.cpp:98
#3  0x00005555584a76c8 in OS::has_feature (this=0x7fffffffd810, p_feature=...) at core/os/os.cpp:633
#4  0x0000555558563de1 in ResourceFormatImporter::_get_path_and_type (this=0x555559e34440, p_path=..., r_path_and_type=..., r_valid=0x0) at core/io/resource_importer.cpp:85

aaronfranke · 2019-05-20T11:50:27Z

When you say "than before", are you comparing to Godot 3.0?

jamie-pate · 2019-05-20T17:40:57Z

Yes, with 3.0 I could use --export with the headless build and export each platform and it did not re-import every resource.

With 3.1 it fails to accept the resource file as up to date and re-imports every resource. (Because VSG::storage->has_os_feature() always returns false on the headless build.)

Mithul · 2019-08-19T18:22:37Z

Facing the same as @jamie-pate for "Linux/X11" as well. Using headless Godot Godot_v3.1.1-stable_linux_headless.64

Command
./Godot_v3.1.1-stable_linux_headless.64 --export "Linux/X11" builds/

Output

ERROR: instance: Class 'EditorSettings' can only be instantiated by editor.
   At: core/class_db.cpp:523.
ERROR: poll: /home/mithul/.config/godot/editor_settings-3.tres:3 - Parse Error: Can't create sub resource of type: EditorSettings
   At: scene/resources/resource_format_text.cpp:561.
ERROR: load: Condition ' err != OK ' is true. returned: RES()
   At: core/io/resource_loader.cpp:208.
ERROR: _load: Failed loading resource: /home/mithul/.config/godot/editor_settings-3.tres
   At: core/io/resource_loader.cpp:285.
WARNING: create: Could not open config file.
   At: editor/editor_settings.cpp:871.
ERROR: copy: Failed to open 
   At: core/os/dir_access.cpp:300.
ERROR: _fs_changed: Project export failed with error code 12.
   At: editor/editor_node.cpp:495.
ERROR: ~List: Condition ' _first != __null ' is true.
   At: ./core/self_list.h:111.
ERROR: cleanup: There are still MemoryPool allocs in use at exit!
   At: core/pool_vector.cpp:70.

Using regular Linux build fails with

> ./godot --export "blah" `readlink -f builds/`
ERROR: initialize: X11 Display is not available
   At: platform/x11/os_x11.cpp:126.

The non-headless server also fails with

ERROR: _load: Method/Function Failed, returning: RES()
   At: core/io/resource_loader.cpp:285.
ERROR: start: Condition ' !scene ' is true. returned: false
   At: main/main.cpp:1739.
WARNING: cleanup: ObjectDB Instances still exist!
   At: core/object.cpp:2095.
ERROR: clear: Resources Still in use at Exit!
   At: core/resource.cpp:425.
ERROR: cleanup: Condition ' allocs_used > 0 ' is true.
   At: core/pool_vector.cpp:70.

Mtaethefarmer · 2019-11-23T08:36:48Z

I was able to get rid of the re-importing issue by being inside the project folder and specifying the 'project.godot' file.

godot project.godot -v -q --export "Linux/X11" ~/project/build/linux/$EXPORT_NAME.x86_64

There is still the

ERROR: ~List: Condition ' _first != __null ' is true.
   At: ./core/self_list.h:111.
ERROR: cleanup: There are still MemoryPool allocs in use at exit!
   At: core/pool_vector.cpp:70.

but that is a different issue all together. #8985

jamie-pate · 2020-03-04T19:51:29Z

Godot_v3.2-stable_linux_headless.64 project.godot --export Linux/X11 '../export/tmp/$EXPORTNAME-linux' does not solve the problem for me.

-q got rid of the re-import, but also failed to actually do any exporting

Calinou · 2020-03-04T21:23:42Z

-q got rid of the re-import

It probably just silenced the reimport as -q enables the quiet mode 😉

jamie-pate · 2020-03-05T04:58:15Z

Well it also silenced the exported files. Are you sure it's not --quit?

Calinou · 2020-03-05T10:49:54Z

~~@jamie-pate --quit doesn't have a shorthand, unlike --quiet whose shorthand is -q.~~

akien-mga · 2020-03-05T12:02:30Z

--quit doesn't have a shorthand, unlike --quiet whose shorthand is -q.

Hm ;)

$ godot-git --help | egrep "\-\-(quit|quiet)"
  --quiet                          Quiet mode, silences stdout messages. Errors are still displayed.
  -q, --quit                       Quit after the first iteration.

Calinou · 2020-03-05T13:02:35Z

@akien-mga Seems like I was wrong then. It's kind of unexpected, -q should probably have been a shorthand for --quiet as it's a more common operation.

fire · 2021-01-12T00:22:27Z

Can we recreate this on the latest stable?

realkotob · 2021-01-12T08:13:45Z

@fire I'll try again in the weekend with the most recent 3.2 release

It was happening to me on CI for version 3.2.2 (or 3.2.3) last spring.

jamie-pate · 2021-12-07T16:12:35Z

still a problem on 3.3.4 when running --export-pack at least

KoBeWi · 2022-05-19T22:55:43Z

Is this maybe fixed in 4.0?

jamie-pate · 2022-12-28T17:11:09Z

#69511 maybe it's worse in godot 4? :D

Still an issue on 3.5.2-rc1 with --export-pack

$ ./test.sh 
Godot Engine v3.5.2.rc1.official.f5f0543ae - https://godotengine.org
 
reimport: begin: (Re)Importing Assets steps: 1
	reimport: step 0: icon-texture.png

repro-headless-png-always-reimports.zip

jamie-pate mentioned this issue Apr 12, 2019

Headless build fails to export and then crashes #27933

Closed

akien-mga added bug topic:editor topic:import labels Apr 15, 2019

akien-mga added this to the 3.2 milestone Apr 15, 2019

jamie-pate mentioned this issue Jan 15, 2020

Export: Properly disable resource preview thread #35113

Merged

akien-mga added the confirmed label Jan 15, 2020

akien-mga self-assigned this Jan 15, 2020

akien-mga changed the title ~~Godot_v3.1-stable_linux_headless.64 re-imports all resources when exporting~~ Command line export with Linux headless always re-imports PNG resources Jan 15, 2020

akien-mga modified the milestones: 3.2, 4.0 Jan 16, 2020

akien-mga added the cherrypick:3.x Considered for cherry-picking into a future 3.x release label Jan 16, 2020

akien-mga removed the cherrypick:3.x Considered for cherry-picking into a future 3.x release label Mar 18, 2021

Calinou mentioned this issue Dec 7, 2021

Make -q CLI argument toggle quiet stdout instead of quitting #55696

Merged

rainerweston mentioned this issue Dec 8, 2021

Godot headless build does not import some assets guysoft/EnergySource#3

Closed

KoBeWi added this to 4.x Priority Issues May 19, 2022

KoBeWi moved this to To Assess in 4.x Priority Issues May 19, 2022

clayjohn modified the milestones: 4.0, 3.x Jan 18, 2023

clayjohn removed this from 4.x Priority Issues Jan 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Command line export with Linux headless always re-imports PNG resources #27963

Command line export with Linux headless always re-imports PNG resources #27963

jamie-pate commented Apr 12, 2019

jamie-pate commented Apr 12, 2019 •

edited

Loading

aaronfranke commented May 20, 2019

jamie-pate commented May 20, 2019 •

edited

Loading

Mithul commented Aug 19, 2019

Mtaethefarmer commented Nov 23, 2019 •

edited

Loading

jamie-pate commented Mar 4, 2020 •

edited

Loading

Calinou commented Mar 4, 2020 •

edited

Loading

jamie-pate commented Mar 5, 2020

Calinou commented Mar 5, 2020 •

edited

Loading

akien-mga commented Mar 5, 2020 •

edited

Loading

Calinou commented Mar 5, 2020

fire commented Jan 12, 2021

realkotob commented Jan 12, 2021

jamie-pate commented Dec 7, 2021 •

edited

Loading

KoBeWi commented May 19, 2022

jamie-pate commented Dec 28, 2022

Command line export with Linux headless always re-imports PNG resources #27963

Command line export with Linux headless always re-imports PNG resources #27963

Comments

jamie-pate commented Apr 12, 2019

jamie-pate commented Apr 12, 2019 • edited Loading

aaronfranke commented May 20, 2019

jamie-pate commented May 20, 2019 • edited Loading

Mithul commented Aug 19, 2019

Mtaethefarmer commented Nov 23, 2019 • edited Loading

jamie-pate commented Mar 4, 2020 • edited Loading

Calinou commented Mar 4, 2020 • edited Loading

jamie-pate commented Mar 5, 2020

Calinou commented Mar 5, 2020 • edited Loading

akien-mga commented Mar 5, 2020 • edited Loading

Calinou commented Mar 5, 2020

fire commented Jan 12, 2021

realkotob commented Jan 12, 2021

jamie-pate commented Dec 7, 2021 • edited Loading

KoBeWi commented May 19, 2022

jamie-pate commented Dec 28, 2022

jamie-pate commented Apr 12, 2019 •

edited

Loading

jamie-pate commented May 20, 2019 •

edited

Loading

Mtaethefarmer commented Nov 23, 2019 •

edited

Loading

jamie-pate commented Mar 4, 2020 •

edited

Loading

Calinou commented Mar 4, 2020 •

edited

Loading

Calinou commented Mar 5, 2020 •

edited

Loading

akien-mga commented Mar 5, 2020 •

edited

Loading

jamie-pate commented Dec 7, 2021 •

edited

Loading