Mingw link issue #3270

dmoody256 · 2019-01-14T04:56:25Z

The test/Win32/mingw.py test on my local system and on Appveyor 2017 image. We run the 2015 image which the test gets skipped in.

These changes allow the test to pass.

Changes in this PR:

Update link.py to validate targets as Nodes
Update mingw.py to remove /nologo flag added by msvc tool
Update appveyor to use 32 bit mingw until this is done: mingw tool should provide method to select version and target platform #3291

Contributor Checklist:

I have updated master/src/CHANGES.txt directory (and read the README.txt in that directory)

…er, and updated mingw to remove MSVC like nologo flag

coveralls · 2019-01-14T08:07:19Z

Coverage decreased (-0.0009%) to 79.998% when pulling 4518d14 on dmoody256:mingw_link_issue into 0f782d0 on SCons:master.

coveralls · 2019-01-14T08:07:19Z

Coverage decreased (-0.0009%) to 79.998% when pulling 4518d14 on dmoody256:mingw_link_issue into 0f782d0 on SCons:master.

coveralls · 2019-01-14T08:07:20Z

Coverage decreased (-0.0009%) to 79.998% when pulling 4518d14 on dmoody256:mingw_link_issue into 0f782d0 on SCons:master.

coveralls · 2019-01-14T08:07:20Z

Coverage decreased (-0.0009%) to 79.998% when pulling 4518d14 on dmoody256:mingw_link_issue into 0f782d0 on SCons:master.

coveralls · 2019-01-14T08:07:21Z

Coverage decreased (-0.02%) to 79.983% when pulling c2dc898 on dmoody256:mingw_link_issue into 0f782d0 on SCons:master.

bdbaddog · 2019-01-14T15:24:00Z

Where is /NOLOGO getting set in the first place?
Wouldn't it be better to not set it?

Also a unit test can check this is working.

dmoody256 · 2019-01-14T17:39:21Z

Where is /NOLOGO getting set in the first place?
Wouldn't it be better to not set it?

I think both maybe, I suppose it matters the order the tools are loaded. It is set from the msvc.py tool, and for the test/Win32/mingw.py test, it creates a Environment(tools=['default', 'mingw']) which I think means the msvc tool runs first, adding the flag, then the mingw tool runs, removing the flag.

Another case could be Environment(tools=['mingw', 'msvc']) which would not remove the flag. I guess the msvc tool should check what tools are loaded and make sure not to add that flag if the mingw tool was loaded? The test/Win32/mingw.py should probably be updated to test this case also.

Also a unit test can check this is working.

Ok, I'll add a unit test so we can see it passing in the Appveyor 2015 image for this PR.

bdbaddog · 2019-01-15T04:55:07Z

Maybe add a unit test, but file a bug which says mingw and msvc stomp on each others flags?
With a short repeater?

bdbaddog · 2019-01-20T21:46:08Z

src/engine/SCons/Tool/mingw.py

@@ -167,7 +167,8 @@ def generate(env):

    env['SHOBJSUFFIX'] = '.o'
    env['STATIC_AND_SHARED_OBJECTS_ARE_THE_SAME'] = 1
-
+    if 'CCFLAGS' in env:
+        env['CCFLAGS'] = SCons.Util.CLVar(str(env['CCFLAGS']).replace('/nologo', ''))


Won't this fail badly if CCFLAGS is a list?

yeah, I was assuming it was CLVar, but its user settable so really should check if its CLVar and if not make it one first before doing the replace.

bdbaddog · 2019-01-27T22:01:16Z

src/engine/SCons/Tool/mingw.py

@@ -150,6 +150,7 @@ def generate(env):

    #... but a few things differ:
    env['CC'] = 'gcc'
+    env['CCFLAGS'] = SCons.Util.CLVar('')


with something like

env=Environment(tools=['mslink','mingw','g++'],CCFLAGS='-myflag')

with the new code CCFLAGS will be ignored?

I was looking at the MSVC tool and it seemed to do the same thing, so I
thought that was ok to remove flags during generate:

scons/src/engine/SCons/Tool/msvc.py

Lines 243 to 244 in e44c771

env['CCFLAGS'] = SCons.Util.CLVar('/nologo')

env['CFLAGS'] = SCons.Util.CLVar('')

bdbaddog · 2019-01-27T22:02:49Z

Seems like the best fix here is probably to fix the tests, and then for mingw to remove any /options ? (or to issue warnings about / options)

dmoody256 · 2019-01-28T03:38:10Z

Seems like the best fix here is probably to fix the tests,

Not sure what you mean by fix the tests, they are explicitly testing that msvc and mingw work together:

scons/test/Win32/mingw.py

Lines 165 to 166 in e44c771

    
           # Test with specifying the default tool to make sure msvc setting doen't ruin it 
        
           # for mingw:

I can add unit test such that fakes out the need for mingw to actually be present, and check that certain things were setup in the environment, like flags that don't have / in them.

and then for mingw to remove any /options ? (or to issue warnings about / options)

It would have to remove it, because the msvc tool is adding it, and seems like the test indicates those tools should be able to be in the same environment.

bdbaddog · 2019-01-28T18:43:16Z

src/engine/SCons/Tool/link.py

@@ -81,6 +81,8 @@ def _lib_emitter(target, source, env, **kw):
    if Verbose:
        print("_lib_emitter: target[0]={!r}".format(target[0].get_path()))
    for tgt in target:
+        if SCons.Util.is_String(tgt):


Curious why this started failing now?

I don't know either, I don't know the last time we had the test/Win32/mingw.py test passing to see what changed since then.

A bit of a weakness of our tests which just skip a test if the tool/setup is not there is if no CI environment is set up for a particular config, it just doesn't get tested. We have potential exposure on mingw, D, docbook, TeX, SWIG, packaging. Some of these have gotten added in recently, others fail to be added (e.g. the WiX toolset for msi packaging: apparently that code has rotted, or WiX has changed out from underneath it).

Indeed. Some tests will create fake tools to run with always and/or if the tool doesn't exist in the environment. However in some cases that's insufficient to test if the scons code works. Ideally we'd get a list of all tools missing and/or causing tests to be skipped at the end of a test run.. but that could be considerable work to make happen.

Any thoughts on how to better address?

Both Appveyor and Travis have good REST APIs for getting the logs. I could fashion a python script to get the logs of the latest master CI runs and parse the No Results at the end. That would make collecting the current list easier.

We could take that list and update a wiki page using the wiki git repository like in this example: https://stackoverflow.com/a/14673994/1644736

Yeah, it will get an exception that 'str' has no member 'attributes' just below. I'm not sure if there is somewhere further up the chain that the target should have been converted.

Still should be some kind of check here even if the real fix is somewhere else. Maybe a try/except? Add a warning message? I think assuming its a file if its a string is a pretty good default case.

Yes. Can you change to try/except and put a comment in the except that we'd expect the target to be a Node at this point. And a # TODO to track down the why at some point?
I don't want to lose this discovery without some reasonable way of remembering to address it at some point if possible.

I chased this down and found that mingw is getting an emitter list of [mslink.py winLibEmitter, link.py shlib_emitter, mingw.py shlib_emitter]. mslink's winLibEmitter never converts its new targets to File's and just leaves them as strings:

scons/src/engine/SCons/Tool/mslink.py

Lines 144 to 154 in 9f09013

extratargets.append(

env.ReplaceIxes(dll,

'%sPREFIX' % paramtp, '%sSUFFIX' % paramtp,

"LIBPREFIX", "LIBSUFFIX"))

# and .exp file is created if there are exports from a DLL

extratargets.append(

env.ReplaceIxes(dll,

'%sPREFIX' % paramtp, '%sSUFFIX' % paramtp,

"WINDOWSEXPPREFIX", "WINDOWSEXPSUFFIX"))

return (target+extratargets, source+extrasources)

The other tools with emitters take care to convert targets to File objects:

scons/src/engine/SCons/Tool/cyglink.py

Lines 100 to 106 in 9f09013

target_strings = env.ReplaceIxes(orig_target[0],

'%sPREFIX' % vp, '%sSUFFIX' % vp,

'IMPLIBPREFIX', 'IMPLIBSUFFIX')

if Verbose:

print("_lib_emitter: target_strings=%r" % target_strings)

implib_target = env.fs.File(target_strings)

So the fix can be applied by converting the the targets in mslink.py's winLibEmitter to File's. Probably still want to have a check in link.py to convert str's as well.

I left the if check instead of a try except because the current code looks cleaner than:

for tgt in target: try: tgt.attributes.shared = 1 except AttributeError: tgt = env.File(tgt) tgt.attributes.shared = 1

I am not sure about the speed up. Let me know your opinion.

dmoody256 · 2019-02-03T04:11:24Z

So now with the AppendENVPath (#3281) fixed, mingw.py test is having an issue because its using the 32 bit MinGW which is first in the tools default paths. Previously it would get pushed to the back and the 64 bit mingw would be next in line. Appveyor sets an environment path for 64 bit mingw, which then fails to run the 32 bit executable. It seems we would need to update the appveyor path to point to 32 bit MinGW.

Which begs the question, what if someone had multiple versions of a tool installed, how would they force scons to make sure the desired one is chosen? Seems like they would need to call PrependENVPath with which ever one they wanted after the tool was loaded.

mwichmann · 2019-02-03T16:40:55Z

Which begs the question, what if someone had multiple versions of a tool installed, how would they force scons to make sure the desired one is chosen? Seems like they would need to call PrependENVPath with which ever one they wanted after the tool was loaded.

I've wondered that... having several java versions installed isn't that uncommon (although maybe multiple jdks is less common). We include JAVA_HOME as an argument to the build to help with that.

bdbaddog · 2019-02-03T17:03:35Z

This is what we added TARGET_ARCH for (well 64 vs 32 bits), but currently it's only used for MSVC. Is there an easy way to determine if the installled mingw is 32 vs 64 bits and could be used to select the correct installed version? Also MINGW_VERSION maybe to say we only want a specific version (also copying from MSVC)?

…

-Bill

On Sun, Feb 3, 2019 at 11:40 AM Mats Wichmann ***@***.***> wrote: Which begs the question, what if someone had multiple versions of a tool installed, how would they force scons to make sure the desired one is chosen? Seems like they would need to call PrependENVPath with which ever one they wanted after the tool was loaded. I've wondered that... having several java versions installed isn't that uncommon (although maybe multiple jdks is less common). We include JAVA_HOME as an argument to the build to help with that. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3270 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFBNJvvX-rb-zJbSWxbEzz1bdgwElplks5vJxEXgaJpZM4Z9nK1> .

dmoody256 · 2019-02-06T03:23:13Z

This is what we added TARGET_ARCH for (well 64 vs 32 bits), but currently it's only used for MSVC. Is there an easy way to determine if the installled mingw is 32 vs 64 bits and could be used to select the correct installed version? Also MINGW_VERSION maybe to say we only want a specific version (also copying from MSVC)?

I am going to write that as a separate issue, and for this PR focus on getting mingw working with the first version it finds. Created #3291 for that functionality.

bdbaddog · 2019-02-15T17:48:19Z

Merging.

dmoody256 added 2 commits January 13, 2019 22:52

fixed mingw emitter to convert str to node before accessing node memb…

fdd3664

…er, and updated mingw to remove MSVC like nologo flag

updated CHANGES.txt

4518d14

dmoody256 changed the title ~~Mingw link issue~~ [WIP] Mingw link issue Jan 14, 2019

dmoody256 mentioned this pull request Jan 20, 2019

Switch to codecov #3275

Merged

bdbaddog reviewed Jan 20, 2019

View reviewed changes

bdbaddog reviewed Jan 27, 2019

View reviewed changes

bdbaddog reviewed Jan 28, 2019

View reviewed changes

dmoody256 added 2 commits January 30, 2019 12:32

check to make sure that the CCFLAGS are a CLVar

b68e505

Merge remote-tracking branch 'origin/master' into mingw_link_issue

dd340d5

dmoody256 force-pushed the mingw_link_issue branch from c2dc898 to dd340d5 Compare January 30, 2019 18:33

Merge remote-tracking branch 'origin/master' into mingw_link_issue

3341dc6

dmoody256 force-pushed the mingw_link_issue branch from 6d2d97f to b3f363b Compare February 3, 2019 06:03

This was referenced Feb 4, 2019

make M4 test cross platform #3286

Merged

update find_program_path to leave the env un-altered #3287

Closed

Add windows default paths to lex tool #3290

Merged

switch to 32 bit mingw in appveyor and add mingw env test

3454ffa

Merge remote-tracking branch 'origin/master' into mingw_link_issue

88621c3

dmoody256 force-pushed the mingw_link_issue branch from b3f363b to 88621c3 Compare February 8, 2019 03:52

dmoody256 changed the title ~~[WIP] Mingw link issue~~ Mingw link issue Feb 8, 2019

bdbaddog merged commit f296600 into SCons:master Feb 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mingw link issue #3270

Mingw link issue #3270

dmoody256 commented Jan 14, 2019 •

edited

Loading

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019 •

edited

Loading

bdbaddog commented Jan 14, 2019

dmoody256 commented Jan 14, 2019

bdbaddog commented Jan 15, 2019

bdbaddog Jan 20, 2019

dmoody256 Jan 21, 2019

bdbaddog Jan 27, 2019

dmoody256 Jan 28, 2019 •

edited

Loading

bdbaddog commented Jan 27, 2019

dmoody256 commented Jan 28, 2019

bdbaddog Jan 28, 2019

dmoody256 Jan 29, 2019

mwichmann Jan 29, 2019

bdbaddog Jan 29, 2019

dmoody256 Jan 29, 2019

dmoody256 Feb 6, 2019 •

edited

Loading

dmoody256 Feb 6, 2019

bdbaddog Feb 6, 2019

dmoody256 Feb 8, 2019 •

edited

Loading

dmoody256 Feb 8, 2019

dmoody256 commented Feb 3, 2019 •

edited

Loading

mwichmann commented Feb 3, 2019

bdbaddog commented Feb 3, 2019 via email

dmoody256 commented Feb 6, 2019 •

edited

Loading

bdbaddog commented Feb 15, 2019

	env['CCFLAGS'] = SCons.Util.CLVar('/nologo')
	env['CFLAGS'] = SCons.Util.CLVar('')

	extratargets.append(
	env.ReplaceIxes(dll,
	'%sPREFIX' % paramtp, '%sSUFFIX' % paramtp,
	"LIBPREFIX", "LIBSUFFIX"))
	# and .exp file is created if there are exports from a DLL
	extratargets.append(
	env.ReplaceIxes(dll,
	'%sPREFIX' % paramtp, '%sSUFFIX' % paramtp,
	"WINDOWSEXPPREFIX", "WINDOWSEXPSUFFIX"))

	return (target+extratargets, source+extrasources)

	target_strings = env.ReplaceIxes(orig_target[0],
	'%sPREFIX' % vp, '%sSUFFIX' % vp,
	'IMPLIBPREFIX', 'IMPLIBSUFFIX')
	if Verbose:
	print("_lib_emitter: target_strings=%r" % target_strings)

	implib_target = env.fs.File(target_strings)

Mingw link issue #3270

Mingw link issue #3270

Conversation

dmoody256 commented Jan 14, 2019 • edited Loading

Contributor Checklist:

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019

coveralls commented Jan 14, 2019 • edited Loading

bdbaddog commented Jan 14, 2019

dmoody256 commented Jan 14, 2019

bdbaddog commented Jan 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmoody256 Jan 28, 2019 • edited Loading

Choose a reason for hiding this comment

bdbaddog commented Jan 27, 2019

dmoody256 commented Jan 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmoody256 Feb 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmoody256 Feb 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmoody256 commented Feb 3, 2019 • edited Loading

mwichmann commented Feb 3, 2019

bdbaddog commented Feb 3, 2019 via email

dmoody256 commented Feb 6, 2019 • edited Loading

bdbaddog commented Feb 15, 2019

dmoody256 commented Jan 14, 2019 •

edited

Loading

coveralls commented Jan 14, 2019 •

edited

Loading

dmoody256 Jan 28, 2019 •

edited

Loading

dmoody256 Feb 6, 2019 •

edited

Loading

dmoody256 Feb 8, 2019 •

edited

Loading

dmoody256 commented Feb 3, 2019 •

edited

Loading

dmoody256 commented Feb 6, 2019 •

edited

Loading