Catching AttributeError which may occur when executing multi-threaded #4424
Conversation
bwuerkner
changed the title
|
That's kind of scary - it means there's a way for the taskmaster to try retrieving the build environment of a task that doesn't really have one. All the rest is a consequence, we shouldn't really be going down the route of constructing a default environment with the initialization that goes with it. I'm suspicious this means something has a |
|
Hmmm... the comment in def retrieve_from_cache(self) -> bool:
"""Try to retrieve the node's content from a cache
This method is called from multiple threads in a parallel build,
so only do thread safe stuff here.
... """Following the thread of execution down to the instantiation of |
|
Can you describe or share your custom Builder/Action? |
|
Python 3.11.4 I'll try to summarize them and give some code snippets. Now for the setup: custom_tool.py def exists(env):
return True
def generate(env):
some_action = Action(internal_function_to_execute_some_command, strfunction=internal_function_for_name)
bld = Builder(action=some_action)
env.Append(BUILDERS={"custom_tool": bld})The Aside from that I have a SConstruct file that contains the loading logic, some error handling and parameter setting and the following: SConstruct env = Environment(
variables = cmdline_params,
tools=['custom_tool'],
toolpath=['path/to/custom/tool'],
ENV=os.environ,
overall_start_time = time.time(),
db = None,
event=None,
execution_log={}
)and for entry in data:
target = env.custom_tool(
source=entry.source,
target=entry.dest
)
targets.append(target) |
I did also find it weird that a function related to the cache is called when there's no cachedir defined. |
|
just for grins, if you're not actually using the Default Environment (that is, your calls are all of the form env.Mybuilder, env.Append, etc.), could you add: DefaultEnvironment(tools=[])This isn't a fix, it's just an experiment which might help because of two factors:
There still something quite interesting going on that it's not yet clear how to drill down on. |
|
Can you expand on what |
That does fix the issue so I guess Experiment succeeded.
No I'm not calling any builders or other SCons related content in that function. It is rather simple and as outlined above the content doesn't actually seem to matter. def internal_function_to_execute_some_command(target, source, env):
return 0and with that I can trigger the issue without a problem. I'm somewhat assuming it has to do with the parallel initialization of the Builders after their initial creation. I didn't clearly state it in my original message but before the error occurs I get I'm also starting to think that this PR is not gonna be necessary since the manual initialization prevents the problem in a way more stable way. |
It's very odd we've never seen this failure mode before, if nothing special about your usage is triggering it. |
I agree, that's odd. I think the code path looks vulnerable, but if it actually is (rather than just in theory), why hasn't it been seen over the last 20 years? It may be time to take a deeper look at revamping the way the Default Environment is initialized. There's a form of deferred initialization now, but as we've talked about in other threads (and as mentioned in #1069 - apparently some work was once started on this) what we probably really want is deferring the tool setup in the default environment. The above suggestion defers tool setup by not doing it at all on that environment, but it seems rather a hack - why do you need to do something special if you know you're not going to call on the environment yourself (and then what if you change your mind?). That's probably food for a separate thread - maybe in a GitHub Discussion, but thought I'd mention it here. |
I don't have a testcase for this because I don't really understand why or how this is happening but I can 100% reproduce it locally with a rather complicated setup.
The setup consists amongst other things of custom Builder and Action implementations.
The error I observe only occurs when running with -j2 or higher, it never occurs single threaded.
It also goes away if I just call SCons again after it failed and then proceeds as expected.
Running a --clean provokes the error again on next invocation.
The error is:
The AttributeError module is not consistent either, I've seen at least:
AttributeError: module 'SCons.Tool.default' has no attribute 'generate'
AttributeError: module 'SCons.Tool.gcc' has no attribute 'generate'
Also I'm not using Tool.gcc at all in this Project
Contributor Checklist:
CHANGES.txt(and read theREADME.rst)