Resource files located in separate folder cannot be found

[ask-almas]

Hi!

I'm trying to use rfswarm in testing app but keep getting resource file doesn't exist error. I have following structure and trying to use agent and manager with -i option with custom ini config files. I have a separate folder where I keep resource files and import it in robot test files. Even with default configs and got the same issue. Is there any issue with my structure which swarm can't work with or is there a mistake in one of the config files? I assume I maybe messed up with scriptdir, scenariodir, scenariofile, agentdir locations. Here's details which may help to reproduce the issue:

Python: 3.11
OS: Windows
rfswarm: 1.3.0

Unfortunately, v 1.3.1 kept giving me infinite errors at the start of the agent but didn't test 1.3.2. If this issue was solved, please let me know.

structure

tests/
|--scenario.rfs
|--agent.ini
|--manager.ini
|--test.robot
|--agent-dir/
|--results/
|--resources/
|   |--config.yaml
|   |--folder-1/
|   |   |--a.resource
|   |--folder-2/
|   |   |--b.resource

manager.ini

[GUI]
win_width = 1280
win_height = 720
graph_list = 

[Plan]
scriptdir = C:\***\tests\
scenariodir = C:\***\tests\
scenariofile = C:\***\tests\scenario.rfs

[Run]
resultsdir = C:\***\tests\results
display_index = False
display_iteration = False
display_sequence = False
display_percentile = 90

[Server]
bindip = 
bindport = 8138

agent.ini

[Agent]
agentname = E-5CG2026KN2
agentdir = C:\***\tests\agent-dir
xmlmode = False
excludelibraries = BuiltIn,String,OperatingSystem,perftest
properties =
swarmmanager = http://localhost:8138/
robotcmd = robot

scenario.rfs

[Scenario]
uploadmode = err
scriptcount = 1
graphlist = 

[Script Defaults]
robotoptions = --variablefile resources/config.yaml

[1]
robots = 2
delay = 0
rampup = 10
run = 60
test = Basic test
script = test.robot

test.robot

*** Settings ***
Resource          resources/folder-1/a.resource
Resource          resources/folder-2/b.resource
Suite Teardown    Close All Browsers

*** Test Cases ***
Basic test

Thank you!

[damies13]

1.3.2 hasn't been released yet, though was planing to release it today, but I'll try to reproduce your issue in 1.3.0 and then add an extra test for it in 1.3.2 to check if it's fixed (It might be, there are a few windows path related fixes in 1.3.2) but if not I'll try to give you a fix

[damies13]

Hi @ask-almas,

I've recreated you file structure and files and see a few issues:

1. You will be impacted by a bug in 1.3.0 where the manager fails to find the 2 .resource files due to a path issue that was fixed in 1.3.1. to see the impact of this, if you navigate to C:\***\tests\agent-dir\scripts\ you will see test.robot but not the resources directory or it's contents. I think from memory you only need the manager to be v1.3.1 for this issue to be resolved, so that may be an option for you.
2. resources/config.yaml doesn't get copied across to the agent's filestructure, this is covered in the RFSwarm documentation under Support Files, and why this is needed is explained in Robot File handling (transfer from Manager to Agent). All you need to do here is add the line below to the *** Settings *** section of test.robot and this will tell the manager to copy this file to the agent.

Metadata    File    resources/config.yaml

I've confirmed with this setup should run as expected with manager v1.3.1 and agent v1.3.0 when you add the Metadata File line

Unfortunately, v 1.3.1 kept giving me infinite errors at the start of the agent

Also can you give more details on what the errors were that you were getting?

[ask-almas]

Thank you for your response, I appreciate it!

I think from memory you only need the manager to be v1.3.1 for this issue to be resolved

I upgraded the version, now I see resources files being copied to the agent scripts directory. And does the script copy resource files based on imports in .robot file or based on this configs in manager.ini? And what is the purpose of these location paths?

[Plan]
scriptdir = C:\***\tests\
scenariodir = C:\***\tests\

Unfortunately, v 1.3.1 kept giving me infinite errors at the start of the agent.

I made a mistake here, it should be manager:

Also can you give more details on what the errors were that you were getting?

To reproduce this issue, I just start agent and manager with -i options. Then I don't start the run but simply click some buttons such as Plan, Agent, Parameters and within 2-3 minutes the GUI becomes (Not Responding) and following errors appear in the manager console window infinitely:

OSError: [Errno 24] Too many open files: 'C:\\***.resource'
Exception in thread Thread-58410 (find_dependancies):
Traceback (most recent call last):
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1038, in _bootstrap_inner
  File "C:\***\venv\Lib\site-packages\rfswarm_manager\rfswarm.py", line 1383, in find_dependancies
    self.run()
    self.run()
    self.run()
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files: 'C:\\***.resource'
Exception in thread Thread-58418 (find_dependancies):
    self.run()
    with open(localpath, 'r', encoding="utf8") as afile:
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
    self.run()
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "C:\***\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
Exception in thread Thread-58424 (find_dependancies)

Thank you!

[damies13]

Manager copies files to the agent based on what it finds in the .robot file. well actually more correctly the manager identifies the files and responds with the list of files to get when the agent requests it, then the agent pulls the files from the manager, it's a pull rather than a push.

The purpose of recreating the file structure in the agent-dir is explained in Robot File handling (transfer from Manager to Agent). It's a about a page of explanation so it'll take a little while to read and digest, but it will make a lot of things about RFSwarm much easier to understand.

The directories in the manager's ini file are just default paths used when opening file dialogues, they aim to make life easier for you:

• scriptdir = C:\***\tests\ - is the default location the dialog will open to when selecting a robot file, when you select a robot it gets updated, if you have 5+ rows in your scenario and all your robot files are in the same directory or all your tests are in the same robot file, this will make life easier, as you don't need to navigate again to find the same robot file
• scenariodir = C:\***\tests\ - same idea here it's the location the the open scenario dialogue opens to and save scenario as dialogue opens to.

To reproduce this issue, I just start agent and manager with -i options. Then I don't start the run but simply click some buttons such as Plan, Agent, Parameters and within 2-3 minutes the GUI becomes (Not Responding) and following errors appear in the manager console window infinitely:

Thanks for that, I'll try to reproduce it, but from the errors you're getting it looks like it's coming from threading.py which is core python code so maybe I can't do anything about it. which specific python 3.11 version are you using? So I can make sure I use the same version.(I was using 3.11.9)

[damies13]

OSError: [Errno 24] Too many open files: 'C:\***.resource'

I haven't been able to reproduce it with your example file structure, was it a.resource or b.resource that triggered it? or do you have many resource files? how many?

From what I found searching this error windows has a default limit of 507 files, where as macos/linux is higher but still can get this error with enough files.

I may need to change the approach of processing the resource files if you have more than 500 resource files. This would also explain why you didn't get this error on 1.3.0, as the manager wasn't able to find the files to open them.

[ask-almas]

or do you have many resource files? how many?

The resources folder is quite large, it contains ~160 .resource files, divided into multiple folders, but also it contains other files with different extensions. In total, ~300 different files in 70 folders. But I see from the scripts folder, 30 .resource files were transferred. Can this be the issue still and there is a need to simplify resources folder?

which specific python 3.11 version are you using?

Mine is Python 3.11.5

[damies13]

It would be good to know if you still get the error after updating your python to the latest version, it could be a python bug that's already been fixed.
30 resource files shouldn't cause a too many open files error, but maybe there was an error where the files weren't being closed by python
I'll create a similar structure with 30 resource files to try and reproduce it tomorrow (it's nearly 1am here now) if the update to 3.11.9 doesn't fix it for you

[damies13]

I uninstalled python 3.11.9 and installed python 3.11.5, created 70 folder and 70 resource files, 35 each linked to a.resource and b.resource and still couldn't reproduce that Too many open files error

All I can suggest is:

• update to python 3.11.9
• then run pip install msvc-runtime (sometimes matplot gives errors on windows if this is missing)
• then run pip install -U rfswarm-manager rfswarm-agent

Hopefully everything will work for you then

[ask-almas]

Hi!

Just tried to upgrade the python version to 3.11.9 and installed msvc-runtime but got the same issue.

the manager identifies the files and responds with the list of files to get when the agent requests it, then the agent pulls the files from the manager, it's a pull rather than a push

Is there a possibility that the issue is coming from the unoptimized imports and cross references of the resource files which lead to infinite cycle? For example this simple structure:

structure

tests/
|--scenario.rfs
|--agent.ini
|--manager.ini
|--test.robot
|--agent-dir/
|--results/
|--resources/
|   |--config.yaml
|   |--folder-1/
|   |   |--a.resource
|   |   |--b.resource
|   |--folder-2/
|   |   |--c.resource
|   |   |--d.resource

test.robot

*** Settings ***
Resource          resources/folder-1/a.resource
Resource          resources/folder-2/c.resource
Suite Teardown    Close All Browsers

*** Test Cases ***
Basic test

a.resource

*** Settings ***
Resource    b.resource
Resource    ../folder-2/c.resource
Resource    ../folder-2/d.resource

b.resource

*** Settings ***
Resource    ../folder-2/c.resource
Resource    ../folder-2/d.resource

c.resource

*** Settings ***
Resource    d.resource
Resource    ../folder-1/a.resource

d.resource

*** Settings ***
Resource    ../folder-1/a.resource
Resource    ../folder-1/b.resource

Maybe this looks odd, but in some places I have similar things, unfortunately, and so this would be my last guess. Could you please check if you get the same issue with this kind of structure?

Thank you!

[damies13]

I don't think it's likely to be the issue, but I will check if i can reproduce it that way.

The reason I say unlikely is the manager resolves the full path of the file (as well as a path relative to the original robot file), then stored both those with the md5 hash of the file as an entry in a python dictionary (the hash is the key, the paths are values under the key) so if it encounters the same file again it will match the hash and not continue adding the file to the list, this is how it knows which files the agents need, it's also how it knows when a file changes (the hash changes)

I'll still check, I'd like to find what's causing this issue for you if I can.

Another option if you can, can you create a test structure that causes the issue for you but doesn't contain any sensitive data and make a zip file of the structure and attach it here?

[damies13]

Well that extra info was the key to reproducing it 👍

Now I know how to reproduce it I'll add a test case and work on a fix, thanks for letting us know about this issue

[damies13]

FYI it seems to be related to the circular nature of your resource references, i.e. a.resource references c.resource and c.resource references a.resource.

As a test to see if it was just a number of files issue, I created a directory with >3500 resource files and referenced it in a copy of test.robot and all the files eventually got copied to the agent (though the agent disconnected from the manager many times, so that's another issue) but this test did not produce the Too many open files error

I have a fix for the circular resource references part of this issue but not yet a fix for the >3500 resource files part
This build has the first part of the fix:
https://github.com/damies13/rfswarm/actions/runs/9837453234

So if you need it urgently you can download this dist file, unzip it and install manager with the python package inside (tell pip to install the whl file):
https://github.com/damies13/rfswarm/actions/runs/9837453234/artifacts/1677045649

If you can wait the fix will be part of v1.3.2 soon

[damies13]

OK I have the zip file, what should be in Library ../../core-common/libs/DatabaseLibrary/? some python files?

Is it OK to put these files into a test case or do you need me to rename them first?

I think I'm close to finishing the fix for this issue, I've figured out the disconnect issue I was getting when there was over 3500 files, I changed the code of the agent to use a download queue like the upload queue it uses when sending the logs back.

My first attempt at fixing the circular resource references stopped the Too many open files error but used lots of memory (over 6GB) so my second attempt fixed the memory usage which is what I linked in my previous post

This issue has been the hardest bug to fix, it's also a bug that has always existed in RFSwram, so I really appreciate you finding it 👍

[ask-almas]

it's also a bug that has always existed in RFSwram, so I really appreciate you finding it

Thank you! Also thank you for your amazing work!

Library ../../core-common/libs/DatabaseLibrary/? some python files?

Yes, these are python libraries that were created.

Is it OK to put these files into a test case or do you need me to rename them first?

If you have time, it would be good if rename it. I wanted to do it, but didn't have much time

[damies13]

OK renamed like this:

• KPIIA --> kiwi
• eea --> egg
• anything that looked like a password --> removedpass
• kubelibrary --> mangolibrary
• vertica --> guava

[damies13]

Everything working well now, merging into v1.3.2