Allow default resource limits to be specified in the artifact definition

[scudette]

Currently resource limits are hard coded but some artifact need longer time or more uploaded bytes. By having these in the artifact definition we can tailor those limits to the right artifacts.

[predictiple]

The timeout value doesn't seem to be applied

result

Also it would be nice if the "New Collection: Specify Resources" screen showed the values if specified, but I guess that might be tricky if multiple artifacts are selected?

[scudette]

Timeout is actually applied for each query (i.e. artifact + source) in a collection.

The GUI probably needs to show something else here (probably "default" rather than 600).

When specifying the timeout in the resources tab then it overrides the timeout in the artifact definition. But if you don't specify then it will be taken as the artifact default, or if there is no default then 600.

You should be able to see the correct timeout set in the request tab.

[predictiple]

Ah thanks, yes it does show the correct timeout value in the Requests tab.

[predictiple]

The resources settings don't seem to apply at all when used in a server artifact.

the artifact definition specifies:

resources:
  timeout: 666666
  max_rows: 999999999
  max_upload_bytes: 99999999999

Incidentally on the New Collection tab the timeout value is shown but not the others:

Running the collection, the Requests tab shows:

"max_row": "1000",
"timeout": "666666"

and the collection times out after 600 seconds:

It would be handy if the resources settings were applied because my use case is taking a (typically large) collection and forwarding it to Elastic on-demand. The server artifact is a simple one and very similar to Elastic.Flows.Upload

[predictiple]

forgot to mention, the above testing is with
version: 0.5.8-rc1
commit: cf9311b

[predictiple]

It seems like the timeout value is ignored for server artifacts, even if entered manually on the Specify Resources tab. Even though the Requests tab shows the correct timeout value, the collection always times out at 600 seconds.

[predictiple]

Testing with a trivial long-running server artifact confirms that the 600sec timeout is hardcoded. Neither the resources setting in the artifact, nor the Specify Resources tab overrides the timeout (which makes one wonder why that page even displays a Max Execution Time setting).

That means post-processing tasks longer than 10 minutes are impossible.

[scudette]

Yes that is correct, as can be seen here

velociraptor/services/server_artifacts/server_artifacts.go

Line 457 in ce78aec

timeout: time.Second * time.Duration(600),

The timeout is fixed.

[predictiple]

Is this likely to change?

It doesn't seem improbable that post-processing tasks could take longer than client collection tasks, especially given that the client collection timeout can be arbitrarily long.

[scudette]

Yes - this is clearly an oversight :-). Let's open another issue for it.

[scudette]

Should be fixed by #1020

[predictiple]

Thanks for attending to this so quickly. I really appreciate it. I've tested successfully with the new build.

One thing that's interesting is that the Requests tab shows a max_row of 1000 even though my artifact specifies max_rows (note the 's') of 999999999.

The collection (server artifact) didn't stop at 1000 rows though, so it does seem to be using the value defined in the artifact. Possibly there are some discrepancies with max_row vs. max_rows, or maybe max_row is another setting that has some other purpose that I'm just not aware of.

[scudette]

Server artifacts don't enforce a row limit at the moment. Do you think they should? The main reason for the client side row limits is the large uploads but this does not really apply to the server artifacts.

…

On Sat, Apr 17, 2021, 03:33 predictiple ***@***.***> wrote: Thanks for attending to this so quickly. I really appreciate it. I've tested successfully with the new build. One thing that's interesting is that the Requests tab shows a max_row of 1000 even though my artifact specifies max_row*s* (note the 's') of 999999999. [image: Screenshot from 2021-04-16 19-13-25] <https://user-images.githubusercontent.com/9843422/115061010-da7a9500-9ee8-11eb-8349-6e03bfd1fe20.png> The collection (server artifact) didn't stop at 1000 rows though, so it does seem to be using the value defined in the artifact. Possibly there are some discrepancies with max_row vs. max_row*s*, or maybe max_row is another setting that has some other purpose that I'm just not aware of. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1011 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5NRIT6DPLHJDKKCKARTW3TJBYGZANCNFSM42QRNWPQ> .

[predictiple]

I don't think there should be any limits at all, but it's working perfectly now with huge limits 🤓

With the max_row / max_rows it just seems potentially misleading if it displays a limit when there is none, and it also seems like max_row vs. max_rows is possibly an older and a newer name for the same setting that could cause trouble in future if one doesn't notice the missing 's'.

[scudette]

Should be fixed in 0.5.9-rc1