scale out mechanism in HTEX not compatible with kubernetes provider

[ZhuozhaoLi]

Currently, the scale_out mechanism in HTEX is to scale out block by block, as in

parsl/parsl/executors/high_throughput/executor.py

Lines 480 to 482 in a00acdd

	for i in range(blocks):
	if self.provider:
	block = self.provider.submit(self.launch_cmd, 1, 1)

This is conflicted with the mechanism in kubernetes provider. In kube provider, the number of blocks refer to the number of pods, and it creates the blocks (pods) directly, as in

parsl/parsl/providers/kubernetes/kube.py

Lines 126 to 131 in a00acdd

	self.deployment_obj = self._create_deployment_object(job_name,
	self.image,
	deployment_name,
	cmd_string=formatted_cmd,
	replicas=self.init_blocks,
	volumes=self.persistent_volumes)

Need to fix to make them compatible.

[ZhuozhaoLi]

I am going to add a check in the executor to let kubernetes provider skip this loop below:

parsl/parsl/executors/high_throughput/executor.py

Lines 480 to 482 in a00acdd

	for i in range(blocks):
	if self.provider:
	block = self.provider.submit(self.launch_cmd, 1, 1)

Any better suggestion for this issue?

[benclifford]

@ZhuozhaoLi so that scale_out does nothing in that case?

[ZhuozhaoLi]

@ZhuozhaoLi so that scale_out does nothing in that case?

@benclifford So the scale_out only does self.provider.submit(self.launch_cmd, 1, 1) once, rather than blocks times for kubernetes provider -- should be break the loop after once.

[benclifford]

Some questions that spring to mind:

Who calls scale_out?

Are they going to be happy that they didn't get the number of blocks they ask for?

What happens if they ask to scale out more blocks by calling scale_out repeatedly because you didn't scale out the right number of blocks?

Should this code throw a ValueError if it cannot do what it was asked to do rather than silently doing something else?

Architecturally it's horrible to put in class type tests all over the place - the suggested change isn't going to work for the not yet implemented SimilarToKubernetesProvider, for example.

[benclifford]

The result of a discussion between myself and @ZhuozhaoLi on slack was that a better representation of blocks in kubernetes might be: each block is a kubernetes pod, managed directly via parsl, rather than via a kubernetes deployment which doesn't allow pod management in the way that our block management is headed.