Expose Experiment metadata to Trial Template

[mickvangelderen]

What you would like to be added?

We can access metadata from the trialSpec in the template as documented here.

I would like to obtain the experiment name (which is generated by K8S for me) and so I would like to have something similar to trialSpec.Name but for experimentSpec.Name.

With my limited understanding of go, it seems like the relevant code is here.

For now I can work around this by making an assumption on how the trial name is generated from the experiment name, but it would be nice to not have to make that assumption.

As a related question, can we make katib provide certain env vars by default (KATIB_EXPERIMENT_NAME, KATIB_TRIAL_NAME, and KATIB_NAMESPACE for example) to the trial programs? This would eliminate the need for the additional experiment spec configuration.

Why is this needed?

I let K8S generate the experiment name. In order to obtain the experiment name, I currently have to query the experiment name from the trial name at run time in my trial program or make assumptions on how the trial name is generated from the experiment name which I would rather not.

My use case is having consistent naming between katib experiments and trials and MLflow experiments and runs. I want to set the MLFlow experiment and run names based on the katib experiment and trial names.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[andreyvelich]

Thank you for creating this @mickvangelderen! Exposing the default env variable to Trial makes sense to me. We already expose KATIB_TRIAL_NAME for push-based Metrics Collection
Actually, we have tracking issue to decide what we should do with metadata parameters: #2390.

Do we have any other option to get Katib Experiment name into Trial's pods ?

[mickvangelderen]

The only other option I can think of is to pass the data as arguments to the program. Injecting them into the container's environment seems like a better option to me.

[Electronic-Waste]

/remove-label lifecycle/needs-triage
/area operator

[Electronic-Waste]

We already expose KATIB_TRIAL_NAME for push-based Metrics Collection

Actually, we expose KATIB_TRIAL_NAME for both pull-based and push-based Metrics Collection:

katib/pkg/webhook/v1beta1/pod/utils.go

Lines 286 to 311 in eb8af4d

	func mutatePodEnv(pod *v1.Pod, trial *trialsv1beta1.Trial) error {
	// Search for the primary container
	index := getPrimaryContainerIndex(pod.Spec.Containers, trial.Spec.PrimaryContainerName)
	if index >= 0 {
	if pod.Spec.Containers[index].Env == nil {
	pod.Spec.Containers[index].Env = []v1.EnvVar{}
	}
	// Pass env variable KATIB_TRIAL_NAME to the primary container using fieldPath
	pod.Spec.Containers[index].Env = append(
	pod.Spec.Containers[index].Env,
	v1.EnvVar{
	Name: consts.EnvTrialName,
	ValueFrom: &v1.EnvVarSource{
	FieldRef: &v1.ObjectFieldSelector{
	FieldPath: fmt.Sprintf("metadata.labels['%s']", consts.LabelTrialName),
	},
	},
	},
	)
	return nil
	} else {
	return fmt.Errorf("Unable to find primary container %v in mutated pod containers %v",
	trial.Spec.PrimaryContainerName, pod.Spec.Containers)
	}
	}

Do we have any other option to get Katib Experiment name into Trial's pods ?

Unfortunately , we only support passing Trial related parameters to env currenlty, like this example: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/trial-template/trial-metadata-substitution.yaml

And I agree with @mickvangelderen. Injecting those parameters directly into container's environment is a better choice, since it is more convenient for developers. And also, we may need to update our doc/website to inform developers that we've already passed some env variables lie KATIB_TRIAL_NAME to containers:)

WDYT👀 @kubeflow/wg-automl-leads @tariq-hasan @helenxie-bit @mickvangelderen

[Electronic-Waste]

I would like to help you pass the experiment name to the env of Pod :)

/assign

[Electronic-Waste]

Hi, I've raise a PR to pass the experiment name to Pod Env. PTAL if you are available👀.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Electronic-Waste]

/remove-lifecycle stale