How to create a kubernetes container and run it from another pod

You do not need an image inside, what you need is a kubernetes client that you will use to create deployment/job/pod. Your pod needs a serviceaccount that has RBAC role/clusterrole allowing for creation of what you need. This way you can create a service that on demand interacts with kubernetes api and creates what you want.

All in all, it's a very similar concept to how operators work, so looking at https://github.com/operator-framework might provide some usefull insights, but even just launching kubectl inside the pod might be good enough for your needs.

It is possible to do exactly what you're asking, you can use a kubernetes client and spin up a pod, wait on its status, get its IP and talk to it and then kill it; however you should really rethink your design. There's a reason you don't want to handle your own resources - if pod A spins up pod B and midway through its work pod A looses connection to pod B, pod B becomes orphaned. It is possible to deal with this too but I'd rather avoid this altogether. Try to design your systems with more flexible coupling, maybe you can use a queue instead of grpc?

I found this good open source project for kubenetes API java-client https://github.com/fabric8io/kubernetes-client with examples of how to do this.

I added this example in response to the good suggestion made below.

It has examples for creating Pods, Deployments, CronJobs etc.

For creating a Deployment the example provided is: https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/DeploymentExamples.java

public class DeploymentExamples {
  private static final Logger logger = 
    LoggerFactory.getLogger(DeploymentExamples.class);

  public static void main(String[] args) throws InterruptedException {
      Config config = new ConfigBuilder().build();
      KubernetesClient client = new DefaultKubernetesClient(config);

      try {
       // Create a namespace for all our stuff
       Namespace ns = new NamespaceBuilder()
             .withNewMetadata()
             .withName("thisisatest")
             .addToLabels("this", "rocks")
             .endMetadata().build();
       log("Created namespace", client.namespaces().createOrReplace(ns));

        ServiceAccount fabric8 = new ServiceAccountBuilder()
            .withNewMetadata()
            .withName("fabric8")
            .endMetadata().build();
        client
            .serviceAccounts()
            .inNamespace("thisisatest")
            .createOrReplace(fabric8);
        for (int i = 0; i < 2; i++) {
           System.out.println("Iteration:" + (i+1));
           Deployment deployment = new DeploymentBuilder()
                .withNewMetadata()
                .withName("nginx")
                .endMetadata()
                .withNewSpec()
                .withReplicas(1)
                .withNewTemplate()
                .withNewMetadata()
                .addToLabels("app", "nginx")
                .endMetadata()
                .withNewSpec()
                .addNewContainer()
                .withName("nginx")
                .withImage("nginx")
                .addNewPort()
                .withContainerPort(80)
                .endPort()
                .endContainer()
                .endSpec()
                .endTemplate()
                .withNewSelector()
                .addToMatchLabels("app", "nginx")
                .endSelector()
                .endSpec()
                .build();

            deployment = client
                 .apps()
                 .deployments()
                 .inNamespace("thisisatest")
                 .create(deployment);
            log("Created deployment", deployment);

            System.out.println("Scaling up:" + 
                deployment.getMetadata().getName());
            client
               .apps()
               .deployments()
               .inNamespace("thisisatest")
               .withName("nginx").scale(2, true);
            log("Created replica sets:", client.apps().replicaSets()
               .inNamespace("thisisatest").list().getItems());
            System.out.println("Deleting:" + 
               deployment.getMetadata().getName());
            client.resource(deployment).delete();
       }
       log("Done.");
    }finally {
        client.namespaces().withName("thisisatest").delete();
        client.close();
    }

Madhu