cURL -o command won't output to file until I stop the java program executing the cURL command

user_11677147 0
Viewed 450

I am writing a Java program that creates a cURL command and executes it. I haven't had any issues except one command won't allow me to stream the stdout into my program. The stdout has information necessary to execute future cURL commands.

I've been using the -o command to output the stdout to a .txt file, but this doesn't work unless I stop the Java program immediately after executing the cURL command. I've also tried several different streaming techniques but none seem to work.

Unfortunately, the API I'm communicating with requires account information in order to execute many of the commands, but I will share the code that doesn't display my username and password.

This is the command that executes and returns a value that I need for future commands:

curl -o C:\Users\friza\eclipse-workspace\CanaryCoalMine\birdseed.txt -k -J -X POST -v --include -H "X-gts-token:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" https://api.gts.geant.net/taas/api/v2/projects/CanaryCoalMine/types/OneHost/reservations

This is the output to the birdseed.txt file. The five digit value at the very end is what I need:

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Thu, 20 Jun 2019 16:25:07 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha http://www.springframework.org http://java.sun.com/jsp https://www.google.com/jsapi https://www.google.com https://www.gstatic.com code.jquery.com ajax.googleapis.com https://*.bootstrapcdn.com https://*.cloudflare.com; img-src 'self' http://195.113.161.164:* https://www.gstatic.com/recaptcha code.jquery.com; style-src 'self' 'unsafe-inline' https://*.bootstrapcdn.com https://*.cloudflare.com https://fonts.googleapis.com code.jquery.com; font-src 'self' https://themes.googleusercontent.com https://*.bootstrapcdn.com https://*.cloudflare.com; frame-src 'self' 'unsafe-inline' https://www.google.com; object-src 'none'
Expect-CT: enforce; max-age=3600
Strict-Transport-Security: max-age=31622400; includeSubDomains; preload
Referrer-Policy: no-referrer
Feature-Policy: geolocation 'none';midi 'none';notifications 'none';push 'none';sync-xhr 'none';microphone 'none';camera 'none';magnetometer 'none';gyroscope 'none';speaker 'self';vibrate 'none';fullscreen 'self';payment 'none';

10940

This is the Java method I'm using to execute the above code:

public static void createType(String tokenHeader, String dslEncoded) {
        String curlCommand = "curl -o C:\\Users\\friza\\eclipse-workspace\\CanaryCoalMine\\birdseed.txt -k -J -X POST -v --include -H "+tokenHeader
                +" -H \"Content-Type: application/x-www-form-urlencoded\" -d \"script="
                +dslEncoded+
                "\" https://api.gts.geant.net/taas/api/v2/projects/CanaryCoalMine/types";



        try {
            //execute Curl Command to create new type
            Process process = Runtime.getRuntime().exec(curlCommand);

            System.out.println("Creating Type...");

            System.out.println("Sent: " + curlCommand);



        } catch (IOException e) {
            // TODO Auto-generated catch block
            System.out.println("IO EXCEPTION :( ");
            e.printStackTrace();

        }
    }

TL;DR Expected: Once the cURL command executes, I want to see the given output in the file WITHOUT stopping the Java program that executes the command.

Actual Results: Cannot see the above output in the .txt file until I hit STOP on the Java program. Cannot proceed with the program until I'm able to extract the last five digit number from the file. Cannot stream directly for some reason unknown.

curl java
1 Answers

To add to what user207421 said...

It is most likely a buffer reading issue.

Dealing with external processes is quite difficult and if the buffer of the external program fills up, it will stop responding/working until that buffer is cleared.

Therefore, when reading the buffers from an external process, those reading processes, must occur in their own threads.

Here is a simple example from some work my team did a while ago for dealing with external processes.

The main method...

package processbuilder;

import processbuilder.streamreader.PrintStreamStreamReader;

public class CurlProcessBuilder {

    public static void main(String... args) throws Exception {

        ProcessBuilder processBuilder = new ProcessBuilder("curl", "-v", "-s", "-k", "https://localhost:8443");

        Process process = processBuilder.start();

        PrintStreamStreamReader outReader = new PrintStreamStreamReader(System.out, process.getInputStream());
        Thread outThread = new Thread(outReader);
        outThread.start();

        PrintStreamStreamReader errReader = new PrintStreamStreamReader(System.err, process.getErrorStream());
        Thread errThread = new Thread(errReader);
        errThread.start();

        int exitValue = process.waitFor();
        System.out.println("Exit value: " + exitValue);
    }
}

The threads used to read from the process buffer...

package processbuilder.streamreader;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintStream;

public class PrintStreamStreamReader implements Runnable {

    private PrintStream printStream;
    private InputStream inputStream;

    public PrintStreamStreamReader(PrintStream printStream, InputStream inputStream) {

        this.printStream = printStream;
        this.inputStream = inputStream;
    }

    @Override
    public void run() {

        try {

            int c;
            byte[] data = new byte[1024];
            while ((c = inputStream.read(data, 0, data.length)) != -1) {

                printStream.write(data, 0, c);
            }

        } catch (IOException e) {

            e.printStackTrace();
        }
    }
}

Dealing with long running external process is a pain. Here are some other resources explaining the issues. I had to read all these link numerous times myself to figure out what was actually being said and why it was so difficult to get everything 'just right'.

user_429620
Related