How do I log request and response details using Java 11 HttpClient?

If you are using Java 11’s HttpClient and want to log request and response details, you can achieve this by implementing a custom utility. Java 11’s HttpClient provides flexibility to log and inspect both requests and responses using its APIs. Here’s how you can log them:

Full Implementation for Logging Request and Response

Below is an example that demonstrates logging details such as HTTP headers, request body, response status, response headers, and response body.

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class HttpClientLogger {

   public static void main(String[] args) {
      try {
         // Create a sample HTTP GET request
         HttpClient httpClient = HttpClient.newHttpClient();

         HttpRequest request = HttpRequest.newBuilder()
                 .uri(URI.create("https://jsonplaceholder.typicode.com/posts/1"))
                 .GET()
                 .header("Accept", "application/json")
                 .timeout(Duration.ofSeconds(10))
                 .build();

         // Log Request Details
         logRequestDetails(request);

         // Send the request and receive a response
         HttpResponse<String> response = httpClient.send(request,
                 HttpResponse.BodyHandlers.ofString());

         // Log Response Details
         logResponseDetails(response);

      } catch (Exception e) {
         e.printStackTrace();
      }
   }

   private static void logRequestDetails(HttpRequest request) {
      System.out.println("---- HTTP Request Details ----");
      System.out.println("Method: " + request.method());
      System.out.println("URI: " + request.uri());
      System.out.println("Headers: " + request.headers().map());
      request.bodyPublisher().ifPresentOrElse(
              bodyPublisher -> {
                 System.out.println("Request Body: (BodyPublisher details not directly accessible, consider passing it explicitly)");
              },
              () -> System.out.println("Request Body: No body")
      );
      System.out.println("-------------------------------");
   }

   private static void logResponseDetails(HttpResponse<String> response) {
      System.out.println("---- HTTP Response Details ----");
      System.out.println("Status Code: " + response.statusCode());
      System.out.println("Headers: " + response.headers().map());
      System.out.println("Body: " + response.body());
      System.out.println("--------------------------------");
   }
}

Explanation of the Code

  1. Setting up HttpClient:
    • We create a HttpClient instance using HttpClient.newHttpClient().
  2. Building the HttpRequest:
    • Use the HttpRequest.Builder to construct the request. This includes setting the URI, method, headers, and a timeout.
  3. Logging HTTP Request Details:
    • Log details of the request by accessing:
      • HTTP Method: HttpRequest::method
      • URI: HttpRequest::uri
      • Headers: HttpRequest::headers
      • The request body can be logged if you manually supply it during the request creation since BodyPublisher doesn’t provide a content preview.
  4. Sending the Request:
    • Use HttpClient::send to perform the HTTP operation, and specify HttpResponse.BodyHandlers.ofString() to read the response as a string.
  5. Logging HTTP Response Details:
    • Log response information:
      • Status Code: HttpResponse::statusCode
      • Headers: HttpResponse::headers
      • Body: HttpResponse::body

Notes

  1. POST/PUT Requests with a Body:
    • If you are sending POST or PUT requests that include a body (e.g., JSON or a form), you should explicitly log the body content when building the request. Example:
      HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://jsonplaceholder.typicode.com/posts"))
            .POST(HttpRequest.BodyPublishers.ofString("{\"key\":\"value\"}"))
            .header("Content-Type", "application/json")
            .build();
      

      To log the request body, simply store it in a separate variable and print it in logRequestDetails.

  2. Production Logging:

    • Avoid directly printing details to the console in production.
    • Use proper logging libraries like SLF4J with an implementation (e.g., Logback or Log4j) to write logs at different levels like DEBUG, INFO, ERROR, etc.
  3. Sensitive Data:
    • Avoid logging sensitive details like authentication headers or personal data (e.g., Authorization tokens, passwords).

This approach provides a reusable template for logging HTTP requests and responses while using the Java 11+ HttpClient.

How do I monitor performance and latency of Java 11 HTTP requests?

Monitoring the performance and latency of Java 11 HTTP requests is essential when utilizing the java.net.http package introduced with the new Java 11 HttpClient API. It helps identify bottlenecks, optimize network calls, and ensure efficient resource usage for your application.

Here’s how you can do it:

1. Measure Latency Using Timestamps

You can manually measure the time taken to send and receive HTTP requests by recording timestamps before and after executing the HTTP call.

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.time.Instant;

public class HttpPerformanceMonitor {
    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://example.com"))
                .GET()
                .build();

        // Record start time
        Instant start = Instant.now();

        // Send request and get response
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

        // Record end time
        Instant end = Instant.now();

        // Calculate latency
        Duration latency = Duration.between(start, end);
        System.out.println("Response time: " + latency.toMillis() + " ms");
        System.out.println("Response status code: " + response.statusCode());
    }
}

This approach calculates the time taken for the entire HTTP operation, including connection establishment, sending the request, and receiving the response.

2. Use a Custom Executor to Monitor Thread Usage

HttpClient allows you to set a custom Executor for handling its asynchronous operations. You can measure how efficiently the threads are being utilized.

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;

public class HttpClientWithCustomExecutor {
    public static void main(String[] args) throws Exception {
        ExecutorService executor = Executors.newFixedThreadPool(4); // Monitor thread usage

        HttpClient client = HttpClient.newBuilder()
                .executor(executor)
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://example.com"))
                .GET()
                .build();

        long startTime = System.currentTimeMillis();

        client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
                .thenAccept(response -> {
                    long endTime = System.currentTimeMillis();
                    System.out.println("Response time: " + (endTime - startTime) + " ms");
                    System.out.println("Response status code: " + response.statusCode());
                })
                .join(); // Wait for the async call to complete

        executor.shutdown();
    }
}

By setting a custom Executor, you can track thread pool utilization and capture latency.

3. Enable Logging for HTTP Headers and Debugging

Java 11 supports configuring logging for the HTTP Client. You can enable debug-level logging to capture low-level details:

  • Add the following JVM options to enable java.net.http.HttpClient debugging:
-Djdk.httpclient.HttpClient.log=requests,headers,frames:all

This will output detailed log information, including HTTP request/response headers and frames.

4. Monitor HTTP Client Metrics Using Libraries

Libraries that support metrics collection (like Micrometer) can be integrated with Java 11 HttpClient to collect throughput, response times, and errors.
Example: Using Micrometer You can create custom timers and counters to record metrics manually.

package org.kodejava.net.http;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.TimeUnit;

public class HttpClientWithMetrics {
    public static void main(String[] args) throws Exception {
        MeterRegistry registry = new SimpleMeterRegistry(); // Replace with a distributed registry like Prometheus

        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://example.com"))
                .GET()
                .build();

        registry.timer("http.requests").record(() -> {
            try {
                Instant start = Instant.now();
                HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
                Instant end = Instant.now();

                System.out.println("Response time: " + Duration.between(start, end).toMillis() + " ms");
                System.out.println("Response status code: " + response.statusCode());
            } catch (Exception e) {
                e.printStackTrace();
            }
        });

        registry.get("http.requests").timers().forEach(timer -> {
            System.out.println("Timer count: " + timer.count());
            System.out.println("Total time: " + timer.totalTime(TimeUnit.MILLISECONDS) + " ms");
        });
    }
}

5. Use Dependency Injection Frameworks

Combine the Java 11 HttpClient with Spring Boot’s actuator (if you’re already using it). Actuator provides built-in metrics and HTTP tracing capabilities.

6. Profiling with External Tools

You can observe Java application performance (including HTTP requests) using APM tools, such as:

  • Java Flight Recorder (JFR)
    • Use JFR to monitor detailed HTTP request timings and underlying JVM performance.
  • Java VisualVM
    • Profile application threads and network usage.
  • Third-Party APMs
    • Tools like New Relic, AppDynamics, or Datadog can report on HTTP client usage.

Summary of Approaches

Approach Tools/Techniques Notes
Manual Timing Instant + Duration Tracks latency at the request level.
Custom Executors ExecutorService Measures thread usage and async execution.
Logging JVM Debugging (HttpClient.log) Debugs detailed network activity.
Metrics Libraries Micrometer (manual instrumentation) Can feed metrics into observability tools.
Framework Integration Spring Boot Actuator Collects trace and performance metrics.
External Profiling/Monitoring JFR, APM Tools, VisualVM Monitors JVM, HTTP, and app performance.

Choose the approach based on your application’s complexity and monitoring needs!


Maven Dependencies

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>1.15.0</version>
</dependency>

Maven Central

How do I retry failed requests using Java 11 HttpClient?

Retrying failed requests using Java 11’s HttpClient involves implementing a custom retry mechanism. Java 11’s HttpClient doesn’t provide native support for automatic retries, but you can build a retry mechanism into your application by wrapping the logic in a loop.

Here’s how you can do it step by step:

1. Define a Retry Mechanism

You’ll need to decide on:

  • The maximum number of retry attempts.
  • An optional delay between retries.
  • The type of failures that should trigger retries (e.g., IOException).

2. Create a Function to Retry Requests

You can encapsulate the retry logic in a utility method or by using a loop.

3. Full Code Example:

Below is an example implementation:

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.io.IOException;

public class HttpClientWithRetry {

   public static void main(String[] args) {
      HttpClient httpClient = HttpClient.newHttpClient();
      HttpRequest request = HttpRequest.newBuilder()
              .uri(URI.create("https://api.example.com/resource"))
              .timeout(Duration.ofSeconds(10))
              .GET()
              .build();

      int maxRetries = 3;  // Max number of retries
      Duration delay = Duration.ofSeconds(2); // Delay between retries

      try {
         HttpResponse<String> response = sendWithRetries(httpClient, request, maxRetries, delay);
         System.out.println("Response received: " + response.body());
      } catch (Exception e) {
         System.err.println("Request failed after retries: " + e.getMessage());
      }
   }

   public static HttpResponse<String> sendWithRetries(HttpClient client,
                                                      HttpRequest request,
                                                      int maxRetries,
                                                      Duration delay) throws Exception {
      int attempts = 0;

      while (true) {
         try {
            // Attempt to send the request
            return client.send(request, HttpResponse.BodyHandlers.ofString());
         } catch (IOException | InterruptedException ex) {
            attempts++;
            if (attempts > maxRetries) {
               throw new Exception("Maximum retry attempts reached.", ex);
            }
            System.out.println("Request failed, retrying... (" + attempts + "/" + maxRetries + ")");
            Thread.sleep(delay.toMillis()); // Delay before retry
         }
      }
   }
}

Explanation:

  1. HttpClient Initialization:
    • The HttpClient is created using HttpClient.newHttpClient().
  2. Retry Logic:
    • The sendWithRetries method sends the HTTP request and retries on exceptions (IOException or InterruptedException in this example).
    • A loop tracks the number of attempts and retries until either the request succeeds or the maximum retry limit is reached.
  3. Delay Between Retries:
    • A Thread.sleep introduces a delay (defined by a Duration object) between retries.
  4. Failure Handling:
    • If all retries fail, an exception is thrown indicating the request could not be completed.
  5. Configurations:
    • You can adjust the retry count, delay, and consider making the logic more robust by checking specific HTTP status codes for retries (e.g., 5xx server errors).

Future Improvements:

  • Exponential Backoff: Increase the delay between retries exponentially to ease server load.
  • Timeout Handling: Handle custom timeout scenarios with detailed configuration.
  • Customizable Errors: Customize retry logic based on specific HTTP response codes (e.g., retry only for 500–599 responses).

How do I chain asynchronous calls using Java 11 HttpClient and CompletableFuture?

Chaining asynchronous calls using Java 11’s HttpClient and CompletableFuture can be achieved by leveraging the reactive capabilities of CompletableFuture. The sendAsync method of HttpClient supports asynchronous processing, and you can chain multiple calls together using methods like thenApply, thenCompose, or thenAccept. Here’s a step-by-step example:


Key Concepts Used:

  1. CompletableFuture:
    • Allows for async processing and chaining of dependent tasks.
  2. HttpClient and HttpRequest:
    • The async calls are made using the HttpClient.sendAsync method.
  3. Chaining methods:
    • Use thenApply to transform the response or thenCompose to chain dependent async calls.

Example: Chaining Multiple HTTP Requests

Say we need to:

  1. Fetch data using one API.
  2. Use the response data to make another API call.
  3. Process the final response.

Here’s how you can do that:

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.concurrent.CompletableFuture;

public class AsyncChainingExample {

   public static void main(String[] args) {
      HttpClient client = HttpClient.newHttpClient();

      // First API Request
      HttpRequest firstRequest = HttpRequest.newBuilder()
              .uri(URI.create("https://jsonplaceholder.typicode.com/posts/1"))
              .GET()
              .build();

      // First Async Call
      CompletableFuture<Void> future = client.sendAsync(firstRequest, HttpResponse.BodyHandlers.ofString())
              .thenApply(HttpResponse::body) // Extract body from response
              .thenCompose(body -> {
                 System.out.println("First API Response: " + body);

                 // Use data from the first response to make the second API request
                 String secondApiUri = "https://jsonplaceholder.typicode.com/comments?postId=1";
                 HttpRequest secondRequest = HttpRequest.newBuilder()
                         .uri(URI.create(secondApiUri))
                         .GET()
                         .build();

                 return client.sendAsync(secondRequest, HttpResponse.BodyHandlers.ofString());
              })
              .thenApply(HttpResponse::body) // Extract body from second response
              .thenAccept(secondResponse -> {
                 // Final result processing
                 System.out.println("Second API Response: " + secondResponse);
              });

      // Wait for all the tasks to complete
      future.join();
   }
}

Explanation of the Code:

  1. Create the HttpClient:
    • HttpClient.newHttpClient() initializes the HTTP client that will send requests asynchronously.
  2. First API Call:
    • The first API request (firstRequest) is created using HttpRequest.newBuilder.
    • Send the request asynchronously with:
    client.sendAsync(firstRequest, HttpResponse.BodyHandlers.ofString());
    
    • thenApply is used to extract the body of the response.
  3. Second API Call (Chained):
    • In thenCompose, the code prepares and sends the second API request. This ensures that the second API call happens only after the first call completes.
    • The response of this call is again processed by extracting the body.
  4. Response Processing:
    • thenAccept is used at the end of the chain to process the final response.
  5. Waiting for Completion:
    • Since the operations are asynchronous, future.join() blocks the main thread until all the chained calls complete.

Output:

Sample output from the above example (when run):

First API Response: {
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati",
  "body": "quia et suscipit..."
}

Second API Response: [
  {
    "postId": 1,
    "id": 1,
    "name": "id labore ex et quam laborum",
    ...
  },
  ...
]

Key Functions Used in the Chain:

  1. thenApply(Function)
    • Transforms the result of the previous step (e.g., extract the body).
  2. thenCompose(Function)
    • Used for dependent async calls. Ensures one CompletableFuture waits for another.
  3. thenAccept(Consumer)
    • Consumes the result without returning anything.

Advantages of this Approach:

  • No need for manual thread management.
  • Non-blocking I/O.
  • Easily scalable chaining of async calls.

This is a modern, clean solution for handling asynchronous HTTP requests in Java using HttpClient and CompletableFuture.

How do I cancel an ongoing HTTP request using Java 11 HttpClient?

In Java 11, the HttpClient API does not provide an explicit “cancel” method for ongoing HTTP requests, but you can use a java.util.concurrent.Future to achieve cancellation. When you send an asynchronous request using HttpClient in Java 11, the sendAsync method returns a CompletableFuture. You can call cancel on the CompletableFuture to attempt to cancel the ongoing HTTP operation.

Here’s how you can cancel an ongoing HTTP request:

Example Code

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.concurrent.CompletableFuture;

public class HttpClientCancelExample {
    public static void main(String[] args) {
        HttpClient client = HttpClient.newHttpClient();

        // Create the HttpRequest
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://httpbin.org/delay/5"))  // A 5-second delay URL
                .GET()
                .build();

        // Send the request asynchronously
        CompletableFuture<HttpResponse<String>> futureResponse =
                client.sendAsync(request, HttpResponse.BodyHandlers.ofString());

        // Simulate a need to cancel the request
        try {
            // Wait 2 seconds
            Thread.sleep(2000);
            // Attempt to cancel the request
            boolean cancelResult = futureResponse.cancel(true);

            if (cancelResult) {
                System.out.println("Request was cancelled successfully.");
            } else {
                System.out.println("Request could not be cancelled.");
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

        // Handle the original CompletableFuture response (if not cancelled)
        futureResponse.whenComplete((response, throwable) -> {
            if (throwable != null) {
                System.out.println("Request was cancelled or completed with error: " + throwable.getMessage());
            } else {
                System.out.println("Response received: " + response.body());
            }
        });
    }
}

Explanation

  1. Create an Asynchronous Request: Use HttpClient.sendAsync to initiate the HTTP request asynchronously. This returns a CompletableFuture.
  2. Cancel the Request: Use CompletableFuture.cancel(true) to cancel the request. The true parameter interrupts the thread performing the computation if it is running.
  3. Handle Cancellation or Completion: Use whenComplete on the CompletableFuture to handle the scenario where the operation is either cancelled or successfully completed.
  4. Resource Cleanup: If cancellation succeeds, you should be aware that the underlying network connection may not always terminate immediately, as it depends on the implementation.

Things to Note

  1. Cancellation Result: The cancel method returns true if the cancellation was successful and false if the task could not be cancelled, such as if it was already completed.
  2. Non-blocking Behavior: This approach is non-blocking and makes use of asynchronous programming.