How do I use JSch with strict host key checking and known_hosts validation?

When using JSch (Java Secure Channel) to connect to an SSH server, you can enable strict host key checking and validate the server against a known_hosts file. By default, strict host key checking ensures that your application will only connect to SSH servers that are already listed in the known_hosts file. If the server’s key is not present or doesn’t match, the connection will fail.

Here’s how you can implement strict host key checking and configure the use of a known_hosts file with JSch:

Step 1: Enable Strict Host Key Checking and Set Known Hosts

Below is an example of how to configure JSch with strict host key checking:

package org.kodejava.jsch;

import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;

import java.util.Properties;

public class JSchStrictHostKeyCheckingExample {
   public static void main(String[] args) {
      String username = "username";
      String host = "example.com";
      int port = 22; // default SSH port
      String privateKeyPath = "/path/to/your/private/key";
      String knownHostsPath = "/path/to/your/known_hosts";

      try {
         // Initialize JSch
         JSch jsch = new JSch();

         // Set private key if authentication requires it
         jsch.addIdentity(privateKeyPath);

         // Set the known_hosts file for host key verification
         jsch.setKnownHosts(knownHostsPath);

         // Create SSH session
         Session session = jsch.getSession(username, host, port);

         // Set session properties for strict host key checking
         Properties config = new Properties();
         config.put("StrictHostKeyChecking", "yes"); // Enables strict host key checking
         session.setConfig(config);

         // Connect to the SSH server
         session.connect();

         System.out.println("Connected securely with strict host key checking.");

         // Perform your operations (e.g., execute commands, transfer files, etc.)

         // Disconnect from the SSH server
         session.disconnect();
         System.out.println("Disconnected from server.");

      } catch (Exception e) {
         e.printStackTrace();
      }
   }
}

Explanation:

  1. StrictHostKeyChecking:
    • Setting the StrictHostKeyChecking property to "yes" will enforce strict validation of the host’s key against the known_hosts file.
    • If the host is not in the known_hosts file or if the key does not match, the connection will fail.
  2. Known Hosts File:
    • Use jsch.setKnownHosts(knownHostsPath) to specify the path to the known_hosts file. This file stores the public host keys of remote servers that you trust.
  3. Private Key:
    • If the SSH server requires private key authentication, use jsch.addIdentity(privateKeyPath) to add your private key.
  4. Session Configuration:
    • Other common configuration options (set in Properties) may include:
      • PreferredAuthentications: Specify the preferred authentication methods (e.g., publickey,password,keyboard-interactive).
      • UserKnownHostsFile: Alternative way to point to the known_hosts file.
  5. Error Handling:
    • If the server’s host key is not present in the known_hosts file or does not match, you will encounter an error similar to:
    com.jcraft.jsch.JSchException: reject HostKey: your-host
    

    This means the server’s public key either needs to be added to the known_hosts file or matches an incorrect entry.

Step 2: Generating/Updating the known_hosts File

To manually add a host key to the known_hosts file:

Run the following command on any system with SSH installed:

ssh-keyscan -H your-host >> /path/to/known_hosts
  • -H: Hashes the hostname before storing it in the known_hosts file.
  • Replace your-host with the actual hostname or IP address of the server.

Common Issues and Debugging

  1. Host Key Verification Failed:
    • Ensure the server’s public key exists in the known_hosts file.
    • Ensure the correct knownHostsPath is specified in your code.
  2. Permission Denied:
    • Check your username, private key path, and associated permissions.
    • Make sure your private key is readable and properly associated with the user on the server.
  3. Logging Debug Information:
    JSch provides detailed logs for debugging. You can enable verbose logging as below:

    JSch.setLogger(new com.jcraft.jsch.Logger() {
          public boolean isEnabled(int level) { return true; }
          public void log(int level, String message) { System.out.println(message); }
      });
    

By following this approach, you can securely connect to an SSH server while leveraging strict host key checking and known_hosts validation.


Maven Dependencies

<dependency>
    <groupId>com.jcraft</groupId>
    <artifactId>jsch</artifactId>
    <version>0.1.55</version>
</dependency>

Maven Central

How do I transfer files with resume support over SFTP using JSch?

When transferring files over SFTP using JSch, resuming partially transferred files (either uploads or downloads) can be implemented by handling offsets for files that are already partially transferred.

This guide explains how to:

  • Resume downloads by continuing from the last transferred byte of a local file.
  • Resume uploads by appending to a remote file.

Handling Download with Resume Support

To resume a download:

  1. Check the current size of the local file.
  2. Skip already downloaded bytes from the remote file using InputStream.skip().
  3. Append remaining content to the local file.

Code for Resuming Download

package org.kodejava.jsch;

import com.jcraft.jsch.*;
import java.io.*;

public class SFTPResumeDownload {

   public static void main(String[] args) {
      String host = "sftp.example.com";
      String username = "user";
      String password = "password";
      String localFile = "local/path/to/file.txt";
      String remoteFile = "/remote/path/to/file.txt";

      JSch jsch = new JSch();
      Session session = null;
      ChannelSftp sftpChannel = null;

      try {
         // Setup SFTP connection
         session = jsch.getSession(username, host, 22);
         session.setPassword(password);
         session.setConfig("StrictHostKeyChecking", "no"); // Disable key checking
         session.connect();

         Channel channel = session.openChannel("sftp");
         channel.connect();
         sftpChannel = (ChannelSftp) channel;

         // Resume download logic
         File file = new File(localFile);
         long localFileSize = file.exists() ? file.length() : 0;
         long remoteFileSize = sftpChannel.lstat(remoteFile).getSize();

         if (localFileSize >= remoteFileSize) {
            System.out.println("File already fully downloaded.");
            return;
         }

         try (InputStream inputStream = sftpChannel.get(remoteFile);
              OutputStream outputStream = new FileOutputStream(file, true)) {
            inputStream.skip(localFileSize); // Skip downloaded portion

            byte[] buffer = new byte[1024];
            int bytesRead;
            while ((bytesRead = inputStream.read(buffer)) != -1) {
               outputStream.write(buffer, 0, bytesRead);
            }

            System.out.println("Download resumed and completed.");
         }
      } catch (Exception e) {
         e.printStackTrace();
      } finally {
         if (sftpChannel != null) sftpChannel.disconnect();
         if (session != null) session.disconnect();
      }
   }
}

Handling Upload with Resume Support

To resume an upload:

  1. Check the size of the remote file.
  2. Skip already uploaded bytes from the local file.
  3. Use the ChannelSftp.APPEND flag to append remaining bytes to the remote file.

Code for Resuming Upload

public static void resumeUpload(ChannelSftp sftpChannel, String localFile, String remoteFile) throws SftpException, IOException {
    File file = new File(localFile);
    long remoteFileSize = 0;

    try {
        remoteFileSize = sftpChannel.lstat(remoteFile).getSize(); // Check remote file size
    } catch (SftpException e) {
        System.out.println("Remote file does not exist. Starting upload from the beginning.");
    }

    System.out.println("Resuming upload from byte: " + remoteFileSize);

    try (InputStream inputStream = new FileInputStream(file)) {
        inputStream.skip(remoteFileSize); // Skip already uploaded bytes

        // Append mode upload
        sftpChannel.put(inputStream, remoteFile, ChannelSftp.APPEND);
        System.out.println("Resume upload completed.");
    }
}

Explanation of Key Steps

  1. Session Setup:
    • A secure session is established with the SFTP server using user credentials.
    • StrictHostKeyChecking is disabled for simplicity, but proper key validation is recommended for production.
  2. Resume Logic:
    • Download: The remote file is read as an InputStream, skipping already downloaded bytes. The local file is opened in append mode.
    • Upload: The local file is read as an InputStream, skipping already uploaded bytes, and the put method with ChannelSftp.APPEND is used to continue the upload.
  3. Error Handling:
    • If the remote file or local file does not exist, appropriate error handling ensures either the upload/download starts from the beginning or exits gracefully.
  4. File Integrity: To ensure file integrity, consider validating the file with hash checks or checksums after transfer.

Notes

  • Increase the buffer size (byte[] buffer = new byte[1024]) for better performance for larger files.
  • Consider implementing retries or reconnect logic if the SFTP session disconnects during a transfer.
  • Always confirm proper permissions for writing to the destination and reading from the source.

Conclusion

The above solution demonstrates how to implement resumable file transfer via SFTP using JSch. It ensures efficient and reliable file transfers by avoiding redundant retransmission of already transferred data.


Maven Dependencies

<dependency>
    <groupId>com.jcraft</groupId>
    <artifactId>jsch</artifactId>
    <version>0.1.55</version>
</dependency>

Maven Central

How do I secure API requests with OAuth2 using Java 11 HttpClient?

Securing API requests with OAuth2 using Java 11’s HttpClient involves obtaining an access token from the OAuth2 provider and including it in the header of your HTTP requests. Here’s a step-by-step guide:

1. Understand OAuth2 Flow

OAuth2 involves several flows (e.g., Authorization Code, Client Credentials, etc.). For simplicity, we’ll focus on the Client Credentials Grant flow where your application (client) authenticates with the OAuth2 provider and retrieves an access token for API calls.

2. Dependencies

You don’t need external dependencies unless you choose to use a library (like Spring Security). Java 11’s HttpClient can directly process token requests.

3. Steps to Secure API Requests

Obtain Access Token

To obtain an access token, make a POST request to the OAuth2 token endpoint with the required parameters:

  • client_id: Your application’s client ID.
  • client_secret: Your application’s secret key.
  • grant_type: For example, client_credentials.

Use the Access Token in API Requests

Once you have the access token, include it in the Authorization header of your requests.

4. Sample Code

Here’s an example of how to secure API requests with OAuth2 using Java 11’s HttpClient:

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;

public class OAuth2HttpClient {

   public static void main(String[] args) throws Exception {
      // OAuth2 Token Endpoint and Client Details
      String tokenEndpoint = "https://your-oauth2-provider.com/token";
      String clientId = "your-client-id";
      String clientSecret = "your-client-secret";
      String scope = "your-api-scope";

      // Step 1: Obtain Access Token
      String accessToken = getAccessToken(tokenEndpoint, clientId, clientSecret, scope);

      // Step 2: Use Access Token for API Request
      String apiEndpoint = "https://api.example.com/secure-resource";
      sendSecureApiRequest(apiEndpoint, accessToken);
   }

   private static String getAccessToken(String tokenEndpoint, String clientId, String clientSecret, String scope) throws Exception {
      // Create request body
      String requestBody = "grant_type=client_credentials" +
                           "&client_id=" + URLEncoder.encode(clientId, StandardCharsets.UTF_8) +
                           "&client_secret=" + URLEncoder.encode(clientSecret, StandardCharsets.UTF_8) +
                           "&scope=" + URLEncoder.encode(scope, StandardCharsets.UTF_8);

      // Create HttpClient and HttpRequest
      HttpClient client = HttpClient.newHttpClient();
      HttpRequest request = HttpRequest.newBuilder()
              .uri(URI.create(tokenEndpoint))
              .header("Content-Type", "application/x-www-form-urlencoded")
              .POST(HttpRequest.BodyPublishers.ofString(requestBody))
              .build();

      // Send request and parse response
      HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
      if (response.statusCode() == 200) {
         // Extract access token from JSON response
         String responseBody = response.body();
         return extractAccessToken(responseBody);
      } else {
         throw new RuntimeException("Failed to get access token. Status: " + response.statusCode() + ", Body: " + response.body());
      }
   }

   private static String extractAccessToken(String responseBody) {
      // Parse JSON response to extract the "access_token" (you can use a library like Jackson or Gson)
      // For simplicity, assume the response contains: {"access_token":"your-token"}
      int startIndex = responseBody.indexOf("\"access_token\":\"") + 16;
      int endIndex = responseBody.indexOf("\"", startIndex);
      return responseBody.substring(startIndex, endIndex);
   }

   private static void sendSecureApiRequest(String apiEndpoint, String accessToken) throws Exception {
      // Create HttpClient and HttpRequest
      HttpClient client = HttpClient.newHttpClient();
      HttpRequest request = HttpRequest.newBuilder()
              .uri(URI.create(apiEndpoint))
              .header("Authorization", "Bearer " + accessToken)
              .GET()
              .build();

      // Send request and print response
      HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
      System.out.println("API Response Status: " + response.statusCode());
      System.out.println("API Response Body: " + response.body());
   }
}

5. Explanation

  1. Obtaining Access Token:
    • A POST request is sent to the token endpoint with appropriate parameters in the body.
    • The response typically returns an access token in JSON format.
  2. Secure API Request:
    • Include the token in the Authorization header as Bearer <token> in subsequent requests to the secured API.
  3. Error Handling:
    • If the token request or API request fails, handle the error gracefully (e.g., retry or log).

6. Security Tip

  • Never hardcode client_id or client_secret in your code. Store them securely (e.g., environment variables or a secrets manager).
  • If you handle sensitive data, ensure your OAuth2 provider supports HTTPS.

How do I log request and response details using Java 11 HttpClient?

If you are using Java 11’s HttpClient and want to log request and response details, you can achieve this by implementing a custom utility. Java 11’s HttpClient provides flexibility to log and inspect both requests and responses using its APIs. Here’s how you can log them:

Full Implementation for Logging Request and Response

Below is an example that demonstrates logging details such as HTTP headers, request body, response status, response headers, and response body.

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class HttpClientLogger {

   public static void main(String[] args) {
      try {
         // Create a sample HTTP GET request
         HttpClient httpClient = HttpClient.newHttpClient();

         HttpRequest request = HttpRequest.newBuilder()
                 .uri(URI.create("https://jsonplaceholder.typicode.com/posts/1"))
                 .GET()
                 .header("Accept", "application/json")
                 .timeout(Duration.ofSeconds(10))
                 .build();

         // Log Request Details
         logRequestDetails(request);

         // Send the request and receive a response
         HttpResponse<String> response = httpClient.send(request,
                 HttpResponse.BodyHandlers.ofString());

         // Log Response Details
         logResponseDetails(response);

      } catch (Exception e) {
         e.printStackTrace();
      }
   }

   private static void logRequestDetails(HttpRequest request) {
      System.out.println("---- HTTP Request Details ----");
      System.out.println("Method: " + request.method());
      System.out.println("URI: " + request.uri());
      System.out.println("Headers: " + request.headers().map());
      request.bodyPublisher().ifPresentOrElse(
              bodyPublisher -> {
                 System.out.println("Request Body: (BodyPublisher details not directly accessible, consider passing it explicitly)");
              },
              () -> System.out.println("Request Body: No body")
      );
      System.out.println("-------------------------------");
   }

   private static void logResponseDetails(HttpResponse<String> response) {
      System.out.println("---- HTTP Response Details ----");
      System.out.println("Status Code: " + response.statusCode());
      System.out.println("Headers: " + response.headers().map());
      System.out.println("Body: " + response.body());
      System.out.println("--------------------------------");
   }
}

Explanation of the Code

  1. Setting up HttpClient:
    • We create a HttpClient instance using HttpClient.newHttpClient().
  2. Building the HttpRequest:
    • Use the HttpRequest.Builder to construct the request. This includes setting the URI, method, headers, and a timeout.
  3. Logging HTTP Request Details:
    • Log details of the request by accessing:
      • HTTP Method: HttpRequest::method
      • URI: HttpRequest::uri
      • Headers: HttpRequest::headers
      • The request body can be logged if you manually supply it during the request creation since BodyPublisher doesn’t provide a content preview.
  4. Sending the Request:
    • Use HttpClient::send to perform the HTTP operation, and specify HttpResponse.BodyHandlers.ofString() to read the response as a string.
  5. Logging HTTP Response Details:
    • Log response information:
      • Status Code: HttpResponse::statusCode
      • Headers: HttpResponse::headers
      • Body: HttpResponse::body

Notes

  1. POST/PUT Requests with a Body:
    • If you are sending POST or PUT requests that include a body (e.g., JSON or a form), you should explicitly log the body content when building the request. Example:
      HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://jsonplaceholder.typicode.com/posts"))
            .POST(HttpRequest.BodyPublishers.ofString("{\"key\":\"value\"}"))
            .header("Content-Type", "application/json")
            .build();
      

      To log the request body, simply store it in a separate variable and print it in logRequestDetails.

  2. Production Logging:

    • Avoid directly printing details to the console in production.
    • Use proper logging libraries like SLF4J with an implementation (e.g., Logback or Log4j) to write logs at different levels like DEBUG, INFO, ERROR, etc.
  3. Sensitive Data:
    • Avoid logging sensitive details like authentication headers or personal data (e.g., Authorization tokens, passwords).

This approach provides a reusable template for logging HTTP requests and responses while using the Java 11+ HttpClient.

How do I Parse JSON Responses from Java 11 HttpClient Easily?

To parse JSON responses easily using Java 11’s HttpClient, you can use libraries like Jackson or Gson that help in converting a JSON string to Java objects (POJOs) without hassle.

Here’s a step-by-step guide:

Step 1: Create a Java 11 HttpClient request

  1. Use Java’s HttpClient to make an HTTP request and get the JSON response as a string.
  2. Use HttpRequest or HttpResponse as part of the Java 11 HttpClient API.

Step 2: Add Jackson or Gson to Parse JSON

  1. Jackson: Add the dependency to your pom.xml (for Maven):
    <dependency>
       <groupId>com.fasterxml.jackson.core</groupId>
       <artifactId>jackson-databind</artifactId>
       <version>2.15.2</version> 
    </dependency>
    
  2. Gson: Add this dependency:
    <dependency>
       <groupId>com.google.code.gson</groupId>
       <artifactId>gson</artifactId>
       <version>2.10.1</version> 
    </dependency>
    

Step 3: Example Using Jackson

package org.kodejava.net.http;

import com.fasterxml.jackson.databind.ObjectMapper;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;

public class HttpClientJacksonExample {

    public static void main(String[] args) throws Exception {
        // 1. Create an HttpClient
        HttpClient client = HttpClient.newHttpClient();

        // 2. Create an HttpRequest
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://jsonplaceholder.typicode.com/posts/1")) // Example URL
                .GET()
                .build();

        // 3. Send the HttpRequest and get an HttpResponse
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

        // 4. Parse JSON String Response to a Java Object using Jackson
        ObjectMapper mapper = new ObjectMapper();
        Post post = mapper.readValue(response.body(), Post.class);

        // 5. Use the parsed object
        System.out.println("Post Title: " + post.getTitle());
    }

    // Sample POJO to match the JSON structure
    public static class Post {
        private int userId;
        private int id;
        private String title;
        private String body;

        // Getters and Setters
        public int getUserId() {
            return userId;
        }

        public void setUserId(int userId) {
            this.userId = userId;
        }

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getTitle() {
            return title;
        }

        public void setTitle(String title) {
            this.title = title;
        }

        public String getBody() {
            return body;
        }

        public void setBody(String body) {
            this.body = body;
        }
    }
}

Step 4: Example Using Gson

package org.kodejava.net.http;

import com.google.gson.Gson;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;

public class HttpClientGsonExample {

    public static void main(String[] args) throws Exception {
        // 1. Create HttpClient
        HttpClient client = HttpClient.newHttpClient();

        // 2. Create HttpRequest
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://jsonplaceholder.typicode.com/posts/1")) // Example URL
                .GET()
                .build();

        // 3. Send HttpRequest and get HttpResponse
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

        // 4. Parse JSON String response to Java Object using Gson
        Gson gson = new Gson();
        Post post = gson.fromJson(response.body(), Post.class);

        // 5. Use the parsed object
        System.out.println("Post Title: " + post.getTitle());
    }

    // Sample POJO class to match the JSON structure
    public static class Post {
        private int userId;
        private int id;
        private String title;
        private String body;

        // Getters and Setters
        public int getUserId() {
            return userId;
        }

        public void setUserId(int userId) {
            this.userId = userId;
        }

        public int getId() {
            return id;
        }

        public void setId(int id) {
            this.id = id;
        }

        public String getTitle() {
            return title;
        }

        public void setTitle(String title) {
            this.title = title;
        }

        public String getBody() {
            return body;
        }

        public void setBody(String body) {
            this.body = body;
        }
    }
}

Key Points:

  • Serialization and Deserialization: POJO structure must match your JSON’s keys.
  • Why Jackson or Gson?
    They are robust and simplify working with JSON (and even converting nested structures).

How do I Implement Connection Pooling in Java 11 HttpClient?

Java 11 introduced the HttpClient API as part of java.net.http. By default, the HttpClient implementation provides connection pooling, so you don’t usually need to manually enable or build it. However, you do need to configure it effectively to manage connection pooling and ensure it aligns with your performance and scalability requirements.

Here’s how you can implement and configure connection pooling in Java 11’s HttpClient:


1. Default Connection Pooling in Java 11 HttpClient

The HttpClient is built with connection pooling enabled by default. When you create an instance of HttpClient, it manages multiple connections automatically across requests to the same host. However, connection pooling behavior is controlled via the HttpClient.Builder.


2. Configure Connection Pooling Options

You can configure the pooling behavior through settings like timeouts and thread handling. Use the builder pattern to fine-tune:

Example:

package org.kodejava.net.http;

import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.time.Duration;

public class HttpClientConnectionPoolingExample {

    public static void main(String[] args) throws Exception {
        // Create a custom HttpClient with connection pooling
        HttpClient httpClient = HttpClient.newBuilder()
                // Set connection timeout
                .connectTimeout(Duration.ofSeconds(10))
                // Use a custom Executor (optional - controls the threads for async requests)
                .executor(java.util.concurrent.Executors.newFixedThreadPool(10))
                // Enable HTTP/2 (default, but can be set explicitly)
                .version(HttpClient.Version.HTTP_2)
                .build();

        // Create an HTTP request
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://jsonplaceholder.typicode.com/posts"))
                .GET() // Optional, as GET is the default
                .build();

        // Perform the request
        HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());

        // Print the response
        System.out.println("Response Code: " + response.statusCode());
        System.out.println("Response Body: " + response.body());
    }
}

3. Key Concepts for Connection Pooling

  • Reusing HttpClient Instances
    Always reuse a single HttpClient instance across your application, especially for frequent HTTP calls. Each HttpClient has its own connection pool, so creating new instances unnecessarily can result in poor resource management and lack of reuse.
  • Setting a Custom Executor
    By default, the HttpClient uses a default Executor for asynchronous requests. You can configure a custom thread pool (using java.util.concurrent.Executors) for better control over the number of threads used by your application.
  • Testing and Handling Timeouts
    Always set reasonable timeouts for connections and requests to avoid blocking indefinitely when the server is slow or unreachable.
  • Setting Keep-Alive
    The connection pooling mechanism uses HTTP/1.1 or HTTP/2’s persistent connections to keep sockets alive. You don’t need to manually configure Keep-Alive as it’s handled internally by the HttpClient.

Example of Keep-Alive Headers:

If necessary, you can include a custom header in requests to explicitly control Keep-Alive durations:

HttpRequest request = HttpRequest.newBuilder()
        .uri(new URI("https://example.com"))
        .header("Connection", "keep-alive")
        .build();

4. Advanced: Tuning JVM System Properties

The HttpClient supports additional tuning using JVM system properties. For example, you can configure maximum connections per route, or total connections, if necessary.

Examples of commonly used JVM properties:

-Dsun.net.httpclient.defaultConnectTimeout=10000
-Dsun.net.httpclient.defaultReadTimeout=10000
-Djdk.httpclient.allowRestrictedHeaders=connection,keep-alive

5. Asynchronous Requests with Connection Pooling

If you perform a large number of requests and need high concurrency, leveraging asynchronous handling (sendAsync) is a good practice:

Example:

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.concurrent.CompletableFuture;

public class AsyncHttpClientDemo {
    public static void main(String[] args) {
        HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(10))
                .executor(java.util.concurrent.Executors.newFixedThreadPool(10))
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://jsonplaceholder.typicode.com/posts"))
                .GET()
                .build();

        // Send Async request
        CompletableFuture<HttpResponse<String>> responseFuture =
                client.sendAsync(request, HttpResponse.BodyHandlers.ofString());

        // Process the response asynchronously
        responseFuture.thenAccept(response -> {
            System.out.println("Status Code: " + response.statusCode());
            System.out.println("Response Body: " + response.body());
        }).join(); // Optional: Blocks until complete
    }
}

Summary

  • Java 11’s HttpClient comes with built-in support for connection pooling.
  • Always reuse HttpClient instances to automatically take advantage of the pool.
  • Tune parameters such as connectTimeout, executor, and HTTP versions.
  • Use asynchronous APIs (sendAsync) with custom executors for better concurrency.
  • Avoid creating too many HttpClient instances to prevent fragmentation.

This approach ensures that connection pooling is effectively implemented without the need for third-party libraries.

How do I use Java 11 HttpClient with HTTP/2 protocol?

The HttpClient introduced in Java 11 provides an easy-to-use mechanism for sending HTTP requests, and it supports HTTP/2 out of the box. To utilize the HTTP/2 protocol with Java 11’s HttpClient, you simply need to set the version to HttpClient.Version.HTTP_2 while building the client.

Here’s how you can use it:

Step-by-Step Example

Below is an example of how to configure and use HttpClient with HTTP/2 for making a GET request:

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Http2Example {
    public static void main(String[] args) {
        // Create an HttpClient with HTTP/2 enabled
        HttpClient httpClient = HttpClient.newBuilder()
                .version(HttpClient.Version.HTTP_2)  
                .build();

        // Create an HttpRequest to URL that supports HTTP/2
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://http2.golang.org")) 
                .GET()
                .build();

        try {
            // Send the request and get the response
            HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());

            // Print response details
            System.out.println("HTTP Version: " + response.version()); 
            System.out.println("Status Code: " + response.statusCode());
            System.out.println("Response Body:\n" + response.body());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Explanation of the Code

  1. HttpClient Setup:
    • HttpClient.newBuilder() creates a new HttpClient instance.
    • version(HttpClient.Version.HTTP_2) specifies that HTTP/2 should be used as the preferred protocol. If the server cannot support HTTP/2, it will automatically fall back to HTTP/1.1.
  2. HttpRequest:
    • A request is created using HttpRequest.newBuilder() with the desired URI and HTTP method (e.g., GET, POST, etc.).
    • For HTTP/2 compatibility, ensure the server supports HTTP/2.
  3. HttpResponse:
    • httpClient.send(request, HttpResponse.BodyHandlers.ofString()) sends a synchronous request and waits for the server response. It returns an HttpResponse object containing the status code, version used, headers, and body.
  4. Response Details:
    • Use response.version() to inspect the actual protocol version used (HTTP/1.1 or HTTP/2).
    • Retrieve response data using response.body().

Key Features:

  • HTTP/2 is enabled by default if supported by the server.
  • Asynchronous requests can also be made using the sendAsync() method:
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString())
        .thenApply(HttpResponse::body)
        .thenAccept(System.out::println)
        .join();

Requirements:

  • Ensure the server supports HTTP/2. If it doesn’t, the client will fall back to HTTP/1.1.
  • Java 11 or later must be installed to use the java.net.http package.

Tips:

  • Use tools like Wireshark or browser developer tools (via Network tab) to verify that HTTP/2 is being utilized.
  • Use response.version() to check the actual protocol version if in doubt about server support.

This code provides a clean and modern approach to handling HTTP/2 communication with Java

How do I use Java 11 HttpClient in a multithreaded environment?

When using the Java 11 HttpClient in a multithreaded environment, you need to ensure it is used in a thread-safe and efficient manner. Thankfully, the HttpClient introduced in Java 11 is designed to be thread-safe and can be shared among multiple threads without additional synchronization.

Here are the important points about using HttpClient in a multithreaded environment:


1. Create a Single HttpClient Instance

  • HttpClient is thread-safe, so create a single shared instance of HttpClient and reuse it across your application. Do not create a new instance of HttpClient for every request, as it can lead to performance inefficiencies.
  • Use the factory method HttpClient.newHttpClient() to create a default instance, or configure it with a custom builder.

Example:

package org.kodejava.net.http;

import java.net.http.HttpClient;
import java.time.Duration;

public class HttpClientBuilder {
    // Create a shared instance of HttpClient
    private static final HttpClient httpClient = HttpClient.newBuilder()
            .version(HttpClient.Version.HTTP_2) // Support HTTP/2
            .connectTimeout(Duration.ofSeconds(10)) // Set up a timeout
            .build();

    public static HttpClient getHttpClient() {
        return httpClient;
    }
}

2. Use Thread-Safe Methods

  • Use the methods send() (for synchronous requests) or sendAsync() (for asynchronous requests) provided by HttpClient. These methods are thread-safe and can be invoked concurrently by multiple threads.
  • You should handle response callbacks correctly in the case of asynchronous usage since they will be executed by the internal thread pool.

Synchronous Request Example:

package org.kodejava.net.http;

import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;

public class SynchronousRequest {
    public static void main(String[] args) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://jsonplaceholder.typicode.com/posts"))
                .GET()
                .build();

        // Using the shared HttpClient instance
        HttpResponse<String> response = HttpClientBuilder.getHttpClient()
                .send(request, HttpResponse.BodyHandlers.ofString());

        System.out.println("Response code: " + response.statusCode());
        System.out.println("Response body: " + response.body());
    }
}

Asynchronous Request Example:

package org.kodejava.net.http;

import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.util.concurrent.CompletableFuture;

public class AsynchronousRequest {
    public static void main(String[] args) {
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://jsonplaceholder.typicode.com/posts"))
                .GET()
                .build();

        // Asynchronous call
        CompletableFuture<HttpResponse<String>> futureResponse =
                HttpClientBuilder.getHttpClient()
                        .sendAsync(request, HttpResponse.BodyHandlers.ofString());

        // Handle response when it is ready
        futureResponse.thenAccept(response -> {
            System.out.println("Response code: " + response.statusCode());
            System.out.println("Response body: " + response.body());
        }).join(); // Wait for the response to complete
    }
}

3. Internal Thread Pool

  • By default, the HttpClient uses an internal thread pool managed by a ForkJoinPool for asynchronous requests. You can customize this by providing your own Executor when configuring the HttpClient with the builder.

Example: Custom Executor

package org.kodejava.net.http;

import java.net.http.HttpClient;
import java.util.concurrent.Executors;

public class CustomExecutorExample {
    public static void main(String[] args) {
        HttpClient httpClient = HttpClient.newBuilder()
                .executor(Executors.newFixedThreadPool(10))
                .build();

        // Use this HttpClient instance for requests
    }
}

4. Timeouts

  • Set a connection timeout at the HttpClient level using the connectTimeout method.
  • For individual requests, you can specify a request timeout.

Example:

package org.kodejava.net.http;

import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.time.Duration;

public class RequestWithTimeout {
    public static void main(String[] args) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://jsonplaceholder.typicode.com/posts"))
                .timeout(Duration.ofSeconds(5)) // Set request-specific timeout
                .GET()
                .build();

        HttpResponse<String> response = HttpClientBuilder.getHttpClient()
                .send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println("Response body: " + response.body());
    }
}

5. Handle Exceptions

  • Handle exceptions like timeouts, ConnectException, or server errors gracefully when making HTTP calls.
  • Use try-catch blocks for synchronous methods and handle() or exceptionally() for asynchronous methods.

Example:

package org.kodejava.net.http;

import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.util.concurrent.CompletableFuture;

public class HandleExceptionsExample {
    public static void main(String[] args) {
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://invalid.url"))
                .GET()
                .build();

        CompletableFuture<HttpResponse<String>> futureResponse = HttpClientBuilder.getHttpClient()
                .sendAsync(request, HttpResponse.BodyHandlers.ofString())
                .exceptionally(e -> {
                    System.out.println("Request failed: " + e.getMessage());
                    return null;
                });

        futureResponse.thenAccept(response -> {
            if (response != null) {
                System.out.println("Response code: " + response.statusCode());
            }
        }).join();
    }
}

6. Handling Concurrency

  • In multithreaded scenarios, the HttpClient can process concurrent requests efficiently since it is non-blocking, especially when using sendAsync(), and does not require manual synchronization.
  • You can run multiple requests in parallel by delegating each request to a thread or submitting tasks to an ExecutorService.

Example: Multiple Requests in Parallel

package org.kodejava.net.http;

import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;
import java.util.List;
import java.util.concurrent.CompletableFuture;

public class ParallelRequestsExample {
    public static void main(String[] args) {
        List<URI> uris = List.of(
                URI.create("https://jsonplaceholder.typicode.com/posts/1"),
                URI.create("https://jsonplaceholder.typicode.com/posts/2"),
                URI.create("https://jsonplaceholder.typicode.com/posts/3")
        );

        List<CompletableFuture<Void>> futures = uris.stream()
                .map(uri -> {
                    HttpRequest request = HttpRequest.newBuilder().uri(uri).GET().build();
                    return HttpClientBuilder.getHttpClient()
                            .sendAsync(request, HttpResponse.BodyHandlers.ofString())
                            .thenAccept(response -> {
                                System.out.println("Response for " + uri + " : " + response.body());
                            });
                })
                .toList();

        // Wait for all requests to complete
        CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
    }
}

Key Best Practices

  1. Reuse a single HttpClient instance for better resource management.
  2. Use asynchronous methods (sendAsync) for non-blocking and parallel requests.
  3. Configure timeouts to avoid hanging.
  4. Customize the thread pool if your application has specific concurrency requirements.
  5. Properly handle errors and exceptions in both synchronous and asynchronous requests.

By following these guidelines, you can effectively use HttpClient in a multithreaded environment.

How do I monitor performance and latency of Java 11 HTTP requests?

Monitoring the performance and latency of Java 11 HTTP requests is essential when utilizing the java.net.http package introduced with the new Java 11 HttpClient API. It helps identify bottlenecks, optimize network calls, and ensure efficient resource usage for your application.

Here’s how you can do it:

1. Measure Latency Using Timestamps

You can manually measure the time taken to send and receive HTTP requests by recording timestamps before and after executing the HTTP call.

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.time.Instant;

public class HttpPerformanceMonitor {
    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://example.com"))
                .GET()
                .build();

        // Record start time
        Instant start = Instant.now();

        // Send request and get response
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

        // Record end time
        Instant end = Instant.now();

        // Calculate latency
        Duration latency = Duration.between(start, end);
        System.out.println("Response time: " + latency.toMillis() + " ms");
        System.out.println("Response status code: " + response.statusCode());
    }
}

This approach calculates the time taken for the entire HTTP operation, including connection establishment, sending the request, and receiving the response.

2. Use a Custom Executor to Monitor Thread Usage

HttpClient allows you to set a custom Executor for handling its asynchronous operations. You can measure how efficiently the threads are being utilized.

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;

public class HttpClientWithCustomExecutor {
    public static void main(String[] args) throws Exception {
        ExecutorService executor = Executors.newFixedThreadPool(4); // Monitor thread usage

        HttpClient client = HttpClient.newBuilder()
                .executor(executor)
                .build();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://example.com"))
                .GET()
                .build();

        long startTime = System.currentTimeMillis();

        client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
                .thenAccept(response -> {
                    long endTime = System.currentTimeMillis();
                    System.out.println("Response time: " + (endTime - startTime) + " ms");
                    System.out.println("Response status code: " + response.statusCode());
                })
                .join(); // Wait for the async call to complete

        executor.shutdown();
    }
}

By setting a custom Executor, you can track thread pool utilization and capture latency.

3. Enable Logging for HTTP Headers and Debugging

Java 11 supports configuring logging for the HTTP Client. You can enable debug-level logging to capture low-level details:

  • Add the following JVM options to enable java.net.http.HttpClient debugging:
-Djdk.httpclient.HttpClient.log=requests,headers,frames:all

This will output detailed log information, including HTTP request/response headers and frames.

4. Monitor HTTP Client Metrics Using Libraries

Libraries that support metrics collection (like Micrometer) can be integrated with Java 11 HttpClient to collect throughput, response times, and errors.
Example: Using Micrometer You can create custom timers and counters to record metrics manually.

package org.kodejava.net.http;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.TimeUnit;

public class HttpClientWithMetrics {
    public static void main(String[] args) throws Exception {
        MeterRegistry registry = new SimpleMeterRegistry(); // Replace with a distributed registry like Prometheus

        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(new URI("https://example.com"))
                .GET()
                .build();

        registry.timer("http.requests").record(() -> {
            try {
                Instant start = Instant.now();
                HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
                Instant end = Instant.now();

                System.out.println("Response time: " + Duration.between(start, end).toMillis() + " ms");
                System.out.println("Response status code: " + response.statusCode());
            } catch (Exception e) {
                e.printStackTrace();
            }
        });

        registry.get("http.requests").timers().forEach(timer -> {
            System.out.println("Timer count: " + timer.count());
            System.out.println("Total time: " + timer.totalTime(TimeUnit.MILLISECONDS) + " ms");
        });
    }
}

5. Use Dependency Injection Frameworks

Combine the Java 11 HttpClient with Spring Boot’s actuator (if you’re already using it). Actuator provides built-in metrics and HTTP tracing capabilities.

6. Profiling with External Tools

You can observe Java application performance (including HTTP requests) using APM tools, such as:

  • Java Flight Recorder (JFR)
    • Use JFR to monitor detailed HTTP request timings and underlying JVM performance.
  • Java VisualVM
    • Profile application threads and network usage.
  • Third-Party APMs
    • Tools like New Relic, AppDynamics, or Datadog can report on HTTP client usage.

Summary of Approaches

Approach Tools/Techniques Notes
Manual Timing Instant + Duration Tracks latency at the request level.
Custom Executors ExecutorService Measures thread usage and async execution.
Logging JVM Debugging (HttpClient.log) Debugs detailed network activity.
Metrics Libraries Micrometer (manual instrumentation) Can feed metrics into observability tools.
Framework Integration Spring Boot Actuator Collects trace and performance metrics.
External Profiling/Monitoring JFR, APM Tools, VisualVM Monitors JVM, HTTP, and app performance.

Choose the approach based on your application’s complexity and monitoring needs!


Maven Dependencies

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>1.15.0</version>
</dependency>

Maven Central

How do I retry failed requests using Java 11 HttpClient?

Retrying failed requests using Java 11’s HttpClient involves implementing a custom retry mechanism. Java 11’s HttpClient doesn’t provide native support for automatic retries, but you can build a retry mechanism into your application by wrapping the logic in a loop.

Here’s how you can do it step by step:

1. Define a Retry Mechanism

You’ll need to decide on:

  • The maximum number of retry attempts.
  • An optional delay between retries.
  • The type of failures that should trigger retries (e.g., IOException).

2. Create a Function to Retry Requests

You can encapsulate the retry logic in a utility method or by using a loop.

3. Full Code Example:

Below is an example implementation:

package org.kodejava.net.http;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.io.IOException;

public class HttpClientWithRetry {

   public static void main(String[] args) {
      HttpClient httpClient = HttpClient.newHttpClient();
      HttpRequest request = HttpRequest.newBuilder()
              .uri(URI.create("https://api.example.com/resource"))
              .timeout(Duration.ofSeconds(10))
              .GET()
              .build();

      int maxRetries = 3;  // Max number of retries
      Duration delay = Duration.ofSeconds(2); // Delay between retries

      try {
         HttpResponse<String> response = sendWithRetries(httpClient, request, maxRetries, delay);
         System.out.println("Response received: " + response.body());
      } catch (Exception e) {
         System.err.println("Request failed after retries: " + e.getMessage());
      }
   }

   public static HttpResponse<String> sendWithRetries(HttpClient client,
                                                      HttpRequest request,
                                                      int maxRetries,
                                                      Duration delay) throws Exception {
      int attempts = 0;

      while (true) {
         try {
            // Attempt to send the request
            return client.send(request, HttpResponse.BodyHandlers.ofString());
         } catch (IOException | InterruptedException ex) {
            attempts++;
            if (attempts > maxRetries) {
               throw new Exception("Maximum retry attempts reached.", ex);
            }
            System.out.println("Request failed, retrying... (" + attempts + "/" + maxRetries + ")");
            Thread.sleep(delay.toMillis()); // Delay before retry
         }
      }
   }
}

Explanation:

  1. HttpClient Initialization:
    • The HttpClient is created using HttpClient.newHttpClient().
  2. Retry Logic:
    • The sendWithRetries method sends the HTTP request and retries on exceptions (IOException or InterruptedException in this example).
    • A loop tracks the number of attempts and retries until either the request succeeds or the maximum retry limit is reached.
  3. Delay Between Retries:
    • A Thread.sleep introduces a delay (defined by a Duration object) between retries.
  4. Failure Handling:
    • If all retries fail, an exception is thrown indicating the request could not be completed.
  5. Configurations:
    • You can adjust the retry count, delay, and consider making the logic more robust by checking specific HTTP status codes for retries (e.g., 5xx server errors).

Future Improvements:

  • Exponential Backoff: Increase the delay between retries exponentially to ease server load.
  • Timeout Handling: Handle custom timeout scenarios with detailed configuration.
  • Customizable Errors: Customize retry logic based on specific HTTP response codes (e.g., retry only for 500–599 responses).