How do I read website content using HttpClient?

The HTTP Client API can be used to request HTTP resources over the network. This new API was introduced as a new API in Java 11. It supports HTTP/1.1 and HTTP/2 and also support both synchronous and asynchronous programming models. The code snippet below show you how to use the new API to read the content of a website page.

In the code below we start by creating a new instance of HttpClient using the newHttpClient() static method. This is equivalent to calling newBuilder().build(). This give us an instance of HttpClient with default settings like using the “GET” request method the as the default. Then we create an HttpRequest object using the newBuilder() method, set the request URI and call the build() method to build the HttpRequest object.

Next we send the request by calling the send() method of the HttpClient object. This will sends the given request, blocking if necessary to get the response. The returned HttpResponse object contains the response status, headers, and body as handled by given response body handler.

package org.kodejava.httpclient;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;

public class ReadWebsiteContent {
    public static void main(String[] args) throws Exception {
        // Creates HttpClient object with default configuration.
        HttpClient httpClient = HttpClient.newHttpClient();

        // Creates HttpRequest object and set the URI to be requested, 
        // when not defined the default request method is the GET request.
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://httpie.org/hello"))
                .GET()
                .build();

        // Sends the request and print out the returned response.
        HttpResponse<String> response = httpClient.send(request,
                HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));

        System.out.println("Status Code: " + response.statusCode());
        System.out.println("Headers    : " + response.headers().toString());
        System.out.println("Body       : " + response.body());
    }
}

Here is the content of the website that we read using the code snippet above:

Status Code: 200
Headers    : java.net.http.HttpHeaders@2d299ad6 { {:status=[200], cf-cache-status=[DYNAMIC], cf-ray=[5875b78d5df2eb00-LAX], cf-request-id=[023d710c5b0000eb00b738f200000001], content-length=[116], content-type=[text/x-rst;charset=utf-8], date=[Tue, 21 Apr 2020 08:25:53 GMT], etag=["234b9a1fe19f125356a5396c8cc72d54493a2eef"], expect-ct=[max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"], server=[cloudflare], set-cookie=[__cfduid=d5bdb6d828be3bb85d0f1f4c2ff81041c1587457553; expires=Thu, 21-May-20 08:25:53 GMT; path=/; domain=.httpie.org; HttpOnly; SameSite=Lax]} }
Body       : 

Hello, World! 👋
~~~~~~~~~~~~~~~~

Thank you for trying out HTTPie 🥳

I hope this will become a friendship.
Wayan

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.