How do I copy a URL into a file?

The code snippet below shows you how to use the FileUtils.copyURLToFile(URL, File) method of the Apache Commons IO library to help you to copy the contents of a URL directly into a file.

package org.kodejava.commons.io;

import org.apache.commons.io.FileUtils;

import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URL;

public class URLToFile {
    public static void main(String[] args) {
        try {
            URL url = URI.create("https://www.google.com").toURL();
            File destination = new File("google.html");

            // Copy bytes from the URL to the destination file.
            FileUtils.copyURLToFile(url, destination);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Maven Dependencies

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.16.1</version>
</dependency>

Maven Central

How do I get servlet request URL information?

In the code example below we will extract information regarding the HTTP (Hypertext Transport Protocol) from the request object (HttpServletRequest). We will extract the protocol used (http / https), server name and its assigned port number. We will also read our application context path, servlet path, path info and the query string. If we combine all the previously mentioned information we will get something equals to the value returned by request.getRequestURL() method.

Let’s check the code snippet below to see what method of the HttpServletRequest class that we can call to get the information regarding the HTTP request object that we can collect.

package org.kodejava.servlet;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;

@WebServlet(urlPatterns = "/url-info")
public class ServletUrlInformation extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        doPost(request, response);
    }

    @Override
    protected void doPost(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        // Getting servlet request URL
        String url = request.getRequestURL().toString();

        // Getting servlet request query string.
        String queryString = request.getQueryString();

        // Getting request information without the hostname.
        String uri = request.getRequestURI();

        // Below we extract information about the request object path
        // information.
        String scheme = request.getScheme();
        String serverName = request.getServerName();
        int portNumber = request.getServerPort();
        String contextPath = request.getContextPath();
        String servletPath = request.getServletPath();
        String pathInfo = request.getPathInfo();
        String query = request.getQueryString();

        response.setContentType("text/html");
        PrintWriter pw = response.getWriter();
        pw.print("Url: " + url + "<br/>");
        pw.print("Uri: " + uri + "<br/>");
        pw.print("Scheme: " + scheme + "<br/>");
        pw.print("Server Name: " + serverName + "<br/>");
        pw.print("Port: " + portNumber + "<br/>");
        pw.print("Context Path: " + contextPath + "<br/>");
        pw.print("Servlet Path: " + servletPath + "<br/>");
        pw.print("Path Info: " + pathInfo + "<br/>");
        pw.print("Query: " + query);
    }
}

When you access this servlet using the following url http://localhost:8080/url-info?x=1&y=1, you’ll get the following output in your browser:

Url: http://localhost:8080/url-info
Uri: /url-info
Scheme: http
Server Name: localhost
Port: 8080
Context Path:
Servlet Path: /url-info
Path Info: null
Query: x=1&y=1

Maven dependencies

<dependency>
    <groupId>javax.servlet</groupId>
    <artifactId>javax.servlet-api</artifactId>
    <version>4.0.1</version>
</dependency>

Maven Central

How do I get response header from HTTP request?

package org.kodejava.net;

import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
import java.util.List;
import java.util.Map;

public class HttpResponseHeaderDemo {
    public static void main(String[] args) {
        try {
            URL url = new URL("https://kodejava.org/index.php");
            URLConnection connection = url.openConnection();

            Map<String, List<String>> responseMap = connection.getHeaderFields();
            for (String key : responseMap.keySet()) {
                System.out.print(key + " = ");

                List<String> values = responseMap.get(key);
                for (String value : values) {
                    System.out.print(value + ", ");
                }
                System.out.println();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

The result produced by the code above are:

null = HTTP/1.1 200 OK, 
Referrer-Policy = no-referrer-when-downgrade, 
Content-Length = 147566, 
Content-Type = text/html; charset=UTF-8, 
Connection = keep-alive, 
Date = Sun, 26 Sep 2021 03:59:06 GMT, 
Accept-Ranges = bytes, 
Vary = Accept-Encoding, Accept-Encoding, 
Link = <https://wp.me/8avgG>; rel=shortlink, <https://kodejava.org/wp-json/>; rel="https://api.w.org/",  

How do I create a URL object?

package org.kodejava.net;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.nio.charset.StandardCharsets;

public class URLDemo {
    public static void main(String[] args) {
        try {
            // Creating a url object by specifying each parameter separately, including
            // the protocol, hostname, port number, and the page name
            URL url = new URL("https", "kodejava.org", 443, "/index.php");

            // We can also specify the address in a single line
            url = new URL("https://kodejava.org:443/index.php");

            BufferedReader reader = new BufferedReader(
                    new InputStreamReader(url.openStream(), StandardCharsets.UTF_8));

            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
            }

            reader.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

How do I read or download web page content?

You want to create a program that read a webpage content of a website page. The example below use the URL class to create a connection to the website. You create a new URL object and pass the URL information of a page. After the object created you can open a stream connection using the openStream() method of the URL object.

Next, you can read the stream using the BufferedReader object. This reader allows you to read line by line from the stream. To write it to a file create a writer using the BufferedWriter object and specify the file name where the downloaded page will be stored.

When all the content are read from the stream and stored in a file close the BufferedReader object and the BufferedWriter object at the end of your program.

package org.kodejava.net;

import java.io.*;
import java.net.URL;
import java.nio.charset.StandardCharsets;

public class UrlReadPageDemo {
    public static void main(String[] args) {
        try {
            URL url = new URL("https://kodejava.org");

            BufferedReader reader = new BufferedReader(
                    new InputStreamReader(url.openStream(), StandardCharsets.UTF_8));
            BufferedWriter writer = new BufferedWriter(
                    new FileWriter("data.html", StandardCharsets.UTF_8));

            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
                writer.write(line);
                writer.newLine();
            }

            reader.close();
            writer.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}