How do I read / download webpage content?

You want to create a program that read a webpage content of a website page. The example below use the URL class to create a connection to the website. You create a new URL object and pass the URL information of a page. After the object created you can open a stream connection using the openStream() method of the URL object.

Next, you can read the stream using the BufferedReader object. This reader allows you to read line by line from the stream. To write it to a file create a writer using the BufferedWriter object and specify the file name where the downloaded page will be stored.

When all the content are read from the stream and stored in a file close the BufferedReader object and the BufferedWriter object at the end of your program.

package org.kodejava.example.net;

import java.io.*;
import java.net.MalformedURLException;
import java.net.URL;

public class UrlReadPageDemo {
    public static void main(String[] args) {
        try {
            URL url = new URL("http://www.kodejava.org");

            BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
            BufferedWriter writer = new BufferedWriter(new FileWriter("data.html"));

            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
                writer.write(line);
                writer.newLine();
            }

            reader.close();
            writer.close();
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }  catch (IOException e) {
            e.printStackTrace();
        }
    }
}

How do I read a text file using BufferedReader?

The code snippet below is an example of how to read a text file using BufferedReader class from the java.io package. This snippet read a text file called README.md and print out its content.

To create an instance of java.io.BufferedReader we do the following steps:

File file = new File("README.md");
FileReader fileReader = new FileReader(file));
BufferedReader bufferedReader = new BufferedReader(fileReader);

Let’s see the complete code snippet.

package org.kodejava.example.io;

import java.io.*;

public class ReadTextFileExample {
    private static String lineSeparator = System.getProperty("line.separator");

    public static void main(String[] args) {
        File file = new File("README.md");
        try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
            StringBuilder contents = new StringBuilder();
            String text;
            while ((text = reader.readLine()) != null) {
                contents.append(text).append(lineSeparator);
            }

            System.out.println(contents.toString());
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

You can also try to use the following example to read a file, How do I read text file content line by line using commons-io?. To create and write a text file see the following example: How do I create and write data into text file?.