In Java, you can retrieve web content and parse HTML using the URL and URLConnection classes. Here’s a step-by-step guide along with an example:

Steps to Retrieve Web Content

1. Create a URL: Use the URL class to specify the web address.
2. Open a Connection: Use the openConnection() method from the URL object to establish a connection.
3. Read the Content: Use the InputStream from the URLConnection to retrieve the content.
4. Parse the HTML: Once you have the content, you can parse the HTML using libraries like org.jsoup (recommended for HTML parsing in Java).

Example Code

package org.kodejava.net;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.StandardCharsets;

public class WebContentReader {
   public static void main(String[] args) {
      try {
         // Step 1: Create a URL object
         URL url = new URL("https://example.com"); // Replace with your URL

         // Step 2: Open a connection
         URLConnection connection = url.openConnection();

         // Step 3: Read content using InputStream and BufferedReader
         BufferedReader reader = new BufferedReader(
                 new InputStreamReader(connection.getInputStream(), StandardCharsets.UTF_8));
         StringBuilder content = new StringBuilder();
         String line;

         while ((line = reader.readLine()) != null) {
            content.append(line).append("\n");
         }
         reader.close();

         // Step 4: Print or process the HTML content
         System.out.println(content);

         // Optional: Parse the content with Jsoup (external library)
         //org.jsoup.nodes.Document document = org.jsoup.Jsoup.parse(content.toString());
         //System.out.println("Title: " + document.title());
      } catch (Exception e) {
         e.printStackTrace();
      }
   }
}

Explanation

1. URL and URLConnection:
   • URL represents the web resource.
   • URLConnection allows you to retrieve the data from the specified URL.
2. BufferedReader + InputStreamReader:
   • Used to read the incoming data line by line.
   • UTF-8 encoding ensures proper handling of characters.
3. StringBuilder:
   • Accumulates the content in memory to be processed further.

Parsing HTML with Jsoup

If you parse the HTML, libraries like Jsoup make it easy to work with HTML documents. Here’s what you can do after retrieving the web content:

1. Add Jsoup dependency to your pom.xml (if using Maven):

   <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.16.1</version>
   </dependency>
   

2. Parse the HTML content using Jsoup:

   import org.jsoup.Jsoup;
   import org.jsoup.nodes.Document;
   
   ...
   
   Document document = Jsoup.parse(content.toString());
   System.out.println("Title: " + document.title()); // Extract the title
   

Important Notes

• Error Handling: Always handle exceptions like MalformedURLException, IOException, etc., as network operations can fail.
• Timeouts: Use HttpURLConnection (subclass of URLConnection) if you want more control, like setting timeouts.
• Avoid Blocking: For large content or real-time web scraping, consider asynchronous I/O or libraries like Apache HttpClient or OkHttp.

This approach is simple but effective for learning how to retrieve and process web content in Java.