How do I break a text or sentence into words?

By Wayan in Core API, Text Package 0 Comment

At first, it might look simple. We can just split the text using the String.split(), the word is split using space. But what if a word ends with questions marks (?) or exclamation marks (!) instead? There might be some other rules that we also need to care.

Using the java.text.BreakIterator makes it much simpler. The class’s getWordInstance() factory method creates a BreakIterator instance for words break. Instantiating a BreakIterator and passing a locale information makes the iterator to breaks the text or sentence according the rule of the locale. This is really helpful when we are working with a complex language such as Japanese or Chinese.

Let us see an example of using the BreakIterator below.

package org.kodejava.text;

import java.text.BreakIterator;
import java.util.Locale;

public class BreakIteratorExample {
    public static void main(String[] args) {
        String data = "The quick brown fox jumps over the lazy dog.";
        String search = "dog";

        // Gets an instance of BreakIterator for word break for the
        // given locale. We can instantiate a BreakIterator without
        // specifying the locale. The locale is important when we
        // are working with languages like Japanese or Chinese where
        // the breaks standard may be different compared to English.
        BreakIterator bi = BreakIterator.getWordInstance(Locale.US);

        // Set the text string to be scanned.
        bi.setText(data);

        // Iterates the boundary / breaks
        System.out.println("Iterates each word: ");
        int count = 0;
        int lastIndex = bi.first();
        while (lastIndex != BreakIterator.DONE) {
            int firstIndex = lastIndex;
            lastIndex = bi.next();

            if (lastIndex != BreakIterator.DONE
                    && Character.isLetterOrDigit(data.charAt(firstIndex))) {
                String word = data.substring(firstIndex, lastIndex);
                System.out.printf("'%s' found at (%s, %s)%n",
                        word, firstIndex, lastIndex);

                // Counts how many times the word dog occurs.
                if (word.equalsIgnoreCase(search)) {
                    count++;
                }
            }
        }

        System.out.println("Number of word '" + search + "' found = " + count);
    }
}

Here are the program output:

Iterates each word: 
'The' found at (0, 3)
'quick' found at (4, 9)
'brown' found at (10, 15)
'fox' found at (16, 19)
'jumps' found at (20, 25)
'over' found at (26, 30)
'the' found at (31, 34)
'lazy' found at (35, 39)
'dog' found at (40, 43)
Number of word 'dog' found = 1

How do I split up string using regular expression?

By Wayan in Regex 0 Comment

This code snippet uses the java.util.regex.Pattern.split() method to split-up input string separated by commas or whitespaces (spaces, tabs, new lines, carriage returns, form feeds).

package org.kodejava.regex;

import java.util.regex.Pattern;

public class RegexSplitExample {
    public static void main(String[] args) {
        // Pattern for finding commas, whitespaces (spaces, tabs, new lines,
        // carriage returns, form feeds).
        String pattern = "[,\\s]+";
        String colors = """
                Red,White, Blue   Green        Yellow,
                Orange Pink""";

        Pattern splitter = Pattern.compile(pattern);
        String[] results = splitter.split(colors);

        for (String color : results) {
            System.out.format("Color = \"%s\"%n", color);
        }
    }
}

The result of our code snippet is:

Color = "Red"
Color = "White"
Color = "Blue"
Color = "Green"
Color = "Yellow"
Color = "Orange"
Color = "Pink"

How do I split a string using Scanner class?

By Wayan in Core API, Util Package 0 Comment

Instead of using the StringTokenizer class or the String.split() method we can use the java.util.Scanner class to split a string.

package org.kodejava.util;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

public class ScannerTokenDemo {
    public static void main(String[] args) {
        // This file contains some data as follows:
        // a, b, c, d
        // e, f, g, h
        // i, j, k, l
        File file = new File("data.txt");
        try {
            // Here we use the Scanner class to read file content line-by-line.
            Scanner scanner = new Scanner(file);
            while (scanner.hasNextLine()) {
                String line = scanner.nextLine();

                // From the above line of code we got a line from the file
                // content. Now we want to split the line with comma as the 
                // character delimiter.
                Scanner lineScanner = new Scanner(line);
                lineScanner.useDelimiter(",");
                while (lineScanner.hasNext()) {
                    // Get each split data from the Scanner object and print
                    // the value.
                    String part = lineScanner.next();
                    System.out.print(part + ", ");
                }                
                System.out.println();
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }
    }
}

How do I split a string?

By Wayan in Core API, Lang Package 0 Comment

Prior to Java 1.4 we use java.util.StringTokenizer class to split a tokenized string, for example a comma separated string. Starting from Java 1.4 and later the java.lang.String class introduce a String.split(String regex) method that simplify this process.

Below is a code snippet how to do it.

package org.kodejava.lang;

import java.util.Arrays;

public class StringSplit {
    public static void main(String[] args) {
        String data = "1,Diego Maradona,Footballer,Argentina";
        String[] items = data.split(",");

        // Iterates the array to print it out.
        for (String item : items) {
            System.out.println("item = " + item);
        }

        // Or simply use Arrays.toString() when print it out.
        System.out.println("item = " + Arrays.toString(items));
    }
}

The result of the code snippet:

item = 1
item = Diego Maradona
item = Footballer
item = Argentina
item = [1, Diego Maradona, Footballer, Argentina]

How do I use StringTokenizer to split a string?

By Wayan in Core API, Util Package 0 Comment

The code below is an example of using StringTokenizer to split a string. In the current JDK this class is discouraged to be used, use the String.split(...) method instead or using the new java.util.regex package.

package org.kodejava.util;

import java.util.StringTokenizer;

public class StringTokenizerExample {
    public static void main(String[] args) {
        StringTokenizer st =
            new StringTokenizer("A StringTokenizer sample");

        // get how many tokens inside st object
        System.out.println("Tokens count: " + st.countTokens());

        // iterate st object to get more tokens from it
        while (st.hasMoreElements()) {
            String token = st.nextElement().toString();
            System.out.println("Token = " + token);
        }

        // split a date string using a forward slash as delimiter
        st = new StringTokenizer("2021/09/14", "/");
        while (st.hasMoreElements()) {
            String token = st.nextToken();
            System.out.println("Token = " + token);
        }
    }
}

Here is the result of this sample code:

Tokens count: 3
Token = A
Token = StringTokenizer
Token = sample
Token = 2021
Token = 09
Token = 14