Class Tokenizer


  • public class Tokenizer
    extends java.lang.Object
    Utility class to help with tokenizing strings.
    Since:
    1 Sep 2005
    Author:
    Mark Taylor
    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static java.lang.String[] tokenizeLines​(java.lang.String text)
      Splits a string up into lines, separated by semicolons or newlines.
      static java.lang.String[] tokenizeWords​(java.lang.String line)
      Chops up a line of text into tokens.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Method Detail

      • tokenizeLines

        public static java.lang.String[] tokenizeLines​(java.lang.String text)
                                                throws uk.ac.starlink.task.UsageException
        Splits a string up into lines, separated by semicolons or newlines. Semicolons may appear inside quoted strings without terminating a line.
        Parameters:
        text - input string
        Returns:
        array of lines
        Throws:
        uk.ac.starlink.task.UsageException
      • tokenizeWords

        public static java.lang.String[] tokenizeWords​(java.lang.String line)
                                                throws uk.ac.starlink.task.UsageException
        Chops up a line of text into tokens. Works roughly like the shell, as regards quotes, whitespace and comments.
        Parameters:
        line - line of text
        Returns:
        array of words corresponding to line
        Throws:
        uk.ac.starlink.task.UsageException