/* $DOC$
   $AUTHOR$
      2016 Pete D. <pete_westg@yahoo.gr>
   $TEMPLATE$
      Function
   $NAME$
      hb_ATokens()
   $CATEGORY$
      API
   $SUBCATEGORY$
      Strings
   $ONELINER$
      Parses a complex string (e.g. a sentence or multi-line text) into individual
      tokens (words or other string chunks depending on delimiter used).
   $SYNTAX$
      hb_ATokens( <cString>, [<cDelim>|<lEOL>], [<lSkipStrings>], ;
                  [<lDoubleQuoteOnly>] ) --> aTokens
   $ARGUMENTS$
      <cString> Complex string to be parsed.

      <cDelim>|<lEOL Character(s) used as delimiter of separate tokens.
                     If <lEOL> flag defined instead of <cDelim>,
                     then end of line marker(s) will be used as delimiter.

      <lSkipStrings> Boolean flag indicating if quoted substrings
                     will be tokenized or not.

      <lDoubleQuoteOnly> Boolean flag indicating that only double-quoted
                         substrings will be tokenized.
   $RETURNS$
      <aTokens> A character array, filled with the individual tokens found.
   $DESCRIPTION$
      This function analyses the complex string <cString> and splits it
      into separate sub-strings (tokens) that are delimited by <cDelim>
      or by space character, if no <cDelim> delimiter is passed, or by EOL marker
      if <lEOL> instead of <cDelim> is specified.

      The located tokens, are stored in an array which is returned by the function.

      If <lSkipStrings> is .T. (default: .F.), the quoted sub-strings (if any)
      within <cString> are not tokenized. If <lDoubleQuoteOnly> is .T.
      only the double quote `"` is recognized as a quote sign.
      This argument is meaningful only when <lSkipStrings> is .T.

   $NOTES$
      1) the tokenization process is case sensitive, in the (rare) case
         where <cDelim> is an alphabetic character.

      2) The delimiters are removed (trimmed) from tokens.
   $EXAMPLES$
      LOCAL cString := "Harbour is proven to be stable, robust and efficient."
      LOCAL aTokens := hb_ATokens( cString )
      AEval( aTokens, {| token, n | QOut( hb_ntos(n), token ) } )
      ?
      aTokens := hb_ATokens( cString, "," )
      AEval( aTokens, {| token, n | QOut( hb_ntos(n), token ) } )
   $STATUS$
      R
   $COMPLIANCE$
      H
   $PLATFORMS$
      All
   $FILES$
      Library is core
   $SEEALSO$
      SubStr()
   $END$
 */