fronttoken/3, str_toklist/2

fronttoken(String,String,String)

str_toklist(String,TokenList)

Break the string into tokens

Remarks

fronttoken/3 operates as if it were defined by the equation:

String = (the concatenation of Token and RestString)

str_toklist/2 performs one step conversion of the string into a list of tokens

A group of one or more characters constitutes a token in the following cases:

• The text group constitutes a <name> according to Prolog syntax.
• The group constitutes a valid string representation of a integer or real (a preceding sign is returned as a separate token).
• The group is a single character, but not the ASCII space character (ASCII 32), Horizontal Tab (ASCII 9 - '\t' <HT>) or other white-space character (Line Feed (ASCII 10 <LF>), Vertical Tab (ASCII 11 <VT>), Form Feed (ASCII 12 <FF>), Carriage Return (ASCII 13 (CR)), SUBstitute (ASCII 26 <SUB> or <EOF> - Ctrl+Z) ).

Notice that in DOS-like strings, new lines ('\n') are represented as the couples of the <CR> and the <LF> characters. Contrary, in UNIX -like strings, new lines are represented as the single <LF> character (see also str_dosstr/2 predicate).

Example

fronttoken("all kids do fine",TOK,REST)

TOK=all, REST= kids do fine

1 Solution

fronttoken("all+kids do fine",TOK,REST),

fronttoken(REST,TOK1,_)

TOK=all, REST=+kids do fine, TOK1=+

1 Solution

str_toklist("all kids do fine",L)

L=[all,kids,do,fine]

1 Solution