+1 vote
860 views
in Compilers & Operating Systems by

1 Answer

+1 vote
by
selected by (user.guest)
 
Best answer

TOKENS, PATTERNS AND LEXEMES:

A token is a pair consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a kind of lexical unit, e.g., a particular keyword, or a sequence of input characters denoting an identifier. The token names are the input symbols that the parser processes. In what follows, we shall generally write the name of a token in boldface. We will often refer to a token by its token name. 

A pattern is a description of the form that the lexemes of a token may take [ or match]. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens, the pattern is a more complex structure that is matched by many strings.

A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token. 

Example: In the following C language statement ,

printf ("Total = %d\n‖, score) ;

both printf and score are lexemes matching the pattern for token id, and "Total = %d\n‖

is a lexeme matching literal [or string].


Related questions

0 votes
1 answer 585 views
+1 vote
1 answer 481 views
+1 vote
1 answer 274 views
asked May 24, 2021 in Compilers & Operating Systems by anonymous
+1 vote
1 answer 377 views
+3 votes
1 answer 8.7k views
+1 vote
1 answer 277 views
+1 vote
1 answer 155 views
0 votes
1 answer 77 views
+1 vote
1 answer 36 views
+1 vote
1 answer 3.6k views
+1 vote
1 answer 188 views
+1 vote
1 answer 198 views
+1 vote
1 answer 1.1k views
Welcome to CPENTalk.com
Solution-oriented students of computer engineering on one platform to get you that

ONE SOLUTION

...