So we're going to take a look at the interpretation process by looking at how to parse and indeed evaluate
simple numeric expressions.
So the kind of thing that we want to evaluate is something like the following.
So we might want to take a string like for example let's say 13 minus four minus 12 plus 1.
And what we want to do is we want to interpret the string turn it into first of all a sequence of tokens.
This is what we're going to do in this lesson.
And then in the next lesson we're going to parse the tokens and turn them into some sort of object oriented
structure.
So it's a two step process and we're going to do the first step right now.
So the idea is that we have this expression that we want to split it into separate tokens and these
tokens would be for example the opening round bracket.
Then de-value 13 then the minus then the value of 4 and so on and so forth.
So we do want to keep the numbers together.
We don't want to split the team into the tokens 1 and 3 because if we do them we will have to coalesce
them back into a single value later on.
So that's what we're going to do and we're going to start by of course defining the kind of tokens that
we can have in our system.
So let's actually have some sort of structure called tokin.
So this is going to represent a single token that is actually acquired from the expression.
Let me just move this a little bit like so.
OK so the token can be an integer.
Since we have expressions involving integers only but it can also be a plus or minus or left parenthesis
or a right parenthesis.
So let's actually enumerate the types.
So will have integer them plus minus left parenthesis and right parenthesis.
OK so these are the token types.
We're also going to have the text of the token itself and the reason why we want this is because for
example if you have an integer just knowing that it's an integer isn't enough to tell you what kind
of integer it is what kind of value it actually has.
We're going to keep the text of the actual tokens as we kind of get them.
So let's have a constructor which initializes both of these.
OK so it doesn't give me.
Of course we haven't made a variable here.
So this is going to be a parable called type.
So hopefully a second time lucky OK we have the constructor.
Let's also have a stream output operator where I'll be going to print the text may be delimited by some
backwords so that we can subsequently verify that the tokens we are getting are actually correct So
let me implement the stream output operator a like.
So we're only interested in the text to really and I really hate it when we do this get an include for
O stream which I don't really need.
So what we're going to output is we'll outputs of the back quote and then token tax and then another
Backwell just so we can see the sort of tokens of where they start and where they end because who knows
they might have whitespace in them.
Not in our design obviously but in general you might encounter something yourself.
Now let's go on to the lexing process.
We're going to have a single function called leks which is going to take an input string that will be
the string right here and we're going to return a vector of tokens.
So it was going to be called leks.
And here is the input.
OK.
So let's make a vector for our result and that's what we're going to return later on.
And now what we need to do is we need to go through the string and actually look at each of the characters
and determine whether this character is actually a self-contained token or if you for example if you
encounter a one a one by itself is not a self-contained token and you have to look beyond the one to
get additional characters.
So I'm going to make a for loop.
So in the for loop we're going to go we're going to use a counter so we'll use the index i going from
zero to the dot size.
That's the length of the input string and we're going to do a switch on the character position.
And we're going to try to determine what it is.
So most of these cases are actually very simple because essentially if you get a plus for example that
means that you add a token which is a plus
actually do I need the plus prefix and I can probably just do it like this.
And it's called pushback.
So if I encounter a plus then I'm going to add the token and yes I think I need to prefix in here a
token plus because it's a global function not related to our tokens so we add the plus and the text
is obviously a plus as well.
So this is how we would define this particular case and of course in addition to plus we also have the
minus the opening parenthesis the closing parentheses.
Now all of these are more or less the same but we cannot just do one generic case for all of them.
So here is every single one.
Nothing exciting here the exciting part is getting to identify the integers and I'm going to stick it
into the default sections.
So if it's not one of these operators then it's going to be an integer default.
OK.
So what we're going to do is we're going to basically try to parse the try to read the integer a character
at a time.
I'm going to make a buffer for putting in all the characters so that we can subsequently kind of convert
them into a single string so I'll have an o string stream called buffer.
And then I will add the first character because oh well we know if it's not one of the operators It
has to be a number.
We're going to add that to the buffer.
And then we're going to go through the rest of the characters so I'll have a for loop here.
So use the index j for this one and it's going to once again start at E-plus Plus once those starting
out the next character and going all the way until the end of the string all the way until input the
size and what we're going to do here is we're going to once again verify whether or not the character
at position J is in fact addition because if it is we need to add that to the buffer just like the previous
digit.
So if his digit input and J then what we do is we it to the buffer input and J.
And remember we also have to increment the index because i's the global index the outer index which
we are going through.
So as we kind of increment the J We also have to increment the eye here as well.
Now Otherwise what happens if we don't get an integer is a we take everything from a buffer and we simply
added to the overall set of tokens.
So if we didn't get the integer we just take the buffer and we say result that we push back and we make
a new token.
And this time the token is an integer and it's actual text is what we have in the buffer so buffer Dauth
asked for that would go.
So we add the token and then we do a break.
So we jump around from this in a for loop and we can continue the overall execution.
So that's pretty much it for the lexing and we can now try to verify that what we got in terms of lexing
this string is actually correct.
Let's say auto tokens equals Lex..
Lex the input and then I'm going to print all of them.
So I'm going to say for otos see in tokens.
Let's see out see.
Followed by maybe some spaces.
And then let's put a line break at the end of it all.
There we go.
OK.
So we can now run this and see whether the lexing process actually works correctly or not.
All right so as you can see we're getting the right results here are the tokens so the opening round
bracket the number 13 minus 4 and so on and so forth.
So the lexing part is finished and we can move on to the parsing of the lexical tokens that we got.