Eliminate some recursion from children of _expand_token_list
Previously, both _expand_node and _expand_function would always make
mutually recursive calls into _expand_token_list. This was unnecessary
since these functions can simply return unexpanded results, after which
the outer iteration will next attempt expansion of the results.
The only trick in doing this is to arrange so that the active list is
popped at the appropriate time. To do this, we add a new token_node_t
marker to the active stack. When pushing onto the active list, we set
marker to last->next, and when the marker is seen by the token list
iteration, we pop from the active stack.
ir_dereference_array always operates on an r-value
ir_dereference_array::array is always an r-value. If the dereference
is of a varaible, that r-value will be an ir_dereference_variable.
This simplifies the code a bit.
Factor out common sub-expression from multi-line-comment regular expression.
In two places we look for an (optional) sequence of characters other
than "*" followed by a sequence of on or more "*". Using a name for
this (NON_STARS_THEN_STARS) seems to make it a bit easier to
understand.
Fix multi-line comment regular expression to handle (non) nested comments.
Ken reminded me of a couple cases that I should be testing. These are
the non-nestedness of things that look like nested comments as well as
potentially tricky things like "/*/" and "/*/*/".
The (non) nested comment case was not working in the case of the
comment terminator with multiple '*' characters. We fix this by not
considering a '*' as the "non-slash" to terminate a sequence of '*'
characters within the comment. We also fix the final match of the
terminator to use '+' rather than '*' to require the presence of a
final '*' character in the comment terminator.
We were nicely constructing a new expression for the implicit type
conversion, but then checking that the previous types matched instead
of the new expression's type. Fixes errors in Regnum Online shaders.
ir_constant_variable: New pass to mark constant-assigned variables constant.
This removes a bunch of gratuitous moving around of constant values
from constructors. Makes a shader ir I was looking at for structure
handling almost readable.
ir_expression_flattening: Fix breakage from hierarchichal visitor.
Similar to other situations where the visitor pattern doesn't fit, in
this case we need the pointer to the base instruction in the
instruction stream for where to insert any new instructions we
generate (not the instruction in the tree we're looking at). By
removing the code for setting the base_ir, flattened expressions would
end up, for example, before the function definition where they had appeared.
Implement comment handling in the lexer (with test).
We support both single-line (//) and multi-line (/* ... */) comments
and add a test for this, (trying to stress the rules just a bit by
embedding one comment delimiter into a comment delimited with the
other style, etc.).
To keep the test suite passing we do now discard any output lines from
glcpp that consist only of spacing, (in addition to blank lines as
previously). We also discard any initial whitespace from gcc output.
In neither case should the absence or presence of this whitespace
affect correctness.
Fix #if-skipping to *really* skip the skipped group.
Previously we were avoiding printing within a skipped group, but we
were still evluating directives such as #define and #undef and still
emitting diagnostics for things such as macro calls with the wrong
number of arguments.
Add a test for this and fix it with a high-priority rule in the lexer
that consumes the skipped content.
The take-2 branch started over with a new grammar based directly on
the grammar from the C99 specification. It doesn't try to capture
things like balanced sets of parentheses for macro arguments in the
grammar. Instead, it merely captures things as token lists and then
performs operations like parsing arguments and expanding macros on
those lists.
We merge it here since it's currently behaving better, (passing the
entire test suite). But the code base has proven quite fragile
really. Several of the recently added test cases required additional
special cases in the take-2 branch while working trivially on master.
So this merge point may be useful in the future, since we might have a
cleaner code base by coming back to the state before this merge and
fixing it, rather than accepting all the fragile
imperative/list-munging code from the take-2 branch.
Add three more tests cases recently added to the take-2 branch.
The 071-punctuator test is failing only trivially (whitespace change only).
And the 072-token-pasting-same-line.c test passes just fine here, (more
evidence perhaps that the approach in take-2 is more trouble than it's
worth?).
The 099-c99-example test case is the inspiration for much of the rest
of the test suite. It amazingly passes on the take-2 branch, but
doesn't pass here yet.
Add test and fix bugs with multiple token-pasting on the same line.
The list replacement when token pasting was broken, (failing to
properly update the list's tail pointer). Also, memory management when
pasting was broken, (modifying the original token's string which would
cause problems with multiple calls to a macro which pasted a literal
string). We didn't catch this with previous tests because they only
pasted argument values.
Previously '=' was not included in our PUNCTUATION regeular expression,
but it *was* excldued from our OTHER regular expression, so we were
getting the default (and hamful) lex action of just printing it.
The test we add here is named "punctuator" with the idea that we can
extend it as needed for other punctuator testing.
Tweak test 25 slightly, (so the non-macro doesn't end the file).
This isn't a problem here, but on the take-2 branch, it was trickier
at one point to make a non-macro work when the last token of the file.
So we use the simpler test case here and defer the other case until
later.
Perform macro by replacing tokens in original list.
We take the results of macro expansion and splice them into the
original token list over which we are iterating. This makes it easy
for function-like macro invocations to find their arguments since they
are simply subsequent tokens on the list.
This fixes the recently-introduced regressions (tests 55 and 56) and
also passes new tests 60 and 61 introduced to strees this feature,
(with macro-argument parentheses split between a macro value and the
textual input).
Simplify calling conventions of functions under expand_token_list_onto.
We previously had a confusing thing where _expand_token_onto would
return a non-zero value to indicate that the caller should then call
_expand_function_onto. It's much cleaner for _expand_token_onto to
just do what's needed and call the necessary function.
Stop interrupting the test suite at the first failure.
This behavior was useful when starting the implementation over
("take-2") where the whole test suite was failing. This made it easy
to focus on one test at a time and get each working.
More recently, we got the whole suite working, so we don't need this
feature anymore. And in the previous commit, we regressed a couple of
tests, so it's nice to be able to see all the failures with a single
run of the suite.
Remove blank lines from output files before comparing.
Recently I'm seeing cases where "gcc -E" mysteriously omits blank
lines, (even though it prints the blank lines in other very similar
cases). Rather than trying to decipher and imitate this, just get rid
of the blank lines.
This approach with sed to kill the lines before the diff is better
than "diff -B" since when there is an actual difference, the presence
of blank lines won't make the diff harder to read.
This test was tricky to make pass in the take-2 branch. It ends up
passing already here with no additional effort, (since we are lexing
integers as string-valued token except when in the ST_IF state in the
lexer anyway).
To do this correctly, we change the lexer to lex integers as string values,
(new token type of INTEGER_STRING), and only convert to integer values when
evaluating an expression value.
Add a new test case for this, (which does pass now).
Provide support for empty arguments in macro invocations.
For this we always add a new argument to the argument list as soon as
possible, without waiting until we see some argument token. This does
mean we need to take some extra care when comparing the number of
arguments with the number of expected arguments. In addition to
matching numbers, we also support one (empty) argument when zero
arguments are expected.
Add a test case here for this, which does pass.
Make two list-processing functions do nothing with an empty list.
This just makes these functions easier to understand all around. In
the case of _token_list_append_list this is an actual bug fix, (where
append an empty list onto a non-empty list would previously scramble
the tail pointer of the original list).
Avoid treating an expanded comma as an argument separator.
That is, a function-like invocation foo(x) is valid as a
single-argument invocation even if 'x' is a macro that expands into a
value with a comma. Add a new COMMA_FINAL token type to handle this,
and add a test for this case, (which passes).
Carl Worth