The steps one shall do when checking into the NetBeans CVS
repository
Having write access to the NetBeans CVS repository is a privilege
but
also a responsibility: responsibility to the quality of the code
and responsibility to other people working on NetBeans. In order to
ensure that the quality of the code base is high and that you will not
disturb anybody else's work, here are a few guidelines that will help
you to validate your integration before you commit it into the NetBeans
CVS
repository.
- Necessary step is to get the right sources.
In contrast to a case when you just watch the sources you need
more modules than just standard configuration. Execute
cvs co -P nbbuild xtest
ant -f nbbuild/build.xml checkout
to be able to validate your changes before integration.
- When your modification is ready to be integrated do a clean build
to make sure that it does not negatively influence other
modules. Execute
cd nbbuild; ant clean
to clear the sources and
cd nbbuild; ant build-nozip
to compile the NetBeans IDE. Currently we are building with JDK 1.4, so
you
can use all its methods, but be careful not to depend on anything
newer.
- Execute commit validation test suite which is designed
to test the basic functionality of NetBeans and shall affirm you
that your integration is not completely broken. Execute
cd nbbuild; ant commit-validation
Of course no test shall fail otherwise there is something wrong -
probably with your
integration. You should evalute
why it fails.
- Execute unit tests that might be influenced by your changes. For
example when checking into openide
module its test shall be run by
cd openide/test; ant
and shall not fail.
-
If you use Linux, you can run a premade script that does the
clean build and test suites for you. Try nbbuild/misc/full-build-linux.sh
(see the shell script comments for usage instructions).
- If the IDE compiles, tests pass you are ready for integration.
Check what you have changed by evaluating the output of
cvs
diff -u and if the output really contains what you have
modified do cvs ci -m "description msg".
Evaluate commit
validation results
Once you have executed commit validation test suite you have to
evaluate its results. If you run it from command line, at the end it is
printed out summary information about pass ratio of the suite. There is
also provided a link to HTML presentation of the results. If something
fails, you should follow this link and check error messages and log
files to determine what is wrong with your build.
Main page of the results is split into two frames. The left navigation
frame shows you links to results of particular modules and also list of
modules which failed. The right frame shows summary of results in
several levels.
The shortest way to see the reason of failure is to click on module
name in Failures report in the left navigation frame. Then you can see
name of failed test case and error message. If you need more details,
click on its name. You will see test cases for whole test suite and
failed test in red color. To see exception stack trace follow the link
in Status column. There might be also additional proprietary log files
in working directory of the test case. In that case there should be
"Yes" link in Workdir column. For UI functional tests (module ide) are
useful jemmy.log with trace messages of performed operations and
screen.png containing B&W screen shot at the time of the failure.
If you are not able to identify reason of failure, or you think failure
is not caused by your changes.
How to quickly find useful things from results main page
- messages.log and similar
- click on module name, then testbag name - there should be link
IDE Logs which points to messages.log
- ant logs
- follow the link Logs from build scripts - you can find there
compilation errors etc.
- test case error message
- click on module name, then testbag name, suite name - you will
see list of executed test cases and failed one in red color with error
message
- exception stack trace
- click on module name, then testbag name, suite name, link in
Status column - you will see stack trace of exception which
caused that test failed
- jemmy.log
- click on module name, then testbag name, suite name, link in
Workdir column, jemmy.log file - it contains trace messages of
performed operations and it can be useful to know the last operation
before the failure
- screen.png
- click on module name, then testbag name, suite name, link in
Workdir
column, screen.png file - it contains B&W screen shot at the time
of the failure
Add a new test case into commit validation suite
To add a test case into the commit validation suite you have to modify
module's tests config file and also xtest's master config if module is
not already included in the suite. If you need more information about
XTest, please visit the
site.
Reasons to add a test
The
commitValidation suite shall serve as a global check
suite, that
will try to catch possible regressions spread over wide range of
functionality.
For example a modification in
org.openide.text.Line can
affect
the correctness of debugging of JSPs. It would be the best to write
correct unit
test of the
Line but if that is not possible, than the
JSP
module developer is encouraged to add a test for the correct
JSP
debugging functionality into the
commitValidation tests
of
the
JSP module. That way he will help the core developer
recognize
that there is something wrong before the commit is done.
Another example might be related to correct garbage collection. The
whole
system is designed to free itself from memory when not used, but small
mistake
in any module can prevent this feature to function correctly. As this
can be
broken by any commit in any module, this is a reasonable test candidate
for a
commitValidation suite.
In no way the commitValidation is a replacement for unit
tests. Each
module is responsible for its own set of tests for basic functionality
that
shall be executed before any change of the module code. The
commitValidation suite cannot contain such tests as it would
not scale
and we need it to execute quickly as it shall be run before every
commit.
Think about that before extending it - if it is not reasonable to
provide
validation of an important functionality on a level of unit tests,
then commitValidation is the right place. But try to minimize
such
cases for the sake of scalability.
Modify module's config file
Create a new test bag with attribute
commit and add test cases in it
which you want to be executed in commit validation suite. The test bag
should look like this:
<testbag
testattribs="commit" executor="code" name="Before Commit Validation">
<testset dir="unit/src">
<patternset>
<include
name="org/openide/loaders/*Folder*.class"/>
<include name="org/openide/windows/*Test.class"/>
</patternset>
</testset>
</testbag>
Above example is taken from
in openide module. If test bag with
commit attribute already exists
in config file, you only need to add your test cases by adding an
appropriate
include
element.
Modify XTest's master config file
You have to modify master config file only if your module is not
already included in commit validation suite. In that case simply add a
line to config named
commit-validation-nb.
Master config is located in
and commit validation part looks like this:
<config
name="commit-validation-nb">
<module name="core" testtypes="unit" attributes="commit"/>
<module name="openide" testtypes="unit" attributes="commit"/>
<module name="ide" testtypes="qa-functional" attributes="commit"/>
</config>
Tips and tricks
-
If you participate in a development, you should be subscribed to
.
Failures in a daily builds as well as continuous ones are notified
there and if your integration breaks something, you can react
quickly. If you are going to fix such problem please reply to the
mail, so others know that it is being solved.
-
Test and test tools are also just a software this means if
you find bug in them please fill it. If you don't now which
Issuezilla module it is then go to
where you can find more info,
- If you don't know how to evaluate commit validation suite
results, read above.
-
Is there a way do a simple baseline to get a before/after
diff of your patches (unless you do it after every cvs update)?
Yes, there is look at cvsps.
-
The automatic tests opens the IDE and popup and close various
windows which prevents me from work. How can I prevent it? Easy on
Unix - start new X server (Xnest is useful) and execute
the
validation test there. Using the full-build-linux.sh
driver script, just pass spawndisplay=yes.
-
Usually one is commiting change that has a record in Issuezilla
(bug, some task, enhancement). In such case mention the issue number
in the commit message.
cvs ci -m "#31043: Fixing the screen resolution"
and include the output of the commit into the description field
when resolving the issue. Then one can it is possible to find out
more about the commit in the issuezilla and it is possible to find
the sources that were modified to resolve the issue.
Common mistakes
- Do not check in debug code that depends on a modified API
with something made public that isn't Solution: Always
do a sanity diff
- Check in a class that also contains a change that's part
of a different bug fix Solution: Always do a sanity
diff
