AutomatedTests: Difference between revisions

From Request Tracker Wiki
Jump to navigation Jump to search
(Point readers to RT's official docs on testing)
 
(add docs)
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Automated Testing in RT =
== Test Prerequisites ==
[http://bestpractical.com/rt/docs/latest/hacking Please see the official RT documentation for information on running the test suite. This information is incomplete] (and outdated; it's "make test" not "make regression").


== Introduction ==
The best way to set up RT for running tests (both its own and extensions') is by running the following from a source checkout:


When working with RT (and any large software system) one should really have a large set of automated tests. Best Practical likes patches that come with regression tests, etc etc etc. So write tests.
<pre>./configure.ac --with-db-type=TYPE --enable-layout=inplace [other options ...]
mkdir -p var</pre>


== Basic overview of test framework ==
Then make sure all your dependencies are in place by running <code>make testdeps</code>. (If you need help with that part, check out [[ManualInstallation]].) If you're at that point and just want to get running as fast as possible, skip down to "Environment Variables to Control Tests." The next couple of sections give more background about what's going on.


The tests are kicked off by <code> make regression</code> this runs a full test suite, which may be more than you want.
=== Why a real database? ===


Tests all use the <code> Test::More</code> perl module.
RT does include database-specific tests to check that various integrations work as expected. There's no good way to abstract these, so the test suite selects and runs the tests appropriate for your database.


Tests from the "old" code are stored in the modules source code, in specific pod sections. They are extracted before being executed.
=== Why an inplace layout? ===


However, the preferred way of writing tests is now writing test scripts in <code> lib/t/regression/</code>).
The RT tests read the configuration file <code>RT_Config.pm</code> located in the directory named at <code>$RT::Generated::EtcPath</code>. If you use the default relative layout, or another layout intended for production like FHS, the tests will try to read your production configuration. That's probably not what you want.


Depending on which tests you run, you'll need different amounts of RT installed. Most things should need a working database. Some things will need a webserver.
Note that the tests '''only''' read <code>RT_Config.pm</code>. They do '''not''' read <code>RT_SiteConfig.pm</code> or files under <code>RT_SiteConfig.d</code>. You cannot change the behavior of tests by editing configuration there.


Tests run smoothly as root, but this may be a problem for some users. Non privileged testing is achievable but your mileage may vary for some parts (tests related to mail gateway and web server).
(This is another reason you need to specify a database type on the configure line: so it gets put into <code>RT_Config.pm</code>. There are environment variables to control most database parameters you need, but not the database type.)


== Details about writing tests ==
== Environment Variables to Control Tests ==


Using <code> Test::More</code> is pretty simple. <code> perldoc Test::More</code> should give you lots of examples, but here's a very simple test:
=== Test RT Configuration ===


<nowiki>#!/usr/bin/perl
Except on SQLite, the tests need permission to create new databases; to create new users in the database; and to grant rights to those users. To run the tests at all, you '''must''' set the environment variables <code>RT_DBA_USER</code> and <code>RT_DBA_PASSWORD</code> with the username and password, respectively, of a database account that has all those rights.
 
  use strict;
  use warnings;
  use Test::More plan =&gt; 2;
 
  is(1, 1, "one should be one");
  isnt(1, 2, "one is not two");
 
  </nowiki>


Writing tests is pretty much just like using the API. You call some whatever you're testing, then you use <code> is</code> or <code> isnt</code> to check that the result is what you expect. I believe <code>12-search.t</code> should be a pretty clear example.
The tests write a small <code>RT_SiteConfig.pm</code> it uses to configure itself. You can control some of the settings with environment variables. Refer to [https://docs.bestpractical.com/rt/latest/RT_Config.html the RT_Config documentation] for full details.


== Tips and tricks ==
{| class="wikitable" style="width: 100%;"
! Environment variable
! Corresponding RT_SiteConfig setting
|-
|<code>RT_TEST_DB_SID</code>
|<code>$DatabaseName</code> (intended for Oracle only)
|-
|<code>RT_TEST_DB_HOST</code>
|<code>$DatabaseHost</code>
|-
|<code>RT_TEST_RT_HOST</code>
|<code>$DatabaseRTHost</code>
|-
|<code>RT_TEST_DEVEL</code>
|<code>$DevelMode</code>
|-
|<code>RT_TEST_PLUGINS</code>
|<code>@Plugins</code> (split on spaces)
|}


The full regression suite takes a while to run, and depends on lots of things you may not have installed. For example, when hacking on the search related tests, I really didn't want to install enough email pieces to run the email tests. So I made a <code> seph-test</code> target that depended on <code> regression-install regression-reset-db run-regression</code> I also moved most of the tests in lib/t/regression/ aside, so only my test was being run.
=== Test Selection ===


The regression tests will use the first perl in your path. So set <code> $PATH</code> appropriately or set <code> $PERL</code> to specify which <code> perl</code> to use.
A few tests are skipped unless you do/n't set an environment variable.
 
{| class="wikitable" style="width: 100%;"
! Environment variable
! Controls tests
|-
|<code>RT_TEST_HEAVY</code>
|Set a string here to run tests that require relatively expensive database queries.
|-
|<code>RT_TEST_SMIME_REVOCATION</code>
|Set a string here to run SMIME revocation tests. Doing so downloads a certificate revocation list (CRL) from the public Internet.
|-
|<code>SKIP_GPG_TESTS</code>
|Set a string here to skip tests that run GnuPG
|}
 
=== Test Run Environment ===
 
{| class="wikitable" style="width: 100%;"
! Environment variable
! Controls tests
|-
|<code>RTHOME</code>
|Path to the RT source code checkout you use for testing. The core RT tests don't use this, but extensions usually do.
|-
|<code>RT_TEST_PARALLEL</code>
|Set a string here to notify the test suite that you're running tests in parallel. This changes some of the logic around database creation and elsewhere to avoid contention. You can set this even if you only have a single test runner.
You may also find some references to <code>RT_TEST_PARALLEL_NUM</code>. This is used by RT's Makefile to control how many test runners run in parallel. If you don't go through RT's Makefile, then nothing else uses that variable.
|-
|<code>RT_TEST_APACHE</code>
|Path to the <code>httpd</code> executable on your system
|-
|<code>RT_TEST_APXS</code>
|Path to the <code>apxs</code> executable on your system
|-
|<code>RT_TEST_APACHE_MODULES</code>
|Path to the directory where your httpd module <code>.so</code> files live
|}
 
== Running the Tests ==
 
Once you've set and exported all the environment variables you need, run the tests with <code>prove</code> from your source directory:
 
<pre>RT_TEST_PARALLEL=1 prove -w --lib --recurse --jobs=NUMBER t</pre>
 
{| class="wikitable" style="width: 100%;"
! Option
! Purpose
|-
|<code>-w</code>
|Enable warnings
|-
|<code>--lib</code>
|Load libraries from the <code>lib</code> directory (i.e., your source checkout)
|-
|<code>--recurse</code>
|Recurse through the path argument(s) to find and run all test <code>.t</code> files
|-
|<code>--jobs=NUMBER</code>
|Run <code>NUMBER</code> test runners in parallel
|-
|<code>t</code>
|Run tests under this path. You can specify other directories (some extensions have tests in <code>xt</code>), specific subdirectories, or even individual files.
|}

Latest revision as of 17:05, 17 December 2021

Test Prerequisites

The best way to set up RT for running tests (both its own and extensions') is by running the following from a source checkout: 

./configure.ac --with-db-type=TYPE --enable-layout=inplace [other options ...]
mkdir -p var

Then make sure all your dependencies are in place by running make testdeps. (If you need help with that part, check out ManualInstallation.) If you're at that point and just want to get running as fast as possible, skip down to "Environment Variables to Control Tests." The next couple of sections give more background about what's going on.

Why a real database?

RT does include database-specific tests to check that various integrations work as expected. There's no good way to abstract these, so the test suite selects and runs the tests appropriate for your database.

Why an inplace layout?

The RT tests read the configuration file RT_Config.pm located in the directory named at $RT::Generated::EtcPath. If you use the default relative layout, or another layout intended for production like FHS, the tests will try to read your production configuration. That's probably not what you want.

Note that the tests only read RT_Config.pm. They do not read RT_SiteConfig.pm or files under RT_SiteConfig.d. You cannot change the behavior of tests by editing configuration there.

(This is another reason you need to specify a database type on the configure line: so it gets put into RT_Config.pm. There are environment variables to control most database parameters you need, but not the database type.)

Environment Variables to Control Tests

Test RT Configuration

Except on SQLite, the tests need permission to create new databases; to create new users in the database; and to grant rights to those users. To run the tests at all, you must set the environment variables RT_DBA_USER and RT_DBA_PASSWORD with the username and password, respectively, of a database account that has all those rights.

The tests write a small RT_SiteConfig.pm it uses to configure itself. You can control some of the settings with environment variables. Refer to the RT_Config documentation for full details.

Environment variable Corresponding RT_SiteConfig setting
RT_TEST_DB_SID $DatabaseName (intended for Oracle only)
RT_TEST_DB_HOST $DatabaseHost
RT_TEST_RT_HOST $DatabaseRTHost
RT_TEST_DEVEL $DevelMode
RT_TEST_PLUGINS @Plugins (split on spaces)

Test Selection

A few tests are skipped unless you do/n't set an environment variable.

Environment variable Controls tests
RT_TEST_HEAVY Set a string here to run tests that require relatively expensive database queries.
RT_TEST_SMIME_REVOCATION Set a string here to run SMIME revocation tests. Doing so downloads a certificate revocation list (CRL) from the public Internet.
SKIP_GPG_TESTS Set a string here to skip tests that run GnuPG

Test Run Environment

Environment variable Controls tests
RTHOME Path to the RT source code checkout you use for testing. The core RT tests don't use this, but extensions usually do.
RT_TEST_PARALLEL Set a string here to notify the test suite that you're running tests in parallel. This changes some of the logic around database creation and elsewhere to avoid contention. You can set this even if you only have a single test runner.

You may also find some references to RT_TEST_PARALLEL_NUM. This is used by RT's Makefile to control how many test runners run in parallel. If you don't go through RT's Makefile, then nothing else uses that variable.

RT_TEST_APACHE Path to the httpd executable on your system
RT_TEST_APXS Path to the apxs executable on your system
RT_TEST_APACHE_MODULES Path to the directory where your httpd module .so files live

Running the Tests

Once you've set and exported all the environment variables you need, run the tests with prove from your source directory: 

RT_TEST_PARALLEL=1 prove -w --lib --recurse --jobs=NUMBER t
Option Purpose
-w Enable warnings
--lib Load libraries from the lib directory (i.e., your source checkout)
--recurse Recurse through the path argument(s) to find and run all test .t files
--jobs=NUMBER Run NUMBER test runners in parallel
t Run tests under this path. You can specify other directories (some extensions have tests in xt), specific subdirectories, or even individual files.
Retrieved from "https://rt-wiki.bestpractical.com/index.php?title=AutomatedTests&oldid=27041"