View source for TroubleShooting

= Troubleshooting RT =

== Introduction ==

I'm starting this section on the wiki in order to try to gather some of the various troubleshooting information that I've been running across from various locations. The reason I want to regurgitate it here is that I am finding many of the links from the search engines are starting to get stale and many of them are dead-ending. Hopefully we can preserve them in this repository.

- [[MichaelErana]]

== Perl Related ==

=== Can't locate object method "boot" via package "mod_perl" ===

I recently ran into the following error:

[Tue Jun 21 09:09:30 2005] [[error]] Can't locate object method "boot" via package "mod_perl" at /usr/lib/perl5/Apache/Constants.pm line 8. Compilation failed in require at /usr/lib/perl5/Apache.pm line 6. BEGIN failed--compilation aborted at /usr/lib/perl5/Apache.pm line 6. Compilation failed in require at /usr/share /perl/5.8/CGI.pm line 191. Compilation failed in require at /usr/share/request- tracker3.4/libexec/webmux.pl line 56. BEGIN failed--compilation aborted at /usr/share/request-tracker3.4/libexec/webmux.pl line 56. Compilation failed in require at (eval 3) line 1.

[Tue Jun 21 09:09:30 2005] [[error]] Can't load Perl file: /usr/share/request-tracker3.4/libexec/webmux.pl for server rt.company.com:0, exiting...

This was just after some updates to a Debian Stable installation. I read several posts about some problems with mod-perl2 on Apache2 and the suggested fix was to do a search and replace to change all occurances of "Apache::" to "Apache2::"

I really wanted to avoid code mods so I went for the lowest common denominator. I renamed the old Apache.pm and then just linked /usr/lib/perl5/Apache.pm to /usr/lib/perl5/Apache2.pm. This got the server back up and working. I suspect that there will be issues with some module specifics later but I am hoping that the RT Dist will also come up with ways to cope with that internally.

- [[MichaelErana]]

Change occurrences in which files?

=== Can't locate Apache/Response.pm in @INC ... Can't load Perl file: bin/webmux.pl for server localhost:0, exiting... ===

Found this issue in Aurora SPARC Linux 2.0 (Fedora Core 3 for SPARC) when trying to install... turns out that after installing all the modules for rt, I '''still''' had an ancient verion of CGI.pm, which was not intended to correctly detect the mod_perl2 modules vs. regular mod_perl. Upgrading CGI via CPAN quickly resolved this problem after an embarrassing number of hours troubleshooting.

If Apache refuses to start after you add [[PerlRequire]] bin/webmux.pl to your config and you're sure you've got a valid mod_perl2 installation, try upgrading CGI to the latest version.

- Ben

=== Can't locate Apache2.pm in @INC ... Can't load Perl module Apache2 for server localhost:0, exiting... ===

A transitional version of <code>mod_perl2</code> (version 1.99) broke many things on its way to being stable. Some instructions scattered throughout this wiki (including some on this page) still refer to the <code>Apache2</code> package when configuring the Apache web server. This package has since become stable and usable, but compatibility options are no longer necessary if the applications that depend on them have been updated to use <code>Apache2</code> modules instead of <code>Apache</code> modules. RT versions since 3.4.4 (feel free to correct this if I'm mistaken) have been updated to take advantage of <code>Apache2</code> with <code>mod_perl2</code>, so references to it in the Apache configuration are no longer necessary.

Making certain you '''avoid''' using the following lines in the RT section of your Apache configuration will eliminate this error.

 PerlModule Apache2
 PerlModule Apache::compat

- Ed Eaglehouse

=== Search Page Fails, Other Things Work, after Perl update ===

After installing a Perl security upgrade (via RPM from "up2date" on RHEL 4), the RT search query builder fails with the following error:

*Can't locate object method "new" via package "RT::Interface::Web::[[QueryBuilder]]::Tree" at /home/local/rt-3.4.4/share/html/Search/Build.html line 509.*

Everything had previously been working, and the Perl version was 5.8.5 before and after the upgrade -- all that changed was the build number, so one would think that wouldn't break much.

Reinstalling [[DBIx]]::[[SearchBuilder]] as recommended elsewhere in this Wiki for a similar problem did not help.

It took a lot of digging, but the root cause of this problem was found to be an error in [[RedHat]]'s building of the Perl binary, in that they left out support for "weak references", a feature that had been in their previous build. The CPAN-installed Tree::Simple package, used by RT::Interface::Web::[[QueryBuilder]]::Tree, could not construct an instance of itself because it had previously been [http://www.annocpan.org/~STEVAN/Tree-Simple-1.15/lib/Tree/Simple.pm configured to support weak references]. The problem is also described in [http://qs321.pair.com/~monkads/?node_id=481322 another web article].

I fixed the problem by deleting (from /usr/lib/perl/site_perl/5.8.5/) the "Tree" directory, then using CPAN to reinstall the packages therein. Apparently the reinstallation detected that weak references were no longer valid, and configured Tree::Simple appropriately. I then restarted Apache to make the change effective. You may have to reinstall more than just this one module; I'm not enough of a Perl expert to be sure, and I had already reinstalled other Perl packages in unsuccessful attempts to fix this problem. Thus, I can't be sure if the last change, had it been the '''only''' change, would have done the job by itself.

- [[ScottCourtney]]

Same problem, Red Hat Enterprise Linux AS release 4 (Nahant Update 4), same solution, excepting that after deletion of the module, I reinstalled it with:

 &lt;installdir&gt;/rt3/sbin/rt-test-dependencies --with-mysql --with-fastcgi --install

(Adjust options to suit, obviously...)

- Benji Wakely

Same problem with Red Hat Enterprise Linux AS release 4 (Nahant Update 4) but using the script did not work. The solution is to go to cpan and download Tree-Simple-1.14.tar.gz. The newst Tree-simple-1.16 will not compile on RHEL4 with updates but 1.14 compiled and works no problem.

--Piotr Misztal

=== RT will not send mail ===

This is related to the Search Page fails... problem above in that it is another case where missing weak references make RT crash.

RT 3.6.1 would not send mail because of the references to Scalar::Util::weaken in the code. On a RH AS 4 system, the Scalar::Util module exists but the weaken() function is defined elsewhere in an XS library. The library could be installed via CPAN with

 perl -MCPAN -e 'install "G/GB/GBARR/Scalar-List-Utils-1.18.tar.gz"'

--[[JasonLander]]

=== Syslog error 'MDA returned nonzero status 75' when using fetchmail, procmail or similar ===

Email was no longer being forwarded to the RT queue giving the above error in /var/adm/messages (logged by syslog). I had recently changed the name of one of the queues.

This is because I had not updated the .fetchmailrc file to reflect the change in queue name. So when the rt-mailgate was called it was going to the old queue which no longer existed.

If you are using a program such as fetchmail, procmail, maildrop etc. to deliver email to your queue, check the commandline arguments to see if they are still configured for the old queue.

This may seeme to be a no-brainer for many of you but it was driving me crazy for hours!

- [[BrendanArnold]]

A good way to debug this is to run your rt-mailgate incant by hand from the terminal and pipe a single mail into it. By adding the <code>--debug</code> flag, you can get verbose output about what's going wrong. One cause of this can be Apache redirecting rt-mailgate's request somewhere.

=== 'The page cannot be found' error in Internet Explorer when using SSL connections ===

IE has problems with certain types of SSL connections, especially during POST and GET operations (i.e. updating a ticket). The following in the mod_ssl FAQ gives a solution.

http://www.modssl.org/docs/2.8/ssl_faq.html#ToC49

- [[BrendanArnold]]

=== Initiation error when starting httpd service, Starting httpd: No root path(s) specified at /usr/bin/webmux.pl line 98 ===

--Setup on Centos 4.2 Server per...

 http://wiki.bestpractical.com/index.cgi?RPMInstall

...using (see INSTALLATION PROCEDURE below)...

 yum install perl-DBD-Pg rt rt-mail-dispatcher mysql-server lynx

--then after making local configuration changes (see FILE CHANGES and INSTALLATION PROCEUDURE below)...

 /usr/sbin/rt-setup-database --action init --dba root --dba-password contribs.org

...results in error message...

 Possible unintended interpolation of @network in string at /etc/rt/RT_SiteConfig.pm line 34.

...and...

 service httpd start

...results in error message...

 Starting httpd: No root path(s) specified at /usr/bin/webmux.pl line 98

--then, after navigating to a private (local) ip and successfully logging on to...

 http://rt.mynetwork.us

...all links (erroneously?) xref as if [[WebBaseURL]]="", example...

 Preferences links to...
  http://User/Prefs.html

...versus...

 http://rt.mynetwork.us/User/Prefs.html

--INSTALLATION PROCEDURE

*** install from repository '''*'''

0.-setup-from-rt-repo.sh

 <nowiki>#!/bin/bash
  
         yum update
  
         wget -nc -nd -S
         http://campus.fct.unl.pt/paulomatos/rt/repository/3.4.x/rt-3.4.x.repo
  
         /bin/mv --force rt-3.4.x.repo /etc/yum.repos.d/
  
         yum install perl-DBD-Pg rt rt-mail-dispatcher mysql-server lynx
  
         exit 0
  
  </nowiki>

1.-backup-files-before-editing.sh

 <nowiki>#!/bin/bash
  
  /bin/cp -p /etc/aliases{,.orig}
  /bin/cp -p /etc/hosts{,.orig}
  /bin/cp -p /etc/rt/RT_SiteConfig.pm{,.orig}
  /bin/cp -p /etc/httpd/conf.d/rt.conf{,.orig}
  /bin/cp -p /etc/httpd/conf/httpd.conf{,.orig}
  /bin/cp -p /etc/sysconfig/httpd{,.orig}
  /bin/cp -p /etc/mail/sendmail.mc{,.orig}
  /bin/cp -p /etc/mail/virtusertable{,.orig}
  /bin/cp -p /etc/mail/local-host-names{,.orig}
  /bin/cp -p /var/rt/home/.procmailrc{,.orig}
  
  exit 0
  </nowiki>

*** make file changes (make manual edits per FILE CHANGES listed below) '''*'''

*** incorporate file changes (run following scripts) '''*'''

2.-setup-users-sendmail-etc.sh

 <nowiki>#!/bin/bash
  
         pushd /etc/mail
         make
         popd
  
         service sendmail restart
  
         adduser -u 0 -o -g 0 -d /root -s /bin/bash -c admin admin
         echo contribs.org | passwd --stdin admin
         passwd -S admin
  
         ln -s /usr/bin/rt-mailgate /etc/smrsh
  
         newaliases
  
  </nowiki>

3.-setup-mysql-rt-database.sh

 <nowiki>#!/bin/bash
  
         service mysqld start
         mysqladmin -u root password contribs.org
  
         chkconfig mysqld on
         chkconfig --list | grep mysqld
  
         /usr/sbin/rt-setup-database --action init
                 --dba root --dba-password contribs.org
  #      /usr/sbin/rt-setup-database --action init
  #              --dba root --prompt-for-dba-password
  
  </nowiki>

--HERE ARE SOME OTHER USEFUL SCRIPTS...

90.-drop-rt-database.sh

 <nowiki>#!/bin/bash
  
         /usr/sbin/rt-setup-database --action drop --dba root --dba-password contribs.org
  
  </nowiki>

93.-make-patches.sh

 <nowiki>#!/bin/bash
  
  diff -uN /etc/aliases{.orig,} &gt; patch.aliases
  diff -uN /etc/hosts{.orig,} &gt; patch.hosts
  diff -uN /etc/rt/RT_SiteConfig.pm{.orig,} &gt; patch.RT_SiteConfig.pm
  diff -uN /etc/httpd/conf.d/rt.conf{.orig,} &gt; patch.rt.conf
  diff -uN /etc/httpd/conf/httpd.conf{.orig,} &gt; patch.httpd.conf
  diff -uN /etc/sysconfig/httpd{.orig,} &gt; patch.httpd
  diff -uN /etc/mail/sendmail.mc{.orig,} &gt; patch.sendmail.mc
  diff -uN /etc/mail/virtusertable{.orig,} &gt; patch.virtusertable
  diff -uN /etc/mail/local-host-names{.orig,} &gt; patch.local-host-names
  diff -uN /var/rt/home/.procmailrc{.orig,} &gt; patch..procmailrc
  
  
  </nowiki>

--FILE CHANGES: MADE AFTER BASE CENTOS 4.2 SERVER INSTALL...

 <nowiki>--- /etc/hosts.orig     2006-01-20 15:26:15.000000000 -0500
  +++ /etc/hosts  2006-01-20 00:23:45.000000000 -0500
  
  +192.168.62.21          rt.mynetwork.us rt
  +192.168.62.21          www.mynetwork.us www
  
  --- /etc/aliases.orig   2006-01-20 15:17:05.000000000 -0500
  +++ /etc/aliases        2006-01-20 15:15:07.000000000 -0500
  
  +
  +# specify a program to run when mail comes to address correspond
  +correspond: "|/usr/bin/rt-mailgate --queue General --action correspond --url http://rt.mynetwork.us/
  
  --- /etc/sysconfig/httpd.orig   2006-01-05 13:34:38.000000000 -0500
  +++ /etc/sysconfig/httpd        2006-01-19 11:31:49.000000000 -0500
  -#OPTIONS=
  +OPTIONS=-DPerlStatus
  
  --- /etc/httpd/conf/httpd.conf.orig     2006-01-05 13:34:38.000000000 -0500
  +++ /etc/httpd/conf/httpd.conf  2006-01-19 13:17:41.000000000 -0500
  
  +ServerName www.mynetwork.us:80
  
  --- /etc/mail/local-host-names.orig     2005-02-21 11:14:29.000000000 -0500
  +++ /etc/mail/local-host-names  2006-01-20 14:38:25.000000000 -0500
  
  +rt.mynetwork.us
  +www.mynetwork.us
  
  --- /var/rt/home/.procmailrc.orig       2004-05-10 14:52:51.000000000 -0400
  +++ /var/rt/home/.procmailrc    2006-01-17 04:52:04.000000000 -0500
  
  -MAILDOMAIN=rt.yourdomain.com
  +MAILDOMAIN=rt.mynetwork.us
  
  -RT_URL="http://rt.yourdomain.com/"
  +RT_URL="http://rt.mynetwork.us/"
  
  --- /etc/httpd/conf.d/rt.conf.orig      2006-01-12 23:04:31.000000000 -0500
  +++ /etc/httpd/conf.d/rt.conf   2006-01-19 23:04:12.000000000 -0500
  
  -&lt;VirtualHost your.ip.address:80&gt;
  -    ServerName your.rt.server.hostname
  +&lt;VirtualHost 192.168.62.21:80&gt;
  +    ServerName rt.mynetwork.us
  
  -&lt;VirtualHost your.ip.address:443&gt;
  -    ServerName your.rt.server.hostname
  +&lt;VirtualHost 192.168.62.21:443&gt;
  +    ServerName rt.mynetwork.us
  
  --- /etc/rt/RT_SiteConfig.pm.orig       2006-01-12 23:04:30.000000000 -0500
  +++ /etc/rt/RT_SiteConfig.pm    2006-01-19 23:39:33.000000000 -0500
  
  -Set( $rtname, 'example.com');
  +Set($rtname, "network_us");
  
  +Set($Organization , "rt.mynetwork.us");
  +Set($Timezone, 'US/Eastern');
  +Set($WebBaseURL, "http://rt.mynetwork.us");
  +Set($WebPath, "/");
  +Set($WebURL , $WebBaseURL . $WebPath . "/");
  +
  +Set($CorrespondAddress, 'correspond@rt.mynetwork.us');
  +Set($CommentAddress, 'comment@rt.mynetwork.us');
  +Set($SendmailPath, "/usr/sbin/sendmail");
  +Set($SendmailArguments , "-oi -t");
  +
  +Set($LogToSyslog, '');
  +Set($LogToFile, 'debug');
  +Set($LogDir, '/var/log/rt');
  +Set($LogToFileNamed , "rt.log");
  +
  +Set($OwnerEmail, "admin@mynetwork.us");
  +Set($MyTicketsLength, 20);
  +
  +Set($DatabasePassword , 'contribs.org');
  +
  +Set($AmbiguousDayInPast , 0);
  +
  
   1;
  
  --- /etc/mail/sendmail.mc.orig  2005-02-21 11:14:29.000000000 -0500
  +++ /etc/mail/sendmail.mc       2006-01-17 04:27:10.000000000 -0500
  
  -DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl
  +dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl
  
  --- /etc/mail/virtusertable.orig        2005-02-21 11:14:29.000000000 -0500
  +++ /etc/mail/virtusertable     2006-01-17 04:56:41.000000000 -0500
  
  +#
  +# This will handle all aliases for all queues. Note that this implies
  +# that all queue address are @rt.mynetwork.us
  +#
  +# Here put your exceptions. Next one made this domain RFC822 compliant.
  +postmaster@rt.mynetwork.us   postmaster
  +
  +# leave root as 'root'
  +root@rt.mynetwork.us   root
  +
  +# everything else goes to user 'rt'
  +@rt.mynetwork.us   rt
  
  
  </nowiki>

-Hasan Muhammad

=== Initiation error when starting httpd service, Starting httpd: No root path(s) specified at /usr/bin/webmux.pl line 98 ===

I just wanted to add to this. At least in my case (Linux OS, apache-perl, Apache1/modperl1), this error was being caused by the following line in webmux.pl:

 rmtree([ bsd_glob("$RT::MasonDataDir/obj/*") ], 0, 1);

"$RT::[[MasonDataDir]]/obj" is a directory, in my case it is this:

 /var/cache/request-tracker3.4/mason_data/obj

On startup, there are no non-hidden files/directories in this directory. Apparently bsd_glob returns the empty result, and rmtree chokes on that with the "No root path(s)" error. As a workaround, I edited webmux.pl to replace that rmtree line with this:

 my @adir = bsd_glob("$RT::MasonDataDir/obj/*");
 if (@adir) {
     rmtree([ @adir ], 0, 1);
 }

That got rid of the "No root path(s)" error message.

- [[MaryaDoery]]

3.6.0RC3 has it fixed.

=== Continuous login ===

I encountered a problem with RT asking me to log in every time I clicked a link. After doing some research, it seemed that the cookie was being sent and set properly. There were two possibilities mentioned:

* A [[MySQL]] 5.x issue: check your mysql error logs. The session module has been seen to try to connect as [mailto:rt_user@localhost rt_user@localhost] or [mailto:rt_user@127.0.0.1 rt_user@127.0.0.1]<nowiki> instead of rt_user@'' </nowiki>
* A table structure issue: take a look at the "sessions" table in the RT database. There should be a column named "a_session". This column must have the type "longblob" not "longtext". Also note that if there are any records in the table, the entire table should be emptied (either before or after making the change), otherwise the problem will persist.

I personally hit the table structure problem while installing RT 3.6.3 on Gentoo Linux, but since fixing that, everything has been fine.

- kwgagel

I hit this problem after upgrading from 3.8.4 to 3.8.7. Kevin Falcone from Best Practical suggested the removal of my /local/html/autohandler file. I renamed the file and restarted the httpd service and the problem vanished.

- [[DaveIngram]]

I hit the table structure issue with 3.6.5 on Fedora 8, deleting the table and recreating as longblob fixed. Thanks Dave.

I found the same thing after running the upgrade script from 3.6.1 &gt;&gt; 3.8.2, change the data type in the db and set the default value to NULL and now is working. Thanks Dave!

-- [sorry the html goes nuts here: will fix shortly. Rob101]

The [[MySQL]] SQL I used to correct this problem with RT 3.6.7 on Gentoo was:

 <nowiki>mysql&gt; desc sessions;
   +-------------+-----------+------+-----+-------------------+-------+
   | Field       | Type      | Null | Key | Default           | Extra |
   +-------------+-----------+------+-----+-------------------+-------+
   | id          | char(32)  | NO   | PRI | NULL              |       |
   | a_session   | longtext  | YES  |     | NULL              |       |
   | LastUpdated | timestamp | NO   |     | CURRENT_TIMESTAMP |       |
   +-------------+-----------+------+-----+-------------------+-------+
   3 rows in set (0.00 sec)
   
   mysql&gt; alter table sessions modify a_session longblob default NULL;
   Query OK, 215 rows affected (0.02 sec)
   Records: 215  Duplicates: 0  Warnings: 0
   
   mysql&gt; delete from sessions;
   Query OK, 215 rows affected (0.00 sec)
   
   mysql&gt; desc sessions;
   +-------------+-----------+------+-----+-------------------+-------+
   | Field       | Type      | Null | Key | Default           | Extra |
   +-------------+-----------+------+-----+-------------------+-------+
   | id          | char(32)  | NO   | PRI | NULL              |       |
   | a_session   | longblob  | YES  |     | NULL              |       |
   | LastUpdated | timestamp | NO   |     | CURRENT_TIMESTAMP |       |
   +-------------+-----------+------+-----+-------------------+-------+
   3 rows in set (0.00 sec)
   
   mysql&gt;
   
   </nowiki>

Sorry if dumping this here was too verbose!

Rob101 --

It sure wasn't Rob. I confirmed that this was the answer to a problem for me going from 3.6.x to 3.8.x ... it's feasible that something about my upgrade broke the database upgrade scripts. Either way -- if you are having problems with RT logins happening too much -- the above seems to be the fix! -- [[MGMead]] (PS - I cleaned up the MYSQL session so you can read it more easily)



Happenned to me after restoring a (Postgres) dump from another computer (same RT version, same PG version). Restarting Apache was the solution. RT 4.0.7 on Postgres 9.1, apache2/modperl.

-- JC Boggio

=== The dreadful MySQL server has gone away error ===

I was seeing this error almost every time I create a ticket in one of my queues. If I insisted at least 5 times, the ticket was finally created.

Searching about this I discovered that the most common reason for this kind or error is data base corruption. I tried checking, repairing and optimizing the database using the mysqlcheck command, but none solved the problem. To be sure I dumped all rt3 database (using mysqldump) e recreated the whole database, but that too did not solve the problem. Nevertheless, some people really got rid of this error using those methods and those could work for you.

After enabling debug logging on [[RT SiteConfig|RT_SiteConfig]], I saw that the error was occurring much sooner than I thought, because the web interface was only showing the error when the Apache::Session::Lock::[[MySQL]] object was being destroyed.

Actually, the problem was occurring right after an Scrip [[SendMail]] action and in the logs I could see RT complain about sendmail exiting with status 16384. I checked the scrip saw that the template used had the following in it's first line: To: [mailto:someemail@somedomain.com someemail@somedomain.com];[mailto:otheremail@somedomain.com otheremail@somedomain.com]

I removed the “;[mailto:otheremail@somedomain.com otheremail@somedomain.com]” part and the problem was gone. I found that to be super strange and totally unrelated, but if you got that error, don't forget to check your templates. I guess there is really a reason for the [[SendMail]] Scrip Action not being enabled by default.

- dante.

=== Config file /etc/request-tracker/RT_SiteConfig.pm is locked ===

(BTW are _all_ of these Perl-related? Or are we just putting them under that heading by default?)

Got this error ("Config file /etc/request-tracker/RT_SiteConfig.pm is locked") when I fired up request-tracker this morning. Since I had no idea what the cause could be, I asked Google. All the forum entries it pointed me to said that it's because request-tracker can't access MySQL. But on my system this wasn't the case -- MySQL was running fine.

Checking the logs, I found this intriguing message:
<pre>The WebBaseURL config option requires no trailing slash (/usr/lib/perl5/vendor_perl/5.16.0/RT/Config.pm:708)
</pre>

I checked my <b>RT_SiteConfig.pm</b> and, sure enough, I had a trailing slash in the URL. After removing it and restarting Apache, the error went away.

- smithfarm