iVia

Chapter 16: Troubleshooting

This chapter discusses some of the problems reported with the iVia software and how they are solved. The following table describes their symptoms, with links to the likely solutions below.

For each solution, we describe the level of access required; this will either be "root" (i.e. you require root privileges) or "iVia user" (i.e. you need to be logged in as the user who installed iVia).

Inverted Index Corruption

Access required: iVia user

If every "canned_search" returns zero results but the regular web pages and "What's new" pages continue to work then there is almost certainly a problem with the inverted indexes. Here’s what you should try to fix things as user iVia.

See Chapter 12: The iVia Databases for a description of the inverted index databases.

1. Reset the Lock and Memory Pool Files

The most common problem with the inverted indexes at the moment is that the lock and memory pool files can become corrupt or have permission problems. The files in question are
/home/ivia/locks/__db.001,
/home/ivia/locks/__db.002 and
/home/ivia/locks/__db.003.

If these files don't exist, it may be because they have recently been "cleaned up" by a maintenance program. In this case, you can recreate them by running the data dumper command (e.g. word_level_data_dumper tree.)

The $HOME/bin/check_berkely_env.sh program checks these files and attempts to fix them if there is a error. If the searches start working again, then your problems are solved.

Heavy-traffic sites may want to set up a cron script to run this program every 3 minutes, so these problems often fix themselves after a minute or two. (Memory pool corruption is one of the biggest sources of instability on the INFOMINE site. It apparently occurs because the Berkely DB shared memory pool management doesn't behave well under high loads, and can run out of memory, in which case it simply dies, leaving an unusable mess behind.)

If the problem persists, the first thing to try is to simply delete the __db.* files. They will be recreated automatically.

2. Replace the Inverted Index Files from Backups

If repairing the __db.* files doesn't work, then there is probably an error with the inverted index files themselves. In order to quickly get iVia working again, you can copy an old version of the indexes over the current version. You can find the previous backup by changing to the backup_dir identified in the etc/iVia.conf file, and looking in the 1 subdirectory.

You should then immediately run check_berkely_env.sh.

If you do elect to replace the existing files with the backups, use a sequence of operations like these:

 cd ~/data
 cp Backup/1/stemmed.db temp.db
 mv temp.db stemmed.db
 cp Backup/1/unstemmed.db temp.db
 mv temp.db unstemmed.db
 rm __db.*

This sequence is necessary because we don’t want the web server trying to access a database when it is half copied (with "cp"), and a move (with "mv") prevents this from happening.

This is a temporary solution that will get the searches running again. In order to fix things properly, you will need to rebuild the inverted indexes.

3. Rebuild the Inverted Index Files

When all else fails, you may have to rebuild the inverted index files. See Chapter 12: The iVia Databases for a description of the databases. Note that the inverted indexes are normally rebuilt every night (this is when the backup described above is created).

To regenerate the inverted index (as the ivia user), run the create_inverted_index.sh command. Afterwards, the inverted index files the data directory will be replaced and the lock/memory pool files will hopefully have been reset for you.

Problems with Commands and Scripts

Access required: iVia user

If you’re trying to run commands like create_inverted_indexes.sh or fix_file_permission.sh and they will not run, then it may be that the iVia bin directory is not on the path. The iVia bin directory is required because many of the scripts make use of two helper applications: ivia-config and read_ini_file. These are used to find the parameters of your installation, and if they are not available, the scripts usually exist with an error.

Apache Configuration Problems

Access required: root

If the CGI scripts are not working, you will have to figure out what is wrong and fix apache. The best tests for this condition are to click on "What’s new" (which runs cgi-bin/whats_new) or a category name (runs cgi-bin/search) from the front page, and check the apache error log.

MySQL not set up Correctly

There are several reasons why the iVia user may be denied access to connect to the server:

1. The MySQL server may not be listing on a network port for database connections. iVia uses the network port even when the MySQL server and iVia are installed on the same machine. In MySQL prior to version 3, you must make sure the "skip-networking" line in /etc/mysql/my.conf is commented out. MySQL version 4 has a new option, bind_address.
2. If you have a separate database server, the inetd daemon (which controls connections across the network to a machine) may be preventing the iVia server from reaching the MySQL machine. To test this, try using the telnet program to connect from the iVia server to the MySQl port (look in the MySQL configuration file, my.conf, so see what port this is). If you cannot connect at all, you may need to instruct inetd to allow the iVia server to connect to the MySQL machine by updating its /etc/hosts.allow file.
3. The iVia user may be denied access to the iVia database on the grounds that they do not have permission to connect. At that point, I recommend you find a book on securing your MySQL server.

MySQL Database Problems

Access required: root

If the view_record or whats_new programs fail, or if the canned_search program outputs the top part of the page including the number of results but does not output any result data, then there is likely to be a problem with iVia's connection to the MySQL database.

The first thing to check is that your MySQL installation is running at at all. If it is not, restarting it will probably solve your problem.

Next you should make sure that MySQL and iVia are configured correctly so that the iVia user can connect to the MySQL database; this is discussed in the Installation Chapter and a couple of problems are discussed above.

If MySQL is running, and the iVia user can connect, but MySQL appears to be hanging (no responses are forthcoming), then the database or table may be locked. To find out if this is the case, log into the MySQL server and run mysqladmin processlist. If many processes are listed and marked "Locked" then MySQL is waiting for some other query or update to finish before responding to the other commands. You can generally see from the processlist which command is causing the problem; you have the option of killing it with mysqladmin kill.

Sometimes (particularly in crawlers or the site checker, you may see errors like this one:

mysql_real_connect failed (host: [your MySQL page cache server], username: ivia, database: ivia): Too many connections.

This is caused by iVia attempting to make too many connections to MySQL. In some programs, iVia may try to make hundreds of connections in parallel to speed up its page cache download processes (usually there will be a configurable parameter for such programs that is called "web_page_fanout" or similar). The solution is to update your MySQL server to accept more connections. For heavily used installations, we use the following lines in the /etc/mysql/my.conf file:

max_connections      = 10000
max_user_connections = 8000

Login Verification Problems

Access required: root

The programs used by the adders (available through http://[hostname]/adders, and stored in the secure_scripts directory) are password protected and require a login. This login is performed by checking the adders user name and password against the adders_info table of the main iVia database on the installations MySQL database server.

If people are unable to log in, the apache error log and the MySQL error log will tell you why. Generally, it means either that their password is being entered incorrectly or there is a database problem verifying it.

In the case that it is a password problem, you can fix it through the Adders Web Site if you have User management access.

Database verification problems sometimes happen when iVia is looking on the wrong server for the MySQL database. In Apache 1.2, this server is usually set near the end of /etc/apache/httpd.conf in the section describing the secure_scripts directory, where these lines appear:

 
 # Apache 1.2 configuration.
 Auth_MySQL_Info db.example.org www-data [encrypted-password]

 [...]

 AuthName "DB Management"
 AuthType Basic
 require valid-user

 Auth_MySQL_DB ivia
 Auth_MySQL_Password_Table  adders_info
 Auth_MySQL_Username_Field  remote_user
 Auth_MySQL_Password_Field  password
 Auth_MySQL_Empty_Passwords off

You need to make sure that the AuthMySQLInfo is set correctly. Password problems are a real headache. Remember passwords have to be encrypted with the MySQL PASSWORD command, not with the standard Unix crypt.

Stray record_editor_submit Processes

Access required: root

You may log in to the iVia server and notice lots of old record_editor_submit processes that have not finished. There may also be record_editor_delete and replace_field processes. When the user "submits" a new/changed record in the Record Editor, the record is added to the main iVia database, then a background process is forked off to update the inverted index database. This background processes will sometimes hang if the inverted index is corrupted.

The first thing you can do is to get rid of the processes. Log in as root and run killall record_editor_submit. You can get rid of any record_editor_delete and replace_field processes in the same way.

Next, you have to figure out what is causing the processes to hang. Generally, if canned_search can be run, then the problem was with the inverted index and has since been solved. (The hanging processes should expire after six hours.)

Compilation

Access required: iVia user

If iVia or libiViaCore will not compile, make sure you’re using g++ version 3 or later, but not g++ version 4 (yet). To force configure to use a particular compiler, you can use the CXX variable before running configure. For example, in bash, you could try running export CXX=g++-3.3 first.

We compile with -Werror, which means that warnings are treated as errors. Some compilers may generate warnings that we have not observed, and treat them as errors. (For example, there were problems with early versions of g++ version 3.2 and no_return attributes.) To work around this problem, remove -Werror in the definition of CXXFLAGS in configure.ac, run congfiure again, and then make install as usual.

© 2007 University of California, Riverside