I downgraded the version of OpenLDAP on KE to cure a segfault problem in the PAM LDAP module.

The PAM LDAP module periodically caused a segfault in any program that uses it when it was paired with the 2.0.21 version of OpenLDAP. I grabbed the package from HEIWA and force-removed and downgraded it on KE, and the segfault problem seems to have gone away.

The known way to reproduce the problem was using sudo -v. This would always segfault with 2.0.21, and it never segfaults with 2.0.14.

Since we’re not using OpenLDAP on these servers for anything besides its library for the PAM module, I believe this is safe enough.

I have compiled and installed the pam_ldap module on KE to help us with the authentication difficulties.

FreeRADIUS was failing under the load of authentications this morning, since it was running in single threaded mode on SHANTI. To get around this, I found and compiled the FreeBSD package for the pam_ldap PAM module. This shifts the bulk of our network authentications (e-mail) from RADIUS to directly querying LDAP.

Unfortunately, it seems that the SSL portions of pam_ldap aren’t happy on KE, even though it worked fine on my workstation. Nevertheless, I believe we have a relatively stable authentication system at the moment.

Update

SSL is working fine now. It required the setting host directory.earlham.edu rather than the IP address so that it could verify the certificate. I tested this on HEIWA, and now it, too, is using pam_ldap in place of RADIUS.

RADIUS is now entering passwords into Samba for those that have never reset their passwords to log in to the Samba Windows domain.

Due to numerous complaints from people who hadn’t ever changed their passwords, and thus cweren’t ever active in the Samba password file, I rewrote part of the rlm_smb module in RADIUS. It now checks against LDAP first, and if it succeeds there, it changes the Samba password with the supplied password.

The addition to the RADIUS module is a couple of lines of C code that call a perl script. The script does the LDAP checking and Samba password updating. It’s a somewhat kludgy system, but it is working. I put the RADIUS changes in the FreeRADIUS package that I installed on SHANTI, and the perl script is /usr/local/libexec/radldapsmb.pl, which must be called as root (and thus the RADIUS server must run as root, which it was anyway).

As a result, anyone successfully authenticating now can log in to the Windows 2000 computers without having to go through a password reset.

Update

While RADIUS is still working in the mode described here on SHANTI, I have shifted the bulk of authentications (which are e-mail logins from KE) to querying LDAP directly with the pam_ldap module. The above modifications required FreeRADIUS to run in single threaded mode, which was insufficient for the load placed on it by KE. As a result, it was dropping RADIUS request packets and causing login failures. I will keep RADIUS running in this mode for the time being, as it is still useful to have the Samba auto-update feature. However, I suspect we will see a shift away from RADIUS authentication towards direct LDAP authentication.

Oh, right. The UPS has a new emergency bypass switch.

Not much to say here. Lee Higgs from Melling installed it, and Mike Kammer from Liebert verified it. The UPS twitched slightly a couple of times when large loads were put on it (RAID arrays or the Black Diamond), but is doing fine now.

Be sure to turn the back switch from UPS to BYPASS before turning the bypass switch on the wall. Otherwise, everything's quite nice.

After booting this morning, PAX complained of bad memory and failing drives on the PowerVault array.

We took out the first four 256MB DIMMs and moved the remaining four into the first slots. The first four came with the system and the remaining four were added some time later. It was happy with the memmory at that point, but the PV drives were still unhappy.

The PERC BIOS claimed that drives 3 and 5 on the PV220S were “different” somehow, and put the container into critical. This was unacceptable, since there were no alert indications on the PowerVault unit itself and two drives out of a RAID-5 system is bad news. Poking at the PERC BIOS didn’t seem to change things. We talked with Dell tech support for quite a while, during which they had us verify the drives from the PERC BIOS. The two “failed” drives were fine, but drive 8 had a couple of bad sectors. We left tech support with the suggestion to upgrade the PERC firmware and see if that made a difference. If not, our only hope was to delete the container, recreate it, and restore from tape.

Upgrading to version 2.7 of the PERC BIOS changed the error. Now it stated that drive 8 (the one with bad sectors) was changed or missing, or somesuch. It allowed us to verify the drive and start a rebuild of the array. The rebuild is currently ongoing and the data is intact.

Update

The PowerVault array finished rebuilding successfully at approximately 7:00 AM Sunday morning.

Samba and FreeRADIUS are now using LDAP (on ASHTI) as their authentication store.

The biggest things here were the migration of all accounts in /etc/passwd to LDAP on ASHTI. This went through pretty easily with the scripts in my directory: ~rowan/smbldap/MigrationTools-44.

Upgrading Samba to LDAP support was straightforward. The new configuration variables in smb.conf are required, and I changed the Unix passwd sync to the resetpass program, which changes the LDAP password and the Seminary password as well as the Unix password. Then Samba can go ahead and change the Samba password. This works well except when the seminary server is unavailable, in which case the Unix and LDAP passwords are changed, but Samba isn’t (and the seminary one isn’t of course, as well).

FreeRADIUS is at version 0.7 currently, but we seem to be having issues with the LDAP module periodically losing its connection to ASHTI. I‘m working on this. Version 0.8.1 of the server might help if I can’t get anything else to work. I changed from TLS enabled LDAP to unencrypted. I might also try using the SMB authentication module, which I happened to compile into the package I installed. I am now currently using the SMB module to authenticate against Samba on SHANTI (which is authenticating against the Samba account fields in LDAP). I’m still not sure what’s wrong, but I can’t leave it periodically dying.

As you can tell, MovableType is now at version 2.63.

I rewrote the changes I made for the authentication system, making it slightly more extensible. We can now choose between IMAP, LDAP, and LDAPS for authentication back ends in the mt.cfg file.

I also added a bunch of new plugins. I'll detail these later.

I installed 1 GB RAM and updated versions of MIMEDefang, SpamAssassin, and PHP on KE today.

• RAM Addition: I installed 1 GB of RAM, increasing the total to 2 GB and using up all four DIMM slots. 512 MB of the new RAM is being used as a memory filesystem mounted on /var/spool/MIMEDefang. This is speeding up mail submissions considerably, since all MIMEDefang work and Vexira antivirus scanning is now essentially done in RAM. The 512 MB size of the filesystem is fixed due to limitiations in FreeBSD, however this should be plenty for these purposes (by comparison, the old spool directory had approximately 600 MB free and much of the space used there is for the /var/log directory).

• I upgraded MIMEDefang to version 2.33. I apparently hadn’t installed the latest version of my mimedefang-filter script, so I copied that from my workstation (to get native Vexira support).

• SpamAssassin is now at version 2.54, which should be much more accurate than the year-old previous version. This also includes the Bayes extensions, which are currently turned off.

• I upgraded to PHP 4.3.1 (the latest version in the FreeBSD ports tree). This required the addition of the pth package. I also changed the default configuration to have register_globals be ON (to keep SquirrelMail happy).

The firmware on the Cyclades TS2000 terminal server was upgraded from 1.3.4 to 1.3.6 today.

Some problems arose with the RADIUS authentication: the files /etc/raddb/server, /etc/pam.conf, and /etc/motd were not in the config_files list and were not saved. Changing these back to appropriate versions worked.

Point to remember: it is a good idea to tar up the /etc directory and save it on another system before upgrading the firmware. This gives a good backup of all the configuration files in case the upgrade procedure touches more than the release notes say it will.

Rather than using the FTP download for the upgrade images, I downloaded them to my workstation (via FTP) and then copied them to the TS2000 using scp from the console.

I have done the final OS install and directory server setup on ASHTI. It is ready for migration.

ASHTI (SunFire V120) is going to be the new LDAP directory server. I have performed the final Jumpstart using Solaris 9, Directory Server configuration. See ASHTI's TWiki page for specifications of this machine.

The Sun ONE Directory Server is configured as server ID ashti and has an SSL certificate for directory.earlham.edu (which is going to be the official CNAME for LDAP). It is all ready to go; at this point we need to finalize the import procedure from SHANTI's flat files before we can do the final cutover.

Other things that need to be set up:

• AMANDA backups (make it join the ROJ cluster, most likely)
• Automatic exports of the directory server. This can probably be set up as a cron job.

resetpass is being updated to support the seminary servers and add support for LDAP password changes.

The new version adds several things:

1. Remote password changing program: the remote program is a simple command line password changer that is run on a remote host through an SSH tunnel. This requires null-passphrase SSH keys to be set up between SHANTI and the remote server (in this case Bathsheba). Note that the SSH keys and host information must be in root's .ssh directory since resetpass runs as root under sudo.
2. LDAP password changing: it is possible to specify an LDAP server and the appropriate DN and password to reset or change someone's password in an LDAP directory. If LDAP support is specified, the initial authentication check is done via LDAP binds rather than getpwnam lookups.
3. New password specification: this script will take the place of passwd in Samba's system password synchronization, and as such needs to be able to accept a new password specified by the user. The -n argument signals this. If given, resetpass will prompt for the new password rather than generating a random one.

Other minor new features include a command line argument to specify the configuration file (necessary to support two configuration files: one for password resets via the web interface and one for password changes initiated from Samba), checking of the $SUDO_USER environment variable (necessary to work out some kinks with the passwd account and admin users restarting Samba with sudo), and support for password change notifications (another change module: uses a local program to "notify" password changes - could, for example, send e-mail with the user's username to a specified address, or other nasty insecure things - don't use it).

Zach and Steve Spyker have been testing this version for seminary changes with good success. I suspect it will go live on Monday for general purpose resets. It will go live for Samba resets when we move to LDAP authentication (and put in the new Samba).

I'm doing some exploration of mail routing using LDAP.

The Email and LDAP chapter of O'Reilly's LDAP System Administration book is available free in PDF format. It covers how to set up e-mail clients to access LDAP servers for address book information and also how to configure three popular MTAs to use LDAP for map and routing information.

The chapter assumes that one is working with OpenLDAP. As we will be using the Sun ONE Directory Server, some things need to change. Particularly for the Sendmail routing information, the object class we should use is already built in to the DS: mailRecipient. This object class does not define the attribute mailLocalAddress, but I believe we can simply use the mail attribute and put the same value in it (it should be the canonical e-mail address of the user, in either case).

The routing support may come in handy when (if?) we move seminary users to receiving mail on the seminary servers.

I just reset /etc/fstab to mount all of PAX's NFS exports on HEIWA using TCP rather than UDP.

TCP should give us better reliabilty (not that it's been a problem, except when we put packet filters in the Black Diamond...).