Changes To Generic NQS v3.50.0

Abstract

This document contains a summary of the changes made to produce Generic NQS v3.50.0.

Contents

• Introduction
• Administrators: New Features
• Administrators: Changes To Existing Features
• Administrators: Fixes For Problems
• List Of Changes For Developers
• Other Credits

Introduction

We are most grateful for the contributions made by other individuals and organisations.

About This Release

• Portability
• Robustness
• Ease of installation

We recommend that all sites running Monsanto NQS, or Generic NQS v3.4x, should upgrade to this version. There will be no support for any previous versions of Monsanto-NQS or Generic NQS available from us.

Compatibility

• All message logging (and debugging output) now goes via the syslog mechanism. Generic NQS currently uses `local0' as the facility it logs to (this is configurable at compile-time). The old NQS logdaemon (and its logfile) are no longer used or supported.

  System administrators will have to ensure that their syslog daemon does not discard messages which come from the `local0' facility.

• Pipe queues and queue complexes have been modified so that a pipe queue can be a member of a queue complex. Sites which use either pipe queues or queue complexes will have to remove their existing NQS installations, and re-install from scratch.

  This is an unfortunate side-effect of the way NQS stores its own information. Hopefully, this will be the last file format change for a very long time.

• Support for resource limits in terms of words, kilo-words, mega-words, and giga-words, has been removed.

  The size of a `word' varies so much from host to host that a meaningful comparision is not possible.

• The number of inodes used to hold transaction state information has been doubled.

  The side effect of this change is that any requests which have been queued will be corrupted when you install this version. Our advise is to use pipe queues to syphon queued requests to another NQS node; shutdown NQS, upgrade to this release, then use pipe queues to move the requests back onto this machine (courtesy of Thomas Eifert).

Project Conclusion

The University of Sheffield has agreed to continue to host the Generic NQS World-Wide Web site, and FTP site, at least until the end of June, 1997. In addition, they will continue to maintain the existing Mailbase mailing lists.

Stuart Herbert is leaving the University for a job in industry. However, thanks to the loan of equipment from the University, he will continue to act as world-wide maintainer of Generic NQS and its related technologies. This will be done in his spare time, and it is plainly obvious that he won't have anywhere near as much time to spend on Generic NQS as he did when he was employed full-time to support it, so please bear with us for a month or two until things settle into a new routine.

Administrators: New Features

SETUP includes a number of automatic tests, to determine a number of compile-time constants which previously were supplied by Makefiles. One of these tests is to actually determine what type of computer you are trying to install Generic NQS on.

This software is undergoing testing.

All platforms are affected.

This code was contributed by Stu.

Cluster-Wide Dynamic Scheduling Added

To use dynamic scheduling, add all of the pipe queues on your scheduling host to a queue complex, and set the user_limit for that queue complex to something suitable. Users who submit more jobs than the complex's user_limit will find their excess jobs being deferred in preference to jobs from other users, as is the case with dynamic scheduling across batch queues.

This code has not been tested.

All platforms are affected.

This code was contributed by Stu, and is based on Dave Safford's dynamic scheduling for batch queues.

Processor Set Support For Digital UNIX Added

These changes have not been tested.

Platforms affected :

> [ ] AIX 3			[ ] AIX 4
> [ ] DYNIX/PTX			[ ] FUJITSU
> [ ] HPUX 8			[ ] HPUX 9
> [ ] HPUX 10			[ ] IRIX 5
> [ ] IRIX 6			[ ] LINUX
> [ ] NCR			[x] OSF/1
> [ ] SOLARIS 2			[ ] SUNOS 4
> [ ] ULTRIX			[ ] UNICOS

Prologue/Epilogue Scripts Feature Added

One possible use of this facility is support for restricted processors on IRIX; the prologue script could ensure that, for a given queue, a given processor has been restricted (using mpadmin), and the epilogue script could unrestrict the processor once the request has run to completion.

This code has not been tested.

All platforms affected.

Code contributed by Stu.

Support For Dynix Added

This change has been tested.

Support For HP-UX v10 Updated

These changes have not been tested.

Platforms affected :

> [ ] AIX 3			[ ] AIX 4
> [ ] DYNIX/PTX			[ ] FUJITSU
> [ ] HPUX 8			[ ] HPUX 9
> [x] HPUX 10			[ ] IRIX 5
> [ ] IRIX 6			[ ] LINUX
> [ ] NCR			[ ] OSF/1
> [ ] SOLARIS 2			[ ] SUNOS 4
> [ ] ULTRIX			[ ] UNICOS

Administrators: Changes To Existing Features

I recommend that you update your syslog.conf file so that all messages from `local0' are logged to a single file. I have the following entry in my syslog.conf file (on Linux) :

>  local0.*		/var/log/nqs

All platforms are affected.

This code was contributed by Stu.

Debugging Levels Rationalised

• Level 0: Fatal errors only.
• Level 1: Level 0 + information messages.
• Level 2: Level 1 + high-priority debugging messages.
• Level 3: Level 2 + medium-priority debugging messages.
• Level 4: Level 3 + low-priority debugging messages.
• Level 5: Level 4 + temporary debugging (trace) messages.

Upon installation, Generic NQS defaults to level 1. You can change the debugging level using the `qmgr set debug' command.

This code has been tested.

All platforms are affected.

This code was contributed by Stu.

More Machine ID's Supported

This code has not been tested.

Code contributed by Stu, thanks to Mark Loveridge.

Pipe Queues Can Now Be Part Of Queue Complexes

This code has not been tested.

All platforms are affected.

This code was contributed by Stu.

Support For Word-Size Limits Removed

This code has not been tested; requests which contain one of these units may cause NQS to crash. Further work is anticipated before this change is complete.

All platforms are affected.

This code was contributed by Stu.

Staging Support For Pre-Releases

This code has not been tested.

All platforms are affected.

This code was contributed by Stu.

Administrators: Fixes For Problems

> [ ] AIX 3			[ ] AIX 4
> [ ] DYNIX/PTX			[ ] FUJITSU
> [x] HPUX 8			[ ] HPUX 9
> [ ] HPUX 10			[ ] IRIX 5
> [ ] IRIX 6			[ ] LINUX
> [ ] NCR			[ ] OSF/1
> [ ] SOLARIS 2			[ ] SUNOS 4
> [ ] ULTRIX			[ ] UNICOS

Contributed by Michael Andrews.

Compilation Problems On HP-UX v9

Platforms affected :

> [ ] AIX 3			[ ] AIX 4
> [ ] DYNIX/PTX			[ ] FUJITSU
> [x] HPUX 8			[x] HPUX 9
> [ ] HPUX 10			[ ] IRIX 5
> [ ] IRIX 6			[ ] LINUX
> [ ] NCR			[ ] OSF/1
> [ ] SOLARIS 2			[ ] SUNOS 4
> [ ] ULTRIX			[ ] UNICOS

Compilation Problems On IRIX

> [ ] AIX 3			[ ] AIX 4
> [ ] DYNIX/PTX			[ ] FUJITSU
> [ ] HPUX 8			[ ] HPUX 9
> [ ] HPUX 10			[x] IRIX 5
> [x] IRIX 6			[ ] LINUX
> [ ] NCR			[ ] OSF/1
> [ ] SOLARIS 2			[ ] SUNOS 4
> [ ] ULTRIX			[ ] UNICOS

Jobs Stuck In The Arriving State

Code in the machine id database library has been replaced in order to attempt to resolve this problem.

This code has not been tested.

All platforms are affected.

This code has been contributed by Stu, based on a contribution by Mark Whidby (M.Whidby@mcc.ac.uk).

Incorrect Resource Limits

The implementation of resource limits has been completely re-written from scratch; the new implementation should cure the problem. UNICOS users will, however, have to wait for the next pre-release before their resource limits are correctly supported.

This code has been partially tested.

All platforms are affected.

This code was contributed by Stu; the UNICOS support is based on the previous support written by Dave Safford.

Device Queues And Queue Complexes

The implementation of device queues and queue complexes has been completed, so that all operations on device queues and queue complexes are safe.

This code has not been tested.

All platforms are affected.

This code has been contributed by Stu.

Memory Leak On HP-UX

Platforms affected :

> [ ] AIX 3			[ ] AIX 4
> [ ] DYNIX/PTX			[ ] FUJITSU
> [x] HPUX 8			[x] HPUX 9
> [x] HPUX 10			[ ] IRIX 5
> [ ] IRIX 6			[ ] LINUX
> [ ] NCR			[ ] OSF/1
> [ ] SOLARIS 2			[ ] SUNOS 4
> [ ] ULTRIX			[ ] UNICOS

Code contributed by Stu, based on points raised by Michael Andrews.

Incorrect TMPDIR Support

All platforms are affected.

Contributed by Stu.

Updated Transaction Handling Code

Our original intention was to remove the use of inodes completely; however, this approach requires more time than we can afford to spare at this stage of the project.

Instead, the transaction handling code has been updated to use twice as many inodes, and to store 16-bit values into each inode. This allows the existing transaction mechanisms to work unmodified on all known UNIX platforms, and still leaves open the option of completely replacing this part of NQS in the future.

This code has been tested.

All platforms are affected.

Contributed by Stu.

List Of Changes For Developers

A full description of the source tree layout, and its management through the SETUP software, will be included in the Developer's Manual I'll shortly be working on; in the meantime, if there is anything you want to know, drop me an email.

Library Re-organisations

• The old npsn_compat stuff has been moved into its own libnmap space. Header files are now included from &ltlibnmap/>. The nmapmgr program itself has been moved into it's own directory.

• The old lib stuff has now been moved into its own libnqs space. Header files are now included from &ltlibnqs/>. You MUST include &ltlibnqs/license.h> before including any other header files.

  I've also broken down the library into groups related by common functionality. The `unwelcome' group contains functions which require radical modification or removal.

• The filename handling stuff (originally contributed by Boeing) has been moved into a separate libnqs_a library. This code was used by both libnmap and libnqs, and needs to go into a separate library to allow things to link correctly.

• A new library has been added - libsal, the `System Abstraction Layer'. Future portability work for GNQS will be to hide ALL platform-specific code in this library; the idea is that porting GNQS will become a task of porting libsal to new architectures.

Other Credits

• Michael Andrews
• Ulrich Bernhard
• Mark Loveridge