OS Integrator Assistant requirements document

David Fifield

Background

Did you ever wonder what happens after you submit an OS fingerprint at the submission page? Someone looks at every one of those submissions and uses tools in the nmap-dev area of the Subversion repository to build them into the OS database. This document explains how this process works now and gives requirements for a simple new tool to make it easier.

The current tools

There are three primary tools for OS integration: fingermatch, fingerdiff, and fingerfix.

fingermatch: Takes a raw submission and shows which fingerprints in the database it most closely matches. The output looks like this:

    Found **1 PERFECT MATCHES** for entered fingerprint in nmap-os-db:              
    Accu Line# OS (classification)
    100%  7165 IBM OS/400 V5R2 (IBM | OS/400 | V5 | general purpose)
    **ADDITIONAL GUESSES** for entered fingerprint in nmap-os-db:
    Accu Line# OS (classification)
     97%  7148 IBM OS/400 V5R1M0 (IBM | OS/400 | V5 | general purpose)
     96%  7131 IBM OS/400 V4R5 (IBM | OS/400 | V4 | general purpose)
     95%  6995 IBM i5/OS V5 (IBM | i5/OS | V5 | general purpose)

fingerdiff: Shows how a submission fails to match a database entry. If it matches only 96%, what accounts for the 4%? The output looks like this (for the meaning of the codes see the OS detection article):

    SEQ.SP: "AE" NOMATCH "0-5" (25 points)
    SEQ.ISR: "B2" NOMATCH "0-5" (25 points)
    T3.R: "N" NOMATCH "Y" (80 points)
    Accuracy of the two prints is 96% -- see differences above.

fingerfix: Takes a submission and makes some adjustments to make it suitable for a new entry in the database. For example, it automatically includes multiples of the SEQ.GCD field and widens the ranges for tests like SEQ.SP and SEQ.ISR. It can also merge a submission into an existing database entry, causing it to match more. Its output is identical to an entry in nmap-os-db.

How I do OS integration now

Fyodor emails me mbox files containing the new submissions and corrections. I put them in my nmap-dev working directory. I svn update in my nmap working directory and copy nmap-os-db into nmap-dev. I open two big xterm windows, each as tall as the screen. In the left xterm, I open Vim on nmap-os-db. In the right xterm, I have GNU Screen running with two screens: One has Mutt open on the mbox file, and the other is a shell where I run the finger* tools. Having these in the same place on the screen reduces the amount of mouse movement I have to do. in the background I open a web browser with Google so I can look up model numbers.

I look at a submission. If it doesn't have enough information, I skip it without thinking twice. Otherwise, I highlight the submission with the mouse, go to the shell and paste it into fingermatch immediately, even before looking at the submission in detail. If I get one (and only one) 100% match, I check to see if the OS in the submission is the same as what's in the database. If it is, I move on to the next submission. If not, like when a fingerprint matches a new Linux micro revision, I mouse over to the Vim window and make the change. Then I go to the next submission.

If there are no 100% matches, I have to find out what doesn't match. That means I have to run fingerdiff against the closest match at least, and usually the first two or three. This is the part that's really cumbersome. fingermatch returns a list of database line numbers that must be given to fingerdiff. I type "^rdiff" to get a line like ./fingerdiff nmap-os-db:12345 at the prompt, then I replace the line number with the correct one. I have to paste the submission once again into the program. The diff usually takes up most of the screen, so I scroll the xterm back to see the line number of the next closest match, remember it, and give it to fingerdiff again. By this time the terminal is cluttered and it's hard to find my match results, so I run fingermatch again. I repeat this process until I get enough information. I would really like to run fingerdiff on the closest five matches for every submission, but that just takes too long.

If there are no close matches, I create a new fingerprint by running fingerfix with no arguments. I highlight the new fingerprint and paste it into Vim. Then to check, I go back and highlight the submission (again) and run fingermatch to make sure I have a 100% match.

If there's a close match, I just make an adjustment to an existing fingerprint. You can do this by giving fingerfix a line number, but more commonly I just edit the database with Vim so I can better control the results. Then I run fingermatch again to make sure I have a match.

If I find an unusable submission for a new operating system, I limit the view in Mutt to find other submissions for the same OS and handle them right away. If I'm not sure about what to do with a submission, I flag it (F key in Mutt) to look at it later.

Why a new tool would make things better

A significant fraction of my OS integration time is spent carefully highlighting submissions with the mouse. This takes more dexterity than it should, especially when I'm pasting the same submission multiple times into the finger* tools. Ten or twenty times during every integration session I paste a submission into the shell by mistake and get a screen full of Bash errors. An integration tool would know that I'm essentially working on one submission at a time, and I won't have to keep reminding it which one I'm working on.

I estimate that we get about 1500 submissions every three months. By looking at my log, I see that I can do between 40 and 80 submissions an hour. Estimating an average of 60 submissions an hour, that means 25 hours of work every three months. If we can save one second per submission, that saves 25 minutes for the whole batch. Cutting off ten seconds per submission saves more than four hours.

The biggest difficulties in OS integration are copying and pasting and typing in line numbers. A tool that could eliminate those things could make it a lot easier.

What I envision

I see a small graphical (or semi-graphical, like with Ncurses) application that runs a fingermatch for me and allows me to interactively do a fingerdiff on the top matches. I want to be able to hit a key in Mutt and have the submission piped into this little program, which will appear on the screen. I can hit a key to create a new fingerprint or merge with an existing one, and whatever it is gets copied to the clipboard automatically. I still see line numbers so I can hack the database manually if I need to. When I'm done the application disappears and sets me back at Mutt.

Requirements

Before starting, you must work through a few dozen submissions, with assistance from David or Fyodor, to get a feel for the important issues.
The application must show
1. The unwrapped submitted fingerprint (like osprintsunwrap.pl),
2. A selectable list of matches, and
3. A diff for the currently selected match.
The application must be able to be controlled mostly with the keyboard, with keybindings similar to Mutt's. Especially scrolling up and down and exiting must be keyboard-controlled.
The must be buttons or keyboard controls to do fingerfix's two functions: creating a new fingerprint and merging an existing fingerprint. The fingerprint will be automatically copied to the clipboard. It is not necessary to show the new fingerprint on the screen, because I'm just about to paste it anyway, but the application should give some indication of what it has done: "XXX bytes copied to clipboard."
The application must use Nmap's fingerprint matching functions—it is not okay to reimplement them. If you write the application in a language other than C++ you must somehow bind to those functions. You can do this just by parsing the output of fingermatch/fingerdiff, or you can link with the Nmap functions in osscan.cc. However those functions will probably need some work to make them easily usable in a program like this.
The application must start up and appear quickly, not much longer than running a fingermatch and ten fingerdiffs.
The application must work on GNU/Linux. This is a developer tool, not a tool for general release. Do not be concerned with portability if it hinders any other part of development. The program may be written in any free language of your choice as long as it is not too hard to set up on Linux.
As a developer tool, the application may have rough edges. It's okay, for example, to do as the current tools do and link directly with Nmap files in a hardcoded directory (../nmap). However, the program must have good internal documentation (i.e., comments) and follow good coding practices. Also write with an eye towards easy maintenance.

Other ideas

How about a batch script that runs non-interactively on the mailbox and inserts a header field in each message showing the percentage of the highest match? That way I can highlight and quickly work through all the 100% matches just by checking if the OS name is the same.

Questions

You can email me at david@bamsoftware.com.

OSIntegratorAssistantRequirements