StatusReports

• Status report requirements

Status report #1 of 13

May 28, 2007

Accomplishments

• Studied the Nmap code and took notes on how it works: http://www.bamsoftware.com/wiki/Nmap.
• Submitted a bug report about a traceroute assertion failure.
• Reviewed the reason patch and made some suggestions.
• Did a subversion merge to the soc07 branch from the trunk. Wrote a merging mini-howto.
• Submitted a bug report and a patch to keep from installing .svn directories.
• Found and fixed a bug in the top ports patch that stopped ping and list scans from working.
• Made graphs of include file dependencies in order to start paring them down: http://www.bamsoftware.com/wiki/Nmap/BuildDependencies.
• Wrote a program to compare the dependencies in two different makefile.deps, in order to measure dependency-reduction progress.

Priorities

1. Put NMAP_VERSION in only one place, probably a header file. I'll submit a small proposal to make sure that it will work for everyone (Windows folks in particular).
2. Show where data files were loaded from, probably only in verbose mode.
3. Find and remove unnecessary header includes. A lot of dependencies exist where they shouldn't.
4. Review other developers' code. I'd like to look at the reason and top ports patches more closely if I have time.
5. Add a --services option to allow using a different nmap-services file without setting up a special datadir.

Status report #2 of 13

June 4, 2007

Accomplishments

• Reduced the number of build dependencies. Over 600 dependencies were eliminated overall, a decrease of over 16 per target on average. A summary is at http://www.bamsoftware.com/wiki/Nmap/BuildDependencies.
• Put NMAP_VERSION in just one file.
• Added the --servicedb and --versiondb options, along with a mechanism to add similar options easily.
• Modified the Windows makefile to keep from copying .svn directories into the zip distribution.
• Found a bug with a Subversion keyword in nmap-service-probes.

Priorities

1. Send proposals for verbose data file path reporting to nmap-dev, and write a patch.
2. Code review.

Status report #3 of 13

June 11, 2007

Accomplishments

• Sent ideas for verbose data file path reporting to nmap-dev and committed a patch.
• Tracked down and fixed the use of some deallocated memory.

Priorities

1. Learn about OS detection database maintenance.
2. Move basename and dirname replacements into nbase.
3. Remove INTERACTIVE_NAMES functionality.

Status report #4 of 13

June 18, 2007

Accomplishments

• Rewrote in C and moved dirname and basename replacements into nbase.
• Reported a bug with the --script-updatedb option (fixed by Stoiko).
• Removed INTERACTIVE_NAMES.
• Made fingerfix (one of the OS database utilities) keep from displaying empty values for tests that were not present in an observed fingerprint. Also made the OS database utilities warn if they see such an empty value.
• Integrated about 90 OS fingerprint submissions (more to come).

Priorities

1. OS database submission integration.
2. Handle the case when a TTL is > 0xFF in OS detection.
3. Merge identical OS detection lines into one, even if they are not output consecutively.

Status report #5 of 13

June 25, 2007

Accomplishments

• Made Nmap notice and mark a fingerprint as bad when an initial TTL is calculated to be greater than 255. Later reverted the change after it was found that the old behavior was sometimes useful. However, the discussion led to some enhancements in the fingerprint utilities and a clarification in the documentation.
• Made Nmap mark fingerprints with negative network distances as bad.
• Refactored the OS database utilities' fingerprint checking code and made other small fixes and enhancements.
• Integrated about 570 OS detection signature submissions.

Priorities

1. Merge identical OS detection lines into one, even if they are not output consecutively (this should be done before the next release).
2. Finish integrating OS submissions (about 650 to go up through June 13).

Status report #6 of 13

July 2, 2007

Accomplishments

• (Nearly) finished integrating OS submissions (about 650 done this week). Fewer than 25 tricky ones remain, which I'm going to work on with Fyodor.
• Made merging of OS fingerprints for submission work more intelligently even when some tests are omitted or out of order.

Priorities

1. Top ports code review.
2. Discuss including tests with the source ("make test" perhaps?).
3. See about starting one of the big projects (fixed-rate scanning or proxy scanning). Has anyone been looking into these?

Status report #7 of 13

July 9, 2007

Accomplishments

• Reviewed the top ports patchset and wrote a short report.
• Worked through the last few OS submissions with Fyodor.
• Sent my ideas about massping migration to nmap-dev.
• Created a page with my notes on the massping migration: http://www.bamsoftware.com/wiki/Nmap/MasspingMigration.
• Created a massping migration branch at svn://svn.insecure.org/nmap-exp/david/nmap-massping-migration.

Priorities

1. Finish a running prototype of at least one form of host discovery. I aim to accomplish this by Friday.

Status report #8 of 13

July 13, 2007

Accomplishments

• Wrote a prototype (supporting only TCP SYN ping scan) of my idea for implementing massping using ultra_scan.

Priorities

1. Discuss the prototype when I get back.
2. Add support for other ping types. In rough order: TCP ACK, UDP, ICMP, TCP connect.

Status report #9 of 13

July 23, 2007

Accomplishments

• None this week.

Priorities

1. Discuss the ping scan prototype.
2. Make ping scan bail out once the state of a host is known.
3. Add support for other ping scan types. In rough order: TCP ACK, UDP, ICMP, TCP connect.
4. Figure out how ultra_scan timing differs from that of massping.

Status report #10 of 13

July 30, 2007

Accomplishments

• Made ping scan quit scanning a host once it's known to be up.
• Added support for the remaining ping scan types: TCP ACK, UDP, ICMP, and TCP connect.

Priorities

1. Test and tune the new host discovery code. Right now it detects fewer hosts than massping did.
2. Think of a plan for refactoring scan_engine.cc.

Status report #11 of 13

August 6, 2007

Accomplishments

• Started investigating why the new host discovery code is slower and less accurate than the old. Sent a message to nmap-dev asking for help.
• Talked with Fyodor about the best way to proceed.
• Did a few systematic port scans, though I haven't looked at the results in detail yet.

Priorities

1. Make the new host discovery code find all the hosts the old code did on a 500-host random ping scan.
2. Find what's holding back the new host discovery's speed.
3. Formulate an exit strategy for the summer (I don't want to leave any work unfinished).

Status report #12 of 13

August 13, 2007

Accomplishments

• Did more benchmarking of host discovery. See http://www.bamsoftware.com/wiki/Nmap/HostDiscoveryBenchmarks.
• Found and fixed a performance bottleneck that arose when scanning many hosts.
• Made a change to keep track of all scanned hosts even after their probes expire, so probes have a chance to arrive late. I think this was the biggest thing hurting the accuracy of host discovery.

Priorities

1. Measure the effect that using timing pings during host discovery has.
2. Finish bringing ultra_scan-based host discovery up to speed.

Status report #13 of 13

August 20, 2007

Accomplishments

• Made ping scan use timing pings, which keep timing information up to date and make scans go a lot faster.
• Made ping scan send ICMP messages with a non-zero identifier.
• Made ping scan mark up those hosts that send destination unreachable messages.
• Fixed a bug that caused TCP connect ping scan to hang sometimes.
• Did a lot of careful benchmarking (http://www.bamsoftware.com/wiki/Nmap/HostDiscoveryBenchmarks).

Priorities

1. Try tuning ultra_scan's performance variables (congestion control and quick increments, and congestion control threshold).