| Tools Developed, Supported, Sponsored and/or Released by KoreLogic
Below are some of the tools that KoreLogic staff have developed which we have released as open source (either GPLv2, BSD, or dual-license; contact us if you need clarification for any particular tool).
FTimes - An open-source system-baseline and forensic evidence collection and is a lightweight tool in the sense that it doesn't need to be "installed" on a given system to work on that system; it has a small footprint and provides only a command line interface. FTimes is written in C and ported to many popular operating systems, both *nix and Windows.
The FTimes tool suite has been used in many scenarios, clearly establishing its usefulness across a broad spectrum:
- eDiscovery: FTimes is being used to support enterprise-scale eDiscovery searches (i.e., find sensitive data such as PII, PHI, clear text passwords, etc., in production environments).
- Compliance: FTimes was used to help a government agency scan 10,000 Windows systems in a day to verify that known malicious files were not present.
- Security Testing: Used to evaluate a vendor's network gateway appliance by opening the appliance, imaging the hard drive, and using FTimes to analyze the image for clues (credentials, keys, code, DB accounts, etc.) that could be used to attack the appliance.
- Web application testing: FTimes was used to search all files for encryption keys in web applications using client-side plugins.
- Research: FTimes was the key tool suite used to win the 2006 DFRWS Digital File Carving Challenge.
To get a third party opinion of FTimes, please read ISSA Journal Article (Dec 2008 Issue)
- Think of WebJob as is an open source framework that provides the automation and security to support a variety of security "jobs" such as system/integrity monitoring, enterprise search, configuration management, compliance verification, automated analysis, etc. WebJob downloads a program or script from a remote WebJob server and executes it in one unified operation. Any output produced by the program/script is packaged up and sent to a remote, possibly different, WebJob server.
WebJob is currently used in Fortune 500 and government IT production environments to address a wide range of requirements such including:
- eDiscovery: WebJob supports enterprise-scale eDiscovery. WebJob, in concert with tools such as FTimes, is used to automate searches across an arbitrarily large set of machines.
- Security Compliance: WebJob is configured to automatically harvest security configurations to assess security posture and patch level, or automatically apply operating system and application patches as needed.
- Enterprise IDS Management, providing a more labor-efficient way to perform IDS system administrative tasks without sacrificing the availability needed for these security-critical devices. WebJob is used to support the following: System health monitoring, IDS software upgrades, configuration changes.
WebJob has also been used to accomplish the following:
- Automatically harvest argus, ifconfig, lsof, netstat, ndd, patch, ps, tcpdump, (name your utility), etc. data.
- Automatically update cron tabs, DNS records, password files, snort rules, web sites, (name your application), etc.
- Automatically update system binaries when their MD5s do not match expected values.
- Conduct massive searches for credit card numbers, social security numbers, and suspect hashes.
- Harvest system information to perform security audits or compliance verification.
- Implement a virtual evidence locker (VEL).
- Implement/maintain a distributed malware test harness.
- Perform integrity monitoring with FTimes.
To get a third party opinion of WebJob, please read ISSA JournalArticle (Jan 2009 Issue)
MASTIFF - KoreLogic is pleased to announce the release of their latest open-source project: MASTIFF. The process of extracting and analyzing the key characteristics of files, known as static analysis, is one of the first steps that is performed in response to a suspected malware incident. Currently, static analysis is performed manually using available tools or by submitting samples to publicly available services. If any automation is desired, analysts must create their own automation scripts and procedures. This approach leads to a number of problems:
- Manual analysis is slow and error-prone.
- If an analyst does not know about a specific tool or technique, that tool or technique will not be used.
- Public malware analysis systems provide no guarantee of availability or results, do not support analysis environment customizations, and make the analysis results publicly available. This is undesirable in some cases.
- Manual analysis is not consistent as analysts rarely run the same tools in the same order every time.
The MASTIFF framework was created to alleviate these problems. Developed through the DARPA Cyber Fast Track program (http://www.cft.usma.edu/), MASTIFF automates the static analysis process by automatically determining the type of file being analyzed and performing the correct techniques against that file. This allows analysts to quickly and consistently extract key characteristics from a file and analyze the results faster.
To ensure the framework remains flexible and extensible, a community-driven set of plug-ins are utilized for file-type detection and data extraction. By using plug-ins for this functionality, anyone can expand the framework to add new files that can be analyzed or new techniques to be performed.
Download the 0.6.0 release of the MASTIFF source code here [sign], or visit the MASTIFF Git page to access the development code tree. Questions or comments on the project can be sent to firstname.lastname@example.org. [key]
KL-EL - The KoreLogic Expression Language Library (libklel) is a C library that provides a simple expression language that can be embedded in other programs. It does not implement a full programming language, but rather a simpler expression language called KL-EL that is designed to provide arithmetic and logic operations in situations where embedding a full programming language would be overkill. KL-EL can access functions and variables exported from the embedding program, and is statically and strongly typed, which helps ensure that expressions are valid before they are executed. The embedding API is easy to use, and the library itself is very small.
KL-EL source code and PGP signature can be downloaded at http://sourceforge.net/projects/libklel. Questions or comments on the project can be sent to email@example.com. [key]
IDS Filters [sign] - A tool for creating and managing IDS filtering rules highly tailored to a given organization's network, installed services, etc. This allows tuning an IDS deployment to greatly reduce false-positives, raising the effectiveness of the deployment and the efficiency of the IDS analysts. Includes sample rules to eliminate the most common sources of false-positive noise in a typical internal network. Currently supports the Dragon IDS, but adaptable to support others if they have a sufficiently powerful filtering mechanism.
HAP-Haqs - A collection of useful security patches and tools. The bits worthy of publication, such as Dragon IDS addons and signatures, log-summarizing tools, and others are available locally, down below.
HAP-Linux - A collection of security-related Linux kernel patches for the 2.0 and 2.2 kernel series. These patches have been integrated into the GRSecurity project patch set for the 2.4, 2.6, and 3.x kernel series. KoreLogic supports, and highly recommends the GRSec patches.
HashDig - A collection of utilities designed to help practitioners automate the process of resolving MD5 and SHA1 hashes.
PaD - A Payload and Delivery (PaD) file is a self-extracting executable that can be packaged as either a script or a program.
John the Ripper Tools :
rockyou.chr[sign] rockyou-lanman.chr[sign] README
- Updated .chr files for use with the John the Ripper password cracking tool. John uses .chr files to inform its "smart brute force" mode, generating password candidates using the most frequent characters first (rather than "aaaaaaa", "aaaaaab", "aaaaaac", etc). A large online database was recently compromised and passwords leaked; these .chr files are built using this large, current sample. See the README for more information and usage notes.
Miscellaneous Tools :
ddp [md5] [sign] [sign-key]
- DDP, short for Dd-Delta-Patch, is a tool for diffing and patching equal-sized dd(1) images. This tool was written as a companion to dd(1), and its purpose is to reduce the amount of storage required to maintain multiple snapshots of a given partition or disk image. Note: Version 1.0.0.ds29 is a development release.
klogtail [md5] [sign] [sign-key] - KLogTail is a log file tailing program designed to work on UNIX and Win32 platforms. This tool was inspired by Craig Rowland's logtail.
probe_summ [sign] - A perl script to summarize Linux firewall logs--show the top N ports being scanned, the top destination IPs being scanned, the top source IPs sending traffic which is being blocked, and the hours during which the most packets were dropped. Should work for logs from both ipchains and iptables Linux boxes; you may have to tweak some of the tunables depending on how your firewall rules are set up.
tallyho [md5] [sign] [sign-key] - TallyHo is a libpcap-based sniffer program that counts (or tallies) various traffic attributes such as protocol/port/type distributions and byte counts.
weblogicdecode [sign] - WebLogic servers in Cluster mode leak the IP addresses of the backend servers in their session cookies. For some versions of WebLogic, you can manipulate this cookie and trick the webserver into port-scanning the backend network for you. This script decodes a number of different WebLogic cookie formats (they have changed with different WL versions).
www-version-harvest [sign] - Given a starting URL, this script finds/guesses all webservers linked off that page, and grabs Server: tokens from each (webserver software and version information). Useful for "taking the temperature" of a project's mirror servers--if a large percentage of them are running old, vulnerable webserver versions, you should be more suspicious of the software you download from any of them. Especially if the project does not PGP-sign their releases.
yaas [sign] yaas.cfg.sample [sign] imap-login [sign] - Yet Another Auth-before-Something. Watches syslogs of imapd logins and dynamically adds/removes /etc/hosts.allow file entries to allow access to other services, such as smtp, sshd, etc. (You need to create /etc/hosts.deny entries with ALL for each service for which you are using yaas to build /etc/hosts.allow entries.) Also, a small shell script to manually log in over IMAP-SSL, if you want to "register" your IP in the hosts.allow file without actually using a mail client.
LoadLimit.pm [sign] - A proof-of-concept perl module to mitigate application-level denial of service attacks for perl CGIs. Call from within perl CGIs (with or without mod_perl) to check the system load against configurable thresholds, and returns an error if the server is overloaded, short-circuiting potentially expensive operations.
Dragon IDS Tools :
- A shell script that checks on the status of Dragon sensor, replicator, and policy-management processes and connections, and restarts Dragon if things do not look right. This has been sufficient to catch the error-states we've seen most often on our sensors. Suitable to be run from cron or via WebJob.
dbcat [sign] - A perl script which reads a Dragon .db binary format log file and spit out a dragon.log style text output (suitable to feed to alarmtool, SQL-import tools, etc), or print out any readable packet data, suitable for doing tail -f dragon.db | dbgrep "something" | dbcat to watch sessions in real-time.
dbgrep [sign] - A perl script which reads a Dragon .db binary format log file, applies filters (similar to grep, or the Dragon 'drep' tool) and spits out a filtered .db binary format log file. Capable of filtering on source or dest IP or port, eventname, sensorname, or by regular expression in the bodies of captured packets. Supports following the TCP session of any event printed (such as following the session for all FTP:USER-ROOT events, rather than just one session at a time using Dragon's mksession tool). Supports negative filtering as well (-v, similar to grep). Since dbgrep's output is also a binary .db file, any number of dbgrep commands can be chained together to achieve an arbitrarily complex result.
decode_contentidpost [sign] - A perl script which reads the uploads from MarketScore spyware and decodes the contents, so you can easily review what potentially sensitive information has been uploaded to the mothership. Usage details in the script; requires both dbgrep and dbcat.
extract_web-get-exes [sign] - A perl script which reads WEB:GET-EXE events (a signature looking for any GET of an .EXE file; see MISC_LIGHT.lib) and extracts the Host: header and the full file path being requested. This can be used to profile your network and look for anomalous downloads. Windows networks will make many WEB:GET-EXE requests to Microsoft Windows Update servers, and there will likely be some regular downloads from dell.com, antivirus updates, and other presumably legitemate, reputable sites. But the remainder is very likely to be malware or spyware-infected machines, phoning home, downloading additional malware, etc. Usage details in the script; requires both dbgrep and dbcat.
Dragon IDS Signatures :
IRCBDCHAN.lib [sign] - This group contains custom IRC-backdoor-trojan Dragon NIDS signatures. These signatures detect clients registering with a number of IRC channel names commonly used as the remote-control channel for trojans/worms which register themselves after infection to await further instruction, report on progress compromising more machines, etc. NOTE they currently use port 6667, the default IRC port. It would be better to define a local COMPLEX port entry, such as 'I', which listed a wider range of ports commonly used by IRC servers, such as perhaps 6666-7000.
IRCBDCMDS.lib [sign] - This group contains custom IRC-backdoor-trojan Dragon NIDS signatures. These signatures detect the types of command strings, or phone-home reporting progress compromised machines, etc. NOTE they currently use port 6667, the default IRC port. It would be better to define a local COMPLEX port entry, such as 'I', which listed a wider range of ports commonly used by IRC servers, such as perhaps 6666-7000.
MISC_HUNGRY.lib [sign] - This group contains a few custom Dragon NIDS signatures which are known to be CPU-hungry. They should not be enabled on heavily loaded sensors if possible.
MISC_LIGHT.lib [sign] - This group contains a few custom Dragon NIDS signatures which should be fairly low-CPU-cost, and high-value.
ORACLE.lib [sign] - These sigs detect unusual, possibly malicious Oracle database queries/traffic. NOTE they currently use port 1521. That is the default port for Oracle's main listener, but it might easily have been changed by local DBAs. It would be better to define a local COMPLEX port entry, such as 'O', which listed all ports used by Oracle servers within your organization.
SPYWARE.lib [sign] - This group contains Dragon signatures for various known spyware / malware websites and tools.
SQL.lib [sign] - This group contains Dragon signatures for various SQL-injection attacks in HTTP requests, particularly in POST bodies. For the most part these should never triger on legitimate traffic, although handful of false-positives may be triggered by exceptionally insecure web applications.
WEBLOGIC.lib [sign] - These sigs detect unusual, possibly malicious WebLogic application server queries/traffic. NOTE: This library currently uses the 'W' COMPLEX rule to watch for traffic on known Web ports. However, this is sub-optimal: often WebLogic servers run on alternate ports such as 5000, 7001, etc, so these signatures will not monitor the right traffic. Some of these signatures are likely to false-positive on generic Web traffic, whereas if they were watching purely WebLogic sessions, their false positive rate would be very low; those signatures are currently disabled (re-enable them only after assigning them to a WebLogic-specific COMPLEX rule).