Writing your own Preprocessors

Snort Preprocessors Development Kickstart
Last updated: November 29th 2005
Authors: Andrés Felipe Arboleda Torres, Charles Edward Bedón Cortázar


What is a Snort Preprocessor?
Preprocessors are plugable components of Snort, introduced since version 1.5. They're "located" just after the module of protocol analysis and before detection engine and do not depend of rules. They are called whenever a packet arrives, but JUST ONCE, the detection plugins, in the other hand, do depend of rules and may be applied many times for a single packet. SPP's can be used in different ways: They can look for an specific behavior(portscan, flowportscan), to be support for further analysis(is this the expression? help us) like flow, or just collect certain information, like perfmonitor.
More information can be found at http://www.snort.org/docs/snort_htmanuals/htmanual_2.4/node11.html
When should I code a Snort Preprocessor and not a Detection Plugin?
/*TODO*/
A Hello World Preprocessor
These are the steps you should follow to make your first preprocessor(supossing the directory for unzipped Snort is $SNORT_DIR):

1. Open the templates for your code at $SNORT_DIR/templates: 2 files--> spp_template.c and spp_template.h
2. Make your own files, trying to follow the convention of using the prefix "spp_" before the name of the preprocessor. Those files should be created at $SNORT_DIR/src/preprocessors
3. The header file:
spp_helloworld.h

/*To keep you from repeat this anywhere else*/
1 #ifndef __SPP_HELLOWORLD_H__
2 #define __SPP_HELLOWORLD_H__
/*Demostrative(won't be used in this example later):You can define structs to share info if for example, this preprocessor will be used for another one. */
4 typedef struct _helloStruct{
5 int x;
6 char *string;
7 }helloStruct;
/*Demostrative*/
9 int helloCounter=0;
/*This is the actually important prototype. This function will be called when initializating SPP's*/
11 void SetupHelloWorld(void);
/*Demostrative. Remember: important if this preprocessor is intended to be support for another one later*/
13 void myHelloWorldFunction(char *str);
/*Demostrative*/
15 extern helloStruct myHelloWorldStruct;
16 #endif
4. The source file:
spp_helloworld.h

/*config.h is generated by "autoheader", when configuring the project(information about OS, libs available ....)it's located at $SNORT_DIR. Useful things for you to know the what can do and what cannot*/
#ifdef HAVE_CONFIG_H
#include "config.h"
#endif

/*Definition of u_char*/
#include

/*Functions Add* and RegisterPreprocessor */
#include "plugbase.h"

/*Definition of Packet*/
#include "decode.h"

/*Our header file */
#include "spp_helloworld.h"

/*Demostrative */
helloStruct myHelloWorldStruct;

/*Prototypes of internal functions*/

/*Initialization Function(Usually, take parameters from config file, call a parseargs function and initializes local variables) */
void HelloInit(u_char* args);
/*What the pp actually does*/
void HelloFunc();
/*What to do when a term order is received*/
void HelloCleanExitFunction();
/*What to do when a restart order is received*/
void HelloRestartFunction();

Called from plugbase.c
void SetupHelloWorld(void){
/* Here you could (in fact you *should*) use the macro DEBUG_WRAP defined in debug.h, or if you really want this message to appear anyway(not only in debug mode, the place depends on the type of messaging log, it could even be registered in the syslog) use the function logMessage at util.h
NOTE: this function is called always, even if this preproc has not been included in the config file*/
printf("Let's see if Hello World Preprocessor is in the config file...\n");
/*Receives the name of your pp(the one in the config file) and a function pointer to the initialization code */
RegisterPreprocessor("helloworld", HelloInit);
}

/*NOTE: This function is called once, and depends on the addition of the preproc in the config file. See coments above*/
void HelloInit(u_char* args){
printf("Hello world Preprocessor is being initialized...\n");
/*"args" is a string with the arguments in the config file(remember the preprocessor definition syntax preprocessor:). Arguments can have any separator, at $SNORT_DIR/msplit.h, there is a good collection of functions to get the right values, eg, msplit. This file contains some functions not included in libc.*/
printf("Those arguments were received --> %s\n",args)
/*Add this pp to pp list. The argument is a pointer function to the code to be executed everytime a packet arrives*/
AddFuncToPreprocList(HelloFunc);
/*What to do when a term order is received*/
AddFuncToCleanExitList(HelloCleanExitFunction, NULL);
/*What 2 do when a restart order is received*/
AddFuncToRestartList(HelloRestartFunction, NULL);
}

/*NOTE: This function is called when a packet is received*/
void HelloFunc(Packet* p){
printf("Hello world Preproccessor is being called...\n");
/*Just a little demostration. Is the packet TCP?*/
printf((p->tcph==NULL)?
"This packet does not have any TCP header\n":"This a valid TCP packet\n");
}

/*TODO*/
void HelloRestartFunction(){
}

/*Called when Ctrl+C is hit*/
void HelloCleanExitFunction(){
/* Free pointers and other resources, for example. You can also present some stats on stdout or generate more logs*/
}

5. Edit the file $SNORT_DIR/src/plugbase.c: Add these lines-->
plugbase.c
In the includes section:
/* built-in preprocessors */
#include "preprocessors/spp_portscan.h"
(...)
/*Our preprocessor*/
#include "preprocessors/spp_helloworld.h"
(...)

In the function InitPreprocessors:
void InitPreprocessors(){
(...)
SetupHttpInspect();
/*Our preprocessor*/
SetupHelloWorld();
SetupFlow();
(...)
}
6. Edit $SNORT_DIR/src/preprocessors/Makefile (if already generated) or $SNORT_DIR/src/preprocessors/Makefile.in.(before configure)
Makefile or Makefile.in
In the libspp_a_SOURCES section:

(...)
libspp_a_SOURCES = spp_arpspoof.c spp_arpspoof.h spp_bo.c spp_bo.h \
(...)
#Our preprocessor
spp_helloworld.c spp_helloworld.h \
(...)
#And this for the linker in the section "am_libspp_a_OBJECTS"
(...)str_search.$(OBJEXT) spp_portscanai.$(OBJEXT)
(...)
7. Compile ... (nothing is over til it's over)
8. Add our pp in the config file:
etc/snort.conf
(...)
preprocessor flow: stats_interval 0 hash 2
#Our preprocessor
preprocessor helloword: hello world args

preprocessor stream4_reassemble
preprocessor stream4
(...)

Stuff Used

* Snort 2.2.0(successfully used under v2.4.1)
* Anjuta IDE 1.2.2(KDevelop 3.4 too)
* Debian Sarge 3.1 kernel 2.6.13

Bibliography

* http://www.snort.org/docs/

To Do

* Complete the sections marked as "TODO" je je ... excuse us we're a bit impatient, and we can't take til finishin' all work before release the docs... ;)
* Include a little FAQ with information about commonly used functions and data structures
* Do an example with the colaborative work between two or more spp

Happy Snortin'!!!!

Stream5 on Snort ( Good Article by Richard Bejtlich)

It's important for value-added resellers and consultants to understand how Snort detects security events. Stream5 is a critical aspect of the inspection and detection equation. A powerful Snort preprocessor, Stream5 addresses several aspects of network-centric traffic inspection. Sourcefire calls Stream5 a "target-based" system, meaning it can perform differently depending on the directives passed to it. These directives tell Stream5 to inspect traffic based on its understanding of differences of behavior in TCP/IP stacks. However, if Stream5 isn't configured properly, customers may end up with a Snort installation that is running but not providing much real value. In this edition of Snort Report I survey a specific aspect of Stream5, found in Snort 2.7.x and 2.8.x.

If you've never read Insertion, evasion and denial of service: Eluding network intrusion detection by Thomas Ptacek and Timothy Newsham, then you absolutely must do so now. Rather than repeat the theory and consequences of their landmark paper, I'll show how features of the most current version of Snort addresses problems identified by the authors almost ten years ago.

Snort 2.7.0.1 installation

First, I install Snort 2.7.0.1 on FreeBSD 6.2.

Next, I edit my snort.conf file to include the changes indicated by the diff below. Note the deletion of all of the rule locations and the addition of a single rule for testing purposes.

With the changes made to snort.conf, I run Snort in testing mode to ensure I didn't forget anything.

Since Snort operated normally, I'm ready to trigger an alert.

Testing Snort on normal traffic

I start Snort and tell it to listen to the local interface while sending alerts to the screen.

taosecurity:/usr/local/snort-2.7.0.1# bin/snort -i lnc0 -c snort.conf -A console

On a second machine I connect to the Snort sensor's OpenSSH port and enter a text string to trigger the alert.

tws:~# nc -v 192.168.2.103 22
192.168.2.103: inverse host lookup failed: Unknown host
(UNKNOWN) [192.168.2.103] 22 (ssh) open
SSH-2.0-OpenSSH_4.2p1 FreeBSD-20060930
This is a test of the emergency broadcast system.
Protocol mismatch.

Snort reports seeing the traffic as I expected.

09/03-14:31:34.315118 [**] [1:2000001:0] TEST String to Port 22 TCP [**] [Priority: 0] {TCP} 192.168.2.106:2099 -> 192.168.2.103:22

The traffic looks similar to this:

Notice the entire test message is contained in one segment. This is normal traffic, and it will serve as a baseline for future scenarios.

Using Fragroute

I use Fragroute to exercise the new stream preprocessor, Stream5, present in Snort 2.7.x and beyond. I discuss Fragroute in Detect events without Snort IDS rules so I won't repeat the introduction here.

I create a fragroute.conf file with these contents:

tcp_seg 16
print

These directives tell Fragroute to chop up TCP segments into 16 byte fragments and print the traffic to the console.

Next, I start Fragroute on a Debian 4.0 Linux host with IP address 192.168.2.106. For all subsequent cases, 192.168.2.106 will be the "attacker" or client, and 192.168.2.103 will be the "victim" and also the machine inspecting traffic with Snort.

tws:~# fragroute -f fragroute.conf 192.168.2.103
fragroute: tcp_seg -> print

When I repeat my test, connecting to port 22 and entering the "This is a test..." string, Snort doesn't alert. What could be wrong? The traffic was chopped into segments as expected.

Notice the test message is broken into four segments.

The key is found in the Snort startup messages. Look at this excerpt:

Stream5 TCP Policy config:
Reassembly Policy: FIRST
Timeout: 30 seconds
Min ttl: 1
Options:
Static Flushpoint Sizes: YES
Reassembly Ports:
21 client (Footprint)
23 client (Footprint)
25 client (Footprint)
...truncated...

Port 22 isn't being reassembled, although ports 21, 23, 25 and others are. To correct this problem I modify the entry for stream5_tcp in snort.conf to look like this:

preprocessor stream5_tcp: policy first, use_static_footprint_sizes, \
ports both 22

Repeating my test, I find my alert as expected.

09/03-16:29:07.070801 [**] [1:2000001:0] TEST String to Port 22 TCP [**] [Priority: 0] {TCP} 192.168.2.106:2415 -> 192.168.2.103:22

This is our first clue that it's important to understand how your inspection product is configured and how it makes decisions.

Overlapping segments

It's time to make life more interesting. I edit fragroute.conf to add overlapping fragments.

tcp_seg 16 new
print

Here "new" means Fragroute will send garbage first, then correct data last. If the recipient TCP/IP stack favors new data over old, then the connection will be OK because the garbage fragmented data will be discarded prior to being sent to the OpenSSH daemon. (The OpenSSH daemon will ignore it anyway, but if a legitimate OpenSSH connection attempt was being fragmented it would survive the "new" directive.)

At this point I am really interested to see if Snort and Stream5 tell me that someone is potentially attempting an insertion or evasion attack by introducing overlapping fragments.

When I run Snort and the test again, I don't get any alerts from Snort. What could be wrong this time?

Let's return to the stream5_tcp directive:

preprocessor stream5_tcp: policy first, use_static_footprint_sizes, \
ports both 22

Here "policy first" is the problem. According to the Stream5 README, "policy first" means "favor first overlapped segment." We want behavior that BSD supports, since the OpenSSH server is running FreeBSD. In other words, Snort is passively reassembling the traffic it sees to the OpenSSH server, but it's making decisions that don't match how the FreeBSD OS on the OpenSSH server reassembles overlapping fragments.

If we change the stream5_tcp directive we will have better results.

preprocessor stream5_tcp: policy bsd, use_static_footprint_sizes, \
ports both 22

Sure enough, when we re-run our test we get an alert:

09/03-16:44:40.729492 [**] [1:2000001:0] TEST String to Port 22 TCP [**] [Priority: 0] {TCP} 192.168.2.106:3340 -> 192.168.2.103:22

This time, Snort generates an alert because Stream5 reassembled the fragmented traffic in the same way that the underlying FreeBSD OS did. Incidentally, it's irrelevant that Snort is running on the FreeBSD OS itself. Snort makes its own TCP fragmentation and overlap reassembly decisions; it does not look to the underlying OS for answers.

Generic overlapping fragment detection

What if you want to detect any suspicious use of overlapping fragments? In my experience it's uncommon for normal TCP traffic to contain overlapping segments like the following:

That traffic is similar to the packets manipulated by Fragroute in the previous example. As you can see, the last packet contains the beginning of the test message. However, the first packet contains garbage ("b6reOo34j0rojpB.") that overlaps with bytes 17-32 (16 bytes) of the real test message. Because the FreeBSD OS of the OpenSSH server favored new data (the last packet) over old data (the first packet), the correct message was passed to layer 7.

It would be helpful to receive a message from Stream5 whenever odd behavior like this occurs on the wire.

We can modify the stream5_tcp directive one more time to detect this sort of odd activity.

preprocessor stream5_tcp: policy bsd, use_static_footprint_sizes, \
ports both 22, detect_anomalies, overlap_limit 1

Notice the inclusion of the "detect_anomalies, overlap_limit 1" option. (Thank you to beenph and VictorK in IRC for passing this information to me!) These options will cause Stream5 to alert when it sees at least one overlapping TCP segment.

In case one, I send the following string but Snort remains silent:

This is a test test test test test test test test test test

In contrast, the following string causes Snort to trigger:

This is a test test test test test test test test test test test

Stream5 produces an alert like this:

09/03-16:58:22.263117 [**] [129:7:1] Limit on number of overlapping TCP packets reached [**] [Priority: 3] {TCP} 192.168.2.106:4556 -> 192.168.2.103:22

It's good to get at least one alert on overlapping TCP fragments, but why did we only see one alert?

Why one and not the other?

Why did Snort stay quiet for case one, but react for case two?

The difference lies in the number of packets generated. To demonstrate this, I reproduced the critical elements of the two cases as captured by TCPdump. In each listing, the first number is a packet number for easy reference. Next is the P for the "Push" TCP flag. The number1:number2(number3) field means "sequence number of the first byte in this segment:sequence number of the first byte in the NEXT segment(number of bytes of data in this segement". In each case "ack 40" means the sender expects the other side of the conversation to next send data whose first byte will be number 40. All of these numbers are relative (not absolute) sequence numbers.

(Note that number2 is NOT the sequence number of the last byte of data in the segment. For packet 1 below, 32 is the last number. Count with me if you're confused: 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32. 33 would be the 17th byte, which is wrong. This is a common misconception in books on TCP/IP.)

Here are the sequence numbers for the traffic sent in the first case.

1. P 17:33(16) ack 40
2. P 33:49(16) ack 40
3. P 49:61(12) ack 40
4. P 1:33(32) ack 40

Packet 4 is the only overlapping fragment, since it contains bytes 17-32 which were already sent in fragment 1.

Here are the sequence numbers for the traffic sent in the second case.

1. P 17:33(16) ack 40
2. P 49:65(16) ack 40
3. P 65:66(1) ack 40
4. P 1:33(32) ack 40
5. P 33:65(32) ack 40M

Packets 4 and 5 are overlapping fragments. Packet 4 contains bytes 1-32; packet 1 already sent bytes 17-32. Packet 5 contains bytes 33-64; packet 2 already sent bytes 49-64. Therefore, unlike the previous case, this trace contains two overlapping fragments. Because Stream5's "overlap_limit 1" option was set and two fragments (case two) is greater than one fragment (case one), case two triggered Stream5 while case one did not.

I hope this edition of the Snort Report motivated you to think about how you can use Stream5 to detect Fragroute-type TCP fragment overlap attacks. Like most every Snort Report, this edition only scratched the surface of another powerful component of Snort.