Archive for the ‘HowTo’ Category

Set 3com SuperStack 3 Switch 4500 Port Mirroring (SPAN)

Saturday, May 24th, 2008

I do not have the intention to promote any products in this blog. This post is for me to record down the setup and it is pretty simple to do it in console or remote session.

<4500> system-view
[4500] interface Ethernet 1/0/1
[4500-Ethernet1/0/1] mirroring-port ?
  both      Mirror the inbound and outbound packets of the interface
  inbound   Mirror the inbound packets of the interface
  outbound  Mirror the outbound packets of the interface
[4500-Ethernet1/0/1] mirroring-port both
[4500-Ethernet1/0/1] quit
[4500] interface GigabitEthernet 1/0/27
[4500-GigabitEthernet1/0/27] monitor-port
[4500-GigabitEthernet1/0/27] quit
[4500] display mirror
 Monitor-port:
   GigabitEthernet1/0/27
 Mirroring-port:
   Ethernet1/0/1              both

Both inbound and outbound traffic will be mirrored to Gigabit Ethernet 1/0/27, and you are able to get your monitoring server to receive the traffic from this monitoring port.

That’s all for today.

Ourmon DNS Blacklist

Saturday, April 26th, 2008

There is a new module in Ourmon, Topn DNS that shows the DNS traffic statistic. It comes with a DNS blacklist feature that needs to be configured. What I am going to show here are how to configure it (which is quite easy), and automate the update by using simple scripting with Shell, Perl.

First of all, this is the how to for DNS blacklist configuration:

The dns_include file config line allows a DNS name based blacklist to be configured into the ourmon probe. Any number of files may be given. The format of individual entries in the DNS blacklist is as follows:

lsass.exploited.org A

One entry should be given per line.

DNS query responses are unwrapped and if the question is found to match the DNS name in the blacklist, an ourmon event log message is generated.

That means, we need to create a blacklist file with the format above, and a list of black listed domains.

There is a project called DNS-BH, they create and maintain a listing of domains that are known to be used to propagate malware and spyware. So, with the free malware domains list, we are able to create a list and include it to the DNS blacklist.

I have created 2 scripts, 1 Perl script (domain-filter.pl) to grab the domains from the malware domains list and write them to a new file with format that Ourmon accepts, for example:

From
guti.my        unknown   guti.my/ourmon-dns-blacklist      2008apr
to
guti.my A

and 1 Shell (getbldns.sh) script to process the update, so that I am able to set it as cron job.

For those who are interested to try it out, you may download it at http://www.gutizz.com/scripts/ourmon-dns-blacklist.bz2.
md5sum: http://www.gutizz.com/scripts/ourmon-dns-blacklist.bz2.md5
There is a Readme file included the package as well. You need to edit the path of dns_include, and the path in the getbldns.sh shell script if your installation is in different directory.

This is the sample of the event log after blacklist has been configured, so you need to check why there were DNS requests to malware domains from these servers. (Click to enlarge the image.)
DNS Blacklist Event Log

There are 2 blacklist features in Ourmon as well, which are IRC blacklist and IP blacklist. Ourmon team has the IRC blacklist script written and put them in Ourmon src/scripts/, and I am still working on IP Blacklist which will fetch the black list IP from Harimau watchlist, the main problem is the support of netblock in the Ourmon IP Blacklist, which I will consider to remove those netblock addresses.

If you concern on the CPU and memory usage, here is the resource usage of Ourmon, with 20290 domains in DNS blacklist, and list of IP from EmergingThreat botcc rules in IRC blacklist loaded, and over 10k/s packets at peaks.

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
59799   root          1   -58    0  48552K 28348K  bpf    4   3:44   2.49%  ourmon

If you have any other recommendation on DNS blacklist, please let me know.

OpenSSL creates CA serial file

Saturday, April 12th, 2008

Sguil Logo I have encountered error below when I followed the Sguil OPENSSL.README to generate a certificate with a local CA for my Sguil 0.7.0 installation on FreeBSD 7.0 Release.

[[email protected]]# openssl x509 -req -in sguild.req -CA CA.pem -CAkey privkey.pem -CAserial file.sr1 -out sguild.pem
Signature ok
subject=/C=MY/ST=PG/O=Sguil/OU=Security/CN=servername
Getting CA Private Key
Enter pass phrase for privkey.pem:
file.sr1: No such file or directory
82464:error:02001002:system library:fopen:No such file or directory:/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/bio/bss_file.c:352:fopen('file.sr1','r')
82464:error:20074002:BIO routines:FILE_CTRL:system lib:/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/bio/bss_file.c:354:

OpenSSL Logo From the error message, it is obvious that I did not have the file.sr1 there. Since this was the first time I used the CA to sign the certificate, I would need to create serial key containing serial key. So I run -CAcreateserial as below:

[[email protected]]# openssl x509 -req -in sguild.req -CA CA.pem -CAkey privkey.pem -CAcreateserial -out sguild.pem

This created a new file (CA.srl) containing a serial number. The next time I have to use the -CAserial option when I create new certificate, and specify the path to this file name. The serial number will be incremented each time a new certificate is created.

Grace - XY plotting tool

Saturday, April 5th, 2008

I have collected packets of a DDOS attack on one machine recently. It is just pure SYN attack to destination port 80, with over 5000 packets from different IP in 1 second. I try to create a graph with this packet by using Grace, a WYSIWYG 2D plotting tool for the X Window System and M*tif. Grace runs on practically any version of Unix-like OS. As well, it has been successfully ported to VMS, OS/2, and Win9*/NT/2000/XP.

First, I used Argus client - ra to show unidirectional RMON stat, with only source port and destination port selected, piped it to awk to make it readable data by Grace, and then convert the number for destination port. For example, X=0, Y=2224 (source port), then X=1, Y=80 (destination port):

[[email protected] /]# ra -nr syn.argus -M rmon -s sport | awk '{ print 0,$1 }' | sed -e 's/0 80/1 80/g' > syn.dat

The output of syn.dat looks like this:

[[email protected] /]# head -n 6 syn.dat
0 2224
1 80
0 2236
1 80
0 2242
1 80

Then use grace to plot it.

[[email protected] /]# xmgrace6 syn.dat &

This is the graph of 23 seconds DDOS from source port to destination port.
DDOS Ports Grace Graph

If I use afterglow to show the connection from source to destination port, the graph looks like this.
DDOS Ports Afterglow Graph

This is the graph after I converted the source and destination IP to decimal, for example, from 192.168.1.123 to 3232235899.
DDOS IP Grace Graph

1 second source port to destination port graph, approximate 5000 SYN per second:
DDOS Ports Grace 1 Second Graph

From the port graph, I guess the DDOS was launched from 1 host with spoofed source IP and large bandwidth pipe, what do you think?

Sguil 0.70 and Wireshark 1.0.0 Released

Monday, March 31st, 2008

Sguil LogoIt seems a bit late to announce both releases. After a long development process and beta testing, Sguil 0.7.0 has finally been released. It does not take long for modsec2sguil to work with the latest release as well. I have not upgraded my Sguil installation to 0.70 yet, but it won’t take a long time for me to do that because I plan to upgrade the hard disk as well, so every thing will be fresh installation, including the OS - FreeBSD 7.0.

Wireshark LogoIn my opinion, every network analysts should get familiar with these 2 tools, especially Wireshark. In this new version, Wireshark team has fixed few security vulnerabilities, and there is an experimental package for Mac OSX Intel as well.

Other than these releases, I have done a small upgrade from FreeBSD 6.2R to FreeBSD 6.3R recently, which I refer to here. This is only applicable if you are running GENERIC FreeBSD kernel.

[[email protected] /]# fetch http://people.freebsd.org/~cperciva/freebsd-update-upgrade.tgz

[[email protected] /]# fetch http://people.freebsd.org/~cperciva/freebsd-update-upgrade.tgz.asc

[[email protected] /]# gpg --verify freebsd-update-upgrade.tgz.asc freebsd-update-upgrade.tgz

[[email protected] /]# tar -xf freebsd-update-upgrade.tgz

[[email protected] /]# sh freebsd-update.sh -f freebsd-update.conf -r 6.3-RELEASE upgrade
Looking up update.FreeBSD.org mirrors... 1 mirrors found.
Fetching metadata signature for 6.2-RELEASE from update1.FreeBSD.org... done.
Fetching metadata index... done.
Fetching 1 metadata files... done.
Inspecting system... done.

WARNING: This system is running a "nsm-smpkernel-generic" kernel, which is not a
kernel configuration distributed as part of FreeBSD 6.2-RELEASE.
This kernel will not be updated: you MUST update the kernel manually
before running "freebsd-update.sh install".

The following components of FreeBSD seem to be installed:
src/base src/bin src/contrib src/crypto src/etc src/games src/gnu
src/include src/krb5 src/lib src/libexec src/release src/rescue src/sbin
src/secure src/share src/sys src/tools src/ubin src/usbin world/base
world/catpages world/manpages

The following components of FreeBSD do not seem to be installed:
kernel/generic kernel/smp world/dict world/doc world/games world/info
world/proflibs

Does this look reasonable (y/n)? y

Fetching metadata signature for 6.3-RELEASE from update1.FreeBSD.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 6233 patches.....10....20....30.................6230. done.
Applying patches... done.
Fetching 647 files... done.

The following files will be removed as part of updating to 6.3-RELEASE-p1:
/usr/share/examples/netgraph/bluetooth/rc.bluetooth
/usr/share/man/cat3/archive_read_set_bytes_per_block.3.gz
/usr/share/man/cat3/archive_write_prepare.3.gz
/usr/share/man/cat4/kame.4.gz
/usr/share/man/man3/archive_read_set_bytes_per_block.3.gz
/usr/share/man/man3/archive_write_prepare.3.gz
/usr/share/man/man4/kame.4.gz
/usr/share/zoneinfo/Africa/Asmera
......

[[email protected] /]# sh freebsd-update.sh -f freebsd-update.conf install
[[email protected] /]# shutdown -r now

[[email protected] /]# sh freebsd-update.sh -f freebsd-update.conf install
[[email protected] /]# shutdown -r now

Ourmon to detect UDP Flood

Saturday, March 15th, 2008

Using graph to detect anomaly has been one of the great features in Ourmon. This is an example why it is great.

If you have been following this blog, you will notice I have 2 posts (including this) that have the strange traffic spike in the nighttime, and it also means that it is good time to attack when everyone is sleeping.
So, there was UDP flood attack on my network in the nighttime yesterday. By checking on Ourmon graphs and reports, I could identify the attack almost immediately without running any other tools.

This is how I identified the UDP flood attack, normally my network will not generate high UDP traffic.

UDP Flood Graph

Do you notice the TCP traffic spike from 2100 to 2200 which is marked in Pink color? I used the same method which I have mentioned previously to check the traffic spike with Argus. It was a inter network file transfer which was running at load 19,001,224 (bits/sec).

Notice the high port ICMP unreachable?

UDP Flood ICMP Unreachable Graph

From UDP Top report, I could see when and which IP launched UDP flood attack, and the time of the attack was stopped.

UDP Top Report

From UDP summarization Top Source IP report:

UDP Top Source IP Report

It is nice that we have not seen any large UDP traffic coming to our network after these IPs were blocked in Firewall.

Argus to check traffic spike

Saturday, March 8th, 2008

When I checked my Ourmon graph today, I noticed same spike happened at the same time in 2 days.
Usually I don’t really pay attention if there is little spike in the nighttime, but it is not normal when it happened continuously at the same time in 2 days.

Late night traffic spike

Since I have full network traffic log, I converted the tcpdump traffic to Argus flow.

 [[email protected] /nsmdir/2008-03-08]# argus -r snort.log.time -w argus.out 

The traffic spike was neither generated from port 25 nor port 80, I used racluster to merge the status records from the same flow without traffic of port 25 and 80, then piped to rasort to sort (-m) the total count of the packet transaction. -s option is to select and show the fields you want to print, -L0 is used to show the labels. Here is the result:

[[email protected] /nsmdir/2008-03-08]# racluster -nnr argus.out -w - - not port 80 and not port 25 | rasort -nnr - -m pkts -s +1ltime +load +bytes +rate -L0 | less

        StartTime           LastTime    	 Flgs   Proto            SrcAddr  Sport    Dir       DstAddr  Dport     TotPkts   TotBytes       State     Load   TotBytes         Rate
   03:27:47.764771    03:33:46.183595  e sD        6     192.168.134.56.39448     ->     10.2.3.102.46743   2501219 1031277512   FIN 23018378 1031277512  6978.480957
   03:33:46.708627    03:37:43.800844  e sD        6     192.168.134.56.40071     ->     10.2.3.102.16396    526755  543955768   FIN 18354234  543955768  2221.730469
   03:31:51.398870    03:33:15.015036  e sD        6     192.168.134.56.39880     ->     10.2.3.102.32666    212332  219148920   FIN 20967134  219148920  2539.365479
   03:21:43.412374    03:41:50.372550  e sD        6       192.168.200.77.4105    < ? >  10.2.3.102.38327     68916   56532894   CON 374712.5   56532894    57.098816

10.2.3.102 is one of the hosts in my network, so from the packet count and total bytes information, I could identify the spike was generated by which host. You may sort based on other criterias to get the result you want, as there might be flows from different IP with little packet count, with large bandwidth transfer rate (load).

To check the traffic, I used ra to show the flow:

[[email protected] /nsmdir/2008-03-08]# ra -nnr argus.out -L0 - host 192.168.134.56 | less

         StartTime    Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  TotPkts   TotBytes State
   03:27:46.938889  e           6     192.168.134.56.39446     ->     10.2.3.102.21           16       1464   CON
   03:27:47.537244  e           6     192.168.134.56.39447     ->     10.2.3.102.21           22       1964   CON
   03:27:47.764771  e s         6     192.168.134.56.39448     ->     10.2.3.102.46743     36705   15095192   CON
   03:27:52.764922  e sD        6     192.168.134.56.39448     ->     10.2.3.102.46743     30281   12522592   CON
   03:27:57.765092  e iD        6     192.168.134.56.39448     ->     10.2.3.102.46743     14032    5782634   CON
   03:28:02.765262  e sD        6     192.168.134.56.39448     ->     10.2.3.102.46743     41270   17023976   CON

This seems like a passive FTP transfer for me (by checking the src ports), so I logged on to server to check for the FTP log to verify:

Mar  8 03:33:10 server ftpd: ([email protected]@192.168.134.56) [NOTICE] /home/xxx/public_html/mm//xxx.com_2008.03.08_03-31-02 uploaded  (207309250 bytes, 2444.40KB/sec)
Mar  8 03:33:42 server ftpd: ([email protected]@192.168.134.56) [NOTICE] /home/xxx/backup//xxx.com_2008.03.08_03-21-02 uploaded  (891289600 bytes, 2428.22KB/sec)
Mar  8 03:37:39 server ftpd: ([email protected]@192.168.134.56) [NOTICE] /home/xxx/backup//xxx.com_2008.03.08_03-21-02.001 uploaded  (514473734 bytes, 2121.01KB/sec)

The log information explains all. There was a new job which was setup recently, running at every 3.30 AM to transfer the backup files to host 10.2.3.102.

I do have traffic grapher to check which ports generated the spike but that is another story which is not worth for a blog post.

Ourmon 2.8 on FreeBSD 7.0-RELEASE

Monday, March 3rd, 2008

FreeBSD LogoFreeBSD Team announced the release of FreeBSD 7.0-REL recently which introduces many new features and improvements. There is significant performance improvement for FreeBSD on those multi core systems and it has 15% better performance than best performing Linux kernel. I can’t wait to turn all my NSM sensors which are running on FreeBSD 6.2-REL to 7.0-REL.

Before that, Ourmon, the network monitoring and anomaly detection system had released version 2.8, which introduces a few exciting features as well. The most wanted THREADED (for me) feature is supported now. Although it is still under experimental and only x86(amd compatible) primitive is used for spinlocks, I will definitely give it a try on Quad core NSM sensor(AMD64) which has Ourmon disabled due to massive packets dropped during peak hours.

Ourmon graph:
Ourmon

The installation is pretty straight forward and easy since you just have to run configure.pl and change the home net address. Some people may want to change their network interface and web directory during installation, but the rest settings are default or being auto detected.

This is my installation of Ourmon 2.8 on FreeBSD 7.0-REL.
1. Install libpcre from ports. (/usr/ports/devel/pcre)
2. Install libpcap from ports. (/usr/ports/net/libpcap)
3. Install rrdtool from ports (/usr/ports/databases/rrdtool)
4. Install apache from ports. (/usr/ports/www/apache13, 20, or 22)
5. Run ourmon’s configure.pl.

Here is the output.

# ./configure.pl
configuration script to install ourmon.
note: default is suggested like so: [default]
note: just hit carriage-return for default actions
---------------------------------------------------
Would you like to install the ourmon probe? [y]
Front-end configuration phase started ####################
pcap in general needs to be reinstalled from www.tcpdump.org before ourmon install
found pcap lib in /usr/local/lib/libpcap.a - which is a good thing
hit CR to continue:

Would you like to compile/install ourmon? [y]
ourmon build: using make -f Makefile.bsd LIBS=/usr/local/lib/libpcre.a /usr/local/lib/libpcap.a
`ourmon' is up to date.

Next we determine the ourmon config/filter file to use.
By default, we use the local /usr/local/mrourmon/etc/ourmon.conf to provide input filters to ourmon.
WARNING: you should read/edit/understand ourmon.conf!
Do you want to use another ourmon.conf file in some other directory than /usr/local/mrourmon/etc? [n]

Next we suggest one modification to the ourmon.conf file.

If this is a default install, you should change the following config directive:

        topn_syn_homeip network/netmask

and set it to your home network and mask (A.B.C.D/maskbits style)
Do you want to change the topn_syn home network address? [y]
note: the home net address may be a subnet or host address (/32).
enter a home net address and mask. [127.0.0.1/32] 192.168.1.0/24
netmask: 192.168.1.0/24

Do you want to install the ourmon startup script in the ourmon bin? [y]
WARNING: the default for the interface may not be what you want.
WARNING: use #ifconfig -a to determine interfaces.
Please enter the input interface name to sniff from: [le0]
input interface is le0

Please enter directory for probe output files (mon.lite, etc.): [/usr/local/mrourmon/tmp]
probe output directory name is: /usr/local/mrourmon/tmp

Creating bin/ourmon.sh driver for startup of ourmon.
ourmon.sh placed in ourmon bin for ourmon front-end/probe startup
./ourmon.sh start

copy the startup script (bin/ourmon.sh) to /usr/local/etc/rc.d for boot startup? [y]
ourmon front-end install complete
ourmon front-end build worked

You should now run /usr/local/mrourmon/bin/ourmon.sh to start ourmon

e.g., # /usr/local/mrourmon/bin/ourmon.sh start

You can use ourmon.sh stop to stop ourmon

part 2: install the back-end, omupdate.pl, etc. (web part)? [y]
Back-end configuration phase started ######################################
We need a local web directory for generated web output.
hint: the webpath given here is a guess: give the CORRECT base web directory with /ourmon at the end
enter absolute web server web path directory: [/usr/local/www/data/ourmon] /usr/local/www/ourmon
your output web path is: /usr/local/www/ourmon

Do you want to create the web directory for ourmon?
HINT: good idea if it doesn't exist. [y]
mkdir: /usr/local/www/ourmon: File exists
ln: web.pages: File exists
cp bard/* /usr/local/www/ourmon/bard
cp batchip.sh batchipall.sh omupdate.sh /usr/local/mrourmon/bin
cp ombatch*.pl wormtolog.pl daily.pl monbackup.pl /usr/local/mrourmon/bin
cp omupdate.pl tcpworm.pl irc.pl /usr/local/mrourmon/bin
cp mklogdir.sh /usr/local/mrourmon/bin
chmod +x /usr/local/mrourmon/bin/*.sh
chmod +x /usr/local/mrourmon/bin/*.pl

INFO only: also setting up logging directory (if needed)
creating log rrddata tmp dirs, if necessary, in /usr/local/mrourmon
hit CR to continue:

If different, enter front-end output file directory absolute path: [/usr/local/mrourmon/tmp]
probe output file path (back-end input/s) is /usr/local/mrourmon/tmp

Now we copy supplied .html files to the web directory for later editing
do you want to copy base web files to the web directory? [y]

INFO only: setting up local rrdbase directory at /usr/local/mrourmon/rrddata
your runtime rrds get stored in this directory, along with the rrd error log file
if you create new BPF filters, check rrdbase/ourmon.log for errors.
hit CR to continue:

We need a UDP weight threshold for UDP scan alerts
what should the weight be (default is given): [10000000]

Install backend crontab commands in /etc/crontab (default answer y)?: [y]
y

ourmon system config complete
see INSTALL for post-config sanity checking

This is the error when I tried to start the ourmon.

[[email protected] /usr/local/etc/rc.d]# ./ourmon.sh start
sysctl: unknown oid 'debug.bpf_bufsize'
sysctl: unknown oid 'debug.bpf_maxbufsize'

When I opened the ourmon.sh, I found these lines under start_om():

sysctl -w debug.bpf_bufsize=8388608
sysctl -w debug.bpf_maxbufsize=8388608

In FreeBSD, they are called net.bpf.bufsize and net.bpf.maxbufsize. So just change them to these will do.

sysctl -w net.bpf.bufsize=8388608
sysctl -w net.bpf.maxbufsize=8388608

If you have your own value set in /etc/sysctl.conf, just comment these 2 lines.

I will post up as soon as Ourmon with threaded support is successfully setup on AMD64 system.