Recovering from Suricata Gone Wild

Recently I tried interacting with one of my lab Security Onion sensors running the Suricata IDS. I found the Sguil server was taking a really long time to offer services on port 7734 TCP. Since I hadn't worked with this lab system in a while, I guessed that there might be too many uncategorized events in the Sguil database. I dusted off an old blog post titled More Snort and Sguil Tuning from 2006 and took a look at the system.

First I stopped the NSM applications on the server.

sudo service nsm stop
Stopping: securityonion
  * stopping: sguil server                                [  OK  ]
Stopping: HIDS
  * stopping: ossec_agent (sguil)                         [  OK  ]
Stopping: Bro
stopping ds61so-eth1-1 ...
stopping proxy ...
stopping manager ...
Stopping: ds61so-eth1
  * stopping: netsniff-ng (full packet data)              [  OK  ]
  * stopping: pcap_agent (sguil)                          [  OK  ]
  * stopping: snort_agent (sguil)                         [  OK  ]
  * stopping: suricata (alert data)                       [  OK  ]
  * stopping: barnyard2 (spooler, unified2 format)        [  OK  ]
  * stopping: prads (sessions/assets)                     [  OK  ]
  * stopping: sancp_agent (sguil)                         [  OK  ]
  * stopping: pads_agent (sguil)                          [  OK  ]
  * stopping: argus                                       [  OK  ]
  * stopping: http_agent (sguil)                      
Next I ran a query to look for the top uncategorized events.
$ mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1639
Server version: 5.5.29-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use securityonion_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
+---------+----------------------------------------------------------------------------------+
| count   | signature                                                                        |
+---------+----------------------------------------------------------------------------------+
| 2299160 | SURICATA STREAM Packet with invalid ack                                          |
| 2298505 | SURICATA STREAM ESTABLISHED invalid ack                                          |
| 1777530 | SURICATA STREAM ESTABLISHED packet out of window                                 |
|   38700 | SURICATA STREAM ESTABLISHED retransmission packet before last ack                |
|   24181 | SURICATA STREAM TIMEWAIT ACK with wrong seq                                      |
|    5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management |
|    3160 | SURICATA STREAM Last ACK with wrong seq                                          |
|     753 | ET POLICY Dropbox.com Offsite File Backup in Use                                 |
|     637 | SURICATA HTTP unknown error                                                      |
|     626 | SURICATA STREAM SHUTDOWN RST invalid ack                                         |
|     505 | SURICATA STREAM FIN1 FIN with wrong seq                                          |
|     494 | SURICATA HTTP request field too long                                             |
|     448 | ET POLICY PE EXE or DLL Windows file download                                    |
|     315 | ET RBN Known Malvertiser IP (22)                                                 |
|     270 | ET POLICY iTunes User Agent                                                      |
|     266 | SURICATA STREAM CLOSEWAIT ACK out of window                                      |
|     237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                   |
|     219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                      |
|     217 | SURICATA STREAM 3way handshake with ack in wrong dir                             |
|     151 | SURICATA STREAM FIN2 FIN with wrong seq                                          |
+---------+----------------------------------------------------------------------------------+
20 rows in set (15.24 sec)
Wow, that's a lot of SURICATA STREAM events. I need to categorize them as non-issues to recover the Sguil server.

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Query OK, 6443375 rows affected, 65535 warnings (3 min 4.89 sec)
Rows matched: 6443375  Changed: 6443375  Warnings: 6443375
Let's see what the database thinks now.
mysql> SELECT COUNT(signature)as count, signature FROM event WHERE status=0 GROUP BY signature ORDER BY count DESC LIMIT 20;
+------+-----------------------------------------------------------------------------------------+
| cnt  | signature                                                                               |
+------+-----------------------------------------------------------------------------------------+
| 5430 | ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management        |
|  753 | ET POLICY Dropbox.com Offsite File Backup in Use                                        |
|  637 | SURICATA HTTP unknown error                                                             |
|  494 | SURICATA HTTP request field too long                                                    |
|  448 | ET POLICY PE EXE or DLL Windows file download                                           |
|  315 | ET RBN Known Malvertiser IP (22)                                                        |
|  270 | ET POLICY iTunes User Agent                                                             |
|  237 | ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)                          |
|  219 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard                             |
|  133 | ET INFO PDF Using CCITTFax Filter                                                       |
|  106 | ET POLICY Pandora Usage                                                                 |
|   97 | ET CHAT Facebook Chat (buddy list)                                                      |
|   93 | ET POLICY MOBILE Apple device leaking UDID from SpringBoard via GET                     |
|   58 | ET POLICY Internal Host Retrieving External IP via whatismyip.com - Possible Infection  |
|   41 | PADS New Asset - ssl TLS 1.0 Client Hello                                               |
|   39 | SURICATA HTTP response header invalid                                                   |
|   39 | ET CURRENT_EVENTS Exploit Kit Delivering Compressed Flash Content to Client             |
|   36 | ET POLICY Python-urllib/ Suspicious User Agent                                          |
|   36 | ET MALWARE Possible Windows executable sent when remote host claims to send a Text File |
|   28 | ET POLICY Http Client Body contains pw= in cleartext                                    |
+------+-----------------------------------------------------------------------------------------+
20 rows in set (0.03 sec)
That's much better.

Before restarting the NSM services, I edit the autocat.conf file to add the following.

none||ANY||ANY||ANY||ANY||ANY||ANY||%%REGEXP%%^SURICATA STREAM||1
This will auto-categorize any SURICATA STREAM alerts as non-issues. I want to keep adding events to the database for testing purposes, but I don't want to see them in the console.

Now I restart the NSM services.

sudo service nsm start
Starting: securityonion
  * starting: sguil server                                                                [  OK  ]
Starting: HIDS
  * starting: ossec_agent (sguil)                                                         [  OK  ]
Starting: Bro
starting manager ...
starting proxy ...
starting ds61so-eth1-1 ...
Starting: ds61so-eth1
  * starting: netsniff-ng (full packet data)                                              [  OK  ]
  * starting: pcap_agent (sguil)                                                          [  OK  ]
  * starting: snort_agent (sguil)                                                         [  OK  ]
  * starting: suricata (alert data)                                                       [  OK  ]
  * starting: barnyard2 (spooler, unified2 format)                                        [  OK  ]
  * starting: prads (sessions/assets)                                                     [  OK  ]
  * starting: pads_agent (sguil)                                                          [  OK  ]
  * starting: sancp_agent (sguil)                                                         [  OK  ]
  * starting: argus                                                                       [  OK  ]
  * starting: http_agent (sguil)                                                          [  OK  ]
  * disk space currently at 22%
I check to see if port 7734 TCP is listening.
sudo netstat -natup | grep 7734
tcp        0      0 0.0.0.0:7734            0.0.0.0:*               LISTEN      10729/tclsh
Now the Sguil server is listening. I can connect with a Sguil client, even the 64 bit Windows .exe that I just found this morning. Check it out at sourceforge.net/projects/sguil/

Comments

Unknown said…
Congratulations Richard Bejtlich! Thank you so much for taking the time to share this exciting information.

IT engineers
Anonymous said…
Hello,
part of this line is missing in your article:

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature


could you pleas complete it?

thanks!
Maybe

mysql> UPDATE event SET status=1, last_modified='2013-02-24 16:26:00', last_uid='sguil' WHERE event.status=0 and event.signature LIKE 'SURICATA STREAM%';
Said said…
Dude you just saved my life today.
Cheers!

Popular posts from this blog

Zeek in Action Videos

New Book! The Best of TaoSecurity Blog, Volume 4

MITRE ATT&CK Tactics Are Not Tactics