Transport layer for stealth and evil, 2014 edition

A few years ago, I tested some obscure transport protocols on Ubuntu 10.04 and whether you could find processes listening for them on your system. Answer was - it depends.

Fast forward a few years, RHEL7 and CentOS 7 appears. Long-deprecated netstat is gone from the default install.

So let's repeat the tests for weird backdoors.

SCTP

yum install socat
socat SCTP-LISTEN:8080 TCP-CONNECT:22

ss(8) is the official replacement for netstat(8).

So what does ss see?

# ss -anp | grep socat
#

Nothing. netstat used to see this on Red Hat-derived distros, so a slight regression you might say. But ss didn't see it in 2010 either.

Lsof has improved, though, and identifies it correctly this time.
 # lsof | grep socat
socat 10790 root 3u sock 30501 0t0 SCTP ENDPT:ffff88003b23df00 0.0.0.0[8080]
#

DCCP

Well, first we need to get netsend up and running. Basically yum install gcc, and get the source from the ubuntu package. The original berlios upstream seems to be gone.

# netsend dccp receive

This sets up a listener at dccp port 6666.

# ss -anp | grep netsend
tcp LISTEN 0 0 *:6666 users:((netsend,11050,3))
#

Um, yes, it shows up ... but looks like a TCP socket? WTF?

telnet localhost 6666
gives connection refused as expected. This will definitely confuse somebody.

# ss -anpd
LISTEN 0 0 *:6666
users:((netsend,11050,3))

So with the -d option (for dccp) it shows sane data.

What about lsof then?
# lsof | grep netsend
netsend 11050 root 3u sock 0,6 0t0 33020 protocol: DCCP

Well, better than before. It at least identifies the protocol, if not the details.

UDP-Lite

This massively popular protocol then?

# netsend udplite receive

Sets up a listener at port 6666, again.

# ss -anp | grep netsend
#

Blank, as in 2010. And there's no udplite option.

lsof then?

# lsof | grep netsend
netsend 17664 root 3u IPv4 38944 0t0 UDPLITE *:6666
#

Yeah, no problem, identifies it perfectly.

Conclusion

Hiding a backdoor in plain sight with an uncommon protocol is still viable, though local firewalls will mitigate. ss(8) still doesn't give you all the info it ideally should, probably because of some missing plumbing. lsof(8) is still a useful swiss army knife slowly getting more tools.

a bit of selinux for noobs

So, once again, you've added an initscript that works fine in test to a prod server. It fails. And you get permission denied to write to a directory - but waitaminute, the process starts as root?

The obvious suspect: selinux. Turning it off is the stupid way out and not an option.

[root@gnuff ~]# grep denied /var/log/audit/audit.log
type=AVC msg=audit(1389617233.067:1073936): avc:  denied  { write } for  pid=34988 comm="tcpdump" name="ts" dev=dm-2 ino=1572884 scontext=unconfined_u:system_r:netutils_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir

So tcpdump isn't allowed to write to a var_t directory. Fair enough, really. What is it allowed to write to, from a confined user?

sesearch to the rescue! (yum install setools-console)

[root@gnuff ~]# sesearch -T -s netutils_t
Found 3 semantic te rules:
   type_transition netutils_t tmp_t : file netutils_tmp_t;
   type_transition netutils_t tmp_t : dir netutils_tmp_t;
   type_transition netutils_t abrt_helper_exec_t : process abrt_helper_t;

This searches for allowed type transitions from the netutils_t context. Aha!

[root@gnuff ~]# chcon -t netutils_tmp_t /var/cache/ts

And restart ... and voila! No more permission denied.

[root@gnuff ~]# semanage fcontext -a -t netutils_tmp_t /var/cache/ts

makes it permanent. (yum install policycoreutils-python if you have no semanage)

Better do it via puppet, of course. (selinux_fcontext)

PS. The strictly correct option is rather:
 [root@gnuff ~]# sesearch -A -s netutils_t | grep dir | grep write
   allow netutils_t netutils_tmp_t : dir { ioctl read write create getattr setattr lock unlink link rename add_name remove_name reparent search rmdir open } ;
   allow netutils_t dirsrv_var_run_t : sock_file { write getattr append open } ;
   allow netutils_t tmp_t : dir { ioctl read write getattr lock add_name remove_name search open } ;

which searches for allow rules. But the type transition rule search gets far fewer hits and is worth a try as a first approximation when doing a sysadmin fix-it-fast search. Few apps transition to a type they can't read/write.

([root@gnuff ~]# sesearch -A -s netutils_t | wc -l
365
)

htrosbif alpha 5 - years late edition

So, somebody recently reminded me that htrosbif has languished for a while. And that's true, but I've had a bunch of changes pending since alpha 4.

So I got off my ass and packaged an alpha 5. Get it while it's... slightly warmer than Oslo in May.

If you've followed the git repo, you've had most of this for ages. ( git clone http://anduin.net/~knan/htrosbif.git/ )

Wishlist: anyone got a buildroot/openwrt-style build-webserversoftware-and-a-chroot-for-it script? That would speed up testing and signature generation awesomely.

Shortlog:

Erik Inge Bolsø (38):
      add monkey sig (insert obvious jokes here)
      xs-httpd added to testsuite. nicely old-school.
      userfriendliness bugfix (slashdot 1z broken! eek!)
      slash fiction
      add continuation tests, split into first/second/third-class tests
      basic first-class matching
      implement overlay test promotion
      fix some more hashref buglets, split classify_test from test run
      fix overlay matching bug if header list was empty
      add more modifiers, for better haproxy matching in strange circumstances
      minor pound detection bugfixes
      relax and prettify our json file handling
      don't overcompensate for lost tests
      varnish matching fixes
      refresh sigs, prettier now
      more json prettifying and relaxation
      sig_deref.pl cleanup
      standardize header names - yes, this throws away useful info, I may regret it later
      add fixme while I remember
      varnish=>zope,monkey,bauk quirks
      a bit of overlay debug code
      yes, varnish hides some interesting version-specific lighty bugs
      refresh old-cups & iis sigs
      add get_11_duplicate_contentlength test
      use caller function for debug tracing
      refresh sigs for new test
      one more varnish quirk
      some eccm, and fixed an overlay matching with limited info bug
      improve scoring with limited information
      launch flares, tune overlay detection threshold (hi kwy)
      add comotion-httpd sig (rare!)
      disable countermeasures by default, add --eccm option, add monkey-0.11.1 sig
      lighty 1.4 hasn't changed signature lately
      a random resin sig, and tomcat sigs
      refresh iis sig
      cleanup signatures a bit, preparing for squid
      minor signature generation bugfix - could drop http/0.9 style responses completely
      refresh sigs for latest changes

Kacper Wysocki (1):
      usability: try to autosense the url

Unbreaking the web with greasemonkey

So, a certain MMO recently managed to turn off their wiki, upgrade their forums and in the process break almost all their links from anywhere. Even intra-forum links broke utterly. And the in-game big friendly Help button.

I'd really not want to responsible for that debacle. Breaking seven years worth of web forum google searches and intralinking is utter insanity from a professional standpoint.

But a ten-minute greasemonkey script can fix that for me personally.

DDO forum fixup

Well done irritating your subscribers, Turbine. Whatever will they think of next...

Netbooting NetBSD in KVM

So, I feel the need to netboot various OSes again. Because I can.

Output to serial is most convenient, my MIPS box is serial only, and besides I have no space for a bunch of extra monitors.

In the IA32 case, the (very very nice, though old) Diskless NetBSD howto directs me to use "pxeboot_ia32_com0.bin".

Problem #1: since NetBSD 4 or so, this isn't in base.tgz anymore.

David Laight points out that it can be created like so using the netbsd installboot(8) command. Though I have no NetBSD yet.

Ah well. I'll boot a more normal NetBSD on KVM and create that, then. Quick DHCPD setup:

subnet 10.50.1.0 netmask 255.255.255.0 {
    range 10.50.1.200 10.50.1.250;
}

host kvm-netbsd-ia32 {
  hardware ethernet 22:22:22:22:00:00;
  next-server 10.50.1.1;                        # TFTP / NFS host
  filename "/netbsd-5/pxeboot_ia32.bin";        # Path on TFTP server

  option root-path "/storage/nfs/ia32/netbsd-5.1.2/root"; # Path on NFS server
                                                          # FreeBSD pxe can use IP:path here, I believe, NetBSD just uses next-server.
}

... not so fast.

ISC DHCPD, current KVM's iPXE netboot firmware and NetBSD's pxeboot doesn't like each other.

* iPXE sends a DHCP request with dhcp_client_identifier (option 61) set to 01-ethernet mac address, loads pxeboot_ia32.bin from tftp and runs it.
* pxeboot_ia32.bin sends a simpler DHCP request without dhcp_client_identifier via PXE_UDP_WRITE. pxeboot_ia32.bin takes care to use the lease it knows about from the PXE environment as source IP, it just wants to ask for some more options.
* ISC DHCPD wants to assign a new address, since this apparently is another client (no dhcp_client_identifier), and assigns that as unicast destination.
* iPXE ignores the return unicast packet apparently sent to somebody else entirely.
* pxeboot_ia32.bin gets no response and gives up.

ISC DHCPD has a special case for BOOTP clients, and will give out an old lease if the hardware address matches even though they don't send any client-id. But we aren't BOOTP, we're just fragile followup-DHCP-in-PXE.

The least icky way out here seems to be patching DHCPD to ignore dhcp_client_identifier mismatches. (CMU DHCPD has a runtime option for it, btw) But oh, the tentacles, ISC source isn't known for its sanity...

hack for isc dhcp 4.2.4

So at this point I'm doing a traditional
  pxe rom (iPXE 1.0 as shipped with modern qemu/kvm, or whatever PXE rom my old VIA Mini-ITX is running)
  => tftp loads pxeboot second stage
  => pxeboot loads kernel via simplified nfs
  => kernel boots a nfsroot system

The smart & modern way is of course to load the netbsd kernel directly from iPXE. But I'm betting that's too easy to be any fun.


Anyway, qemu-kvm in ubuntu precise has fun bugs that doesn't let the NetBSD kernel find any PCI devices. Like, for instance, the network card. So the boot stops with netboot interface not found.

Fixed in current debian and upstream qemu. A rebuild of QEMU seems to be in order.

apt-get source qemu-kvm
apt-get build-dep qemu-kvm
cd qemu-kvm-1.0+noroms/debian/patches
wget -O 'pci-fix-corrupted-pci-conf-index-register-by-unaligned-write.patch' "http://git.qemu.org/?p=qemu.git;a=commitdiff_plain;h=cdde6ffc27517bdf069734fbc5693ce2b14edc75"
echo pci-fix-corrupted-pci-conf-index-register-by-unaligned-write.patch >> series
cd ..
dch -i
cd ..
dpkg-buildpackage -b -uc
cd ..
dpkg -i qemu-kvm_1.0+noroms-0ubuntu14.2eib1_amd64.deb qemu-utils_1.0+noroms-0ubuntu14.2eib1_amd64.deb qemu-common_1.0+noroms-0ubuntu14.2eib1_all.deb

et voila!

Whee, I have a working NetBSD!
Current invocation: kvm -boot n -localtime -net nic,vlan=0,macaddr=22:22:22:22:00:00 -net tap,vlan=0,ifname=tap0,script=no,downscript=no

Now then, in netbsd (After changing root's shell to /bin/sh instead of the traditonal csh abomination)

cd /usr/mdec
cp ./pxeboot_ia32.bin ./pxeboot_ia32_com0.bin
installboot -e -oconsole=com0 -ospeed=0 ./pxeboot_ia32_com0.bin

So far, so good. Drop our new binary in, and boot kvm with no VGA and a serial console:

kvm -boot n -localtime -net nic,vlan=0,macaddr=22:22:22:22:00:00 -net tap,vlan=0,ifname=tap0,script=no,downscript=no -vga none -nographic

knan@sarevok:/storage/current/netboot$ ./kvm-ia32-netbsd.sh

>> NetBSD/x86 PXE Boot, Revision 5.1 (from NetBSD 5.1.2)
>> Memory: 623/391904 k
Press return to boot now, any other key for boot menu
Starting in 0 seconds.    
PXE BIOS Version 2.1
Using PCI device at bus 0 device 2 function 0
Ethernet address 22:22:22:22:00:00
10090076+518916+618576 [521184+509293]=0xbb2988
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.1.2 (GENERIC) #0: Thu Feb  2 17:22:10 UTC 2012
    builds@b6.netbsd.org:/home/builds/ab/netbsd-5-1-2-RELEASE/i386/201202021012Z-obj/home/builds/ab/netbsd-5-1-2-RELEASE/src/sys/arch/i386/compile/GENERIC
[...]
Sun Oct  7 15:54:24 UTC 2012

NetBSD/i386 (netbsd5) (console)

login: root

Yay!

(Todo: train creaky old pxeboot_ia32.bin to retry DHCP via broadcast if unicast fails.)

FreeRADIUS and CRLs

Deploying FreeRADIUS, you may want to use a Certificate Revocation List (CRL) to revoke access to users that has been issued a valid certificate at some point.

How to actually do this isn't completely self-evident. Less so if you use multiple sub-CAs.

FreeRADIUS basically uses OpenSSL's routines and options for this. And OpenSSL doesn't currently refresh CRLs in a running process, so after updating a CRL you need to restart the freeradius daemon (easily scripted).

The important point in freeradius is in /etc/freeradius/eap.conf, specifically check_crl and CA_path in the tls section. Also, comment out CA_file, since CA_path and CA_file are intended for the same purposes, but CRLs only work with the CA_path method.

   #  Check the Certificate Revocation List
   #
   #  1) Copy CA certificates and CRLs to same directory.
   #  2) Execute 'c_rehash '.
   #    'c_rehash' is OpenSSL's command.
   #  3) uncomment the line below.
   #  5) Restart radiusd
   check_crl = yes
   CA_path = /etc/freeradius/certs/CA/

Be sure not to concatenate certificates in your CA_path. One CA, one file. Then, after the c_rehash, your directory should look something like this:

root@radius:~# ls -l /etc/freeradius/certs/CA/
lrwxrwxrwx 1 root freerad    9 2012-04-26 03:15 12345678.0 -> local-sub-ca-1.pem
lrwxrwxrwx 1 root freerad   13 2012-04-26 03:15 12345678.r0 -> sub-ca-1-crl.pem
lrwxrwxrwx 1 root freerad   11 2012-04-26 03:15 23456789.0 -> local-root-ca.pem
lrwxrwxrwx 1 root freerad    8 2012-04-26 03:15 3456789a.0 -> local-sub-ca-2.pem
lrwxrwxrwx 1 root freerad   12 2012-04-26 03:15 3456789a.r0 -> sub-ca-2-crl.pem
-rw-r--r-- 1 root freerad 1360 2012-04-24 15:04 local-root-ca.pem
-rw-r--r-- 1 root freerad 1360 2012-04-24 15:06 local-sub-ca-1.pem
-rw-r--r-- 1 root freerad 1384 2012-04-24 14:58 local-sub-ca-2.pem
-rw-r--r-- 1 root freerad 1251 2012-04-26 03:15 sub-ca-1-crl.pem
-rw-r--r-- 1 root freerad  739 2012-04-26 03:15 sub-ca-2-crl.pem

Thanks to Yasuhiro ABE for pointing me in the right direction. ありがとうございました、あべさん！それはよかったよ！

dnstool - curses-based DNS zone administration

Out now! Get it while the bits are still hot!

https://github.com/Redpill-Linpro/dnstool

The Fine Manual is plain text and even has screenshots, perfect for the old-school lynx enthusiast ;)

If your dns server supports RFC2136 dynamic updates, you might want to look at this. BIND9 does.

Patches always welcome!