Thursday, August 18, 2016

Sed one-liners

I may occasionally publish small notes on clever commands I learn about.  Putting it here helps me store knowledge that my shoddy personal data management practices might otherwise lose...  One such note is a one-line sed command to print out the Linux interface(s) which handles the default route:
sed -n 's/\(^[^\t]*\)\t00000000.*/\1/p' /proc/net/route
An explanation, from left to right: Don't print each line (-n), prepare for a substitution ('s/), look for a string of non-tab characters at the beginning of the line (^[^\t]*) while saving the results (the \( and \) parts surrounding that), followed by a single tab and a string of eight 0s (\t00000000), followed by anything to gobble up the rest of the line (.*), then substitute it all with the non-tabs string saved earlier (/\1/), and print it (p').  The eight 0s represent the default route of

To be really specific the eight 0s specify a route for a network of undetermined size starting at  For the true default route, I should also check for a mask of 00000000, as OpenVPN sometimes adds two net routes ( and to avoid the need to replace the existing default route.  This command will find anything starting at as a default route, which may or may not be what you want...  In reality, since is reserved, if a route starts at zero, it's pretty defaulty anyway..

Friday, July 1, 2016

Forcing SaltStack to "knock harder"

I really like the "knocking harder" technique I developed.  I haven't seen it mentioned in any other places, and it effectively gives the protected service a smart layer of obscurity with minimal effort and complexity.

I also like SaltStack as a remote configuration and management tool.  It connects over two ports, and I was looking for a way to use my technique on this service.  Salt uses ports 4505 and 4506, where 4506 is first to connect and has several short-lived connections as well as a long-lasting session, and 4505 has a single long-lasting session.

I wanted to protect the first connection by requiring multiple SYN packets (a loud knock), but then allow connections to both ports with no delay as long as there's continuous traffic and sessions between them.  To that end, I've come up with the following patch to ufw's after.rules file.

Thursday, June 23, 2016

Functional Programming

Here's my stab at functional programming.  I've written what I perceive to be a typical functional statement:


Here's how I came up with this line.
1. I held down my shift key and mashed a bunch of keys on the top row.
2. I added a couple of common key words: most examples like map and range so I used them.
3. I evened out the parentheses and added a couple of brackets.


That's about as much sense as I can make out of the function language tutorials I've seen.  They always seem to have more obtuse symbols and meta characters than real names and keywords.

So what's up with that?!?

Thursday, April 2, 2015

Port knocking by "knocking harder"

TL;DR: use xt_recent to allow persistent clients rather than block them.

The xt_recent (formerly ipt_recent) module for iptables is well known as a way to limit a malicious attacker's ability to brute force your exposed SSH server.  Just do a google search for iptables brute force for examples.  All of these are reactionary, meaning they only start limiting connection attempts only after a certain number of successful TCP handshakes.

There are user-space programs that are more or less equivalent.  Fail2ban, in particular, watches the auth.log for failed logins, meaning it's not rate limiting raw TCP connections, but rather actual login failures, which is likely what you're really after anyway.  It's still reactionary, not blocking the IP until someone has had a go at it for a while.

I prefer something more proactive, and was looking at various flavors of port-knocking to meet my needs.  They too come in kernel and user space forms, but the kernel solution using xt_recent was way too complicated, in my opinion, involving several states and chains.  Plus, all the solutions require either a separate client or some other way of preceding your real connection attempt with the knock sequence.

Then it hit me.

Why not have the knock sequence just be a few extra knocks on the same port as the service?

Port scanners and brute force tools generally only send a single SYN packet.  If they don't get a response, they assume the port is filtered and move on.  Yet TCP is a reliable protocol.  If a client with a full TCP stack sends a SYN and doesn't get a response, it sends another.  Then another, and another.  While the wait between retries backs off between each attempt, within the first 4-5 seconds, I can reasonably expect the client to have sent 4 identical packets.

With this knowledge in hand, I tested a firewall ruleset which very effectively blocks most port scanners and brute forcers, but lets real clients in with only a 3-4 second delay on each connection attempt.  I call it "knocking harder."

Since I primarily use UFW on Ubuntu 12.04.5, I have edited the /etc/ufw/after.rules file with the following patch.  I put this in after.rules so that the normal user rules can still be used to whitelist or blacklist by source IP (eg. I don't want to have to "knock harder" from certain hosts, or I want to block certain bad IPs that resend SYNs).  Plus, it should still play nice with fail2ban, if you'd like to actively ban IPs with a series of failed service logons.

I've tested with the OpenSSH client on OS X and Ubuntu (12.04.5 and 14.04.1) as well as PuTTY on Windows 7.  It does delay the connection establishment by 3-4 seconds, but if you're using SSH multiplexing, that's only on the first concurrent connection.

In the short time it's been active, I have not seen any sources other than mine actually make it through iptables to the sshd service.  Even the IPs that were previously attempting to brute force me have been stopped.  As far as they know, I've taken the service offline or blocked their IP.

It's as close to an ideal solution as I've found for balancing between invisibility, complexity, and availability.

Monday, August 29, 2011

My favorite robots.txt

I always chuckle when I add this to a website...
User-agent: *

Disallow: /

Disallow: /harm/to/humans
Disallow: /ignoring/human/orders
Disallow: /harm/to/self

I forget where I first ran across it.

Monday, May 9, 2011

Using PowerShell to hash files

I found a way to use the built-in crypto functions to hash files on Windows systems using PowerShell. Paste the following lines into the PowerShell prompt, and specify one or more files when you run md5sum or sha1sum.

Function HashPrint ([byte[]]$hash)
  $result = ""
  foreach ($byte in $hash) {
    $result += "{0:x2}" -f $byte
  Write-Host $result -nonewline

Function md5sum
  $csp = [System.Security.Cryptography.MD5CryptoServiceProvider];
  $hasher = new-object $csp
  for ($count = 0; $count -lt $args.length; $count++ ) {
    [byte[]]$is = Get-Content -encoding byte $args[$count];
    if ( $is.length -gt 0 ) {
    } else {
      HashPrint($hasher.ComputeHash([char[]]"", 0, 0));
    Write-Host " " $args[$count]

Function sha1sum
  $csp = [System.Security.Cryptography.SHA1CryptoServiceProvider];
  $hasher = new-object $csp
  for ($count = 0; $count -lt $args.length; $count++ ) {
    [byte[]]$is = Get-Content -encoding byte $args[$count];
    if ( $is.length -gt 0 ) {
    } else {
      HashPrint($hasher.ComputeHash([char[]]"", 0, 0));
    Write-Host " " $args[$count]

The zero length kludge is the only way I found to work around trying to pass a zero-length string to the ComputeHash function. ComputeHash complains about having only one empty argument, and .length of a zero-length string returns null rather than 0. (!!!) Since that hash is always fixed, I could just let it error out, or fake the result with the expected hash.

Note: this may crash on sufficiently large files, since it stores the file in a variable. I'm sure there's a way to use streams that doesn't have this limitation, anyone care to help out?

Friday, June 11, 2010

Migrating an ext3/4 filesystem to LVM with minimal downtime

There have been a few posts asking about moving a Linux file system from a raw partition onto an LVM. The advantages to using LVMs are numerous, but the advice has generally been to reinstall from scratch.

The problem is, several of the steps must be performed while the file system is unmounted. Specifically, both reducing the ext2/3/4 file system size and moving it from the raw partition onto the logical volume. Additionally, the /boot directory must be moved to a separate physical partition, and the initramfs it contains needs to have the lvm2 tools installed.

The following steps can be followed to migrate from a single monolithic partition (+ swap) to a typical LVM configuration where root and swap are on logical volumes. It may be possible to encrypt the LVM physical volume during this process, and I may explore that in a later post.

/dev/sda1 contains a monolithic / file system
/dev/sda5 contains the swap partition
The boot loader is GRUB2

Step 1: Make a backup

Seriously. Don't have any critical data that's only available on this system. In fact, you may want to take an older, idle system and practice this whole procedure on it. There are plenty of opportunities to screw up, and this guide assumes you're already pretty familiar with the LVM2 and ext tools and HDD partitioning.

Step 2: Prepare the system

Add lvm2 and dependant packages if needed. On Ubuntu, this would be:
# apt-get -y install lvm2

Determine whether you will need a swap file. Take a look at the output of top to see if you've got enough physical memory that you can just disable swap for the duration of the migration. If you will not need swap, skip ahead to plan your partition size. If you will need it, migrate your swap from a partition to a swap file using the following steps (example below creates a 256MB file; increase as you see fit):
# dd if=/dev/zero of=/swapfile bs=1048576 count=256
# mkswap /swapfile
# echo "/swapfile none swap sw 0 0" >> /etc/fstab
# swapon -a
# cat /proc/swaps
Verify that /proc/swaps shows that /swapfile is active and the right size.

Step 3: Make a plan

You need to determine the size and location of a temporary partition to hold your file system while you're making the conversion. You will need the file system "block size" from:
# dumpe2fs -h /dev/sda1
On my test plaform, it is 4096, but may vary depending on your disk size.

You will also need the sector size from:
# fdisk -lu /dev/sda
It is almost always 512. Using these two values, calculate the sectors per block by dividing block size by sector size (eg. 8)

From the fdisk output, you will also need the starting sector of partition 1 (usually either 63 or 2048), and the count of total sectors displayed in the header above the sector size.

You will also need to know the amount of space used on your root filesystem:
# df -h /

Now determine your temporary file system size. Take your used space, and add a comfortable margin of 10-20%, and round up. For instance, if you're using 1.5GB, you may want to make it 3GB. If you're using 45GB, you may want to keep at least 50GB, perhaps more.

In any case, since you'll be duplicating it across your internal disk, your target size should be a little less than 50% of your total disk size. If you cannot shrink the file system to less than 50% of the HDD size, you may have luck with an external eSATA or USB drive, but test it first.

Step 4: Boot into LiveCD

Now we get to the scary part. Since you're bringing the system down, if there are business critical apps running on it, you'll have to do this during a maintenance window.

Step 5. Re-size the file system

This is the main reason the file system must be unmounted. This command is not supported on a mounted ext2/3/4 file system. Substitute the value of 50G in the example below with the size you determined above.
# e2fsck -f /dev/sda1
# resize2fs /dev/sda1 50G

Determine the number of sectors used by the re-sized file system. Do this by multiplying the "Block count" from the following command with the sectors per block calculated above:
# dumpe2fs -h /dev/sda1
Add about 10000 sectors to this for the LVM headers, and subtract it from the drive's total sector count recorded above to determine the starting sector for partition 3.

Step 6. Repartition the drive

This will make room for the temporary partition
# fdisk -cu /dev/sda
Press 'p' to print the current partition table.
Press 'd' and '5' to delete the swap partition, 'd' and '2' to delete the extended partition table, and 'd' to delete the last remaining partition.
Create partition 3 to hold your temporary file system by pressing 'n', 'p', '3', "<start sector P3>", "" (New Partition, Primary, partition 3, starting sector, max size). Substitute the starting sector with what was calculated above.
Recreate partition 1 with the size determined above by pressing 'n', 'p', '1', "<start sector P1>", "" (New partition, Primary, partition 1, starting at 63 (or 2048), max size).
Finally, press 'w' to write your changes to disk and recreate the device nodes.

If ioctl is unable to re-read the partition table after writing to disk, you must reboot into the LiveCD to recreate the proper block devices.

Step 7. Install LVM tools.

You'll need internet connectivity for this, so if you're using a fixed IP, you'll need to first configure the interface and default route manually. Then install the LVM2 tools using a command such as:
# apt-get -y install lvm2

Step 8. Create your logical volume.

Use the following sequence of commands to create a mapped device for your root file system:
# pvcreate /dev/sda3
# vgcreate os /dev/sda3
# lvcreate -l 100%VG -n root os

Verify there are enough sectors to hold existing file system
# blockdev --getsize /dev/mapper/os-root
The number of sectors returned must be more than the file system size calculated above. If not, subtract a few thousand sectors from your partition 3 starting sector, repartition, and try again.

Step 9. Copy your root file system.

This is the other command that must be done while the file system is unmounted:
# dd if=/dev/sda1 of=/dev/mapper/os-root

Step 10. Repartition the drive again

Deactivate the volume group to allow the next command to succeed:
# vgchange -a n os

Up until this point, your system would still boot if you lost power. From this point until GRUB is reinstalled, you'll need the LiveCD to recover your system.
# fdisk -cu /dev/sda
Delete partition 1 by entering 'd', '1'.
Create a new 300MB partition 1 for the /boot file system by entering 'n', 'p', '1', "", "+300M".
Create an extended partition table with the remaining space by entering 'n', 'e', '2', "", "".
Create a logical partition filling the extended partition by entering 'n', 'l', "", "".
Change the file system type value on partition 5 by entering 't', '5', "8e".
Write changes to disk by entering 'w'.

Step 11. Separate /boot to its own partition

The /boot directory contains the kernel and initramfs as well as all of GRUB's modules and config files. It must be available as a separate file system in order to map the logical volume and access root.
# vgchange -a y os
# mount /dev/mapper/os-root /mnt

# mke2fs /dev/sda1
# mkdir /newboot
# mount /dev/sda1 /newboot
# cd /mnt/boot
# tar cf - . | (cd /newboot; tar xvf - )
# echo "/dev/sda1 /boot ext2 defaults 2 0" >> /mnt/etc/fstab

Step 12. Reinstall GRUB

These steps are for GRUBv2. Substitute appropriate commands for GRUB Legacy.
# for mp in dev sys proc; do mount -o bind /$mp /mnt/$mp; done
# chroot /mnt
# update-grub
# grub-install /dev/sda
Handle any errors you see at this point or your system will not boot.

Step 13. Reboot

You should be able to boot into your original system at this point, albeit with reduced file system space.
If you get a GRUB rescue prompt, something went wrong with your GRUB install.
If you get a GRUB error about no disk found, the GRUB configuration was not built correctly.
If you get the {initramfs} prompt, it could not find the root file system in the logical volume, was lvm2 installed before booting into the LiveCD?
I got a blank screen the first boot, but it appeared to be an issue. I hit Ctrl-Alt-Del, and it came up fine upon rebooting. It scared me, but turned out to be nothing.

Step 14. Move your logical volume.

If you're going to encrypt the resulting system, here is where you would prepare /dev/sda5. I may try this in the future, but for now, you're on your own if you attempt this.

These commands add the new space to the volume group and move root to it:
# pvcreate /dev/sda5
# pvscan
# vgextend os /dev/sda5
# pvmove -i 60 -n root /dev/sda3 /dev/sda5

Then you can remove the unused physical volume:
# pvscan
# vgreduce os /dev/sda3
# pvremove /dev/sda3

Step 15. Repartition the drive one final time

Now that we no longer need partition 3, we delete it and extend the logical partition all the way.
# fdisk -cu /dev/sda
Note the starting sector of P2 when you enter 'p'.
Remove partition 3 by entering 'd', '3'.
Remove partitions 5 and 2 by entering 'd', '5', 'd', '2'.
Recreate extended partition 2 by entering 'n', 'e', '2', "", "".
Recreate logical partition 5 by entering 'n', 'l', '5', "", "".
Write changes to disk by entering 'w'.
Note: you will see a warning about ioctl being unable to reload the partition table. This is expected, since the file system is using it.

Step 16. Reboot into your final configuration.

If your reboot is to be delayed, you can buy some time by growing the root logical volume and file system to more usable sizes until you can reboot. If you can reboot now, don't bother with resizing here, as you'll be able to grow it to its final size after rebooting.
# lvresize -l 100%VG /dev/mapper/os-root
# resize2fs /dev/mapper/os-root

Step 17. Grow root to its final size.

Now is your chance to determine how much swap space you really want. If you don't need swap at all, use '-l 100%VG" to take all available space. Otherwise, leave space for a swap volume added in the next step.

Use a combination of the "-l nn%VG" and "-L +n.nG" options to bring your root volume up to the desired size. Do not shrink it unless you know for sure you're not releasing areas that your file system is using.
# pvresize /dev/sda5
# pvs
# lvresize -l 95%VG /dev/mapper/os-root
# lvresize -L +1.24G /dev/mapper/os-root
# resize2fs /dev/mapper/os-root

Step 18. Restore your swap partition

Create your swap partition and restore swap to it.
# lvcreate -n swap -l 100%FREE os

# mkswap /dev/mapper/os-swap
# swapon -a
or, to activate it automatically, use the UUID specified in your fstab file as:
# mkswap -U /dev/mapper/os-swap

Verify your swap partition is active before disabling the swap file with:
# cat /proc/swaps
# swapoff /swapfile
# rm /swapfile

Finally, edit your /etc/fstab file to remove the line starting with /swapfile.

Step 19. Breathe

You've done it! I believe this is the best way to perform this task with the least amount of down time. If you have suggestions or other thoughts, let me know.