Apparently this works a LOT better on USB3 (due to the speed increase), but I only have a USB2 motherboard.
Installing CentOS onto a USB device works just like installing onto any other storage device (damn slow though).
The problem I found is that read/write speeds are very slow under USB2.
In general the system is usable, but any activities that involve writing to disk like yum, are very slow.
It could be that I had a particularly slow USB.  You can check your USB performance using hdparm.
In this example, sda was my USB and sdb was only a standard SATA drive. Big difference.

I ended up playing around with "dirty bytes". I basically set the OS to write things in RAM.  You can make these changes to /etc/sysctl.conf
vm.swappiness = 0
vm.dirty_background_ratio = 20
vm.dirty_expire_centisecs = 0
vm.dirty_ratio = 80
vm.dirty_writeback_centisecs = 0

Writes are syncd to disk at shutdown or when the sync command is run.
This seems to have made a slight improvment.  Overall the system is OK, if you can wait.
Great for rescue media or running a specific app that doesn't require a lot of disk usage.

I have since moved to MX Linux, which runs quite quickly on this same USB device.
MX Linux is a distro that focuses on being live media.
However it can be configured to store data persistently.
Currently playing with this to see how it performs as a hypervisor.