*** Post writing doco finding - OK so after using this on my newly acquired server I had issues with VMs and storage, probably don't try this!
It's great for just a Linux Workstation on the Go though.

Goal - to run Linux from a USB device and be able create VMs on it.
Design - MX Linux running on a USB with Virtual Box installed.
Access from client via SSH and virtualbox GUI over X11 forwarding.
VM disks to be installed on separate storage.

I also considered KVM & VNC as alternatives to VirtualBox & X11.
However the later was simpler, especially after I wasted so  much time trying to get VNC working!

MX is a Debian based Linux distro that is optimised to run as live media.
When you boot MX Linux from a USB you can configure it to store root (/) and home (/home) persistently.
While live it seems to save changes in RAM or some optimised other location.
When you poweroff it commits changes to disk permanently using rsync (this is the slow bit!).

I am used to CentOS so had to learn a couple of Ubuntu commands, some of which detailed below.

Set up automatic updates:
sudo apt install unattended-upgrades apt-listchanges
sudo dpkg-reconfigure -plow unattended-upgrades

Install virtualbox after running an apt-get update
apt-get install virtualbox

Configure firewall to allow SSH
ufw enable
ufw allow ssh

Confirm rules:
ufw status verbose

SSH server not installed by default on MX Linux
apt install openssh-server
service ssh enable

List installed packages
apt list --installed 

netstat equivalent
ss -tulpn

Now I had to launch virtualbox using X11 forwarding so it appeared on my Windows client.

Windows side:
Install Xming so you can do X-Windows stuff.
Configure Putty to talk X11.

Server side:
You need to enable X11 forwarding in your /etc/ssh/sshd_config file.
X11Forwarding yes

Initially I had auth issues when puttying in.  I found that my .Xauthority file in the user's home dir was owned by root.
I had to set this to the user for BOTH user and group ownership on the file.

Success!