You have to register with Inland Revenue (now HMRC), but don't have to register with Companies House. Assuming you don't have a partner, you would register as a sole trader. VAT registration is voluntary if your turnover is under the £67,000 threshold (some hopes!), and for this sort of business would just cause you extra hassle. (Voluntary VAT registration is only useful if you would pay out more VAT per year than you would take in.)
You'll have to undergo the yearly joys of income tax self-assessment, which isn't really all that difficult. The main thing is to keep good records. I prefer using
Collins Essential Accounts Book (the first one on this page), which is very clear to read and easy to use.
The Inland Revenue site has a clear list of what records you are required to keep. You legally have to keep all records for 5 years and 10 months; I generally keep mine for 7 years or so, or until I need the space. It's simplest and most secure to do your income tax online, and also to pay online through your bank.
As self-employed, you will pay National Insurance at a level that makes your nose bleed -- £2.30 a week, plus 8% of profits -- on top of income tax. If you expect very low income for the year (it changes from year to year, currently £4825) you can apply for a "Small Earnings Exception" and not pay National Insurance for that year.
If you have a low to moderately low income (under £18,000 or so), it's worth looking into Working Tax Credits. Doubly so if you have children, are over 50, or are disabled or have a disabled spouse. I've just applied for this myself and hope to get an extra few hundred a year.
I use the "notes" pages at the back of the accounts book to keep my job log. You're likely to have more jobs than me, for a lot more different clients, so you'll probably need a second record book. Give each job a unique reference number, and note down what it is and who it's for at a bare minimum. You're likely to want to keep separate records of client names, contact details, and the sort of thing they want from you.
Use the computer as much as you like, but make sure you have records on paper. Paper files are permanent.
For my job numbering, I started with A100 and am now (about 10 years later) in the A950s. When I pass A999, I'll start with B100 and so on. This keeps the job numbers in the same format and easy to sort by computer if necessary.