As in, an actual office where you would need to buy new equipment? Or things you would already have at home?
Since your starting off, you would probably be better off ordering on demand when it comes to things such as hard drives, CPU's, and RAM... its too expensive to do an initial stock up, even though in the long run you can save money.
Things I find useful when working:
An empty work station with a monitor, keyboard, mouse, Ethernet, and a power plugin. That way, you can grab the PC, and just work on it at that workstation.
I also keep a copy of Ultimate Boot CD at hand for diagnostics.
Tools (just about three sizes of screwdrivers. The little "nose pickers" that you can get for picking up small screws are nice too if you drop one inside the computer)
A flashdrive with a bunch of utilities on it, especially ClamWin for removing viruses since it requires no installer
Business Cards! Propaganda is good! No one said you needed to have a company print business cards for you... if you have a printer at home, look online for a template. Dont use regular printing paper though, find a thicker paper.
Somewhere to back data up to, eg: an external hard drive. Your customer needs things backed up before you reformat their computer...
Disclaimers saying your not responsible for lost data, keep these in check. Get a filing cabinet. Dont throw these forms away, I can dig an official form up for you in the MS-Word format if you remind me to. People arent nice when they go about suing you for something that was probably their fault. Signed paperwork is your shutdown on them.
Once you start getting more of the tech stuff done, keep all your downloaded drivers in a safe place.
I personally keep master copies of drivers for the major models of computers I work on.
In a folder called Drivers, I have things separated by manufacturer, model number, and operating system.
This just reduces time spent on the manufacturer website, as they try to convince you to buy a new computer
If its not inside your home, grab an answering machine... you're going to want to leave hours on the message and have callbacks.
Thats all I can think of.
At my school, I just ghost anything that stops working. Unfortunately, people like to keep their current operating system and files intact... darn them!