If it's not desirable to first put something in the recycle bin, then before you hit "Delete" press and hold the left shift key.
It "Shifts" the delete command, from putting something in the Recycle Bin, to throwing it completely OUT of the computer. OUT The Window, so to speak.
Don't do this with anything that you may ever want to restore.
Good Luck,
The Shadow