The minimum virtual machine that we have installed liferay on has 1GB memory and some swap space, usually as much as the linux setup procedure suggests... This particular machine idles most of the time and user experience or massive traffic is not an issue, there's no heavy-processing stuff going on on that server.
Liferay might run with less memory, but you certainly want to finetune your setup and not use it for more than trying concepts - e.g. either development or production use would be a pain.
Quick solution for you: Add virtual memory (e.g. swap space on disk) and / or lower the memory that's being made available to liferay. If you have the tomcat bundle, look for the file "setenv.sh" in tomcat's bin directory and finetune the memory settings there to fit your machine. When java allocates (and uses) more than the physical memory in your machine prepare for severe slow downs, but at least you should be able to get it to run and continue your exploration from that point on.
On the other hand buying a bit of memory might be cheaper in the (not so) long run than investing your time.
