There are lots of free to use LLMs out there such as Mistral but Llama3 is probably one of the most popular LLMs in use locally today.

You could choose ChatGPT or Google Gemini, but these cloud solutions have limits on their useage and extra cost considerations.

Why not run a local AI on your own hardware? All your information stays secure and you save money at the same time.

Running Llama3 on a CPU alone is an exercise in patience. You need a GPU to help out.

Llama 3 8B runs well on a 12 Series i3 with 16 Gb of RAM and a GeForce 3060. A GPU with 20GB VRAM would be better.

Llama 3 70B is a much larger animal. 32 Gb RAM is the minimum spec, although I think that 64 Gb is a more realistic minimum.

A Core i7 14900K with 32 Gb RAM and an Intel A770 or GeForce 4090 would work well with most small to medium models.

70B works best if you have 160 Gb of VRAM. Even a single 4090 card will not be enough. You will need several GPUs to pull this off. Consider using a g5x.large instance with an NVIDIA A10 if you're serious. The cloud is strictly pay as you go. You can programatically create a configured instance and quickly terminate it when it's not being used.

Even the best Intel socket 1700 motherboard can take at most 2 or 3 GPUs. A threadripper with capacity for several GPUs is another (expensive) alternative for up to 4 large GPUs.

Quantitized Llama 70 models trade of accuracy for useability. Many people get good results with 2 GeForce 4090 cards (48 Gb) or a single Nvidia A5000 (32 Gb) or A6000 (48 GB).

You will need a larger EATX motherboard with extra room for enhanced airflow. Please ensure that your power supply is sufficient. A Nvidia 4090 needs 450W, the A5000 needs 230W and the A6000 needs 300W.

You can pickup older GPUs on the internet. Just consider that many are designed to be fitted to a server chassis. You will need to ensure that they are properly cooled.

There's also the fact that older cards have probably been used 24/7 over the last few years and may not have many years of useful life left.

Drop in for a coffee and we can discuss your AI computing needs.