To set up a reef tank you want to start with picking the largest size tank you can. Reef tanks have a lot of needs, and the more water you have the more forgiving it'll be. You'll have less fluctuations in temperature, pH, calcium. It's just a larger body of water to resist change. And the reef environment is one of the most stable environments on earth. So, trying to replicate this stable environment is easier if you have a larger aquarium.
The types of filtration that you would want include a typical reef sump where you have the water overflowing from the top of your aquarium into an overflow box and then channeling through the reef sump and being pumped back up into the aquarium. While it's in the sump you want that water to be filtered by a micron sock to get a lot of the particulates out of the water. Then you'd want a good protein skimmer to remove a lot of the waste that's dissolved in the water. Then you'd want some carbon in there to remove some of the impurities, the odors, the yellow stains in the water. And then you'd want a good pre-filter sponge to filter out a lot of the fine bubbles and some of the waste before it goes back up into the aquarium.
If you have the room you should also go with a chiller and a refugium. A refugium is going to add just a natural environment for copepods and amphipods to grow which, as they multiply, will become a natural food source for the corals. It also allows macroalgae to grow which will scavenge a lot of the nutrients out of the water preventing undesirable algae from growing in the aquarium itself.
A lot of people nowadays dose vodka into the aquarium. They use a pellet or a biopellet reactor. And the theory behind this is to cultivate large colonies of bacteria which also remove nitrates and phosphates keeping the water as close to natural reef conditions as possible.
All these types of filtration require a large footprint to keep them in. So if you have to pick and choose I would recommend a good reef sump, a good protein skimmer. I would recommend a pellet reactor. If you have the room you can go with a refugium, but if you have a good pellet reactor then you usually don't need a refugium as they're a little bit redundant.
And then a chiller. You know, even if the tank's temperature is operating at a normal temperature all it takes is one afternoon or one night where that heater malfunctions or the temperature hits 100 in the room because the air conditioner broke and that temperature spikes and you could lose everything in the tank. So, it's one of those things you'd rather have it and not need it than need it and not have it.