One key challenge to observing quantum phenomena in a macroscopic mechanical oscillator is reaching its ground state. To achieve the low temperatures required for this, we utilize resolved sideband laser cooling of a few hundred kHz mechanical oscillator with high mechanical Q (a Si3N4 membrane) inside a high finesse optical cavity, in addition to cryogenically reducing the bath temperature. Realizing high Q and high finesse cavity optomechanical devices in a cryogenic environment requires overcoming a number of challenges. In this talk, we describe the design and construction of such a device working at a bath temperature of 300 mK (in a 3He refrigerator) and suited for operation at lower temperatures (in a dilution refrigerator). The design incorporates in-situ commercial piezo actuators to couple externally prepared laser light into the cold optical cavity. The design also incorporates filtering cavities to suppress classical laser noise, and acoustic and seismic isolation of the experiment.