Hi VeqIR,
I'm afraid that the Galileo Board always needs a Linux Image and consequently there will be a time in which the board is boot-loading the Linux Image before starting to execute a stored sketch. Since you need the SD Card in order to store the sketch in the board, I'll recommend you to build a custom BSP Yocto Image in order to reduce that latency you mentioned. You can follow this guide about building a Linux Image: Intel® Quark™ BSP Build and Software User Guide
Let us know if you have more questions. Have a nice day!
Regards,
Diego.