Effective potential of composite operators in the first order region of the QCD phase transition

We propose a method to determine the effective potential of QCD from the gap equation by introducing the homotopy method between the solutions of the equation of motion. Via this method, the effective potential beyond the bare vertex approximation is obtained, which then generalizes the Cornwall-Jackiw-Tomboulis effective potential for the bilocal composite operators. Moreover, the extended effective potential is set to be a function of self-energy instead of the propagator, which is the key point for the potential to be bounded from below. We then investigate the extended effective potential in the cases of phase transition of the QCD vacuum with a small current quark mass, the first order phase transition of QCD at finite temperature, and high baryon chemical potential. In the former case, the effective potential shows as an inflection point at the critical mass where the multiple solutions of the Dyson-Schwinger equation (DSE) vanishes, which is consistent with that obtained by solving the DSE directly. For the latter case, the in-medium properties, such as the latent heat and the difference of trace anomaly, of QCD is obtained.