LaTeX is supported in Pressbooks. When you are working in the text editor, select the “Visual” tab and insert LaTeX inside brackets, like this:
[latex](type your formula)[/latex]
Find examples of e-books using LaTeX in the Appendix.
Read more about LaTeX in Pressbooks.