You can use the performance analyzer and run it alongside a report to start whittling down your list of worst offenders.
You can find documentation here
My suggestion would be to run a report with a large enough number of records that you should see the run time for formulas be significant, and in the report include a handful of formula fields or other entries that you want to investigate and take a look at the performance results. Do this starting with what you might think are the worst offenders such as formula queries that reference large data sets, complex summary fields, lookup fields from large data sets, formulas with complex calls that reference or use multiple data points from different data source etc etc. As you find bad actors that you can fix, you can optimize how you're doing formulas, refactor your queries or find alternatives for how to achieve whatever it is.
