When I try to merge incidents, it succeeds, all the alerts go under one incident, but the other incidents don’t go away? What’s going on?
That sounds like expected behaviour. When merging multiple incidents together, alerts will be consolidated into the selected incident. The other selected incidents will be resolved, with the reason for the resolution listed as “merged”. These incidents will reference the target incident that you selected in the merge dialog.
You can read further about incident merging on our knowledge base article.