In my case, the latest version(5.1.0/5.1.1/5.1.2) always reproduces the same. If I want to delete the currently playing music, a message appears saying that the file cannot be deleted because the file is being used, so I need skip to the next song and then delete it.
Previously(5.0.4), it seems to close and delete the file, but now it seems try to delete without closing.