How to save a transparent frame in After effects?

In AE, composition > Save frame as.. > File

it will add an item to the Render Queue, set to export a still image file. Choose PSD.

(You can choose from a number of still image formats, including PSD, PNG, TIFF, etc. Note that in these cases the alpha channel will be available to Photoshop, but not applied directly as transparency.)

Then you just need to select the alpha channel in the Channels panel in Photoshop, and create a layer mask from the alpha.

To do so.. Ctrl-Click on the Alpha Channel, and then make sure the layer you want the mask on is highlighted, and go to “Layer-Layer Mask” and then you’ll have options for “hide selection” and “reveal selection.”

Of you can click the “New Layer Mask” icon in the layers palette instead of using the “Layer-Layer Mask” menu item.

(Also, it’s very important that in the Render Queue you set the Alpha Channel type as Straght/Unmatted, since that’s what PS handles better. But may cause graphic glitches too, so try both.)