You can create a rectangle with rounded corners in Photoshop using a number of easy steps. Photoshop has not a filter or other tool to create a round corner but Illustrator allows you to apply non-destructive rounded corners effect to pretty much any object. In Effects menu, simply choose Stylize > Rounded Corners.
In this quick tutorial I will show you a simple method to obtain rounded corners in Photoshop using rasterized text, shapes or any other raster image. Let's start with a simple star vector shape with acute angles(it works great for right angles as well).
Now you have to rasterize the shape; right click and choose Rasterize Layer. You will have to do the same if you use Text Layers.
Go to Filter>Blur menu and choose Gaussian Blur. The radius of the Gaussian Blur depend on the size of your image and the result that you want to obtain. I will choose 5 pixels.
Press CTRL and click on the layer thumbnail to make a Photoshop selection. Go to Select menu and choose Refine Edge with the following settings. You can experiment with Smooth and Feather options to see different effects. Leave the Contrast to 100.
Pick the color of the shape, in our case yellow anf Fill the selection with that color. Press SHIFT+F5 to fill with color.
The last step is to add this selection as layer mask and that is all. You have now a star shape with rounded corners.
You can apply this technique for creating text with round corner as you can see in the image below. You can create your own customized font.
If you want to save time you can try this automaded action for making rounded corners in Photoshop with just 1-Click!