There's nothing magical about captions that makes them stay with a picture.
If your picture is In Line With Text and the caption is below it, you can
format the picture as "Keep with next." Or you can put both together in a
table cell, text box, or frame. The only thing you accomplish by using a
Caption is the numbering because the Table of Figures is by default built on
{ SEQ Figure } fields. But you can apply the Caption style to any paragraph
and build the Table of Figures on that style (or any other style used
exclusively for figures).