Doc is Word's 'native' format. It can include all the bell's and whistles of
a Word document -- macros, autotext entries, menu customisations, previous
versions, bookmarks, etc, etc. This format is proprietary to Microsoft and
the details are not published.
RTF stands for Rich Text Format. This is a public standard (do a Google if
you want to see the specs) for formatted text -- eg including font, size,
colouring and so on -- intended to facilitate the transfer of documents from
one application to another. RTF files are stored in plain text so you can
open them in (eg) Notepad. (You can't usefully work on them that way, but
you can see what the contents are like.) There are many applications that
can read and write RTF.
Try creating a complex document. Save it as normal. Then save it as RTF,
close it and re-open. Compare it with the original DOC version to see what's
changed.