Hi Riya,
I have given some examples below for each.
Matrix:
$$\begin{bmatrix}1 & x_{1}\\
1 & x_{2}\\
1 & x_{3}
\end{bmatrix}$$
Vector:
$$\mathbf{u} = 5\vec u+7\hat y$$
Determinant:
$$\begin{vmatrix}a & b\\
c & d
\end{vmatrix}$$
The statements start with $$ followed by \beginmath function (or symbol) and characters. A new line is added using \\ and end the math expression with \end and $$
Run these examples and try yourselves.
For more information, check out this manual.
https://bookdown.org/yihui/rmarkdown/markdown-syntax.html