As nouns the difference between matter and factor
is that
matter is while
factor is (obsolete) a doer, maker; a person who does things for another person or organization.
As a verb factor is
to find all the factors of (a number or other mathematical object) (the objects that divide it evenly).