When you factor out a variable, you subtract exponents. For example, factoring $a^2$ from $a^5$ leaves $a^{5-2}$. Apply this idea to both terms.

After pulling out the common factor, make sure the sign between the remaining terms matches the original expression, and that dividing the coefficients (like $8 \div 4$ and $12 \div 4$) is done correctly.

In Desmos, type something like a=2 and b=3 (you can choose any nonzero values). These will be used to compare the original expression with each choice numerically.

Enter 8a^5*b^3 - 12a^2*b^4 and note the numeric result Desmos gives using your chosen values of $a$ and $b$.

One by one, enter each option, such as 4a^2*b^3*(2a^3 + 3b), 4a^2*b^3*(a^3 - 3b), etc., using the same $a$ and $b$ values. Compare each result to the value from the original expression; the correct choice will give exactly the same value. For extra confidence, change $a$ and $b$ to new values and verify that the same choice still matches.

Look at $8a^5 b^3$ and $12a^2 b^4$.

• Numerical GCF of 8 and 12 is 4.
• For $a$, the smaller exponent between $a^5$ and $a^2$ is 2, so take out $a^2$.
• For $b$, the smaller exponent between $b^3$ and $b^4$ is 3, so take out $b^3$.

So the GCF of the whole expression is $4a^2 b^3$.

Write the original expression as the GCF times a set of parentheses:

Now divide each term by $4a^2 b^3$ to find what goes inside:

• First term: $\dfrac{8a^5 b^3}{4a^2 b^3} = 2a^{5-2} = 2a^3$.
• Second term: $\dfrac{12a^2 b^4}{4a^2 b^3} = 3b^{4-3} = 3b$.

Keep the subtraction sign from the original expression, so the inside becomes $2a^3 - 3b$.

From the factoring steps, the expression becomes

Compare this with the answer choices and select the one that exactly matches: $4a^2 b^3(2a^3 - 3b)$, which is choice D.