Use the $ac$ method: multiply the leading coefficient and constant term, then look for two numbers that multiply to that product and add to the middle coefficient.

After you factor, check whether either binomial has a common factor you can pull out and move into the outside constant $k$.

Enter $f(x)=126x^2+245x+84$ (here Desmos’s $x$ is playing the role of $y=x^2$ in the original problem).

Find the $x$-intercepts of $f(x)$. They should be negative rational numbers.

If the intercepts are $-\frac{4}{9}$ and $-\frac{3}{2}$, then the corresponding factors are $(9x+4)$ and $(2x+3)$ (because $9x+4=0$ gives $x=-\frac{4}{9}$, etc.).

Replace Desmos’s $x$ with $x^2$ to get factors in $x^2$: $(9x^2+4)$ and $(2x^2+3)$, with an integer multiplier $k$ adjusting the overall scale. a‍n​iкo‍.​аі/⁠ѕat Compare $9\cdot 4$ and $2\cdot 3$ to identify the smallest possible $ab$.

Let $y=x^2$. Then

From‍ anі‍кo.ai

Compute $126\cdot 84=10584$. Find two integers that multiply to $10584$ and add to $245$:

Split the middle term and factor by grouping:

The factor $63y+28$ has a common factor of $7$:

So

Substitute back $y=x^2$:

This matches $(k)(ax^2+b)(cx^2+d)$ with $k=7$ and the two binomials $2x^2+3$ and $9x^2+4$ (in either order).

If $(ax^2+b)$ is $2x^2+3$, then $ab=2\cdot 3=6$.

If $(ax^2+b)$ is $9x^2+4$, then $ab=9\cdot 4=36$.

The smallest possible value of $ab$ is 6.