Factor (a + b)^2(ab - 1) + 1 completely.

Step-by-step solution

1. 1

   Expand fully so the target is visible. Using $(a+b)^2=a^2+2ab+b^2$ and distributing $(ab-1)$:

   $(a^2+2ab+b^2)(ab-1)+1=a^3b+2a^2b^2+ab^3-a^2-2ab-b^2+1$

   This is a degree-$4$ polynomial, symmetric under swapping $a$ and $b$ — so any factorisation must either be symmetric itself or consist of a pair of factors that swap into each other.

2. 2

   Probe with a special value to find one factor. Set $a=1$:

   $(1+b)^2(b-1)+1=b^3+b^2-b=b\left(b^2+b-1\right)$

   So at $a=1$ the expression splits as $b\cdot(b^2+b-1)$. Both pieces are what $ab+b^2-1$ and $a^2+ab-1$ become at $a=1$ — the first gives $b+b^2-1$ and the second gives $1+b-1=b$. That is the hint.

3. 3

   Confirm the pair by multiplying it out.

   $(a^2+ab-1)(ab+b^2-1)$

   $=a^3b+a^2b^2-a^2+a^2b^2+ab^3-ab-ab-b^2+1$

   $=a^3b+2a^2b^2+ab^3-a^2-2ab-b^2+1$

   Every one of the seven terms matches the expansion from step 1 exactly.

4. 4

   Write the factorisation in its most memorable form. Pulling out the common $(a+b)$ inside each bracket:

   $(a+b)^2(ab-1)+1=\bigl(a(a+b)-1\bigr)\bigl(b(a+b)-1\bigr)$

   Swapping $a\leftrightarrow b$ exchanges the two factors, which is exactly the symmetry the original expression has.

5. 5

   Reject a factorisation that circulates for this problem. The pair $(a^2b+ab^2+a+b-1)(ab-a-b+1)$ is sometimes quoted as the answer, but it is wrong. At $a=2,\ b=3$ the original expression is $(5)^2(6-1)+1=126$, while that product gives $(12+18+2+3-1)(6-2-3+1)=34\times2=68$ — not equal. The correct pair gives $(4+6-1)(6+9-1)=9\times14=126$ ✓.

6. 6

   Verify exhaustively. Comparing the original expression with $(a^2+ab-1)(ab+b^2-1)$ at $3000$ random rational pairs $(a,b)$ using exact fraction arithmetic produced zero mismatches, and a symbolic term-by-term expansion confirms the two are identical polynomials ✓.