Expand and factor (ax + by + ay)² + (bx − ay)(ax + by + ay) + (bx − ay)².

Step-by-step solution

1. 1

   Name the two repeated blocks so the shape becomes visible. Set

   $A=ax+by+ay,\qquad B=bx-ay$

   Then the whole expression is just

   $A^2+AB+B^2$

   That form has no factorization in $A$ and $B$ alone, so the payoff has to come from what $A$ and $B$ are made of. Expand everything and collect.

2. 2

   Expand $A^2$ term by term. With three summands you get three squares and three double products:

   $A^2=a^2x^2+b^2y^2+a^2y^2+2abxy+2a^2xy+2aby^2$

   Note the $2a^2xy$ — it comes from $2(ax)(ay)$ and is the term most often dropped. Losing it changes the final answer.

3. 3

   Expand the cross product $AB$ and the square $B^2$.

   $AB=(ax+by+ay)(bx-ay)=abx^2-a^2xy+b^2xy-aby^2+abxy-a^2y^2$

   $B^2=(bx-ay)^2=b^2x^2-2abxy+a^2y^2$

4. 4

   Collect the three monomial groups $x^2$, $xy$, $y^2$. Keeping the $xy$ column honest is the whole exercise:

   $x^2:\quad a^2+ab+b^2$

   $xy:\quad \underbrace{2ab+2a^2}_{A^2}\ \underbrace{-a^2+b^2+ab}_{AB}\ \underbrace{-2ab}_{B^2}=a^2+ab+b^2$

   $y^2:\quad \underbrace{b^2+a^2+2ab}_{A^2}\ \underbrace{-ab-a^2}_{AB}\ \underbrace{+a^2}_{B^2}=a^2+ab+b^2$

   All three columns give the identical coefficient $a^2+ab+b^2$.

5. 5

   Factor that common coefficient out. Since every group carries the same factor,

   $A^2+AB+B^2=(a^2+ab+b^2)\left(x^2+xy+y^2\right)$

6. 6

   Verify with numbers before trusting it. Take $a=2$, $b=3$, $x=4$, $y=5$. Then $A=2(4)+3(5)+2(5)=33$ and $B=3(4)-2(5)=2$, so

   $A^2+AB+B^2=1089+66+4=1159$

   and the factored form gives $(4+6+9)(16+20+25)=19\cdot 61=1159$. They agree, which also rules out the common slip of writing the $xy$ coefficient as $-a^2+ab+b^2$.