You cannot mark a cell enclosed. Enclosure is a property of a whole region, and
it is only ever falsified — the moment a walk steps off the edge —
never confirmed until the walk is over. Its complement is confirmable from one cell:
an O on the border escapes, full stop. So invert the question. Seed the border,
flood-mark everything that escapes, then flip whatever the mark did not reach.
X from start
current cell
in the queue
marked — it escapes
captured — flipped to X
"Searching from the inside to see if a region touches the boundary requires buffering all visited cells and a global boolean flag. Instead, invert the question: any 'O' on the border escapes."
"Walk the 4 boundaries. Whenever board[r][c] == 'O', launch BFS/DFS and mark all reachable cells with temporary sentinel '#'."
"Iterate through the whole grid: any cell still reading 'O' was truly captured and flips to 'X'. Any cell reading '#' was border-connected and restores to 'O'."
"Every cell is touched a constant number of times. Total time is $O(m \cdot n)$ and auxiliary space is $O(m \cdot n)$ for recursion stack or queue."
O on the border is known to escape from that one cell, with no region to assemble. Same O(m·n) work, none of the bookkeeping. The temporary # is the only price, and the third pass erases it.