I read in http://cs-www.bu.edu/faculty/homer/complexitybook-vol2-webpg.html that the set $S$ of deterministic $TM$s that operate in polynomial time is not effectively enumerable, since $S = \{i | DM_i \text{operates in polynomial time}\}$ is not computably enumerable (c.e).
A proof used a many-to-one reduction $f(i,w)$ from the set $D=\{<i,w>|DM_i \text{on input w diverges}\}$ to $S$.
$f(i,w)$ works by constructing a $TM$ $F(i,w)$, where $F$ simulates $DM_i$. If $DM_i$ does not halt within $|x|$ steps, then $F$ accepts, and does otherwise. Hence, if $<i,w> \in D$, then $f(i,w) \in S$. Since $D$ diverges, then $F$ is not c.e. by the lemma that if $A \leq_m B$, and $A$ is not c.e., then $B$ is not c.e. In this case, the many-one reduction $\leq_m$ is $f$.
However, what if I construct a reduction $g$ from $S$ to $HALT_{TM}$. The reduction $g$ works by constructing a $TM$ $M$. $M$ simulates an instance of $DM_i$ in the form of $<i,w>$, where $i$ is the Godel number of a $TM$ and $w$ is the input string. If $DM_i$ accepts $<i,w>$, the operation occured in polynomial time in$|x|$ steps, and $M$ accepts. otherwise, $M$ loops. Hence if $<i,w>$ $\in S$, then $<i,w>$ $\in HALT_{TM}$.
However, this should not be possible since $HALT_{TM}$ is c.e., but the set of $TM$s that operate in polynomial time is not c.e., contradicting the lemma that if $A \leq_m B$, and $B$ is c.e., then $A$ should be c.e.
It seems to me that whether or not $S$ is c.e. arbitrarily depends on which reduction function to choose: either $f$ (above, from $D$ to $S$), or $g$ (from $S$ to $HALT_{TM}$), but what am I missing here?
This could be one way to look at the problem:
A $TM$ $i$ operates in polynomial time if it decides all inputs deterministically in $DTIME(n^k)$, i.e. its language is a member of $\textbf{P}$. Following the proof, the reduction $f$ encodes a new $TM$ $F$, which accepts an input $x$ iff the simulated $DM_i$ does not halt within $|x|$ steps (where $DM_i$ receives an input $w$ since it is an instance of $D$). It could be seen that the only way for $F$ to accept any input $x$ and be in $S$ is if $DM_i$ is divergent since if $DM_i$ converges for some steps $\leq |x|$, then $F$ would run forever and not be in polynomial time for some input $x$.
But as for the reduction $g$, the problem with it is how to test if its simulated $DM_i$ is able to accept any input in polynomial time. There is a need to perform this test finitely since $HALT_{TM}$ will still have to perform the subroutine 'accept' at some point in time in order to output $g(x)$. But if ever $DM_i$ accepts an input, there is still no guarantee that it will not loop for other inputs. $g$ can begin testing an infinite number of inputs on $DMI$, but this would already be looping and it would be in $\overline{HALT_{TM}}$ by default for any input, making the reduction invalid.
A trivial reduction would be from $DM_i \in S$ to an instance in the $TERMINATE$ set, where members of the $TERMINATE$ set halt on all inputs. Unlike $g$, this reduction does not need really need to be 'tested' since it trivially follows. However, it is well known that $TERMINATE$ is not c.e., and this does not contradict the first reduction $f$.