What might be the reason for the increased time of execution?

The problem I have is whenever I change the timeout variable, uiMaxWaitMs, from 500 to 2000 I get a delay in my process. However this is just a timeout value and should never actually increase the amount of time I wait in the select() function call. It only allows for additional time to be allowed for a timeout. My question is how does uiMaxWaitMS affect my select() function call if I never hit a timeout in either the 500ms or 2000ms...

When I increase the allotted timeout value my program tends to take longer to send a command through a TCP socket... But from what I've read it should only allow for more time IF needed to but since I never hit a timeout with a 500ms timeout why would increasing the timeout value to 2000ms make my performance worse? I know that the math for the struct variables is weird and just plain wrong for the micro second value (refactoring old code)

int CNPEnetConn::RecvByte (char *pcBuffer, unsigned int uiMaxWaitMs)
{
    fd_set oReadFds;
    struct timeval oTimeout;

    if(uiMaxWaitMs >= 1000)
        oTimeout.tv_sec = uiMaxWaitMs / 1000;
    else
        oTimeout.tv_sec = 1;

    oTimeout.tv_usec = uiMaxWaitMs;

    FD_ZERO (&oReadFds);
    FD_SET (m_hostSocket, &oReadFds);

    //----  wait for read ready, or timeout or error:
    //      (With Winsock, the only way to achieve a timeout is by using select)
    int iStatus = select (0, &oReadFds, NULL, NULL, &oTimeout);

    //then some more code that does not use the timeout value

}

Link to info select function

Link to info timeout struct

500ms timeout

AVG Time: 0.192142794019933
# Deltas > 0.5: 299
Min Time: 4.360000000360742E-4
Max Time: 1.013213000000178
Range Histogram: Total:1505
1.50:   0
1.25:   0
1.00:   14
0.75:   136
0.50:   149
0.25:   149
0.10:   1
0.00:   1056

2000ms timeout

AVG Time: 0.9555644471726187
# Deltas > 0.5: 960
Min Time: 3.8500000005115E-4
Max Time: 1.96373100000028
Range Histogram: Total:1344
1.50:   384
1.25:   182
1.00:   10
0.75:   192
0.50:   192
0.25:   192
0.10:   0
0.00:   192

Every 2 seconds I send a message and the times represent how long it took to create the message and send it. Data was collected with WireShark and analyzed by a simple Java program I created.