This project is read-only.

Problems communicating with a sony FCB 7500 camera

Sep 18, 2015 at 3:53 PM
Hello everyone,

I have been using the SerialPortStream for quite some time now to communicate with Sony cameras that use the VISCA bus system and until now I have never had any problems.

But now I got an FCB-EV7500 camera to work on for a customer and I cannot get it to work correctly with the library. I extracted a simple snippet to test it against the SerialPort class by Microsoft:
var serialPort = new SerialPortStream("COM4", 9600, 8, Parity.None, StopBits.One);

serialPort.Open();
serialPort.ReadTimeout = 2000;
serialPort.WriteTimeout = 2000;
serialPort.Handshake = Handshake.None;
serialPort.RtsEnable = false;
serialPort.DtrEnable = false;

var bytes = new byte[] { 0x81, 0x01, 0x04, 0x00, 0x02, 0xFF };
serialPort.Write(bytes, 0, bytes.Length);

var buffer = new byte[32];
var test = serialPort.Read(buffer, 0, serialPort.BytesToRead);
serialPort.Dispose();
The code snippet above will never yield a response on the stream from the camera, although it receives the packet and acts on it. If I replace the SerialPortStream with the SerialPort class from Microsoft, I (almost) always receive the correct ack and completion packets. I say almost because even with the MS class, sometimes the packets arrive with a certain delay and I get the impression that the camera answers very slowy, even if inserting stuff like Thread.Sleep(2000) between write and read. But with the MS class I can at least wait until the packets arrive. With SerialPortStream they never arrive, BytesToRead is always 0.

What confuses me even more: If I do not use the BytesToRead property, but just a constant or the length of the buffer as parameter for the Read method instead, I sometimes can catch one or both of the packets even with the SerialPortStream class when trying multiple times.

Ideas anyone? I am not the most experienced in serial communication, so I might be doing something wrong. It's just that this is the first camera I had such problems with. In my normal application I use the event model, but they are never raised in this case, either.

Oh and by the way, is it correct behavior that the RtsEnable and DtsEnable Properties revert back to true if you set the HandShake value after setting both of them to false? That kind of confused me, too.
Sep 26, 2015 at 6:04 PM
Hi,

What happens if you set the properties before opening the serial port?
Sep 26, 2015 at 6:16 PM
When running your test program, the following error is obvious.

var buffer = new byte[32];
var test = serialPort.Read(buffer, 0, serialPort.BytesToRead);

You are checking to see how many bytes are in the serial port buffer, which in my test program is zero, so serialPort.Read() is instructed to read zero bytes. You should replace the serialPort.BytesToRead with buffer.Length. Then there should be a timeout of 2000ms (my test runs and dies within 27ms, probably too short for a reply to be sent).
Sep 27, 2015 at 1:26 PM
Hi jmcurl,

thanks for getting back to me.

To answer your first question: If I try to set RtsEnable/DtrEnable to false before opening the stream, I receive an InvalidOperationException stating "Serial port not openend". The other properties I can set safely without opening it.

For the next part:
You are right about the BytesToRead property, I seem to have been using it with a wrong assumption in the test program code. In my real application, I always handle the DataReceived event, and in that case the BytesToRead property is the correct length to use when extracting stuff from the stream, right? So when trying to break down my application just to the basics, I misused the BytesToRead property because there is no more eventing involved.

But my real problem is that sadly, with the FCB camera, the DataReceived event is never raised. Although a response definitely arrives which I can read manually from the stream using a line as you say:
var buffer = new byte[32];
var test = serialPort.Read(buffer, 0, buffer.length);

What could be the problems with the cameras responses and your library? Is there any way I can get you some debug info on what is happening in the background? As I said, using the Microsoft SerialPort library with that camera, the DataReceived event is raised normally...
I am not sure if that is a problem with the hardware, or a very specific serial communication problem. Other sony cameras work just fine with your library and through their responses the DataReceived event is raised as expected. My current workaround is to hide the serial port operations behind an ISerialPort interface, and then register different serial port libraries for different devices in my dependency injection, so that all other devices get your library as serial port implementation, and the FCB camera will use the Microsoft one.
Sep 27, 2015 at 2:02 PM
Let's see if we can get the problem solved.

1) Debugging. You can read the page at http://serialportstream.codeplex.com/wikipage?title=TraceSwitch%20Settings&referringTitle=Documentation which hopefully shows what is going on. If you're running on a 32-bit platform, then the PortMon tool from SysInternals can also log exactly what's going on (and also in more detail). Both of those would be a very good start. You should set tracing to verbose.

2) Rts/Dtr: If you're using no handshake, these properties should have no effect, even though in my code I set RTS to on (so an external device is told to send data). You immediately set it back off. I'm not sure if this has a real effect to your camera, do you use two-wire or four-wire serial cables? Two wire (Rx/Tx and Gnd) would make this point moot. Does it work if you get rid of these two lines?

The DataReceived event is raised by the underlying serial port driver when data is received, so I can't say why it's not triggering in this particular case. Logs might help.
Sep 27, 2015 at 4:33 PM
Edited Sep 27, 2015 at 4:35 PM
1) Have tried for more than an hour now with no results. Tried to activate the tracing, but the application never produces the logfile. I used the exact snippet from your documentation and also tried to use the source name IO.Ports.SerialPortStream_ReadTo and different verbosity levels. Both did not work. This is really strange. I first suspected it could be missing file permissions, but I checked them and also had the application create and write a file. Seems ok on that side. The only other thing I might try is integrating your source code into my application and running it this way. I will try and keep you updated.

PortMon sadly cannot be used because all our PCs are running 64 bit Windows 7 or 8.1 systems. I tried anyway, in different compatibility modes, but I cannot get the capturing to run.

2) I got the cables along with the camera hardware, but the documentation states that the camera only supports the TX, RX and GND pins. So it's probably safe to say its only build for two wire cable. I got rid of the two lines anyway and tested, but it didn't change anything.

If the DataReceived event is raised by the driver, that is indeed strange that your library results in the driver not raising the event but the Microsoft SerialPort class does. Kind of frustrating :)
Sep 27, 2015 at 4:44 PM
Ok, for whatever reason integrating your source code into my application seemed to at least produce the trace results. Here is what is produced with a single run of the test code:

IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: DoWriteEvent: WriteFile(1124, 58823600, 6, ...) == False
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWaitCommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : COM4: CommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWriteEvent: 6 bytes
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWriteEvent: TX-BUFFER empty
IO.Ports.SerialPortStream Verbose: 0 : COM4: OverlappedIO: Stopping Thread
IO.Ports.SerialPortStream Verbose: 0 : COM4: OverlappedIO: Waiting for Thread
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: Thread closing
IO.Ports.SerialPortStream Verbose: 0 : COM4: OverlappedIO: Thread Stopped

I also added an event handler to the data received event, which is never raised even though I have the application waiting for quite some time to be safe.
I also double-checked again using System.IO.Ports.SerialPort which raises the DataReceived event as expected...
Sep 27, 2015 at 4:56 PM
For some reason, there's no RX event occurring. Can you please let me know what device is showing up in Windows Device Manager for the Sony Camera? I predominantly test with FTDI and with a virtual COM port. I've also tested with a PL2303 on Windows 7 (type B, which Win8 doesn't support).

I'm also wondering how I might be able to reproduce this issue on my own PC, but of course i don't have a Sony. I could write a small app that will send out the bytes on the wire if I knew the sequence you're using. Is your test program that you offer here the one which also shows the issue?
Sep 27, 2015 at 5:10 PM
I always use USB to serial adapters in testing and production. I have connected the camera to a USB-to-Serial Adapter with the prolific chipset. It is a newer PL-2303 RA chipset that also runs on Windows 8 and 10 and I am currently using the latest driver version v3.6.78.350, although an older version displayed the same problems I have now.

The byte sequence I used in the test code I posted here is an actual one that switches the camera's power. So the test program produces the issue, you just have to add an event handler to the DataReceived event to complete it.

In case a look at the camera specs might help, here is a link to the official documentation explaining the camera's specs and the serial commands in great detail:
FCB7500 Docs
Sep 27, 2015 at 5:46 PM
There's the method GetReceiveStats() that was added after v1.1.3.0 as per r32875, and a slight change in r32876 that might have changed the read behaviour. If you're saying that you can still read data using my library, just that the event isn't being raised, then there's a logic error somewhere in my code.

If that is the case, those two commits may have broken something. Would you mind retesting with v1.1.3.0? Then I can concentrate on my logic a bit better.

Also. if you could provide logs that would be fantastic. Can you say if the logs above are complete?
Sep 27, 2015 at 5:58 PM
I just integrated your code from version 1.1.3.0 (commit 32778) into my code and tested. Here is the complete log output for one run:

IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: DoWriteEvent: WriteFile(1044, 64263088, 6, ...) == False
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWaitCommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : COM4: CommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWriteEvent: 6 bytes
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWriteEvent: TX-BUFFER empty
IO.Ports.SerialPortStream Verbose: 0 : COM4: OverlappedIO: Stopping Thread
IO.Ports.SerialPortStream Verbose: 0 : COM4: OverlappedIO: Waiting for Thread
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: Thread closing
IO.Ports.SerialPortStream Verbose: 0 : COM4: OverlappedIO: Thread Stopped

The logs from the post above were complete as well.
It seems to be the same as with 1.1.4.0. I also remember having used 1.1.2.0 before upgrading to 1.1.4.0 and having the same issues with the camera. So if there really is a logic error in your code, it must have been there even with 1.1.2.0.
Sep 27, 2015 at 10:53 PM
Edited Sep 27, 2015 at 10:55 PM
I did some testing, till now I cannot recreate the problem. To show you my test, I'm using com0com 3.0.0.0 (may be a source of difference) and wrote a small program that receives data and sends it back (simulating your camera). Com0Com was configured that only Rx and Tx are connected, every other pin is open. That code looks like:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using RJCP.IO.Ports;

namespace SerialResponder
{
    class Program
    {
        static void Main(string[] args)
        {
            SerialPortStream serialPort = new SerialPortStream("CNCB0", 9600, 8, Parity.None, StopBits.One);
            serialPort.Open();

            Console.WriteLine("SerialResponder READY");
            while (true) {
                byte[] data = new byte[32];
                int r = serialPort.Read(data, 0, data.Length);
                if (r > 0) {
                    serialPort.Write(data, 0, r);
                }
                Console.WriteLine("Got: {0} bytes", r);
            }
        }
    }
}
Then I created two test cases to write and wait for a response. That code looks like:
    [TestClass]
    public class CodePlexTesting
    {
        [TestMethod]
        public void WriteReadEvent()
        {
            for (int x = 0; x < 10; x++) {
                using (SerialPortStream serialPort = new SerialPortStream("CNCA0", 9600, 8, Parity.None, StopBits.One)) {
                    serialPort.DataReceived += (s, e) => {
                        Console.WriteLine("Data Received");
                        byte[] buffer = new byte[32];
                        int r = serialPort.Read(buffer, 0, buffer.Length);
                        Console.WriteLine("Received {0} bytes", r);
                    };

                    serialPort.Open();
                    serialPort.ReadTimeout = 2000;
                    serialPort.WriteTimeout = 2000;
                    serialPort.Handshake = Handshake.None;
                    serialPort.RtsEnable = false;
                    serialPort.DtrEnable = false;

                    var bytes = new byte[] { 0x81, 0x01, 0x04, 0x00, 0x02, 0xFF };
                    serialPort.Write(bytes, 0, bytes.Length);
                    System.Threading.Thread.Sleep(1000);
                }
            }
        }

        [TestMethod]
        public void WriteReadSeq()
        {
            for (int x = 0; x < 10; x++) {
                using (SerialPortStream serialPort = new SerialPortStream("CNCA0", 9600, 8, Parity.None, StopBits.One)) {
                    serialPort.Open();
                    serialPort.ReadTimeout = 2000;
                    serialPort.WriteTimeout = 2000;
                    serialPort.Handshake = Handshake.None;
                    serialPort.RtsEnable = false;
                    serialPort.DtrEnable = false;

                    Stopwatch sw = new Stopwatch();
                    sw.Start();
                    var bytes = new byte[] { 0x81, 0x01, 0x04, 0x00, 0x02, 0xFF };
                    serialPort.Write(bytes, 0, bytes.Length);
                    long writeElapsed = sw.ElapsedMilliseconds;

                    byte[] buffer = new byte[32];
                    int test = serialPort.Read(buffer, 0, buffer.Length);
                    long readElapsed = sw.ElapsedMilliseconds;
                    sw.Stop();
                    Console.WriteLine("{0} bytes; Write={1}; Read={2}", test, writeElapsed, readElapsed - writeElapsed);
                }
            }
        }
    }
In the event test case, the event fired every time, and the number of bytes read were six bytes. In the second case, data was read every time and also reported 6 bytes with a round trip time of about 120ms.

I was running on a 32-bit with the head of SVN (test cases were added to the project, the responder uses the project reference, not a DLL).

One particular error that might occur are delays. Because the machine I'm testing on is 32-bit and a Pentium M (8 years old), it might not be fast enough and you could have found a race condition that can't yet be ruled out. More investigation is needed. If at all possible, testing on your side on a 32-bit (even with VMware) running sysmon portmon would be very helpful. That would tell me what the driver is doing.

I'm stumped that a read returns the data, but you're not getting the event. Because your logs show that the driver didn't even report data which means a SerialPortStream.Read() should also return nothing. My Read() method does not access hardware at all, but just reads what ever data already exists in a buffer.

I believe that something is wrong, but I don't know what yet.
Sep 27, 2015 at 11:13 PM
I'll have a look at my code tomorrow, but one interesting note, when I run the tests I get the logs:

IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: DoWriteEvent: WriteFile(756, 43249928, 6, ...) == True
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: ProcessWriteEvent: 6 bytes
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: ProcessWaitCommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: CommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: ProcessWaitCommEvent: TX-BUFFER empty
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: ProcessWaitCommEvent: EV_RXCHAR
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: DoReadEvent: ReadFile(756, 42201320, 1048576) == False
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: ProcessReadEvent: 6 bytes
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: ProcessReadEvent: End=0; Bytes=6
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: CommEvent: EV_RXCHAR
IO.Ports.SerialPortStream Verbose: 0 : CNCA0: SerialThread: DoReadEvent: ReadFile(756, 42201326, 1048570) == False

The ProcessWriteEvent occurs before EV_TXEMPTY in my case. In your case it's reversed.

IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: DoWriteEvent: WriteFile(1044, 64263088, 6, ...) == False
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWaitCommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : COM4: CommEvent: EV_TXEMPTY
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWriteEvent: 6 bytes
IO.Ports.SerialPortStream Verbose: 0 : COM4: SerialThread: ProcessWriteEvent: TX-BUFFER empty
Sep 27, 2015 at 11:56 PM
Hi, and thanks for looking into that problem.

I might be able to investigate more tomorrow evening. If you can point me to the parts of the code that you think might be part of the problem, I could step in with the debugger and have a closer look. It would take me quite some time to dive into your code on my own.
If it is a concurrency problem, that will probably be very hard to find. It might be totally unrelated, but I noticed that quite a lot of code is outside of the locks you use. Could you give me a hint about which parts of your code may execute concurrently and exhibit race conditions, I could also have a closer look at them, too.

Two things I have noticed:
  • In one of your comments, you say that:
    "Because the main event loop handles CommEvents before WriteEvents, it could be that a write event occurs immediately after, actually emptying the buffer". So doesn't that mean that the order of things in my log is as expected? First the CommEvent, then the Write Event. Just to understand it a little better, is the EV_TXEMPTY event supposed to be some kind of acknowledgement or result of writing something to the underlying driver? If that is the case, then the order is definitely wrong in my case.
  • You could make your event trigger methods a little more thread-safe by writing them this way:
    var handler = CommEvent;
    if (handler != null)
    handler(this, new CommEventArgs(e));
This won't resolve the current problem, but is considered some kind of event best practice.
Sep 28, 2015 at 6:49 AM
Hi, r33069 already contained the update for the event handlers (not all, just those that are exposed by the main stream class).

There should be no thread race conditions, per se, as it's based on a single loop using the overlapped I/O model example provided by MS for the Named Pipe example. The main code is rather small, contained in the OverlappedIoThread in the file SerialPortStream.NativeSerialPort.CommOverlappedIo.cs.

I'll be checking the sequence of events again in the order that they're received, perhaps it's related to the ComMEventsMask (although I should only disable the read event in case of a read operation already in progress).
Sep 28, 2015 at 9:57 AM
The behaviour of your instance is expected AFAICS, and I don't see a race condition going into code. If the RX_CHAR was missed for whatever reason, code in OverlappedIoThread() at the beginning of the while(true) loop checks if there are any bytes in the hardware buffer and it will read the data for us (because later readPending is still false, it calls DoReadEvent which checks m_ReadByteAvailable). So we poll in addition to waiting for events (wrokaround for a workaround for the original PL2303). I was thinking of a potential race condition earlier that may be due to the read event being missed. We can cross that off as it's clearly not the case.
if (readPending) {
  UnsafeNativeMethods.SetCommMask(m_ComPortHandle, maskReadPending);
} else {
  UnsafeNativeMethods.SetCommMask(m_ComPortHandle, maskRead);
  if (GetReceiveStats(out bytesInQueue, out eofReceived) && (bytesInQueue > 0 || eofReceived)) {
    m_ReadByteAvailable = true;
    if (eofReceived) m_ReadByteEof |= EofByte.InDriver;
  }
}
We also see that ProcessReadEvent is not being called. So either the hardware driver is saying no data is available, or we've started the overlapped I/O operation in DoReadEvent
bool result = UnsafeNativeMethods.ReadFile(m_ComPortHandle, bufPtr, bufLen, out bufRead, ref overlap);
and the PL2303 is not ever returning. That's not instrumented (and I think it's unlikely, but not impossible).

The logs in your case are also expected when writing occurs. The write is overlapped (pending) as it returns false. The serial comm event EV_TXEMPTY occurs before the write event finishes (to be expected) due to the ordering of the WaitAny() in the main loop and the CommEvent ManualResetEvent occurring first. The code in ProcessWaitCommEvent() handles that by setting m_TxEmptyEvent to true. Code will check that when the write is finished, which occurs on the second loop (remember, all this stuff is running single threaded).

Conclusion: The Read path and the Write path are sufficiently separated that I can't see a race condition here. We poll the bytes in the hardware buffer to see if there could be any data, so I don't see any issue here either.

I'm assuming that readPending is always false in the loop, because I don't see anything explicitly that would indicate a read was started (it could be set only in the condition that bytes really were present, but Read() never returns, or RXCHAR was received and then started which we don't see in logs). So you could instrument DoReadEvent() in more detail.

Second thing you could try is to fiddle with the timeout parameters in the NativeSerialPort.CommOverlappedIo.Start() method.
Sep 28, 2015 at 9:42 PM
Edited Sep 28, 2015 at 9:43 PM
OK, I found out something new and very weird. I am not sure if it is a breakthrough for the problem, but it seems to be related. I only stumbled upon it by accident. Normally I listen to music while coding, but this time my room was completely silent. I first noticed a high pitched sound coming from the camera as soon as it was connected to the serial adapter. I thought this was weird. What was even weirder was that as soon as I started my application, the camera made a clicking noise and changed the pitch of that strange sound. So I investigated further by using the exact same test program and breakpoints for all the testing, I only switched between SerialPort and SerialPortStream during test runs to compare.

Here is what I found out:
  • If I unplug the camera and power it up again, then use the Microsoft SerialPort library, everything works totally fine. I can write to the stream and receive events normally.
  • If I unplug the camera and power it up, but then use the SerialPortStream class, I get that clicking noise as soon as I open the stream, and soon after that the camera sends a packet that was not asked for, but which is a "Network Change" event. In the documentation it says that this event is only routed if power is changed, whatever that may mean. Now if I switch back to the Microsoft SerialPort class without powering down/rebooting the camera, it stops responding to that library as well. As if trying to access the camera using SerialPortStream somehow breaks or confuses something on the camera side. Even the testing software that came with the camera then stops being able to access it.
I investigated more and the clicking sound and pitch change are triggered by this part of your code:
m_ComPortHandle = UnsafeNativeMethods.CreateFile(@"\\.\" + m_Port,
  NativeMethods.FileAccess.GENERIC_READ | NativeMethods.FileAccess.GENERIC_WRITE,
  NativeMethods.FileShare.FILE_SHARE_NONE,
  IntPtr.Zero,
  NativeMethods.CreationDisposition.OPEN_EXISTING,
  NativeMethods.FileAttributes.FILE_FLAG_OVERLAPPED,
  IntPtr.Zero);
The corresponding line from the Microsoft library (at least I think it is), which does not trigger the clicking sound or pitch change:
SafeFileHandle tempHandle = UnsafeNativeMethods.CreateFile("\\\\.\\" + portName, 
  NativeMethods.GENERIC_READ | NativeMethods.GENERIC_WRITE, 
  0,    // comm devices must be opened w/exclusive-access
  IntPtr.Zero, // no security attributes 
  UnsafeNativeMethods.OPEN_EXISTING, // comm devices must use OPEN_EXISTING
  flags,
  IntPtr.Zero  // hTemplate must be NULL for comm devices
); 
I don't know if that helps. Maybe it is a hardware problem after all. Maybe the camera or cable is broken or at least has a glitch that only appears with certain configurations and signals. But I can use the whole testing suite that came with the camera without any problems (I reverse engineered it using dotPeek, and it is just a WinForms application using the SerialPort stream class), so while that test suite will run, the manufacturer will probably admit to nothing :)
Sep 29, 2015 at 9:19 AM
Edited Sep 29, 2015 at 9:20 AM
There is no functional difference between the two lines. But I do believe you have found something significant, as it sounds like some kind of power management from the camera to turn on when it detects the serial port is opened. It's been a long time and I'll have to get out my scope to see what's going on with the Rx/Tx lines when I open a serial port.

Calling SerialPortStream.Open() does the following
  • m_SerialPort.Open() -> This is the CreateFile that you see
  • SetCommState. Might contain some option in the DCB that is causing the issue
  • ClearCommBreak in SerialPortStream.cs line 419
So it's definitely related to how I initialise and how the Camera responds to the state of the serial line. If you can help find which line is causing the problem, we could get closer. Instead of Open(), you could call OpenDirect() and ensure that the properties of the device (in DeviceManager) are exactly that what you need. It's not a solution, but get's rid of a lot of the code for testing.
Sep 29, 2015 at 7:02 PM
Hi,

it is exactly this line which triggers the Camera to click and somehow change state, as I said:

m_ComPortHandle = UnsafeNativeMethods.CreateFile(@"\\.\" + m_Port,
NativeMethods.FileAccess.GENERIC_READ | NativeMethods.FileAccess.GENERIC_WRITE,
NativeMethods.FileShare.FILE_SHARE_NONE,
IntPtr.Zero,
NativeMethods.CreationDisposition.OPEN_EXISTING,
NativeMethods.FileAttributes.FILE_FLAG_OVERLAPPED,
IntPtr.Zero);

Using OpenDirect() did not make a difference because both methods call m_SerialPort.Open() anyway before the if block that gets skipped by OpenDirect().

Sadly, I cannot dig any deeper with the regular Debugger, because it's unmanaged code. Even trying to Disable Just my code and enabling native code debugging in the project would not allow me to dig deeper, and the disassembly did not really mean much to me ;) Is there a way I can get into the native code? Or does this help already?
Sep 29, 2015 at 8:07 PM
Edited Sep 29, 2015 at 8:07 PM
Then I'm completely stumped. The only difference between mine and MS is the flags parameter. I'm using overlapped.
X lpFileName = @"\\.\" + Port in both cases
X dwDesiredAccess = GENERIC_READ | GENERIC_WRITE in both cases
X dwShareMode = FILE_SHARE_NONE in both cases (FILE_SHARE_NONE equals zero)
X lpSecurityAttributes = IntPtr.Zero (or null)
X dwCreationDisposition = OPEN_EXISTING
X dwFlagsAndAttributes
oo Mine is FILE_FLAG_OVERLAPPED
oo MS is 'flags'
X hTemplateFile = IntPtr.Zero

The first idea I have is to use a tool called API Monitor to figure out what MS is giving for 'flags' as this is the only difference from your analysis. Just catch the CreateFile call to the NT kernel. API Monitor is quite powerful and works well (I last tested on Windows 7). This is how I looked into how overlapped worked a couple of years ago.

http://www.rohitab.com/downloads

For the second try, I was investigating today Eltima Serial Port Monitor. I was running it on my x86 laptop Windows 7, but it might also work on x64. Version 6.0 for download has a 14 day trial. That shows exactly the data sent and received as well as all the API calls. The problem here could be after the 14 days if we don't solve it until then.
Sep 29, 2015 at 8:52 PM
Edited Sep 29, 2015 at 8:54 PM
Just used the api monitor, which seems to be a great tool. Lots of info (took me some time to find my way around and I am still confused).

I found the following unmanaged call in the capture log using the MS SerialPort class:
Id Time of Day Thread Module API
10 9:41:23.523 PM 1 clr.dll CreateFileW ( "\.\COM4", GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL )

Return Value Error Duration
0x000001e0 0.0310132

Seems they are the same. This is driving me crazy. I am more and more convinced that the hardware has some sort of issue. But even if it has, that still does not explain why It works perfectly well with the MS library. Maybe it just has a compatible glitch :)
I will not be able to work on it tomorrow sadly, and I will have to return the camera on Friday until further notice, because it was only a device provided for testing and implementation. I will use the API monitor on Thursday to monitor a bit more of the stuff that happens under the hood in case the camera sends a reply. Can you point me to the parts of the code that handle incoming events of the underlying serial driver?
Sep 30, 2015 at 9:45 AM
Hi,

One option is that you provide two independent projects, one with the MS library, the other with my library. Test that the MS library works as expected, by starting the program at least twice (not that there is state between starting the two instances affecting the results). Restart the computer and test twice using the version with SerialPortStream.

Because you say that you have the same behaviour also when only Rx and Tx is used, I might be able t monitor differences in the two using an oscilloscope. I've ordered a PL2303RA, but that takes about a month to arrive via Amazon here in Germany. This is why I thought the "break" code might be causing an effect.

Getting to the code.
  • The most interesting points are in the file SerialPortStream.NativeSerialPort.CommOverlappedIo.cs
  • See the thread OverlappedIoThread()
  • Events are handled when the driver indicates such, by setting the semaphore associated with the WaitCommEvent() overlapped operation in the method DoWaitCommEvent
  • When the semaphore triggers, ProcessWaitCommEvent is then called. It's the first sema in the list of sema's to wait for/checked so it always has the highest priority. If there's an EV_RXCHAR/EV_RXFLAG reported by the driver then I set a flag to start an overlapped read.
  • When ProcessWaitCommEvent is finished, the main loop of OverlappedIoThread() goes back and starts an overlapped read session. Again, the semaphore is triggered when the read operation is finished.
  • Similarly for the write, if there's data in the buffer pending (as put there by the Write() operation) then it starts an overlapped write operation in DoWriteEvent(). When that's finished as the sema for the overlapped I/O signals, ProcessWriteEvent() is called.
Everything around that file is just fluff.

When using API monitor, you'd likely want to watch out for the following calls
  • CreateFile
  • WriteFile
  • ReadFile
  • GetOverlappedResult
  • GetCommProperties
  • GetCommModemStatus
  • GetCommMask
  • SetCommMask
  • WaitCommEvent
  • GetCommTimeouts
  • SetCommTimeouts
  • ClearCommError
  • PurgeComm
  • GetCommState
  • SetCommBreak
  • ClearCommBreak
  • EscapeCommFunction
  • SetupComm
I don't think API monitor will tell you the content that is read/written, only the number of bytes that were read/written.
Oct 1, 2015 at 7:50 PM
Haven't been able to get back to doing anything with the camera, yet. Hopefully I will be able to debug a bit more before having to send them back tomorrow.

Cocerning the adapter: I am from Germany, too, and the adapter I use is this one:
Manhattan-USB-Serial
Oct 2, 2015 at 10:47 PM
Edited Oct 2, 2015 at 11:49 PM
Just a little update: Had to send the camera back and did not have enough time to play around with the settings and the capture as you recommended, because I needed some time to implement the final productive features using the workaround that I load the SerialPort class for that camera and SerialPortStream for other devices.

The hour I spend with trying to dig deeper into what happens when the camera is supposed to send a reply I did not find out much. For some reason, the application always crashed or hung when I had the API monitor active while writing to the stream and trying to debug into the SerialPortStream code. Don't know why that was.

Sadly, I will have to postpone further testing this until I get another camera of the same type, or another device that exhibits the same problem. Might have to integrate serial lamp control into the application soon, maybe I will know more than. But it would be great if you could find out anything with the adapter you ordered. Something must be different between the two libraries after all, or I might just be going mad :)