FSUIPC file memory mapping buffer over run

[LarryJ_KMSO]

Hello Pete,

I have encountered a problem reading large blocks of data using FSUIPC and WideFS. The block I want to read is the AI block which is about 12k. I've created a demo program which illustrates the problem. You can read three blocks of 16K without error. Read the fourth and method "Get"returns an error. However if you read three blocks of 16K followed by FSUIPC open and Init, you can keep reading 3 x 16K bytes without error, apparently indefinitely. I'd send the program but apparently there isn't any way to attach a zip file?

So 16K * 4 is 64K and the IPC buffer is 64K. This suggests that the internal buffer is not circular and that the fourth dump of 16k to it causes the buffer to fill/overflow and is flagged as an error. Apparently, FSUIPC Open and Init resets the buffer indexes (or creates a new buffer).

Is there a way around this?

1) Keep a byte count and perform an FSUIPC open and Init when the count would exceed 64K?

2) I've tried increasing the buffer size to 128K but this just postpones the inevitable error, not a clean solution?

3) Is there some function call I can make to reset the buffer indexes so that I don't have to incur the memory mapping overhead with a call to FSUIPC open/init ?

If you want the demo program, let me know how to send it.

Thank you

Larry Jones

Florence, MT

[Pete Dowson]

I have encountered a problem reading large blocks of data using FSUIPC and WideFS. The block I want to read is the AI block which is about 12k. I've created a demo program which illustrates the problem. You can read three blocks of 16K without error. Read the fourth and method "Get"returns an error.

Sorry, you need to explain this further. Try FSInterrogate, set it to read 64k and it will read them all day. It isn't a good idea, of course, but it produces no errors.

There is a limit of around 30k or so per single "Process" call. An FSUIPC_Read or FSUIPC_Write should return an error if you try more than that, but I cannot swear that all the language examples impose this.

So 16K * 4 is 64K and the IPC buffer is 64K

.

It is nothing to do with that. There's an imposed limit on the amount of data processed by a single FSUIPC_Process call, that's all.

Apparently, FSUIPC Open and Init resets the buffer indexes (or creates a new buffer).

It resets the pointers which FSUIPC_Process would also, once the transfer is complete.

Is there a way around this?

Yes, use FSUIPC_Process calls more often, at least once per 16k.

3) Is there some function call I can make to reset the buffer indexes so that I don't have to incur the memory mapping overhead with a call to FSUIPC open/init ?

Yes, it is called FSUIPC_Process -- you can see the code yourself and work out what is going on.

FSInterrogate is the program to play with to see what is happening. Also my own TrafficLook transfers 16k at a time with no problems.

Regards,

Pete

[LarryJ_KMSO]

Pete,

My read data method DOES perform a call to "process" for each block of data. I've snipped the code below. My three successive calls are Read_FSUIPC_Data(ref buffer, offset, 0x4000); where buffer is an array of bytes large enough to hold 16K and the offset is 0xD000; (like Read_FSUIPC_DATA(ref buf, 0xD000,0x4000)

The fourth call for 16K like that above throws an error to the listBox from the Get method. (including a hex output of the first four bytes of the output array which are always zero) . The Get method is Scott McCrory's C# translation of Bob Scott's FSUIPC VB.net . Works just fine for three calls of 16k then fails on the fourth call for 16K.

If I increase the buffer size to 128k, I can make 128k/16K -1 calls before the error is returned. (With the default IPC buffer size of 64K, I can make 64k/16k -1 passes). That's why I am assuming this is related to the buffer size. If I call for 4k of data, I can make 64k/4k -1 calls successfully before on call number 16 an error is returned from the Get method.

Again, if the fourth call for 16k is preceded by FSUIPC open and Init, the data is correctly read and I can make two more calls for 16K of data.

Obviously, TrafficLook runs all day without error. I must be doing something wrong.

Thanks,

Larry

	private int Read_FSUIPC_Data(ref byte [] buf, int offset, int bytesToRead)
		{
			bool result = false;
			int  result1 = 0, token = 0;



			result = f.FSUIPC_Read(offset, bytesToRead, ref token,ref result1);
			if(result == false)
			{

				listBox1.Items.Add(String.Format("Error in FSUIPC_Read call, Error = {0} ", result1));
				return -1;  // error in Read
			}
				else
				listBox1.Items.Add(String.Format("FSUIPC_Read went OK... Pass = {0} ",pass));

			result = f.FSUIPC_Process(ref result1);
			if(result == false)
			{

				listBox1.Items.Add(String.Format( "Error in FSUIPC_Process call, Error = {0} ", result1));
				return -2;  // error in Process
			}
			else
				listBox1.Items.Add(String.Format("FSUIPC_Process went OK... Pass = {0} ",pass));

			result = f.FSUIPC_Get(ref token, bytesToRead, ref buf);
			if(result == false)
			{

				listBox1.Items.Add(String.Format("Error in FSUIPC_Get callPass = {0} ",pass++));
				listBox1.Items.Add(String.Format("bytes returned =  0x{0:X} 0x{1:X} 0x{2:X} 0x{3:X}",buf[0],buf[1],buf[2], buf[3]));
				return -3;  // error in Get
			}
			else 
			{
				listBox1.Items.Add(String.Format("FSUIPC_Get went OK... Pass = {0} ",pass++));
				return 1;  // success
			}
		}

/[code]

[Pete Dowson]

The fourth call for 16K like that above throws an error to the listBox from the Get method. (including a hex output of the first four bytes of the output array which are always zero) . The Get method is Scott McCrory's C# translation of Bob Scott's FSUIPC VB.net . Works just fine for three calls of 16k then fails on the fourth call for 16K.

I'm sorry, but neither of those is a language I'm familiar with. Can you contact the author for clarification? It sounds like the pointer to the next free position in the memory file map is not being reset in whatever is the equivalent to the FSUIPC_Process call.

I only programmed the C library code, of which the source is provided. I know that works. Also the Delphi package must work, as that is used in FSInterrogate. The VB stuff must work as most of the FSUIPC utilities folks write use that, so it is sounding now like there's a bug in the C# code.

If I increase the buffer size to 128k, I can make 128k/16K -1 calls before the error is returned.

The FSUIPC/WideFS interface doesn't support block sizes larger than around 30-31k including all the red tape that is added. Even if your local C# code is changed to accept such, the Process call will go wrong in WideFS or FSUIPC -- I'm not sure what you'd get, probably just most of the block ignored.

With the default IPC buffer size of 64K

Where do you get this "default" buffer size from? In all my code the maximum block for FSUIPC_Porcess is defined by

#define MAX_SIZE 0x7F00 // Largest data (kept below 32k to avoid // any possible 16-bit sign problems)

(see the IPCuser.c file original Lib_source).

There is no "default 64k buffer" in anything I've provided.

For each individual read/write a number of bytes of red tape are added. See the header files for that.

I must be doing something wrong.

No necessarily. It may be an error in the C# code provided in the SDK. I'm afraid I just provide whatever is provided for me. If you understand C# please check it and let me know. You could compare it with known working code, like the C, VB and Delphi versions.

Regards,

Pete

[LarryJ_KMSO]

Where do you get this "default" buffer size from?

It's in the file fsuipc.cs fom the FSUIPC Developer Kit Library for C#.NET line 78:

public const int IPC_BUFFER_SIZE = 65568;

Well, I've already found a typo in the fsuipc.cs. (Which I communicated to the author Scott McCrory) Not bad for a programmer at my level! I guess I'll print out the Delphi version and compare the two. Then compare with your C version. Maybe I'll be forced to finally understand memory mapped files!

I've been in touch with Bob Scott author of the Visual Basic .NET fsuipc version from which the C# fsuipc.cs was translated by Scott McCrory. (One of the great joys of Managed Code, I understand!) Bob suggesed buffer over run which my tests confirm. Just where and how the overrun occurs or just where the buffer indexes are not being reset, is at this point a mystery.

Larry

[Pete Dowson]

public const int IPC_BUFFER_SIZE = 65568;

If that is used as the limit for the process call then it is too big -- neither WideFS nor FSUIPC will handle that correctly. 32512 is correct (hex 7F00). the value 65568 is rather peculiar in any case, being 33 more than the largest possible 16-bit number.

Just where and how the overrun occurs or just where the buffer indexes are not being reset, is at this point a mystery.

Okay. I hope it is easy to resolve. I look forward to hearing from you when it is! In case it helps, this the is active statement in the C code:

m_pNext = m_pView;

It actually does this BEFORE the call to FSUIPC, but after marking the end of the data with a 32-bit zero word, thus:

ZeroMemory(m_pNext, 4); // Terminator

Because it also then uses the same pointer to process the returns from READs, it resets the pointer to the beginning of the buffer again, just before the exit:

m_pNext = m_pView;

*pdwResult = FSUIPC_ERR_OK;

return TRUE;

Maybe this apparently redundant, but certainly not, repetition is omitted in the C# version?

Incidentally, even this stuff isn't really my code. FSUIPC uses the original interface from FS5IPC/FS6IPC, and the code was written by Adam Szofran (it isn't my style, actually -- mine is rather more obscure! :) ).

Regards,

Pete

[LarryJ_KMSO]

For once, it wasn't something totally stupid in my program that was causing the error!

Turns out the translation that produced fsuipc.cs (C#) from fsuipc. vb.net was done with an earlier version of the vb.NET which had a problem. That problem got carried into the translated code and caused me to go cross-eyed for a few weeks!

So everyone using this routine (C# version of fsuipc.cs), be mindful that a corrected version of the C# FSUIPC class will be published here soon by the vb.NET author Bob Scott. Thanks to Bob for coming to my rescue. If you need the routine before that, I can supply a prelimiinary copy that I have tested. The fsuipc C# routine that is faulty is the FSUIPC_Get method overloaded to fill an array of bytes.

Happy New Year to all

Larry Jones

Florence, MT