Serial protocol pre-design discussion - give us your input!

[FlappySocks]

One: I want data integrity for all writes to the hardware.

No problem. Contained in the header of every message, there are flags which specify if there is error protection, and if an acknowledgment receipt is required. This works in either direction.

Two: I want an easy way to get data out of the hardware for add-on modules.

This is a bit tricky. Either the protocol needs to be watered down or it needs to operate in duel modes, with the ability to switch it mid-stream if necessary (if the user unplugs their digital gauge thingy, and plugs their laptop in for example). This is a bit harder with a serial interface, where there is no logon,logoff.

I have broken my protocol proposal in two parts. The low-level layer takes care of the nuts-n-bolts of the communication, leaving the programmer with the decision what to send, rather than how. All it requires, is somone to port the protocol to each platform. I am happy to write some sample code in Java or C#.

[Fred]

How large is the header?

I'm looking forward to reading this, and I hope others will take a look at it and discuss it too.

Admin.

[FlappySocks]

A minimum of 3 bytes, plus a start and stop byte. So 5 in total. That gets you the payload.

The payload can be anything you want. Binary data or ASCII. If the receiver doesn't understand the message your sending, it can just ignore it and move on to the next message, without breaking anything.

The overhead grows with the complexity of the data your sending. So if your sending down a flood of realtime data, it's not going to get choked up with an unnecessary overhead. But if in the middle of that stream, you need an acknowledge receipt for something critical, then the protocol will take care of it.

[FlappySocks]

Here it is folks. Written in OpenOffice.

Part one is fairly complete.
Part two will be work in progress, as the project grows.

http://www.diyefi.co.uk/FreeEMS%20Seria ... rt%201.odt
http://www.diyefi.co.uk/FreeEMS%20Seria ... rt%202.odt

This is nothing more than a proposal at this stage. The aim is to further the discussion.

I am off to bed, so I expect all issues to be debated and hammered out by morning

Stuart.

[Fred]

Thank you VERY much for taking the initiative on this! I VERY much appreciate your efforts :-) One day we will all be a healthy team of people working on it all! :-)

Thoughts (half formed) :

• I LOVE the thorough way you have laid it out etc, very good.
• I like the start, stop, escape escaping scheme, good idea (probably obvious, but not to me :-) )
• IMO CRC should follow the payload and be just before the stop byte, the header should contain 1 bit (fixed size) or 2 bits (00 = off, 01 = 8bit, 10 = 16bit, 11 = 32bit). I tend to think fixed size is the way we should go and that multiple sizes is overkill. I also tend to think that doing as mtx_man suggested and using a rolling sum during sending is a good idea. That would work most simply with 8 bits, but 16 bits could use a convention such that if only one byte was left, it was either assumed to be high or low in the 16 to add and padded with zeros or ones above or below that. If we want to put it in the header, that means running through all the data BEFORE sending and calculating it in the hope that it doesn't change while we are sending. If it is a large packet, this will not work out well.
• I like the scalable integers for identifiers regardless of my next comment. Good idea.
• Should the protocol carry a version number of it's own, or make it available? Or should we rely on a basic external to the normal protocol scheme of using a particular "command" to request the firmware version number (a byte per part, so max of 255.255.255) and some sort of other unique identifier or similar? Should that be part of level one protocol? I think yes, a version (3 part, or maybe 4 for future proofness) and a name/product identifier string field request too, and possibly one other.
• What about responses as opposed to acknowledgments? i.e. If you ask for xyz, do you need to know that it is THE one you asked for, or just any xyz packet? Could the ack number be dual purpose? as in a bit in the header says "initiator" or "responder" and the ack byte or word is either what is being responded with, or the identifier for a request. perhaps if that bit is 1 for a request the request itself could come back bearing the 1 whereas if it is zero and it is a request. Does this protocol need to be symmetrical?? I think not as the EMS will be responding to data or providing it without being asked (in the case of event notification or data logging continuously being turned on) and the PC (or dash cluster or xyz) will be the one asking for X telling it to do Y, or receiving the unasked for data and processing it. Apologies if this wasn't clear...
• The 20 reserved pay load types, is that enough, too much, not enough? That could be split by using another bit in the header flags section that says 0 = proto 1 = data etc
• What about parity bits, and start and stop bits for each byte? This should be standardised in this document IMO. I vote for : stop, parity, start bits in a 9 bit format. See the details in table 11-14 section 11.4.3 page 495 of the xdp12 reference manual.
• Ping seems like a good idea. Would you say "processor reset request" etc are an appropriate part of the part 1 protocol discussion or part 2?
• I hope firmware doesn't need to be updated through this protocol as the boot loader is there for that, however it is a possibility if the boot loader doesn't work out the best for multi paged data etc. If that is the case and the fault lies with hcs12mem, it would be better to fix hcs12mem. If it lies in the bootloader we would have to live with it and code around it on the cpu in a similar style to kens c64 to c128 bootstrap loader does.
• It's nice that every packet CAN have crc and ack, but none are forced to. I like this.
• Are your escape scheme codes commonly used, or arbitrary? if they are arbitrary, wouldn't CC be a better choice than BB ? maximum clarity of the patterns. What abour 0xFF for XORing for the same reason, ie, notting the value. I suppose it doesn't matter though, or perhaps there is some reason I haven't seen yet.
• I just read part 2 : 20, 21, 22 are reused 3 times, how can this be, is it that there is a header bit I didn't see that controls what basic type it is? If so, how is that different/better than using the numbers sequentially (or in blocks) from 20 (or zero if my 1 bit swap idea is used) up?
• You say that not all devices support all data etc, and in principle I agree, but in practice, the one that all devices would need is likely to end up more complex to handle than it used to be, and more powerful for it. IE a logging packet (is this what is meant by "real time data" ?) should use a mask that defines what each bit is based on the total sequence of bits etc so that it can be user trimmed to include and exclude various parameters. A detailed knowledge of what the data is will be required on any device wishing to use that data because of it's variable form. Using a fixed form would simplify it, but limit it in the future and limit flexibility with regards logging speed/verbosity. Just speaking out loud here, shoot holes in it as you wish :-)
• part 2 is definitely where detailed packet contents spec belongs. something like "parameter change" will need a field of X length (maybe SI again?) to spec what it is or isnt changing, and another field for the data. If we end up doing such things as not using tables, it will need to carry a data structure in full to load to memory pre optimised for indexing at run time. There should be at least "change parameter in memory" with "which parameter" and also "burn memory parameter to flash" with "which parameter" etc
• OK, that's enough out of me, what does everyone else have to say?

Thanks again :-) (esp for tolerating my half thought out comments)

Admin.

[Fred]

Rethink on the notting : XOR with 0xF0 or 0x0F would be better still wouldn't it?

[FlappySocks]

Oops.... regarding 20, 21, 22 in Part 2. I did a copy & pase job on the tables at the last minute. They should be unique numbers. i will fix that now.

I was trying to put the Payload Types into some logical order. so 0-20 are reserved for protocol spacific commans, 21 to 40 might be realtime data etc. in reality, there can be an infinite number of Payload Types.

I feel that Part 1, should be set in stone as early as possible, and if done right wont change much. Part 2 will always be evolving.

[FlappySocks]

http://www.diyefi.co.uk/FreeEMS%20Seria ... rt%201.odt
http://www.diyefi.co.uk/FreeEMS%20Seria ... rt%202.odt

Both documents updated.

I agree the CRC should be at the end.... updated the doc accordingly.

Stuart.

[Fred]

Any chance of including the version of the document in the file name and updating/bumping it with each edit? maybe add a third digit if you don't want to progress past 0.1 :-)

I agree that finalising the design early is a good thing, BUT, nailing ourselves down to something without any practical experience of what features etc might be needed could be detrimental. There will be no need for more than one tool to start with anyway, and by the time the firmware is useful, the serial spec should have properly solidified. Thoughts?

[Fred]

Would it not be better if the header data was part of the checksum? After all, it contains important information too which has just as bad a consequences if corrupted.

The checksum or not bit will tell you if there should be one tagged on the end, and the field length should tell you if the checksum bit has been corrupted or not. (checksum 0 and payload too long by checksum length = corrupt).

You should spec that the payload length doesn't include the optional checksum too.

Admin.