[Pharo-project] TextConverter handling of binary streams

Sven Van Caekenberghe sven at beta9.be
Tue Nov 30 11:46:40 CET 2010


I understand very well that you are very busy, Stéphane, I am not asking for a solution.

It is exactly the same in Squeak.

Like Philippe said, it looks wrong.

On 29 Nov 2010, at 20:12, Stéphane Ducasse wrote:

> sven 
> 
> I'm terribly and more than that busy until mid or more dec.
> Now did you check if the behavior is the same in squeak?
> S.
> 
> On Nov 29, 2010, at 3:23 PM, Sven Van Caekenberghe wrote:
> 
>> Hi,
>> 
>> TextConverter and its subclasses seem to break the contract of #nextFromStream: and #nextPut:toStream: when the stream #isBinary. Consider the following two examples:
>> 
>> ByteArray streamContents: [ :stream | | encoder |
>> 	encoder := UTF8TextConverter new.
>> 	'élève en français' do: [ :each | encoder nextPut: each toStream: stream ] ].
>> 
>> #[233 108 232 118 101 32 101 110 32 102 114 97 110 231 97 105 115]
>> 
>> (String streamContents: [ :stream | | encoder |
>> 	encoder := UTF8TextConverter new.
>> 	'élève en français' do: [ :each | encoder nextPut: each toStream: stream ] ]) asByteArray.
>> 
>> #[195 169 108 195 168 118 101 32 101 110 32 102 114 97 110 195 167 97 105 115]
>> 
>> The first answer is incorrect, the second is correct (as far as I understand it).
>> 
>> This is apparently on purpose, from the implementation of, for example, UTF8TextConverter>>#nextPut:toStream:
>> 
>> nextPut: aCharacter toStream: aStream 
>> 	| leadingChar nBytes mask shift ucs2code |
>> 	aStream isBinary ifTrue: [^aCharacter storeBinaryOn: aStream].
>> 	leadingChar := aCharacter leadingChar.
>> 	(leadingChar = 0 and: [aCharacter asciiValue < 128]) ifTrue: [
>> 		aStream basicNextPut: aCharacter.
>> 		^ aStream.
>> 	].
>> 
>> 	"leadingChar > 3 ifTrue: [^ aStream]."
>> 
>> 	ucs2code := aCharacter asUnicode.
>> 	ucs2code ifNil: [^ aStream].
>> 
>> 	nBytes := ucs2code highBit + 3 // 5.
>> 	mask := #(128 192 224 240 248 252 254 255) at: nBytes.
>> 	shift := nBytes - 1 * -6.
>> 	aStream basicNextPut: (Character value: (ucs2code bitShift: shift) + mask).
>> 	2 to: nBytes do: [:i | 
>> 		shift := shift + 6.
>> 		aStream basicNextPut: (Character value: ((ucs2code bitShift: shift) bitAnd: 63) + 128).
>> 	].
>> 
>> 	^ aStream.
>> 
>> I would say that the contract of #nextPut:toStream: is to take a Character object and write a binary representation using a specific encoding to a stream. However, when given a #isBinary stream, it does no longer do any encoding at all !
>> 
>> The same is true for the other converters as well as for #nextFromStream:. 
>> 
>> Does anyone know why that is the case ?
>> 
>> And if it is by design, how should one do UTF8 encoding on a binary stream ??
>> 
>> Thx,
>> 
>> Sven
>> 
>> PS: If others also think this is strange, I could make an issue, I am just not sure this is a bug.
>> 
>> 
>> 
> 
> 





More information about the Pharo-project mailing list