How Unidecoder Transliterates UTF-8 to ASCII
•
1 like•782 views
Slides of my talk at Paris.rb on 2014-11-07. How does UTF-8 work? How to leverage it to convert chinese, russian or any non-ASCII character to ASCII? Here is what the Unidecoder gem does.
Recommended
More Related Content
More from Simon Courtois
More from Simon Courtois (16)
Recently uploaded
Recently uploaded (20)
How Unidecoder Transliterates UTF-8 to ASCII
- 1. Unidecoder Simon Courtois - @happynoff
- 3. Ni Hao
- 5. How does it work?
- 6. At the beginning there was ASCII
- 7. A 65 B 66 C 67 a 97 b 98 c 99
- 8. A 65 10 00001 64 32 16 8 4 2 1 a 97 11 00001 64 32 16 8 4 2 1
- 9. B 66 10 00010 64 32 16 8 4 2 1 b 98 11 00010 64 32 16 8 4 2 1
- 11. So every country had its own encoding(s)!
- 12. All was fine until…
- 13. The World Wide Web
- 14. UTF-8 to the rescue!
- 15. Everything on 32 bits?
- 16. Bad idea c a f é
- 17. Bad idea c a f é
- 18. Bad idea 0
- 19. A better idea A 65 010 00001 110 XXXXX 10 XXXXXX 1110 XXXX 10 XXXXXX 10 XXXXXX
- 20. A better idea 110 XXXXX 10 XXXXXX 110 10000 10 011111
- 21. A better idea 10000011111 1055 П
- 22. So, how does unidecoder work?
- 23. How do we go from П to P ?
- 24. Start from a string like “П”
- 25. Unpack it “П”.unpack(“U”) [1055] 00000100 00011111 4 31
- 26. 4 x04 x04.yml Ie Io Dj … P 0 1 2 31
- 27. How to obtain 4 and 31 ?
- 28. unpacked = 1055 0000010000011111 unpacked 8 0000010000011111 4
- 29. How to obtain 4 and 3 1 ?
- 30. 31 unpacked = 1055 0000010000011111 unpacked 255 0000010000011111 0000000011111111 0000010000011111
- 31. Brain fried yet? advertising time!
- 32. www.tinci.fr Web Development Software Development Consulting Support @tincihq
- 33. Resources Characters, Symbols and the Unicode Miracle: bit.ly/why-utf8 Unidecoder: github.com/norman/unidecoder Slides: bit.ly/unidecoder
- 34. Thank you! Simon Courtois - @happynoff